On the road to (GNU) debbugs support, some blockers

On the road to (GNU) debbugs support, some blockers

[Jelle Licht]

Thanks for working on piem, and sharing it under a free license!

In getting some local hacks set up, I ran into some inconsistencies and
issues, which may very well be misunderstandings on my part.

The first one:
the public-inbox-config man page (`man 5 public-inbox-config') demonstrates a
url without trailing '/'. Piem's documentation uses an url with trailing '/'.

It seems `piem-b4--get-am-files' assumes this url always has a trailing
'/', which makes it not work for the examples used in the
public-inbox-config manpage (and in particular, my pre-existing
configuration). I'm not unsure if public inbox supports the trailing
'/', but I see no reason to assume it would not.

The second issue; in order to map (GNU) debbugs data to inbox
configurations, the current best way I've worked out is to use the
X-GNU-PR-Package header. 

Concretely, I've had to adjust `piem-gnus-get-inbox' to also take this
particular mail header into account when trying to find matching
configured inboxes. I am not aware of a way for debbugs users to be able
to use this, without messing with non-debbugs users of gnus. Advice
appreciated.

NB, I also added the `:gnu-pr-package' to my configured piem inboxes so
the matching could take place in the first place, of course, but this is
already possible with via {setq,Customize system,$HOME.public-inbox/config}.

Third issue:

The `piem-gnus-mid-to-thread' seems to prepend a line like: "From
mboxrd@z Thu Jan 1 00:00:00 1970" to entries. The issue comes up when we
already have a line that starts with "From " at the top, which after
running the `replace-regexp-in-string' boils down to something like in
my generated m-piem file:
--8<---------------cut here---------------start------------->8---
From mboxrd@z Thu Jan  1 00:00:00 1970
>From unknown Sat May 27 07:31:53 2023
[rest of file]
--8<---------------cut here---------------end--------------->8---

AFAIK, b4 am can't deal with this, as the "From " line needs to be the
first line, and any later lines need to be parsed as email header (which
">From ..." is not). Any solutions appreciated, that (again) don't
impact non-debbugs users of gnus. My local current hack changed the
format string "From mboxrd@z Thu Jan 1 00:00:00 1970\n%s\n" by "%s\n",
and the regex from "^>*From " into "^>+From", but this will of course
break if any of the mails I'm working with contains a line starting with
"From ".

Thanks for any help, and please let me know if you'd be interested
patches for a (clean) integration with (GNU) debbugs.

Kind regards,
Jelle Licht

Re: On the road to (GNU) debbugs support, some blockers

[Kyle Meyer]

Jelle Licht writes:

> Thanks for working on piem, and sharing it under a free license!

You're welcome, and thanks for the feedback.

> the public-inbox-config man page (`man 5 public-inbox-config') demonstrates a
> url without trailing '/'. Piem's documentation uses an url with trailing '/'.
>
> It seems `piem-b4--get-am-files' assumes this url always has a trailing
> '/', which makes it not work for the examples used in the
> public-inbox-config manpage (and in particular, my pre-existing
> configuration). I'm not unsure if public inbox supports the trailing
> '/', but I see no reason to assume it would not.

public-inbox does work fine with the trailing slash, but I agree that
piem should not insist on the slash.  And confusingly I've been
inconsistent about it: the docstring of piem-inboxes says "this value
must end with a slash" but then several spots tack on a slash if needed.

I'll work on updating every spot that retrieves a URL to go through a
helper that appends a slash if needed (but I probably won't get a chance
to do that until this weekend).

> The second issue; in order to map (GNU) debbugs data to inbox
> configurations, the current best way I've worked out is to use the
> X-GNU-PR-Package header.

Have you tried using :listid?  For example, for guix-patches, I use
"guix-patches.gnu.org".

If X-GNU-PR-Package is better than List-Id for some users/cases, I'm
open to adding support for it.  It should be sufficient to teach
piem-inbox-by-header-match to consider that header (along with an
associated piem-inboxes key).

> Concretely, I've had to adjust `piem-gnus-get-inbox' to also take this
> particular mail header into account when trying to find matching
> configured inboxes. I am not aware of a way for debbugs users to be able
> to use this, without messing with non-debbugs users of gnus. Advice
> appreciated.

Hmm, I'm missing how it would mess with non-debbugs messages, at least
in the case of what I mentioned above.  X-GNU-PR-Package would be
considered when mapping to an inbox, but, for non-debbugs lists, the
other headers/criteria can still match.

> Third issue:
> The `piem-gnus-mid-to-thread' seems to prepend a line like: "From
> mboxrd@z Thu Jan 1 00:00:00 1970" to entries. The issue comes up when we
> already have a line that starts with "From " at the top, which after
> running the `replace-regexp-in-string' boils down to something like in
> my generated m-piem file:
> --8<---------------cut here---------------start------------->8---
> From mboxrd@z Thu Jan  1 00:00:00 1970
>>From unknown Sat May 27 07:31:53 2023
> [rest of file]
> --8<---------------cut here---------------end--------------->8---

piem-gnus-mid-to-thread's goal is to construct an mbox (in mboxrd
format) from plain messages.  It looks like your input is already an
mbox.

That's not what I see when using Gnus (Emacs 28.2) with NNTP.  For
example, here's the mbox that I get when applying
<87y1l7fb9j.fsf@gnu.org1> from yhetil.gnu.guix.patches:

  $ grep -c '^From mboxrd@z' /tmp/piem-b4-u1rBSW/m-piem
  37
  $ grep '^From ' /tmp/piem-b4-u1rBSW/m-piem  | grep -cv mboxrd
  0

Can you give more details on your setup? For example, perhaps you're
working with mail rather than NNTP, and gnus-summary-display-article
displays that differently?

> Thanks for any help, and please let me know if you'd be interested
> patches for a (clean) integration with (GNU) debbugs.

Sure, I'd be happy to review patches for that.  Thanks.

Re: On the road to (GNU) debbugs support, some blockers

[Jelle Licht]

Kyle Meyer <kyle@kyleam.com> writes:

> Jelle Licht writes:
>
>> Thanks for working on piem, and sharing it under a free license!
>
> You're welcome, and thanks for the feedback.
>
>> the public-inbox-config man page (`man 5 public-inbox-config') demonstrates a
>> url without trailing '/'. Piem's documentation uses an url with trailing '/'.
>>
>> It seems `piem-b4--get-am-files' assumes this url always has a trailing
>> '/', which makes it not work for the examples used in the
>> public-inbox-config manpage (and in particular, my pre-existing
>> configuration). I'm not unsure if public inbox supports the trailing
>> '/', but I see no reason to assume it would not.
>
> public-inbox does work fine with the trailing slash, but I agree that
> piem should not insist on the slash.  And confusingly I've been
> inconsistent about it: the docstring of piem-inboxes says "this value
> must end with a slash" but then several spots tack on a slash if needed.
>
> I'll work on updating every spot that retrieves a URL to go through a
> helper that appends a slash if needed (but I probably won't get a chance
> to do that until this weekend).
>

Thanks for confirming the issue.

>> The second issue; in order to map (GNU) debbugs data to inbox
>> configurations, the current best way I've worked out is to use the
>> X-GNU-PR-Package header.
>
> Have you tried using :listid?  For example, for guix-patches, I use
> "guix-patches.gnu.org".

I'm using the debbugs.el interface to read the messages, which
may-or-may-not present a very idiosyncratic version of the messages:

--8<---------------cut here---------------start------------->8---
...
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit@debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request@debbugs.gnu.org?subject=help>
X-GNU-PR-Package: guix-patches
...
--8<---------------cut here---------------end--------------->8---

>
> If X-GNU-PR-Package is better than List-Id for some users/cases, I'm
> open to adding support for it.  It should be sufficient to teach
> piem-inbox-by-header-match to consider that header (along with an
> associated piem-inboxes key).

'Better' is a big word, but the X-GNU-PR-Package header is the only one
that is really exposed via the debbugs interface afaics.
>> Concretely, I've had to adjust `piem-gnus-get-inbox' to also take this
>> particular mail header into account when trying to find matching
>> configured inboxes. I am not aware of a way for debbugs users to be able
>> to use this, without messing with non-debbugs users of gnus. Advice
>> appreciated.
>
> Hmm, I'm missing how it would mess with non-debbugs messages, at least
> in the case of what I mentioned above.  X-GNU-PR-Package would be
> considered when mapping to an inbox, but, for non-debbugs lists, the
> other headers/criteria can still match.

Yes, you are correct; it can be a fallback check. I'm currently using
the following el-patch function definition in my init.el:
--8<---------------cut here---------------start------------->8---
(el-patch-defun piem-gnus-get-inbox ()
    "Return inbox name from a Gnus article"
    (when (derived-mode-p 'gnus-article-mode 'gnus-summary-mode)
      (with-current-buffer gnus-original-article-buffer
        (el-patch-wrap 1 1
                       (or (piem-inbox-by-header-match)
                           (piem-debbugs-by-header-match))))))
--8<---------------cut here---------------end--------------->8---

It could also just be added directly to the
`piem-inbox-by-headers-match' instead.

>> Third issue:
>> The `piem-gnus-mid-to-thread' seems to prepend a line like: "From
>> mboxrd@z Thu Jan 1 00:00:00 1970" to entries. The issue comes up when we
>> already have a line that starts with "From " at the top, which after
>> running the `replace-regexp-in-string' boils down to something like in
>> my generated m-piem file:
>> --8<---------------cut here---------------start------------->8---
>> From mboxrd@z Thu Jan  1 00:00:00 1970
>>>From unknown Sat May 27 07:31:53 2023
>> [rest of file]
>> --8<---------------cut here---------------end--------------->8---
>
> piem-gnus-mid-to-thread's goal is to construct an mbox (in mboxrd
> format) from plain messages.  It looks like your input is already an
> mbox.
>
> That's not what I see when using Gnus (Emacs 28.2) with NNTP.  For
> example, here's the mbox that I get when applying
> <87y1l7fb9j.fsf@gnu.org1> from yhetil.gnu.guix.patches:
>
>   $ grep -c '^From mboxrd@z' /tmp/piem-b4-u1rBSW/m-piem
>   37
>   $ grep '^From ' /tmp/piem-b4-u1rBSW/m-piem  | grep -cv mboxrd
>   0
>
> Can you give more details on your setup? For example, perhaps you're
> working with mail rather than NNTP, and gnus-summary-display-article
> displays that differently?

As mentioned, I use debbugs.el to interact with the issue tracker. [1]


Using guix:
`guix shell emacs emacs-debbugs -- emacs':

Evaluate:
`(debbugs-gnu '("serious" "important" "normal") '("guix-patches") nil t)'

Go to any issue, and open a message. Then, in the message windows, run:
`gnus-summary-show-raw-article'

For my chosen message, the buffer starts with:

--8<---------------cut here---------------start------------->8---
From unknown Fri Jun 02 06:31:39 2023
X-Loop: help-debbugs@gnu.org
Subject: [bug#63824] [PATCH 0/3] gnu: openocd: Update to 0.12.0.
...
--8<---------------cut here---------------end--------------->8---

>> Thanks for any help, and please let me know if you'd be interested
>> patches for a (clean) integration with (GNU) debbugs.
>
> Sure, I'd be happy to review patches for that.  Thanks.

[1]: Not entirely relevant to the issue at hand, but I also have all
guix-patches messages available and indexed in my local notmuch
setup. When I use piem-gnus /w my modifcations, in combination with
piem-notmuch to apply patches using `piem-b4-am', the relevant messages
are actually not even downloaded from the yhetil public inbox, since the
messages with the message-ids are already found locally! Really
wonderful design and/or some convenient emergent properties of how piem
does things.

Re: On the road to (GNU) debbugs support, some blockers

[Kyle Meyer]

Jelle Licht writes:

>> Have you tried using :listid?  For example, for guix-patches, I use
>> "guix-patches.gnu.org".
>
> I'm using the debbugs.el interface to read the messages, which
> may-or-may-not present a very idiosyncratic version of the messages:
>
> --8<---------------cut here---------------start------------->8---
> ...
> List-Id: <debbugs-submit.debbugs.gnu.org>
> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
> List-Post: <mailto:debbugs-submit@debbugs.gnu.org>
> List-Help: <mailto:debbugs-submit-request@debbugs.gnu.org?subject=help>
> X-GNU-PR-Package: guix-patches
> ...
> --8<---------------cut here---------------end--------------->8---

Ah, debbugs.el was the piece I was missing.  I've played around with
debbugs.el a bit in the past, but never used it for real and haven't
considered it in the context of piem.

>> If X-GNU-PR-Package is better than List-Id for some users/cases, I'm
>> open to adding support for it.  It should be sufficient to teach
>> piem-inbox-by-header-match to consider that header (along with an
>> associated piem-inboxes key).
>
> 'Better' is a big word, but the X-GNU-PR-Package header is the only one
> that is really exposed via the debbugs interface afaics.

I'll look into debbugs.el a bit more because I don't have a good
understanding of where it's getting the messages from and how it's
interacting with Gnus, but it sounds like the X-GNU-PR-Package header is
the way to go for debbugs.el support.

I'm curious whether it will make sense to have a custom integration
library, perhaps one that builds on top of piem-gnus, but perhaps that
will be clearer once I take a closer look.

> As mentioned, I use debbugs.el to interact with the issue tracker. [1]
> Using guix:
> `guix shell emacs emacs-debbugs -- emacs':
>
> Evaluate:
> `(debbugs-gnu '("serious" "important" "normal") '("guix-patches") nil t)'
>
> Go to any issue, and open a message. [...]

Thanks, those concrete steps are helpful.

> [1]: Not entirely relevant to the issue at hand, but I also have all
> guix-patches messages available and indexed in my local notmuch
> setup. When I use piem-gnus /w my modifcations, in combination with
> piem-notmuch to apply patches using `piem-b4-am', the relevant messages
> are actually not even downloaded from the yhetil public inbox, since the
> messages with the message-ids are already found locally! Really
> wonderful design and/or some convenient emergent properties of how piem
> does things.

I'll claim it's design, unless someone's complaining, then it's just an
unfortunate emergent property :)

But, yeah, the idea is that users can put local retrieval methods in
piem-mid-to-thread-functions, ordered by their preferred sources, and
then those sources are given priority regardless of where piem-mid is
extracting the message ID from.  The main downside is that a given local
source that has the specified message ID doesn't necessarily have all
the thread's messages.

This approach has worked nicely for me because I have all of yhetil
inboxes mirrored locally and indexed with lei but have yet to flesh out
piem-lei enough to move away following lists via NNTP with Gnus [*].

[*] Aside from dev testing, I haven't used or relied on
    piem-gnus-mid-to-thread myself.  My testing prompted by your message
    hit into some encoding issues, so it looks like it needs some work
    beyond the debbugs.el case you raised.

Re: On the road to (GNU) debbugs support, some blockers

[Kyle Meyer]

Jelle Licht writes:

> As mentioned, I use debbugs.el to interact with the issue tracker. [1]
> [...]
> [1]: Not entirely relevant to the issue at hand, but I also have all
> guix-patches messages available and indexed in my local notmuch
> setup. [...]

Because you have the local Notmuch store, perhaps you'd be interested in
debbugs.el visiting the message in Notmuch rather than Gnus.  As I said,
I haven't actually used debbugs.el much, but I stumbled on this in my
config:

  ;; Modified from function in Nicolas Petton's emacs configuration
  ;; (https://gitlab.petton.fr/nico/emacs.d/, 208407f53a)
  (defun km/debbugs-notmuch-select-report (&rest _)
    (require 'debbugs-gnu)
    (let* ((status (debbugs-gnu-current-status))
           (id (cdr (assq 'id status)))
           (merged (cdr (assq 'mergedwith status))))
      (setq merged (if (listp merged) merged (list merged)))
      (unless id
        (user-error "No bug report on the current line"))
      (let ((address (format "%s@debbugs.gnu.org" id))
            (merged-addresses (string-join (mapcar (lambda (id)
                                                     (format "%s@debbugs.gnu.org" id))
                                                   merged)
                                           " ")))
        (notmuch-tree (format "%s %s" address merged-addresses)))))

  (advice-add 'debbugs-gnu-select-report :override
              #'km/debbugs-notmuch-select-report)

Re: On the road to (GNU) debbugs support, some blockers

[Kyle Meyer]

Kyle Meyer writes:

> I'll look into debbugs.el a bit more because I don't have a good
> understanding of where it's getting the messages from and how it's
> interacting with Gnus, but it sounds like the X-GNU-PR-Package header is
> the way to go for debbugs.el support.

debbugs-gnu-select-report (which has built-in support for Gnus or Rmail)
is getting the mbox from

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=%s;mboxmaint=yes

So indeed

 * piem-gnus-mid-to-thread should make sure the input isn't already an
   mbox before inserting the From_ line

 * X-GNU-PR-Package looks like the best (or really only) option to
   connect an mbox from debbugs.gnu.org to an inbox

> I'm curious whether it will make sense to have a custom integration
> library, perhaps one that builds on top of piem-gnus, but perhaps that
> will be clearer once I take a closer look.

In most cases, users will probably be calling piem after selecting the
issue and entering Gnus/Rmail or, if they override the selection
mechanism, perhaps Notmuch (or even lei :>).  So support mostly just
comes from those libraries.

However, I do think it'd be nice to add an integration library for the
Debbugs mode buffer.  With

  (debbugs-gnu-current-status)

we can get the package and message ID, so it'd be straightforward to
define a piem-debbugs-get-inbox and piem-debbugs-get-mid.