Copyright © 2020--2021 all contributors <piem@inbox.kyleam.com>

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license is included in the section entitled ``GNU Free Documentation License''.

piem
****

This manual is for piem version 0.2.0.

Overview
********

piem is a collection of Emacs libraries for working with public-inbox archives.  As much of the hard work here is already done by other Emacs libraries---things like mail clients, news readers, Git interfaces, and even web browsers---piem is mostly about bridging some of these parts for convenience.

public-inbox
============

public-inbox (https://public-inbox.org/README.html) is software for archiving public mailing lists.  Archives can be exposed over HTTP.  As examples, https://public-inbox.org/meta serves public-inbox's own mailing list, and https://lore.kernel.org/lists.html hosts the archives of many Linux development mailing lists.

These web archives are good for searching, particularly if you don't have all of the list's messages on your local machine, or for linking to a message.  On the other hand, the web interface isn't convenient when you want to follow new activity on a list.  To do that, you could of course subscribe to the mailing list, but public-inbox offers a few ``pull methods'' that you can use instead:

  * an atom feed for the list as a whole or for specific searches

  * read-only NNTP

  * read-only IMAP (new in upcoming public-inbox v1.6.0)

Finally, archives are exposed as one or more Git repositories, facilitating replication (see https://public-inbox.org/reproducibility.html).  For example, you can clone the mailing list archives of git.vger.kernel.org with

     git clone --mirror https://lore.kernel.org/git/0 git/git/0.git

After the initial clone, new messages can be retrieved with git fetch.  Unsurprisingly git log is not a pleasant way to read a mailing list; instead this method is useful for mirroring the archive or bulk importing of the messages.  (See https://public-inbox.org/clients.html for a list of some tools designed to work with public-inbox archives.)

Getting started
***************

piem-dispatch transient (see transient) provides an entry point to piem commands. It's recommended to bind piem-dispatch to a key.  However, before most of those commands do anything useful, you need to register inboxes and activate at least one minor mode.

Registering inboxes
===================

A public-inbox archive, referred to as an inbox, is registered by adding an entry to piem-inboxes.  Here's an example entry for the Git project's mailing list:

     ("git"
      :url "https://lore.kernel.org/git/"
      :address "git@vger.kernel.org"
      :listid "git.vger.kernel.org"
      :coderepo "~/src/git/")

The first element is a name for the inbox and will typically match the name at the end of the :url value.  Specifying either :listid or :address is important so that a message in a buffer can be mapped to an inbox in piem-inboxes.

:coderepo points to the working tree of a local Git repository that contains code related to that archive (in the example above, a local clone of https://git.kernel.org/pub/scm/git/git.git/).  This information is required to apply patches from an archive to a local code repository (see Applying patches).

If you mirror some inboxes locally (e.g., for fast local access or for use with lei), you don't need to duplicate the information from your public-inbox configuration (public-inbox-config(5) (https://public-inbox.org/public-inbox-config.txt)). When the option piem-get-inboxes-from-config is non-nil, the function piem-merged-inboxes, which all code should use for accessing the registered inboxes, returns a combined set of inboxes derived from piem-inboxes and public-inbox's configuration. Merging is done at the level of inbox properties (e.g., an inbox's URL may be defined in piem-inboxes and the inbox's address in public-inbox's configuration).  When a value is defined in both sources, the one in piem-inboxes takes precedence.

Properties described for piem-inboxes are constructed by mapping

     [publicinbox $inbox]
     $name = $value

to

       ($inbox :$name $value ...)

The one exception is :coderepo.  In public-inbox's configuration, the value of publicinbox.$inbox.coderepo points to another configuration option, coderepo.$value.dir, which in turn points to a repository's git directory.  The :coderepo of piem-inboxes, however, should be set to the working tree, so :coderepo is derived from the value of coderepo.$value.dir, stripping a trailing /.git if present.

Note that piem-merged-inboxes reads from the public-inbox configuration once, generates the merged set of inboxes, and then caches the result.  If you change piem-inboxes outside the customize interface or change public-inbox's configuration, you need to call the command piem-clear-merged-inboxes to clear the cache.

Enabling integration libraries
==============================

With inboxes defined, the next step is to enable minor modes that teach particular Emacs modes to link a buffer with a registered inbox.  piem currently has libraries to support

  * EWW

  * Elfeed

  * Gnus

  * Notmuch

  * Rmail

For example, if you use notmuch.el to read your mail, you can add support for applying patches from a Notmuch message buffer by enabling piem-notmuch-mode (see Applying patches):

     (piem-notmuch-mode 1)

Help adding support for other modes, especially other mail clients, is welcome.

Applying patches
****************

With piem-inboxes configured and appropriate integration libraries enabled, a buffer that can be linked to an inbox can be mapped to a code repository.  When reading a message in a notmuch-show-mode buffer, for example, the list ID can be used to identify the inbox and thus the associated local code repository.

There are two commands for applying patches:

piem-am-create
     This command tries to extract a patch from the current Notmuch or Gnus message buffer and can handle an inline patch as well as one or more patch attachments.

piem-b4-am
     This command relies on the b4 command-line tool to do more sophisticated processing of the full thread (e.g., pulling out the latest reroll of a series) to generate an mbox that can be fed to git am.  It is only compatible with inline patches.

Applying patches contained in a message
=======================================

M-x piem-am-create RET BRANCH RET BASE
     Apply the patch or patches in the current buffer to the associated code repository.  Before applying, checkout a new branch BRANCH starting at BASE.

You'll be queried for the name of the new branch.  The default name offered is generated by piem-name-branch-who-what-v, which uses the From: and Subject: headers to construct branch names like km/b4-short-subj__v3.  To use a different function to generate the completion default, configure piem-default-branch-function.

Next you'll be queried for the base to use as the starting point for the branch.  If the sender specified a base commit for the series, that will be provided as the default completion candidate.  Entering an empty base signals to use the current branch of the repository as the base.

Rather than applying the patches directly to the associated code repository, you can create a dedicated worktree by setting piem-am-create-worktree to a non-nil value.  Giving a prefix argument to piem-am-create inverts the meaning of piem-am-create-worktree; that is, by default a prefix argument is useful if you generally prefer to work within the configured code repository but would like to trigger the one-off creation of a worktree for a particular call.

If that option is non-nil and Magit is loaded, piem uses Magit for some operations, particularly those that are user-facing.  This includes jumping to the Magit status buffer for a code repository after applying a patch.

Note that the piem-am-create command works only for buffers from which piem-am-ready-mbox can generate an am-ready mbox, which depends on the enabled integration libraries.  Currently piem-notmuch and piem-gnus implement the necessary functionality.

Using b4 to apply patches
=========================

b4 (https://git.kernel.org/pub/scm/utils/b4/b4.git) is a command-line tool for interacting with public-inbox archives.  While useful for public-inbox archives in general, it is written for Linux kernel development and focuses on the public-inbox archives hosted at https://lore.kernel.org.

It's a fast moving target at the moment, but some of its current capabilities include

  * downloading the mbox for a thread based on a given message ID

  * extracting patches from a thread's mbox that can be fed to git am

  * submitting and verifying cryptographic attestation for patches

  * fetching a pull request found in a message ID

  * generating a thanks email for patches

The second item is the focus for piem, though at least some degree of support for all of the above features will likely be added. The entry point to applying patches with b4 is the piem-b4-am transient.  (See transient for more information on using Transient.)

piem-b4-am offers the following actions:

a
M-x piem-b4-am-from-mid-create
     Generate or download a thread's mbox for the current buffer's message ID, process it into an am-ready mbox with b4, and then feed it to git am called within an associated Git repository.  If a message ID of the current buffer is not known (i.e. piem-mid returns nil), one is read from the caller.  The caller is also queried for the branch name and base, as described for piem-am-create (see Applying patches contained in a message).  And, as with piem-am-create, a worktree can be created by configuring piem-am-create-worktree to a non-nil value or by giving a prefix argument.

     To generate the input thread, first any functions in piem-mid-to-thread-functions are tried.  This allows for a thread to be retrieved from a local store (e.g., the Notmuch database).  If that fails, try to download the thread from the piem-inboxes URL associated with the current buffer, provided the current buffer's message ID matches the one piem-b4-am-from-mid-create was called with.  As a last resort, call b4 am without a local mbox to let it download the thread according to its own configuration.

i
M-x piem-b4-am-ready-from-mid
     Call b4 am with a given message ID.  This differs from piem-b4-am-from-mid-create in that it is a direct wrapper around a command-line call to b4 am.  The caller is always queried for the message ID, and the final product is an am-ready mbox.  b4 is responsible for downloading the thread, so the caller must point b4's configuration option b4.midmask to the appropriate public-inbox URL.

b
M-x piem-b4-am-ready-from-mbox
     Like piem-b4-am-ready-from-mid, but process a local mbox rather than identifying the thread based on the specified message ID.

Applying patches without a public-inbox archive
===============================================

Much of the functionality described in the previous sections can work even if messages aren't available in a public-inbox archive. piem-am-create and piem-b4-am-from-mid-create try to generate the am-ready mbox from a local source (e.g., via Notmuch or Gnus) before falling back to downloading the thread from a public-inbox archive.

Also, for those not working with public-inbox archives, it's worth checking out mailscripts (https://git.spwhitton.name/mailscripts/), a nice set of Debian-focused tools by Sean Whitton that provides, among other things, functionality for applying patch series, including b4-inspired patch extraction.

Miscellaneous functionality
***************************

Injecting messages into a Maildir directory
===========================================

public-inbox allows you to follow lists through several mechanisms (see public-inbox).  You may prefer different methods for different projects depending on things like how actively you are following the development and how high traffic the list is.  For a project you maintain, perhaps you want to receive every message as regular mail. For a project you actively follow and occasionally contribute to, you may prefer to not clutter your local mail store and instead follow via read-only NNTP or IMAP in Gnus (which may or may not be your MUA).  And for a project you're new to or are digging into for a particular reason, HTTP via EWW may be all you need.

Depending on your mail setup, a problem with this approach is that it can be inconvenient to start participating in a thread that you aren't reading in your regular MUA (e.g., if you use notmuch.el to read your regular mail but are following a project via NNTP in Gnus).  In this case, you can use the command piem-inject-thread-into-maildir to move the thread's messages into a local Maildir directory specified by the current inbox's :maildir value in piem-inboxes, falling back to piem-maildir-directory.  By default the command downloads the entire thread for the message ID associated with the current buffer.  A prefix argument restricts the download to only the message.

After the messages are injected, each function in piem-after-mail-injection-functions is called with the message ID that was used to identify the thread.  This can be used to pop to the message in your mail client.  For example, Notmuch users may want something like this:

     (defun my/notmuch-new-and-show (mid)
       (message "Running notmuch new")
       (call-process notmuch-command nil nil nil "new")
       (notmuch-show (concat "id:"  mid)))

     (add-hook 'piem-after-mail-injection-functions
               #'my/notmuch-new-and-show)

To prevent duplicate messages from being written on subsequent calls to piem-inject-thread-into-maildir, you can set piem-mail-injection-skipif-predicate to a function that returns non-nil if a message ID is known and should be skipped.  For Notmuch, piem-notmuch provides a function that works for this purpose, piem-notmuch-known-mid-p:

     (setq piem-mail-injection-skipif-predicate
           #'piem-notmuch-known-mid-p)

Copying public-inbox URLs
=========================

When referring to a message from a public-inbox archive, a common format to use is a URL that points to a specific archive and ends with /$INBOX/$MESSAGE_ID, e.g., https://public-inbox.org/meta/20190108015420.GA28903@dcvr. Calling piem-copy-mid-url (available in the piem-dispatch transient) constructs such a URL, using the message ID and inbox asscociated with the current buffer, and then copies the URL to the kill ring.  When a prefix agument is given, browse-url is called after copying the URL.

Note that EWW works nicely with public-inbox's HTTP interface.  If you'd prefer it to be invoked even though it's not your default browser (as configured by browse-url-browser-function), you can set piem-browse-url-browser-function to eww-browse-url.

For notmuch.el users, there's an additional entry point for copying public-inbox URLs: enabling piem-notmuch-mode adds a ``piem'' candidate to archives offered by notmuch-show-stash-mlarchive-link and notmuch-show-stash-mlarchive-link-and-go.

Contributing
************

Patches, bug reports, and other feedback are welcome.  Please send a plain-text email to piem@inbox.kyleam.com.  Messages that include this address are public and available as public-inbox archives at https://inbox.kyleam.com/piem.  Note that this is not a mailing list.  Updates can be followed through one of public-inbox's pull methods (see public-inbox).  This means it is particularly important to not drop participants when replying.

You can, unsurprisingly, use piem to work on piem by adding an entry like this to piem-inboxes.

     ("piem"
      :coderepo "<path/to/local/clone>"
      :address "piem@inbox.kyleam.com"
      :url "https://inbox.kyleam.com/piem/")

The source repository is available at https://git.kyleam.com/piem. Here are some guidelines for sending patches:

  * Please send patches inline rather than as attachments.

     If you're using git send-email, you may want to set sendemail.to to piem@inbox.kyleam.com in your local repository.

  * Specify the base commit.

     This can be done via the --base= option of git format-patch or by configuring format.useAutoBase.

  * Keep rerolls in the same thread.

     In general, prefer to keep iterations of a patch series in the same thread, labeling rerolls with an appropriate version.

  * Consider adding a range-diff to the cover letter of rerolls.

     For a multi-patch series, git range-diff can often provide a nice overview of the changes between the previous and last iteration.  Note that git format-patch has a --range-diff convenience option.

Related projects and tools
**************************

Here's a (short and incomplete) list of tools that you, as someone that cared to look at piem's manual, may be interested in---things that either have some connection to public-inbox or to using an email-based development workflow in Emacs.  Corrections and additions are welcome.

  * git-email provides an Emacs interface for preparing and sending patches.

     https://sr.ht/~yoctocell/git-email/

  * git range-diff is a built-in Git command for comparing iterations of a patch series.  If you use Magit, the magit-tbdiff (https://github.com/magit/magit-tbdiff) extension provides an interface for git range-diff.

     https://kernel.org/pub/software/scm/git/docs/git-range-diff.html

  * grokmirror enables efficient replication of large Git repository collections.  The HTTP interface for public-inbox exposes a grokmirror-compatible manifest.js.gz endpoint.

     https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git

  * mailscripts, mentioned earlier in the manual (see Applying patches without a public-inbox archive), is a ``collection of scripts for manipulating e-mail on Debian'', including b4-inspired patch extraction and an Emacs interface for some of the functionality.

     https://git.spwhitton.name/mailscripts/

A list of more public-inbox-related tools is at https://public-inbox.org/clients.txt.

GNU Free Documentation License
******************************

     [file fdl-1.3.texi not included]