mu4e Hacks

No matter what you take away from this article as the reader, I’d like to note first and foremost, that although this article is borne of frustration, there is no disrespect meant towards the authors and maintainers of mu4e.

In general we (myself and the authors and maintainers of mu4e) suffer from a difference of opinion on what functionality is sensible.

It’s also easily notable that despite my frustration, mu4e is plain awesome. I’m still using it presently despite it needing a hack to conform to my sensibilities. I would in no way want any reader to take away that they should not contribute to, or use, mu4e.

To me, it’s the best way to manage and organize email. Integrations with Emacs Org Mode is a huge plus as well. There are other excellent email management software that integrates with Emacs as well. What sets mu4e apart for me is as follows:

• It relies on IMAP as the source of truth.

  This means that mu4e operates the exact same at any machine that clones an IMAP store without needs to do between machine syncs.

• It searches speedily through emails.

  The manner in how this is done is of no matter between different email management software. I’d probably be happy with naive implementations capable of doing such. The main key is that searches need to be fast. I doubt clients that depend on IMAP or what amounts to file grepping could ever match mu4e’s speed in this regard.

• It aids in the manipulation of a maildir structure.

  It allows for movement of emails between folders (which mirror their IMAP counterparts depedent upon synchronization mechanisms used separately). This also includes deletion of email.

Now, there are other email management software that qualify nearly. I’ve actually tried a few (from graphical, to TUI, to those similar to mu4e). I believe that in the end the way one manage’s their email devolves to what is the best fit for their workflow and what their opinion is on how they want the data organized. There are some systems that require manual synchronization of local tags (gmail-esque) and treat their tags as the source of truth, whilst not supporting movement/deletion in any manner.¹ While I’ve tried the one linked in the footnote, and liked it in general, at the time it’s Emacs interface was found lacking from my own personal standpoint. I moved to mu4e and then realized I would actually prefer IMAP be the source of truth to more easily facilitate managing email via different systems as necessary (such as phones) and at multiple devices (work/home computers) without needing to figure out a synchronization mechanism.

The author of mu4e has a short blurb that gives an idea in a nutshell.

mu/mu4e is an e-mail search engine (mu) and an emacs-based e-mail client (mu4e) written on top of it.

– Dirk-Jan C. Binnema

The only feature I consider missing and sensible to have, but differ in opinion with the author of mu4e, is the capability to list all maildirs in the main view.

In a lot of clients, web or otherwise, you have a list of folders, with each containing mail. When syncing email, dependent upon configuration, one would likely keep the organization structure available at the IMAP server storing the emails.

mu4e, out of the box, has no way to list those in the main view, the first thing you likely see when you open mu4e up in Emacs. It only allows a list of folders you specify manually. Curating and maintaing a list of those manually would be an exceedingly daunting task if you’re a heavy user of folders or psuedo-folders (tags) with your email.

mu4e does offer ways to navigate to those folders, even if it doesn’t outright list them on the main view.

This post will show the history of this feature’s absence and other forms that might have sufficed, along with exemplifying the ease with which it could be added. The only reason it is not added is because the author of mu4e does not like this feature at all, even though it would be entirely opt in.

A few years ago, I sat down, in front of mu4e, having just reconfigured it more to my liking, and wondered, how can I get a list of my maildirs to display in the main view.

There was a extension to mu4e, mu4e-maildirs-extension², that added the feature. Sadly, this project fell to the wayside as mu4e increased it’s version numbers and changed it’s underlying IPC protocols. mu4e-maildirs-extension was also sluggish. It has recently (by a loose definition of recently) picked back up development efforts but still has compatibility issues with the latest stable version of mu4e.³

As mu4e-maildirs-extension fell to the wayside, no longer compatible with mu4e unless you were willing to downgraide your mu4e version, I found myself needing a solution. Even the maintainer of this extension basically says to use mu4e builtin functionality.⁴

One of the key things about this feature is it lists a count like 1000/2000 of unread mail and all messages.

At the time, mu4e had recently included functionality that afforded parts of this feature⁵, but there was a major issue. The author wanted to limit this feature to only displaying the directories listed in a configuration variable for mu4e.⁶

Soon, I submitted an issue⁷, followed up by a PR that added the feature.⁸ The idea behind the PR was that if we support keyless maildir shortcuts, which can point to any arbitrary maildir, we could then have something in our config that generates a the list of maildir shortcuts from the maildirs mu4e knows about. This would then allow for my use case to work with no hacks to mu4e to fix it to support doing so, were this PR to be merged. Sadly, I caught the tail end of development on a stable branch as the developers were prepping to move to a new stable branch with some code restructuring going on and my PR ended up having merge conflicts. That said, I stuck with the patched version I made of mu4e for years because I didn’t want to have to figure all this out again.

The reasoning behind having maildir shortcuts without keys is because there are only 26 characters in the english alphabet, and what do you do if you have more than 26 maildirs? Omit them? Better to have the key that launches a selection over maildirs be the shortcut for just any displayed maildir that is without a key I thought.

Then mid 2022, a contributor picked up the task to bring this about.⁹ It brough compatibility with the newly restructured mu4e portion of the code base and overall looked decent even if I had a few things to point out. I actually jumped in to review the PR and point out where I knew some difficulties might be present. I also would note that aside from the few fixes I recommended to make things visually “accurate”, this committer’s PR was overall the better PR. It introduced a new feature allowing keys to be a custom string. In the end, this committer’s PR was merged. I held tight to my own custom version for a bit longer however.

Recently, I decided, it was time to upgrade a lot of things, mu4e included. I had forgotten about all these things and went to github and checked what the status was, learning that PR#2292 had been merged. I thought all was good, worse case scenario, I’d just generate strings of random letters to solve the more than 26 maildirs problem since that was merged. So I update, read documentation and see documentation disagrees with what the commits in the PR put forth. I was soon to be frustrated, finding out that the changes allowing either keyless or string keys was changed back to be single key in the latest version somewhere along the line. I was fairly upset, and set out on a way to resolve things and have my necessitated feature.

First thing I do is… go back to the comment thread on my issue and look at suggestions from the author himself. The author had this idea (quoted from¹⁰):

I’ve tried allowing keyless maildirs/bookmarks but I don’t like it, so I have to decline this, sorry… If even all alphanumeric keys are not enough, note that they don’t need to be unique (as you mention).

I’m sad to say that this is simply untrue, it was untrue then, and is now.

Even with it all blurred to death, you can see it is overfills the minibuffer with what is basically garbage if you follow the suggestion of using the same key as they are not required to be unique.

I even already explained this in a prior comment that doing so resulted in an abomination of a jumptable display in the minibuffer which is why I had explored the changes I made in my PR. Quoted below¹¹

It would appear if :key is set to ?o all is fine, it’ll still do the same selection process o is supposed to do. So supposedly I could just set all the keys to ’o’ and tolerate the long listing of jumpables that aren’t actually jumpable. Not sure if it’s worth it to anyone to have it displayed correctly as far as jumpability goes, or the Maildirs listing (keep it from showing “jo” as a key sequence for all those maildirs).

That said, if non-unique keys is a bug, were it ever to get fixed I’d be back up the creek having to figure out a good way to go about this.

I see 2 ways to handle the 2 issues I noted (display, jumpability):

• Redefine mu4e~main-maildirs to take care of not displaying a key sequence
• Rebinding the binding for mu4e~headers-jump-to-maildir binding to another function (a hydra) that can read the maildir shortcuts itself and generate an actual list of things with keys, and finally an ’o’ option for other, with a check for uniqueness, along with a refusal to allow the binding of ’o’ to be mapped into the hydra. (All this and maybe it would make better sense to just redo mu4e~headers-jump-to-maildir to be honest.)

The reason a key must be defined is because of the concat operations here in mu4e-main.el. Probably not difficult to have key be either key or a string with one space in it " ".

As for maildir selection, looking at mu4e-utils.el mlist just needs to be built to not include ones without :key or the mapconcat lambda needs to just return an empty string for anything not having a :key. I’m not sure which approach will work (or maybe both will work, in which case the latter I think would work better) due to my lack of expertise in elisp.

The author’s idea of a solution simply would not work. I also came to realize he truly dislikes the keyless shortcuts idea in general. Our opinions differ on this.

I sat down, and got to hacking, found the key areas to fix mu4e to be suitable for myself, and came up with the following results.

(defun mu4e--main-items (item-type max-length)
  "Produce the string with menu-items for ITEM-TYPE.
ITEM-TYPE is a symbol, either `bookmarks' or `maildirs'.

MAX-LENGTH is the maximum length of the item titles; this is used
for aligning them."
  (mapconcat
   (lambda (item)
     (cl-destructuring-bind
         (&key hide name key favorite query &allow-other-keys) item
       ;; hide items explicitly hidden, without key or wrong category.
       (if hide
           ""
         (let ((item-info
                ;; note, we have a function for the binding,
                ;; and perhaps a different one for the lambda.
                (cond
                 ((eq item-type 'maildirs)
                  (list #'mu4e-search-maildir #'mu4e-search
                        query))
                 ((eq item-type 'bookmarks)
                  (list #'mu4e-search-bookmark #'mu4e-search-bookmark
                        (mu4e-get-bookmark-query key)))
                 (t
                  (mu4e-error "Invalid item-type %s" item-type)))))
           (concat
            (mu4e--main-action
             ;; main title
             (format "\t* [@] %s "
                     (propertize
                      name
                      'face (if favorite 'mu4e-header-key-face nil)
                      'help-echo query))
             ;; function to call when activated
             (lambda () (interactive)
               (funcall (nth 1 item-info)
                        (nth 2 item-info)))
             ;; custom key binding string
             (concat (mu4e-key-description (nth 0 item-info))
                     (if (null key) "o" (string key))))
            ;; counts
            (format "%s%s\n"
                    (make-string (- max-length (string-width name)) ?\s)
                    (mu4e--query-item-display-counts item)))))))
   ;; only items which have a single-character :key
   (mu4e-query-items item-type) ""))

I considered submitting yet another PR for keyless maildirs, but I get the feeling that it’ll never make it in, or like the other PR that did get merged, the feature would disappear yet again in a future version. I do believe this feature would be good and maintainable, and for those who don’t want to use it… just populate the :key field in the maildir shortcuts variable. It’s an entirely opt-in process. Having said that, the code above is for those who don’t mind in essence monkey patching mu4e to have the feature. I’ve given up on the idea of mu4e ever including such a feature. I won’t submit a PR unless requested to do so. I just simply feel it will be a waste of my time, not to mention the project’s author, to keep pressing at something he simply does not care for. If you diff my version of the function, it’s an extremely small change.

--- v1  2023-09-20 12:50:30.631111000 -0500
+++ v2  2023-09-20 12:51:29.094897000 -0500
@@ -36,10 +36,12 @@
                (funcall (nth 1 item-info)
                         (nth 2 item-info)))
              ;; custom key binding string
-             (concat (mu4e-key-description (nth 0 item-info)) (string key)))
+             (concat (mu4e-key-description (nth 0 item-info))
+                     (if (null key) "o" (string key))))
             ;; counts
             (format "%s%s\n"
                     (make-string (- max-length (string-width name)) ?\s)
                     (mu4e--query-item-display-counts item)))))))
    ;; only items which have a single-character :key
-   (mu4e-filter-single-key (mu4e-query-items item-type)) ""))
+   (mu4e-query-items item-type) ""))
+

Finally, if you’d like to list all maildirs in the mu4e main view, you can use the below to generate your maildir shortcuts.

  (defun update-maildirs ()
    "When this is called it'll patch `mu4e-maildir-shortcuts' with
all maildirs known to mu4e without a shortcut key assigned,
keeping manually defined ones at the top."
    (let (shortcuts exclusions)
      (dolist (sc mu4e-maildir-shortcuts)
        (push (plist-get sc :maildir) exclusions)
        (push sc shortcuts))
      (dolist (md (mu4e-get-maildirs))
        (unless (-contains? exclusions md)
          (push `(:maildir ,md :key ?o) shortcuts)))
      (setq mu4e-maildir-shortcuts
            (sort shortcuts #'shortcut-comparison))))

  (defun shortcut-comparison (a b)
    (let ((mda (plist-get a :maildir))
          (mdb (plist-get b :maildir))
          (ka  (plist-get a :key))
          (kb  (plist-get b :key)))
      (cond ((and ka (not kb)) ka)
            ((and kb (not ka)) kb)
            (t (string< mda mdb)))))

I guess I will be forever patching mu4e internals in my config, but at least that remains possible so as to remedy the lack of this feature.