Feature Request: Mox for Delta Chat

[s0ph0s-dog]

Hello!

I am working on a hobby project to create an implementation of a chatmail server that is easier to deploy and more user-friendly. My goal is to make it easier for people to use Delta Chat, and hopefully reduce overall reliance on centralized chat platforms like WhatsApp, Telegram, and the like. I was originally going to use Maddy as the email server for this project, but its developer seems to have been inactive on GitHub for a long time. I've stumbled across Mox though, and it seems like a better-maintained choice—one to which I'd be happy to contribute!

In my (limited) testing so far, Mox seems to work well as an email server for Delta Chat:

• It delivers mail relatively quickly
• It implements CONDSTORE (for read state syncing between devices)
• It implements quotas

Chatmail servers also need to do several other things though. Some of them I already have plans to implement on my own, because they don't require any cooperation from the mail server (sign-up web page generation, a P2P data relay, etc.). Here are the things I'd like to discuss adding directly to Mox, because directly building them in might make the most sense:

• Milter support: In order to prevent abuse, chatmail servers need to reject any outgoing mail that isn't PGP encrypted. I've already written a milter plugin for this that works with Maddy, so milter support in Mox should enable that to work as-is.
• External authentication: New accounts on chatmail servers are created automatically when the first login is attempted. I don't see a SASL mechanism (or similar) that I can use to implement external authentication, so that probably requires some work.
• Data retention time limits: Chatmail servers delete unused accounts after a period of time (typically 90 days) and delete emails after a shorter period of time (typically 30 days). I'm not sure how bstore handles concurrent access by different processes, so I'm unsure whether this functionality can be built separately from Mox, or whether it needs to be built in.
• Delivery speed: Email delivery latency to local mailboxes with Mox seems to be slower than Maddy, but only by a few hundred milliseconds (with the junk filter turned off, by very unscientific measurements). I think profiling the code and optimizing this would be a fun challenge!
• TLS ALPN Multiplexing: Chatmail servers offer multiplexing of IMAP, SMTP, and HTTPS over port 443 using TLS ALPN to help users behind aggressive firewalls continue to use Delta Chat. I have some (uncommitted) sketch code that partially implements this alongside Maddy, but it might be better to build the ALPN proxy directly into Mox, given that Mox also runs a web server.

I'm quite happy to help contribute to the development of all of these features, if the help would be welcome! Is this sort of thing something you'd like to see Mox used for? If so, would you be willing to review my pull requests and offer guidance to me as I work on implementing the parts of these that are inside Mox?

Sub-Issues

• https://github.com/mjl-/mox/issues/280
• https://github.com/mjl-/mox/issues/281
• https://github.com/mjl-/mox/issues/282
• https://github.com/mjl-/mox/issues/288
• https://github.com/mjl-/mox/issues/289
• https://github.com/mjl-/mox/issues/290
• https://github.com/mjl-/mox/issues/291
• https://github.com/mjl-/mox/issues/292

[mjl-]

Hi, interesting topic and there certainly is overlap in goals.

It's a goal of mox itself to be easy to install and run. Automating the
creation of dns records, and maintaining them, is on the short list.

About the topics:

• Milter support: Others have asked milter support too. It could be added.
  But is relatively intrusive, with hooks through the smtp process. Especially
  full milter support, which requires being able to change messages
  (add/remove/rewrite headers), which mox cannot do at the moment. From your
  description, you would only run the milter for submission connections? Have
  you considered adding the check for pgp-encryptedness of submitted messages
  directly to mox, without milter? Part of the philosophy of mox is to not
  require many tools to get a running mail server. Mox could get a
  chatmail-mode (either inside mox, or maintained as a fork) that enables
  this functionality.
• External authentication: Mox indeed handles accounts and authentication
  itself. We could add integration with an external auth provider. I think
  it's been suggested before. It's not great for the general case for mox,
  because modern auth mechanisms may not work properly (eg
  scram-sha-256-plus). Another option could be to add autoprovisioning to mox
  itself? Again enabled with the chatmail mode. Are there multiple chatmail
  clients, or is deltachat the only one (in active use)? I presume it does
  plain auth during signup. Will it use better auth mechanisms in later
  connections (eg scram-sha-256-plus)?
  Btw, the mox admin interface is implemented with a simple web api. You
  could reuse that for creating accounts if signups through a website instead
  of signup-through-login are an option.
• Data retention: Bstore database indeed cannot be opened by multiple
  processes. So the cleanup would have to be done inside mox. Could be
  implemented in the chatmail mode.
• Delivery speed: Will certainly be interesting to see why mox is slower
  and ways to improve it!
• TLS ALPN Multiplexing: Indeed makes sense to have this done inside mox. For
  HTTPS, we would have to change the TLS listener to look at the nextproto
  chosen and possibly call a different protocol handler.

The chatmail page mentions push notifications. Mox does implement IMAP IDLE,
but does not send push notifications. Will that be an issue? I don't know
exactly what the latest state is around push notifications on IMAP and
iOS/Android. I've seen a mechanism with a custom imap extension, and I've may
have heard about JMAP push functionality being reused in some way.

I'll be happy to answer questions and the mox code, and give pointers & help
think about approaches. Can also be over email/call etc.

[s0ph0s-dog]

Awesome! I'm glad to hear that this sort of thing would be welcome. I'll reply point-by-point (with some minor re-ordering) below:

It's a goal of mox itself to be easy to install and run. Automating the creation of dns records, and maintaining them, is on the short list.

Oooh, cool! That's a nice bit of overlap :)

Milter support

Others have asked milter support too. It could be added.
But is relatively intrusive, with hooks through the smtp process. Especially
full milter support, which requires being able to change messages
(add/remove/rewrite headers), which mox cannot do at the moment.

Maddy avoids the complexities of full milter support by… also not having it, lol. (See the first three paragraphs of Maddy's documentation of their milter support.) That said, even just being able to accept/reject emails via a standardized interface is a useful feature.

From your description, you would only run the milter for submission connections?

Yep, that's correct.

Have you considered adding the check for pgp-encryptedness of submitted messages
directly to mox, without milter? Part of the philosophy of mox is to not
require many tools to get a running mail server. Mox could get a
chatmail-mode (either inside mox, or maintained as a fork) that enables
this functionality.

This is an interesting possibility. I am initially hesitant to go all-in on building things directly into Mox, because, from Mox's perspective, all of the chatmail stuff is a weird growth stuck to the side of an email server. I wouldn't want to write all of this stuff, upstream it into Mox, and then (heaven forbid) get hit by a bus and leave you with the maintenance burden for code that you may or may not even want to use. If the chatmail mechanisms are separated through standardized interfaces (like milter, Dovecot SASL, etc.), Mox can make good use of those interfaces regardless of what kind of thing is connected to them!

(For the record, I have no plans to disappear—I just want to make realistic plans for the long-term maintenance of the things I build! I have learned the hard way that if I don't make those plans up front, I end up building things that suck to maintain, and which I eventually give up on.)

The major wrinkle in building a standardized interface for all of the pieces here is that I don't know of any existing mechanism for providing external tools access to all of the emails for every account—except for maildir, which would be an enormous rewrite and would sacrifice many performance benefits gained by having a database. I don't think that maildir would be a good idea. I'll describe how I was planning to implement the interfaces for data cleanup and push notifications below.

External Authentication

Are there multiple chatmail clients, or is deltachat the only one (in active use)?

As far as I know, all of the Delta Chat clients (including the forks and third-party ones) use a core library written by the main developers called deltachat-core-rust.

I presume it does plain auth during signup. Will it use better auth mechanisms in later
connections (eg scram-sha-256-plus)?

The deltachat-core library only uses XOAUTH2 or PLAIN/LOGIN SASL mechanisms for authenticating to the server (IMAP code, SMTP code). I don't know for sure, but I think that XOAUTH2 is only used when talking to one of the major email providers (Gmail, Office 365, etc.), because they don't allow anything else. None of the code in the main chatmail server sets up an OAuth identity provider.

We could add integration with an external auth provider. I think
it's been suggested before. It's not great for the general case for mox,
because modern auth mechanisms may not work properly (eg
scram-sha-256-plus).

Maddy accomplishes this by allowing server administrators to delegate authentication to external modules using the Dovecot SASL protocol. Here's the documentation, but I can't pinpoint exactly where in the code this happens.

Mox indeed handles accounts and authentication
itself. […] Another option could be to add autoprovisioning to mox itself? […]
Btw, the mox admin interface is implemented with a simple web api. You
could reuse that for creating accounts if signups through a website instead
of signup-through-login are an option.

All of these could absolutely work! The upstream chatmail server has a /new HTTP API which the client uses to ask the server to hand it a random username and random password. The behavior of this endpoint could quite easily be modified to use the Mox admin API to create the account. That said, I've been trying to stick as close to the same behavior as the upstream, in order to minimize the odds of weird bugs. (Additionally, the trust-on-first-use mechanism prevents broken web crawlers or a miscreant running cURL in a loop from creating thousands of accounts in seconds. Setting up an IMAP connection is a much higher barrier than HTTP, simply by virtue of obscurity.)

Data Retention

Bstore database indeed cannot be opened by multiple
processes. So the cleanup would have to be done inside mox. Could be
implemented in the chatmail mode.

This makes sense. When I was figuring out how to do this with Maddy, I planned to read the message metadata out of the database. It's backed by SQLite (or PostgreSQL), so that is a very easy way to look at message dates and clean up old ones. There was also a "last login" field in the user table that I could use to erase old, unused accounts. SQLite and PostgreSQL both support concurrent access by multiple processes though, so the right choice for Mox might be to build it in.

Delivery Speed

Will certainly be interesting to see why mox is slower and ways to improve it!

I poked at this with the /debug/pprof tools last night, but I haven't figured out anything useful yet. I probably need to spend more time reading the code to understand what the profiling data means :P

TLS ALPN Multiplexing

Indeed makes sense to have this done inside mox. For
HTTPS, we would have to change the TLS listener to look at the nextproto
chosen and possibly call a different protocol handler.

This sounds good—plus it should hopefully mean that the rate limiting code needs no changes!

Push Notifications

The chatmail page mentions push notifications. Mox does implement IMAP IDLE, but does not send push notifications.

The deltachat-core-rust code uses IMAP IDLE when it's running in the foreground, but it also uses APNS/Play push notifications when in the background on mobile.

Will that be an issue? I don't know exactly what the latest state is around push notifications on IMAP and iOS/Android. I've seen a mechanism with a custom imap extension, and I've may have heard about JMAP push functionality being reused in some way.

It shouldn't be an issue—the Delta Chat folks have solved the complexities around IMAP push notifications by stepping sideways around them. The upstream chatmail server sends a "hey there's a new message for asdfghjkl@chatmail-server.example" notification via HTTP to notifications.delta.chat, which maintains a registry of APNS/Play push notification tokens and chatmail email addresses, in order to deliver push notifications. I didn't mention this in the initial list because the solution is approximately the same as the data retention code. Based on what you mentioned above, the best choice seems to be a module which is enabled as part of "chatmail mode" that watches for new emails arriving and makes the required HTTP request to notifications.delta.chat.

When planning to do this with Maddy, I intended to use the IMAP Filter mechanism to send the HTTP request every time a new email was delivered. The "filter" wouldn't actually do any filtering. Instead it would write the email address for the account to a pipe, which the rest of the daemon code could read and then start an HTTP request retry loop. This plan was definitely a hack, and building it in would certainly be cleaner.

I'll be happy to answer questions and the mox code, and give pointers & help think about approaches. Can also be over email/call etc.

Fantastic, thank you! I'll try to ask major planning-related questions here, so that other folks can get involved if they'd like (such as the upstream chatmail server developers, who have expressed interest in my side project), and send more nitty-gritty questions about Mox's internals by email to avoid sidetracking the architectural stuff.

[mjl-]

Good to hear your consideration about adding chatmail-specific vs generic functionality, and the potential vanishing vs finishing. I'm not sure yet what the right approach is. If we end up with many config options that are specific for chatmail, than a single mode would probably have been simpler. It looks like some chatmail-specific thing (notifications and cleaning up) have to be in mox anyway. But some functionality would be good to have in general.

For milter support, indeed a simplified version may be enough. I think milters are intended to be run during each step in the smtp session. But perhaps we can just do the whole exchange at the end. Would simplify implementation quite a bit, I wasn't looking forward to adding it throughout the smtpserver code.

For authentication, if we were to check with another sasl server, we would probably require "plain" as only mechanism. The announced mechanisms would have to be trimmed when such a config option is active. The external server would call back to mox to create the account & email address.
The alternative of building this into mox would be to change the current calls to store.OpenEmailAuth with a new mox.OpenEmailAuth that creates new accounts if they don't exist yet (if autoaccounts are enabled), and calls store.OpenEmailAuth otherwise.

For data retention, a periodic goroutine that goes through all accounts and removes old message is probably easiest. Each account has its own message database that we have to go through. For removing stale accounts, we don't have "last login". The store.LoginSession type is only for web-based logins (from memory). I have been wanting to keep track of more details of active and recent sessions, this may be a good trigger to implement that.

For push notifications, indeed there is no hook for running custom code when a message is delivered...

[s0ph0s-dog]

I've been thinking about this over the holidays. If someone wants to configure Mox to be a chatmail server from scratch, they'd need to:

• Add EnabledOnHTTPS: true to two places in mox.conf
• Enable notifications with an as-yet-unspecified configuration change in domains.conf
• Enable automatic account creation with an as-yet-unspecified configuration change in domains.conf
• Enable automatic account clean-up with an as-yet-unspecified configuration change in domains.conf
• Find a copy of and set up my outgoing-mails-with-PGP-only milter, then add that in one of the configuration files
• Add a whole section to domains.conf which sets up a static webpage for serving the "make account" link
• Find a copy of and set up my generate-random-credentials tool, and add a reverse-proxy configuration to relay requests to it from Mox

This is rather a lot of steps. I'm starting to see your argument for a "chatmail mode" configuration option. Let's say we do build it that way—would the "chatmail mode" functionality be an argument to mox quickstart? How would you imagine that working?

[mjl-]

This is rather a lot of steps. I'm starting to see your argument for a "chatmail mode" configuration option. Let's say we do build it that way—would the "chatmail mode" functionality be an argument to mox quickstart? How would you imagine that working?

Yeah, it makes sense as a parameter to the quickstart. And perhaps we can just make add a single "Chatmail" config section to mox.conf or domains.conf that applies to the entire instance (I don't think you want to mix chatmail-functionality with "regular" operation?). Mox.conf has config values that don't change at runtime, while domains.conf is for settings that can change at runtime. I don't think these config settings will be changed at runtime (without a restart?). Then a single section in mox.conf may be good enough.

[mjl-]

@s0ph0s-dog btw, one more thing i noticed about the autoconfig change is that the port 443 endpoint uses password-cleartext while the first and standard port endpoint uses password-encrypted. i suppose that's because of account autoregistration on port 443? or should it also default to password-encrypted on port 443?

[s0ph0s-dog]

the port 443 endpoint uses password-cleartext while the first and standard port endpoint uses password-encrypted. i suppose that's because of account autoregistration on port 443?

I chose to do that because it matched what was in the upstream autoconfig data, and because that's the only mechanism Delta Chat really uses. I don't think there are any security risks associated with putting that in the autoconfig, because the connection is necessarily over TLS, so it won't expose plaintext passwords to eavesdroppers.

Now that I've gone to check, the Delta Chat autoconfig code doesn't even parse the authentication element. I think it's fine to change that if you'd prefer it be password-encrypted!

And perhaps we can just make add a single "Chatmail" config section to mox.conf or domains.conf that applies to the entire instance.

I think that seems like an attractive choice! One potential wrinkle: does it make sense to have data/account retention settings configured globally, rather than per domain? (Should it be configurable in both places?) That functionality seems like it would have applicability beyond chatmail, and people deploying Mox might find it useful to enable/configure that by itself.

(I don't think you want to mix chatmail-functionality with "regular" operation?)

I can imagine a situation where someone might want to have a chatmail server just for friends or co-workers and which doesn't delete accounts automatically, but it seems like more of a stretch to imagine someone with multiple domains in the same Mox instance with different levels of chatmail functionality enabled. Or, to say it differently, I think someone is likely to want to change the amount of chatmail functionality enabled on all of their domains, but less likely to want differing chatmail settings on multiple domains. I think it would be prudent to design things with an eye towards maybe doing that in the future, but it doesn't make sense to build it that way to start with.

I don't think these config settings will be changed at runtime (without a restart?).

Like I mentioned above, I think the data retention stuff might be changed at runtime and in the web admin interface (probably in an effort to deal with running out of disk space), but the rest of it could quite reasonably be done with a server restart.

[s0ph0s-dog]

Oh also I don't think I've said this yet: my goal currently is to finish figuring out the shape of what needs to be built by discussing it in this issue. Then, we can break it down into sub-issues for each major piece of development work. Now that I've had some exposure to Mox's codebase and you've seen what my patches look like, we can use that experience to decide who should tackle each sub-issue. I'm willing to do basically all of the development work, but I won't mind if you'd prefer to reserve any of the development tasks for yourself!

[mjl-]

does it make sense to have data/account retention settings configured globally, rather than per domain? (Should it be configurable in both places?) That functionality seems like it would have applicability beyond chatmail, and people deploying Mox might find it useful to enable/configure that by itself.

I think someone is likely to want to change the amount of chatmail functionality enabled on all of their domains, but less likely to want differing chatmail settings on multiple domains. I think it would be prudent to design things with an eye towards maybe doing that in the future, but it doesn't make sense to build it that way to start with.

Configuring these settings "per domain" may be too tricky. Or at least, we
should probably at least configure them per account. An account can have
multiple addresses associated with it, also at different domains, and none of
them is the primary domain. If you can mix chatmail and non-chatmail domains
in a mox instance, which settings would apply? It will be confusing if only
some messages in an account would get deleted automatically. In practice, I
wouldn't expect anyone to want to mix chatmail and non-chatmail domains in a
single account, but when it's an option, the code will be confused and it
will be a source of trouble. If we store the auto-cleanup per account,
accounts from autosignups should get chatmail-mode enabled. But at least a
postmaster account that also receives dmarc/tls reports, should probably be a
non-chatmail account, not automatically being removed or its messages
removed. There may still be a flag per domain that says whether a domain is
enabled for auto-signup. So you could also have a non-chatmail domain on the
same server.

Some suggestions for the config:

We could have a top-level Chatmail settings struct in domains.conf, with the
retention settings:

• Enabled boolean
• Days of account inactivity before deletion.
• Days after initial delivery to mailbox to remove old messages.
• Whether to enable notifications (or URL to use for dealing with notifications).

In type "Account" (for domains.conf), we could add:

• A flag indicating chatmail is enabled. So the account and its messages may
  be automatically removed.
• In the future, we could add more options for per-account inactivity periods
  for account/message deletions, but as you said, the global settings will
  probably be enough for now. Likewise, at that time we could also add
  default settings to a domain, to use for accounts created for signups in
  that domain.

Enabling chatmail functionality can be dangerous to do for "regular"
installations. Because accounts and messages may be deleted automatically. I
think the code adding such "dangerous" operations should be guarded by
checking if chatmail is enabled (the Enabled flag in the top-level chatmail
settings struct), along with the per-account/domain check. And we should have
tests that check that those checks work and we can't accidentally trigger it
when not enabled. Assuming admins will typically setup their mox chatmail
instance with the quickstart, it would be enough to add a flag there to
enable it. We wouldn't have to show anything about chatmail in the admin web
interface if it isn't already enabled. Admins that want to enable it later on
would have to edit the domains.conf file manually. This should prevent any
confusion of the non-chatmail users (majority of users).

We could add a simple internal web handler for handling account autosignup. I
wouldn't want to add too much styling to it. The current web interfaces are
relatively spartan and without styling opinion. And they don't include
big/external frameworks, in part because the generated frontend files are
committed to the repository. So the question is whether that will be good
enough. If you want others to easily set up an account autosignup web
interface, we could add it. Then, if you want to have a fancier page, admins
can always choose to not use it and make a nicer looking web frontend on
their own. We could still make it possible to configure some text blocks for
display on the builtin frontend, so you can mention some contact info, rules,
policies, etc. A builtin handler could be just another one of the current
builtin services for the admin/account/webmail interfaces. You can configure
them in the webserver routes. If you want to add a builtin web interface for
account autosignup, we could have add a field to type Domain for domains.conf
that enables autosignup for that domain. The web interface would then show
that domain (or multiple, not sure how useful/common it is for chatmail
servers to have multiple domains) as option to signup for.

For implementing account/message cleanup, I would suggest we just start a
goroutine at startup that wakes up once a day to remove accounts/messages.

For removing accounts, we need to know about its last (successful) activity.
I mentioned before that I wanted to keep track of login attempts. I'll put
that high op the priority list (hope to get to it in the next few weeks, but
this week is busy, with FOSDEM coming up). I plan to add details like
user-agent (eg from browser, or from imap ID command), whether
success/failed, authentication mechanism, protocol (imap/smtp/web), ip
address, timestamp). And store limited history of various combinations. So
you can see both recent successful and failed login attempts, at various
protocols from various ips. Anyway, just the last successful login attempt
should be enough for automatically removing an account.

You can probably test most of the needed changes with mox localserve. My
approach would normally be to make quick & dirty changes for all of these
requirements, to find any blocking problems and to change the approach. And
then write it properly, and finally clean it up and write the tests. If you
don't know where to start for some of the topics, I can point to some places,
or I could write up a hacky diff that at least shows where I would make
changes. Indeed I think the first step is deciding the subtasks, and I would suggest making a quick code sketch of each before completing them individually.

[s0ph0s-dog]

Configuring these settings "per domain" may be too tricky. Or at least, we should probably at least configure them per account. […] There may still be a flag per domain that says whether a domain is enabled for auto-signup. So you could also have a non-chatmail domain on the same server.

Ah yep, that definitely makes more sense. I hadn't discovered that Mox supported a many-to-one address to account mapping yet! Also the postmaster account would need special treatment anyway, so making it an option definitely seems like the right choice.

We could have a top-level Chatmail settings struct in domains.conf, with the
retention settings […] In type "Account" (for domains.conf), we could add…

Yes, these lists seem reasonable. In the top level settings, I'd add:

• Notification URL (defaulting to https://notifications.delta.chat/notify if not specified)
• Block incoming unencrypted email (boolean, default false, outgoing would always be blocked when chatmail mode is enabled, this is a new option added to the upstream chatmail server)

Enabling chatmail functionality can be dangerous to do for "regular" installations. Because accounts and messages may be deleted automatically. I think the code adding such "dangerous" operations should be guarded by checking if chatmail is enabled (the Enabled flag in the top-level chatmail settings struct), along with the per-account/domain check. And we should have tests that check that those checks work and we can't accidentally trigger it when not enabled.

Yeah, I agree. Should it additionally cause a warning/error when an account has chatmail mode enabled, but the global chatmail flag is not enabled?

We could add a simple internal web handler for handling account autosignup. I wouldn't want to add too much styling to it. The current web interfaces are relatively spartan and without styling opinion. And they don't include big/external frameworks, in part because the generated frontend files are committed to the repository. So the question is whether that will be good enough.

I think it would be good enough! I'm plenty happy to stick to the existing web design style—this doesn't need to be a fancy Single-Page Application or anything like that. It needs a title, a QR code (I see Mox already includes a library for this), a dcaccount: link, a /new endpoint that accepts POST requests and returns a random email and password, and yeah, template string blocks for contact info and rules.

For implementing account/message cleanup, I would suggest we just start a goroutine at startup that wakes up once a day to remove accounts/messages.

Seems sensible, yeah. I think the notifications should be a similar goroutine started at startup, which gets notified through a channel whenever a message arrives, and takes care of retries.

For removing accounts, we need to know about its last (successful) activity. I mentioned before that I wanted to keep track of login attempts. […]

Yep! I'll build the other parts of the cleanup machinery and leave that part until last, so that you can implement the last login tracking.

I plan to add details like user-agent (eg from browser, or from imap ID command), whether success/failed, authentication mechanism, protocol (imap/smtp/web), ip address, timestamp). And store limited history of various combinations.

Would it be possible to restrict the data collected to just the last login timestamp divided by 86,400 (one day's worth of seconds) when chatmail mode is enabled? That's the smallest amount of information that's still useful to implement the cleanup functionality, and the upstream project does something similar to reduce the amount of information being stored.

You can probably test most of the needed changes with mox localserve. My approach would normally be to make quick & dirty changes for all of these requirements, to find any blocking problems and to change the approach. And then write it properly, and finally clean it up and write the tests.

Yeah, that's a good idea! "Build one to throw away," and all that.

---

…this week is busy, with FOSDEM coming up.

Ooh, have fun!

[s0ph0s-dog]

While double-checking some things above, I read through the upstream notifications code again. In the first few passes through it, I did not notice that the notifications server wants just a notification token, not an email address. I traced back to where that token comes from, and the client sets it using the IMAP SETMETADATA command (from RFC 5464), which it seems like Mox does not currently implement. I'm going to talk to the upstream project about whether this can be done a different way. Not only is "implement an entire IMAP extension RFC" a significant expansion in scope, but the current metadata keys include vendor.dovecot. This seems like a "we made it work" kind of solution, rather than a standardized one.

With that in mind, the current task list looks something like this:

• TLS ALPN support for thwarting aggressive firewalls
• Disallow incoming (from other servers) and outgoing (to other servers) unencrypted emails
• Automatic account creation on first log-in
• Sign-up webpage with QR code and configurable about/rules text
• Automatic data and account deletion after configured time interval
• (maybe) add (enough) IMAP METADATA extension support
• HTTP POST notifications to the configured notifications URL
• Benchmarking to optimize time between submission and delivery to an idle IMAP connection
• Automatic, ongoing DNS configuration for popular DNS providers

[mjl-]

Should it additionally cause a warning/error when an account has chatmail mode enabled, but the global chatmail flag is not enabled?

I think an error may be good, but not entirely sure. It can help prevent accidentally disabling chatmail mode, or making it seem an account/domain has it enabled while it's actually globally disabled.

Would it be possible to restrict the data collected to just the last login timestamp divided by 86,400 (one day's worth of seconds) when chatmail mode is enabled?

I had a feeling you would want to keep as little information as possible. Probably best to just store this information in a db file (the new auth.db) in a new type. It can be tracked separately from the more detailed login attempt information, which would be disabled for accounts in chatmail mode. I can add the field. Do you want to make a fork where I can make PRs to?

IMAP SETMETADATA

I can implement it, doesn't seem too complicated, haven't had a need for it yet. But it would be good to know if that key triggers some kind of non-standard functionality provided by dovecot.

[s0ph0s-dog]

I think an error may be good, but not entirely sure. It can help prevent accidentally disabling chatmail mode, or making it seem an account/domain has it enabled while it's actually globally disabled.

I can foresee frustration where someone tries to turn off chatmail mode and then needs to go turn off the setting on 300 accounts too, but I think "turning a chatmail server into a regular email server" is unlikely enough that it shouldn't be an issue. Error it is!

I had a feeling you would want to keep as little information as possible.

Oh no, I'm predictable :P

Probably best to just store this information in a db file (the new auth.db) in a new type. It can be tracked separately from the more detailed login attempt information, which would be disabled for accounts in chatmail mode. I can add the field. Do you want to make a fork where I can make PRs to?

That separation of data sounds reasonable to me! I've added you as a collaborator to my fork. I think it makes sense to use the main branch in my fork as the trunk for chatmail features. (Though that would break the ability to use GitHub's "related issues" feature to automatically close the sub-issues here when PRs are merged to main in my fork.)

I can implement it, doesn't seem too complicated, haven't had a need for it yet. But it would be good to know if that key triggers some kind of non-standard functionality provided by dovecot.

It does trigger a non-standard feature: https://doc.dovecot.org/2.3/configuration_manual/dict/#proxy

From what I can tell, the upstream project uses this to run arbitrary Python code upon get/set for specific keys. Since we're building support for all this directly into Mox, that level of extensibility is unnecessary.

I haven't gotten an answer to my request here yet, but assuming my understanding is correct, I'll need to ensure

• all reads of /shared/vendor/deltachat/irohrelay return a specific string from the configuration file
• all writes of /private/devicetoken\tuser@host.tld add to a set of values per-user (probably stored in one of the databases)

If you're willing to implement the metadata extension, go ahead!

---

In the next couple days, I'll write up the sub-issues and then get started on sketch code!