COmanage Users List

[Duncan Brown <>]

One other thing I tried that didn't work: I tried setting up ARC signing in 
mailman following the docs:

<https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/handlers/docs/arc_sign.html>

No matter what I did in the configuration, mailman wouldn't ARC sign the 
messages. One thing I did *not* try was to strip any incoming ARC headers 
before mailman and see if that allowed ARC signing. The DMARC/SPF/DKIM 
solution seemed to work, so I just left ARC off and ignored it. It's also not 
completely clear to me how ARC fits into the DMARC ecosystem anyway...

Cheers,
Duncan.

> On Feb 9, 2022, at 6:13 PM, Duncan Brown <> 
> wrote:
> 
> Hi Scott, Jim, Warren,
> 
> I've been having issues with spam filters junking mail from my 
> comanage+mailman instances on cosmicexplorer.org and np3m.org. I spent some 
> time digging into this and I thought that I'd share what I discovered, 
> incase it is useful to others or if you spot something that I've done that 
> seems bad. I'll use np3m.org as the example here, but Cosmic Explorer sees 
> the same thing.
> 
> NP3M runs a comanage instance on roster.np3m.org (really the docker 
> container np3m-roster.phy.syr.edu running on the host 
> np3m-services.phy.syr.edu) and a mailman instance on mail.np3m.org (really 
> the docker comanage-registry-docker containers running on np3m-services) 
> The MX record for np3m.org points to smtp-ext.syr.edu and Rich routes mail 
> to port 25 on np3m-mail.phy.syr.edu which routes to the container running 
> postfix. Outgoing mail from mailman is routed via the postfix container to 
> port 25 on smtp-host.syr.edu which routes to the outside world. comanage 
> itself sends also mail to port 25 on smtp-host.syr.edu.
> 
> The two main problems are:
> 
> 1. One class of users has problems completing enrollment flows as the 
> confirmation emails (and other emails) from  
> and sent by np3m-roster.phy.syr.edu get junked. There are some universities 
> (e.g. msu.edu) that will junk and reject the email even is the user tries 
> to whitelist the np3m.org in outlook.
> 
> 2. Mailman. There's a world of pain with mailman and DMARC with lots of 
> tales of woe on the internets of mail servers servers junking mail from 
> mailman, but not a lot of good recipes on how to solve it.
> 
> Digging into how DMARC works, I discovered the following: to pass the DMARC 
> spam test, a mail must 
> 
> (EITHER: pass the SPF check, which checks that the message comes from an ip 
> address that the domain claims that it sends from in a DNS record; OR: pass 
> the DKIM check which signs the message with a private key whose public key 
> is published in the domain's DNS record) AND (has From field in the mail 
> header is the same as to the MSG FROM sender domain in the SMTP envelope).
> 
> The clause after the AND is critical and will cause a DMARC rejection, even 
> if SPF and/or DKIM pass.
> 
> I solved problem 1 by setting up a DMARC record and and SPF record in the 
> DNS for np3m.org. I created a txt record in the DNS with the name _dmarc 
> that contains the string
> 
> v=DMARC1; p=reject; sp=reject; rua=; 
> ruf=; fo=1; rf=afrf; pct=100; ri=86400
> 
> This is basically the Syracuse DMARC record and I'm using the SU URIs for 
> XML feedback (rua) an forensic reports (ruf). Then I created a txt record 
> for the top-level domain (@ in GoDaddy) that contains the string:
> 
> v=spf1 ip4:128.230.21.177 ip4:128.230.21.178 ip4:128.230.21.179 
> ip6:fe80::250:56ff:fead:e75b ip6:fe80::250:56ff:fead:805a 
> ip6:fe80::250:56ff:fead:b06f include:syr.edu -all
> 
> This includes the ip4 and ip6 addresses of the machines that can send email 
> from np3m.org and includes the syr.edu SPF record, as we relay though 
> smtp-host.syr.edu.
> 
> That seemed to fix the problem where e.g. MSU would bounce enrollment flow 
> emails from comanage.
> 
> Next I tried to fix mailman. Oh boy, as Sam Beckett might say.
> 
> SPF is supposed to compare the domain in the email's Envelope From with the 
> record in the DNS. I configured my mailman lists to turn on the DMARC 
> mitigation option "Replace From: with list address" and mitigate 
> unconditionally so that all mails come from the mailing list e.g. 
> .
> 
> However, this did not fix mailman delivery for everything. For reasons I 
> don't understand, SPF validation sometimes seems to be done on the hostname 
> of the first IP address found in the Received: from headers. To get around 
> this, I configured postfix to strip all the Received: from headers from the 
> incoming mailing list mail before delivering it to mailman. This ensures 
> that the first Received: from header in the outgoing mail, as well last the 
> X-Originating-IP header:
> 
> <https://github.com/cosmic-explorer/comanage-registry-docker/blob/master/comanage-registry-mailman/postfix/main.cf#L81>
> 
> <https://github.com/cosmic-explorer/comanage-registry-docker/blob/master/comanage-registry-mailman/postfix/header_checks>
> 
> This was sufficient to get SPF to pass on a bunch of different hosts. 
> However, some hosts also seemed to want DKIM to keep the mail out of spam, 
> even though SPF is supposed to be enough...
> 
> To get around this, I created a public/private key pair for DKIM
> 
> <https://github.com/np3m/ce-it-infrastructure/blob/master/mail/build-mailman.sh#L80>
> 
> I used the selector mailman022022 to name the key (this is an arbitrary 
> string, just has to be a valid in a domain name) and published it into 
> GoDaddy as a txt record for the host mailman022022._domainkey
> 
> v=DKIM1; k=rsa; 
> p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArZ7zl5yRwK3pBuXxxWunkwd8dX+EqA310shWZ49qLbr5FmzELUD/edaqmKuvY4lmPPE2eysWN9imWMByM0d6LeWwxpOt9G/5NJViZUKeRMc13hfvlB2c6L0b7q774p9BGGAGIailAFb0alk+3hyRaxRJAJ/+bGrCdiz6U+DHUqJBrmxrWOMFDylnO8e49H/8G56erpz1P2Zj5wXubKWnXQTE73Ns51yM6ZfyeEesPMZ0LlpNpJirUouusUlPh5SMIzIn+UrxZMs/z9+UgWzq+g1UHnefU3vyYMY6xxrp3aCE/H/XUSOq595mY8i/IiA1mO8/2dtBxmZLBXiWbd5lwQIDAQAB
> 
> I had to configure mailman to strip the DKIM headers from inbound messages, 
> as apparently some servers don't like it if there is more than one DKIM 
> signature in the headers of a message:
> 
> <https://github.com/cosmic-explorer/comanage-registry-docker/blob/master/comanage-registry-mailman/core/docker-entrypoint.sh#L163>
> 
> I then installed and configured OpenDKIM in the postfix docker container. 
> Take a look at
> 
> <https://github.com/cosmic-explorer/comanage-registry-docker/tree/master/comanage-registry-mailman/postfix>
> 
> for changes to the Dockerfile, supervisord.conf and OpenDKIM config files. 
> The file TrustedHosts has to contain the IP of the internal address of the 
> mailman container (for me, this is 172.30.100.7)
> 
> <https://github.com/cosmic-explorer/comanage-registry-docker/blob/master/comanage-registry-mailman/postfix/TrustedHosts>
> 
> and the SigningTable configures OpenDKIM to sign all messages sent by this 
> host that match From: *@np3m.org with the key I created:
> 
> <https://github.com/cosmic-explorer/comanage-registry-docker/blob/master/comanage-registry-mailman/postfix/SigningTable>
> <https://github.com/cosmic-explorer/comanage-registry-docker/blob/master/comanage-registry-mailman/postfix/KeyTable>
> 
> Finally, postfix is considered to used OpenDKIM as a milter to sign mail 
> that passes through it:
> 
> <https://github.com/cosmic-explorer/comanage-registry-docker/blob/master/comanage-registry-mailman/postfix/main.cf#L84>
> 
> Since I used mailman's "Replace From: with list address," all mail coming 
> from mailman comes from , so OpenDKIM signs all list 
> emails on their way out to smtp-host.syr.edu. This happens after mailman 
> munges with the message and headers, so 
> 
> That seems to be the magic needed to minimize DMARC rejections. There might 
> be an easier way of doing this, but this works...
> 
> Caveats: 
> 
> 1. You have to use mailman's "Replace From: with list address" feature for 
> all messages. If you don't then envelope from doesn't match the header from 
> and DMARC will fail even if SPF and DKIM pass.
> 
> 2. Some users who forward their institutional mail to gmail are screwed 
> whatever you do. gmail won't let users specify trusted domains, so if your 
> institution changes the envelope from when it forwards to gmail, DMARC will 
> fail. This is a widely known problem with the solution "don't forward your 
> mail to gmail."
> 
> 3. Apple mail has a nasty feature where it caches the From: and Reply-To: 
> fields of mailman mailing lists in its previous recipients tab complete. 
> This means that if you start typing 
> 
> Duncan....
> 
> it might complete to
> 
> Duncan Brown via PIs <>
> 
> and go to the list rather than
> 
> Duncan Brown <>
> 
> which would just go to me. Because Apple Mail hides the real email in the 
> blue box, you need to watch for the "via." There's no way to disable this 
> cacheing in Apple Mail, unfortunately. This could result in embarrassment.
> 
> Hope this is useful to others. Happy to corrected if I did something crazy.
> 
> Cheers,
> Duncan.
> 
> p.s. hello to me when I find this email googling some related problem six 
> months from now...
> 
> -- 
> 
> Duncan Brown                              Room 263-1, Physics Department
> Charles Brightman Professor of Physics     Syracuse University, NY 13244
> http://dabrown.expressions.syr.edu                     (+1) 315 443 5993
> 
> 

-- 

Duncan Brown                              Room 263-1, Physics Department
Charles Brightman Professor of Physics     Syracuse University, NY 13244
http://dabrown.expressions.syr.edu                     (+1) 315 443 5993