COmanage Users List

[Duncan Brown <>]

Hi Scott, Jim, Warren,

I've been having issues with spam filters junking mail from my 
comanage+mailman instances on cosmicexplorer.org and np3m.org. I spent some 
time digging into this and I thought that I'd share what I discovered, incase 
it is useful to others or if you spot something that I've done that seems 
bad. I'll use np3m.org as the example here, but Cosmic Explorer sees the same 
thing.

NP3M runs a comanage instance on roster.np3m.org (really the docker container 
np3m-roster.phy.syr.edu running on the host np3m-services.phy.syr.edu) and a 
mailman instance on mail.np3m.org (really the docker comanage-registry-docker 
containers running on np3m-services) The MX record for np3m.org points to 
smtp-ext.syr.edu and Rich routes mail to port 25 on np3m-mail.phy.syr.edu 
which routes to the container running postfix. Outgoing mail from mailman is 
routed via the postfix container to port 25 on smtp-host.syr.edu which routes 
to the outside world. comanage itself sends also mail to port 25 on 
smtp-host.syr.edu.

The two main problems are:

1. One class of users has problems completing enrollment flows as the 
confirmation emails (and other emails) from  
and sent by np3m-roster.phy.syr.edu get junked. There are some universities 
(e.g. msu.edu) that will junk and reject the email even is the user tries to 
whitelist the np3m.org in outlook.

2. Mailman. There's a world of pain with mailman and DMARC with lots of tales 
of woe on the internets of mail servers servers junking mail from mailman, 
but not a lot of good recipes on how to solve it.

Digging into how DMARC works, I discovered the following: to pass the DMARC 
spam test, a mail must 

(EITHER: pass the SPF check, which checks that the message comes from an ip 
address that the domain claims that it sends from in a DNS record; OR: pass 
the DKIM check which signs the message with a private key whose public key is 
published in the domain's DNS record) AND (has From field in the mail header 
is the same as to the MSG FROM sender domain in the SMTP envelope).

The clause after the AND is critical and will cause a DMARC rejection, even 
if SPF and/or DKIM pass.

I solved problem 1 by setting up a DMARC record and and SPF record in the DNS 
for np3m.org. I created a txt record in the DNS with the name _dmarc that 
contains the string

v=DMARC1; p=reject; sp=reject; rua=; 
ruf=; fo=1; rf=afrf; pct=100; ri=86400

This is basically the Syracuse DMARC record and I'm using the SU URIs for XML 
feedback (rua) an forensic reports (ruf). Then I created a txt record for the 
top-level domain (@ in GoDaddy) that contains the string:

v=spf1 ip4:128.230.21.177 ip4:128.230.21.178 ip4:128.230.21.179 
ip6:fe80::250:56ff:fead:e75b ip6:fe80::250:56ff:fead:805a 
ip6:fe80::250:56ff:fead:b06f include:syr.edu -all

This includes the ip4 and ip6 addresses of the machines that can send email 
from np3m.org and includes the syr.edu SPF record, as we relay though 
smtp-host.syr.edu.

That seemed to fix the problem where e.g. MSU would bounce enrollment flow 
emails from comanage.

Next I tried to fix mailman. Oh boy, as Sam Beckett might say.

SPF is supposed to compare the domain in the email's Envelope From with the 
record in the DNS. I configured my mailman lists to turn on the DMARC 
mitigation option "Replace From: with list address" and mitigate 
unconditionally so that all mails come from the mailing list e.g. 
.

However, this did not fix mailman delivery for everything. For reasons I 
don't understand, SPF validation sometimes seems to be done on the hostname 
of the first IP address found in the Received: from headers. To get around 
this, I configured postfix to strip all the Received: from headers from the 
incoming mailing list mail before delivering it to mailman. This ensures that 
the first Received: from header in the outgoing mail, as well last the 
X-Originating-IP header:

<https://github.com/cosmic-explorer/comanage-registry-docker/blob/master/comanage-registry-mailman/postfix/main.cf#L81>

<https://github.com/cosmic-explorer/comanage-registry-docker/blob/master/comanage-registry-mailman/postfix/header_checks>

This was sufficient to get SPF to pass on a bunch of different hosts. 
However, some hosts also seemed to want DKIM to keep the mail out of spam, 
even though SPF is supposed to be enough...

To get around this, I created a public/private key pair for DKIM

<https://github.com/np3m/ce-it-infrastructure/blob/master/mail/build-mailman.sh#L80>

I used the selector mailman022022 to name the key (this is an arbitrary 
string, just has to be a valid in a domain name) and published it into 
GoDaddy as a txt record for the host mailman022022._domainkey

v=DKIM1; k=rsa; 
p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArZ7zl5yRwK3pBuXxxWunkwd8dX+EqA310shWZ49qLbr5FmzELUD/edaqmKuvY4lmPPE2eysWN9imWMByM0d6LeWwxpOt9G/5NJViZUKeRMc13hfvlB2c6L0b7q774p9BGGAGIailAFb0alk+3hyRaxRJAJ/+bGrCdiz6U+DHUqJBrmxrWOMFDylnO8e49H/8G56erpz1P2Zj5wXubKWnXQTE73Ns51yM6ZfyeEesPMZ0LlpNpJirUouusUlPh5SMIzIn+UrxZMs/z9+UgWzq+g1UHnefU3vyYMY6xxrp3aCE/H/XUSOq595mY8i/IiA1mO8/2dtBxmZLBXiWbd5lwQIDAQAB

I had to configure mailman to strip the DKIM headers from inbound messages, 
as apparently some servers don't like it if there is more than one DKIM 
signature in the headers of a message:

<https://github.com/cosmic-explorer/comanage-registry-docker/blob/master/comanage-registry-mailman/core/docker-entrypoint.sh#L163>

I then installed and configured OpenDKIM in the postfix docker container. 
Take a look at

<https://github.com/cosmic-explorer/comanage-registry-docker/tree/master/comanage-registry-mailman/postfix>

for changes to the Dockerfile, supervisord.conf and OpenDKIM config files. 
The file TrustedHosts has to contain the IP of the internal address of the 
mailman container (for me, this is 172.30.100.7)

<https://github.com/cosmic-explorer/comanage-registry-docker/blob/master/comanage-registry-mailman/postfix/TrustedHosts>

and the SigningTable configures OpenDKIM to sign all messages sent by this 
host that match From: *@np3m.org with the key I created:

<https://github.com/cosmic-explorer/comanage-registry-docker/blob/master/comanage-registry-mailman/postfix/SigningTable>
<https://github.com/cosmic-explorer/comanage-registry-docker/blob/master/comanage-registry-mailman/postfix/KeyTable>

Finally, postfix is considered to used OpenDKIM as a milter to sign mail that 
passes through it:

<https://github.com/cosmic-explorer/comanage-registry-docker/blob/master/comanage-registry-mailman/postfix/main.cf#L84>

Since I used mailman's "Replace From: with list address," all mail coming 
from mailman comes from , so OpenDKIM signs all list emails 
on their way out to smtp-host.syr.edu. This happens after mailman munges with 
the message and headers, so 

That seems to be the magic needed to minimize DMARC rejections. There might 
be an easier way of doing this, but this works...

Caveats: 

1. You have to use mailman's "Replace From: with list address" feature for 
all messages. If you don't then envelope from doesn't match the header from 
and DMARC will fail even if SPF and DKIM pass.

2. Some users who forward their institutional mail to gmail are screwed 
whatever you do. gmail won't let users specify trusted domains, so if your 
institution changes the envelope from when it forwards to gmail, DMARC will 
fail. This is a widely known problem with the solution "don't forward your 
mail to gmail."

3. Apple mail has a nasty feature where it caches the From: and Reply-To: 
fields of mailman mailing lists in its previous recipients tab complete. This 
means that if you start typing 

Duncan....

it might complete to

Duncan Brown via PIs <>

and go to the list rather than

Duncan Brown <>

which would just go to me. Because Apple Mail hides the real email in the 
blue box, you need to watch for the "via." There's no way to disable this 
cacheing in Apple Mail, unfortunately. This could result in embarrassment.

Hope this is useful to others. Happy to corrected if I did something crazy.

Cheers,
Duncan.

p.s. hello to me when I find this email googling some related problem six 
months from now...

-- 

Duncan Brown                              Room 263-1, Physics Department
Charles Brightman Professor of Physics     Syracuse University, NY 13244
http://dabrown.expressions.syr.edu                     (+1) 315 443 5993