COmanage Users List

[Duncan Brown <>]

Hi Warren,

Thanks, that makes sense. Some Outlook servers in particular seem to be 
adding ARC records, but there's no transparency into any decisions being made 
by the ARC headers, as far as I can see. At lest the Authentication-Results 
header gives me some insight into DMARC.

But, yes, this whole issue seems like a huge PITA. Cosmic Explorer is 
starting to face a LIGO.org scale solution with the number of institutions 
and addressing every edge case seems impossible.

Cheers,
Duncan.

> On Feb 9, 2022, at 6:40 PM, Warren G Anderson <> wrote:
> 
> I feel your pain. Spammers and the countermeasures to combat them have made 
> email one of the least reliable communication methods these days. 
> 
> We do not use mailman, but I have gone through much of the same pain with 
> the LIGO.ORG sympa mailing lists. We do not use ARC, DMARC has been 
> sufficient. My understanding is that ARC is a protocol that allows SMTP 
> endpoints to evaluate email that passes through intermediate SMTP services 
> that resend and ruin the SPF and/or DKIM checks. 
> 
> But the real issue, from my perspective, is that each SMTP service can 
> implement any of SPF, DKIM, DMARC and/or ARC, and can set whatever policy 
> they want (reject, quarantine, flag, etc) based on each of them. Also, as 
> you have seen, when comparing domains, there is leeway in from where the 
> SMTP service grabs the domain in the headers.  As a resender, like our 
> mailing lists, or LIGOs vanity email service which forwards email sent to 
> personal ligo.org addresses, you have to care about what every SMTP 
> endpoint to which you send does and set a policy that tries to satisfy all 
> of them. I do not think there is guaranteed to be such a policy.
> 
> It's a mess.
> 
> Warren
> 
> Warren G Anderson, Ph.D.
> Leonard E Parker Center for Gravitation, Cosmology and Astrophysics
> From:  
> <> on behalf of Duncan Brown 
> <>
> Sent: Wednesday, February 9, 2022 17:18
> To: Duncan Brown <>
> Cc: Duncan Brown <>
> Subject: Re: [comanage-users] Comanage, Mailman, DMARC, DKIM, and SPF
>  
> One other thing I tried that didn't work: I tried setting up ARC signing in 
> mailman following the docs:
> 
> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.mailman3.org%2Fprojects%2Fmailman%2Fen%2Flatest%2Fsrc%2Fmailman%2Fhandlers%2Fdocs%2Farc_sign.html&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=HA6yyM0Qy4ayMaGMu81sSXlDGU64k1HSIDvK03sriL0%3D&amp;reserved=0>
> 
> No matter what I did in the configuration, mailman wouldn't ARC sign the 
> messages. One thing I did *not* try was to strip any incoming ARC headers 
> before mailman and see if that allowed ARC signing. The DMARC/SPF/DKIM 
> solution seemed to work, so I just left ARC off and ignored it. It's also 
> not completely clear to me how ARC fits into the DMARC ecosystem anyway...
> 
> Cheers,
> Duncan.
> 
> > On Feb 9, 2022, at 6:13 PM, Duncan Brown <> 
> > wrote:
> > 
> > Hi Scott, Jim, Warren,
> > 
> > I've been having issues with spam filters junking mail from my 
> > comanage+mailman instances on cosmicexplorer.org and np3m.org. I spent 
> > some time digging into this and I thought that I'd share what I 
> > discovered, incase it is useful to others or if you spot something that 
> > I've done that seems bad. I'll use np3m.org as the example here, but 
> > Cosmic Explorer sees the same thing.
> > 
> > NP3M runs a comanage instance on roster.np3m.org (really the docker 
> > container np3m-roster.phy.syr.edurunning on the host 
> > np3m-services.phy.syr.edu) and a mailman instance on mail.np3m.org 
> > (really the docker comanage-registry-docker containers running on 
> > np3m-services) The MX record for np3m.org points to smtp-ext.syr.edu and 
> > Rich routes mail to port 25 on np3m-mail.phy.syr.edu which routes to the 
> > container running postfix. Outgoing mail from mailman is routed via the 
> > postfix container to port 25 on smtp-host.syr.edu which routes to the 
> > outside world. comanage itself sends also mail to port 25 on 
> > smtp-host.syr.edu.
> > 
> > The two main problems are:
> > 
> > 1. One class of users has problems completing enrollment flows as the 
> > confirmation emails (and other emails) from 
> >  and sent by np3m-roster.phy.syr.edu get 
> > junked. There are some universities (e.g. msu.edu) that will junk and 
> > reject the email even is the user tries to whitelist the np3m.org in 
> > outlook.
> > 
> > 2. Mailman. There's a world of pain with mailman and DMARC with lots of 
> > tales of woe on the internets of mail servers servers junking mail from 
> > mailman, but not a lot of good recipes on how to solve it.
> > 
> > Digging into how DMARC works, I discovered the following: to pass the 
> > DMARC spam test, a mail must 
> > 
> > (EITHER: pass the SPF check, which checks that the message comes from an 
> > ip address that the domain claims that it sends from in a DNS record; OR: 
> > pass the DKIM check which signs the message with a private key whose 
> > public key is published in the domain's DNS record) AND (has From field 
> > in the mail header is the same as to the MSG FROM sender domain in the 
> > SMTP envelope).
> > 
> > The clause after the AND is critical and will cause a DMARC rejection, 
> > even if SPF and/or DKIM pass.
> > 
> > I solved problem 1 by setting up a DMARC record and and SPF record in the 
> > DNS for np3m.org. I created a txt record in the DNS with the name _dmarc 
> > that contains the string
> > 
> > v=DMARC1; p=reject; sp=reject; rua=; 
> > ruf=; fo=1; rf=afrf; pct=100; ri=86400
> > 
> > This is basically the Syracuse DMARC record and I'm using the SU URIs for 
> > XML feedback (rua) an forensic reports (ruf). Then I created a txt record 
> > for the top-level domain (@ in GoDaddy) that contains the string:
> > 
> > v=spf1 ip4:128.230.21.177 ip4:128.230.21.178 ip4:128.230.21.179 
> > ip6:fe80::250:56ff:fead:e75b ip6:fe80::250:56ff:fead:805a 
> > ip6:fe80::250:56ff:fead:b06f include:syr.edu -all
> > 
> > This includes the ip4 and ip6 addresses of the machines that can send 
> > email from np3m.org and includes the syr.edu SPF record, as we relay 
> > though smtp-host.syr.edu.
> > 
> > That seemed to fix the problem where e.g. MSU would bounce enrollment 
> > flow emails from comanage.
> > 
> > Next I tried to fix mailman. Oh boy, as Sam Beckett might say.
> > 
> > SPF is supposed to compare the domain in the email's Envelope From with 
> > the record in the DNS. I configured my mailman lists to turn on the DMARC 
> > mitigation option "Replace From: with list address" and mitigate 
> > unconditionally so that all mails come from the mailing list e.g. 
> > .
> > 
> > However, this did not fix mailman delivery for everything. For reasons I 
> > don't understand, SPF validation sometimes seems to be done on the 
> > hostname of the first IP address found in the Received: from headers. To 
> > get around this, I configured postfix to strip all the Received: from 
> > headers from the incoming mailing list mail before delivering it to 
> > mailman. This ensures that the first Received: from header in the 
> > outgoing mail, as well last the X-Originating-IP header:
> > 
> > <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcosmic-explorer%2Fcomanage-registry-docker%2Fblob%2Fmaster%2Fcomanage-registry-mailman%2Fpostfix%2Fmain.cf%23L81&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=%2BppBpuMdDfdETCAzT7YXS8kVY5xT%2B8uSrYJx86E15Ns%3D&amp;reserved=0>
> > 
> > <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcosmic-explorer%2Fcomanage-registry-docker%2Fblob%2Fmaster%2Fcomanage-registry-mailman%2Fpostfix%2Fheader_checks&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=a14kFW3USclWCp%2BalJodU9JAaZPzJuT5An%2BVzppFyxI%3D&amp;reserved=0>
> > 
> > This was sufficient to get SPF to pass on a bunch of different hosts. 
> > However, some hosts also seemed to want DKIM to keep the mail out of 
> > spam, even though SPF is supposed to be enough...
> > 
> > To get around this, I created a public/private key pair for DKIM
> > 
> > <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnp3m%2Fce-it-infrastructure%2Fblob%2Fmaster%2Fmail%2Fbuild-mailman.sh%23L80&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=VTTaSk7glDL80znalF7TfPYNTKi0usts7%2Bc2lrtUkZI%3D&amp;reserved=0>
> > 
> > I used the selector mailman022022 to name the key (this is an arbitrary 
> > string, just has to be a valid in a domain name) and published it into 
> > GoDaddy as a txt record for the host mailman022022._domainkey
> > 
> > v=DKIM1; k=rsa; 
> > p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArZ7zl5yRwK3pBuXxxWunkwd8dX+EqA310shWZ49qLbr5FmzELUD/edaqmKuvY4lmPPE2eysWN9imWMByM0d6LeWwxpOt9G/5NJViZUKeRMc13hfvlB2c6L0b7q774p9BGGAGIailAFb0alk+3hyRaxRJAJ/+bGrCdiz6U+DHUqJBrmxrWOMFDylnO8e49H/8G56erpz1P2Zj5wXubKWnXQTE73Ns51yM6ZfyeEesPMZ0LlpNpJirUouusUlPh5SMIzIn+UrxZMs/z9+UgWzq+g1UHnefU3vyYMY6xxrp3aCE/H/XUSOq595mY8i/IiA1mO8/2dtBxmZLBXiWbd5lwQIDAQAB
> > 
> > I had to configure mailman to strip the DKIM headers from inbound 
> > messages, as apparently some servers don't like it if there is more than 
> > one DKIM signature in the headers of a message:
> > 
> > <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcosmic-explorer%2Fcomanage-registry-docker%2Fblob%2Fmaster%2Fcomanage-registry-mailman%2Fcore%2Fdocker-entrypoint.sh%23L163&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=eFRBwbycZIlGrUseKONIB7wQn33POVwSsTDMVTMqFUg%3D&amp;reserved=0>
> > 
> > I then installed and configured OpenDKIM in the postfix docker container. 
> > Take a look at
> > 
> > <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcosmic-explorer%2Fcomanage-registry-docker%2Ftree%2Fmaster%2Fcomanage-registry-mailman%2Fpostfix&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=8q0uRUrOBXWofYeHJ%2Fcj02BWU9beAWTliEsQqKrGha0%3D&amp;reserved=0>
> > 
> > for changes to the Dockerfile, supervisord.conf and OpenDKIM config 
> > files. The file TrustedHosts has to contain the IP of the internal 
> > address of the mailman container (for me, this is 172.30.100.7)
> > 
> > <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcosmic-explorer%2Fcomanage-registry-docker%2Fblob%2Fmaster%2Fcomanage-registry-mailman%2Fpostfix%2FTrustedHosts&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=X9fsyD93FLvelR4XTM03ehurtkDy99h4xypMyinJB9A%3D&amp;reserved=0>
> > 
> > and the SigningTable configures OpenDKIM to sign all messages sent by 
> > this host that match From: *@np3m.org with the key I created:
> > 
> > <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcosmic-explorer%2Fcomanage-registry-docker%2Fblob%2Fmaster%2Fcomanage-registry-mailman%2Fpostfix%2FSigningTable&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=6LTUNksjCgSG6NU09pZ2e6TzIeEohiI7Lwnusa5rAbc%3D&amp;reserved=0>
> > <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcosmic-explorer%2Fcomanage-registry-docker%2Fblob%2Fmaster%2Fcomanage-registry-mailman%2Fpostfix%2FKeyTable&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=YXrd9tR%2BBAnB3OoXAcA5Y1zSf1nttUEjSXJOUYvO%2B2o%3D&amp;reserved=0>
> > 
> > Finally, postfix is considered to used OpenDKIM as a milter to sign mail 
> > that passes through it:
> > 
> > <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcosmic-explorer%2Fcomanage-registry-docker%2Fblob%2Fmaster%2Fcomanage-registry-mailman%2Fpostfix%2Fmain.cf%23L84&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=0JDtPqdtae7er8POjkR2NznTcH28zEGpI%2BpZjY54kRU%3D&amp;reserved=0>
> > 
> > Since I used mailman's "Replace From: with list address," all mail coming 
> > from mailman comes from , so OpenDKIM signs all list 
> > emails on their way out to smtp-host.syr.edu. This happens after mailman 
> > munges with the message and headers, so 
> > 
> > That seems to be the magic needed to minimize DMARC rejections. There 
> > might be an easier way of doing this, but this works...
> > 
> > Caveats: 
> > 
> > 1. You have to use mailman's "Replace From: with list address" feature 
> > for all messages. If you don't then envelope from doesn't match the 
> > header from and DMARC will fail even if SPF and DKIM pass.
> > 
> > 2. Some users who forward their institutional mail to gmail are screwed 
> > whatever you do. gmail won't let users specify trusted domains, so if 
> > your institution changes the envelope from when it forwards to gmail, 
> > DMARC will fail. This is a widely known problem with the solution "don't 
> > forward your mail to gmail."
> > 
> > 3. Apple mail has a nasty feature where it caches the From: and Reply-To: 
> > fields of mailman mailing lists in its previous recipients tab complete. 
> > This means that if you start typing 
> > 
> > Duncan....
> > 
> > it might complete to
> > 
> > Duncan Brown via PIs <>
> > 
> > and go to the list rather than
> > 
> > Duncan Brown <>
> > 
> > which would just go to me. Because Apple Mail hides the real email in the 
> > blue box, you need to watch for the "via." There's no way to disable this 
> > cacheing in Apple Mail, unfortunately. This could result in embarrassment.
> > 
> > Hope this is useful to others. Happy to corrected if I did something 
> > crazy.
> > 
> > Cheers,
> > Duncan.
> > 
> > p.s. hello to me when I find this email googling some related problem six 
> > months from now...
> > 
> > -- 
> > 
> > Duncan Brown                              Room 263-1, Physics Department
> > Charles Brightman Professor of Physics     Syracuse University, NY 13244
> > https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdabrown.expressions.syr.edu%2F&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=bM1N3PdeUpqjoTTG6ovmJKtVgtbIiNxIDn927lRtCv8%3D&amp;reserved=0
> >                      (+1) 315 443 5993
> > 
> > 
> 
> -- 
> 
> Duncan Brown                              Room 263-1, Physics Department
> Charles Brightman Professor of Physics     Syracuse University, NY 13244
> https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdabrown.expressions.syr.edu%2F&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=bM1N3PdeUpqjoTTG6ovmJKtVgtbIiNxIDn927lRtCv8%3D&amp;reserved=0
>                      (+1) 315 443 5993