COmanage Users List

[Warren G Anderson <>]

I feel your pain. Spammers and the countermeasures to combat them have made email one of the least reliable communication methods these days. 

We do not use mailman, but I have gone through much of the same pain with the LIGO.ORG sympa mailing lists. We do not use ARC, DMARC has been sufficient. My understanding is that ARC is a protocol that allows SMTP endpoints to evaluate email that passes through intermediate SMTP services that resend and ruin the SPF and/or DKIM checks. 

But the real issue, from my perspective, is that each SMTP service can implement any of SPF, DKIM, DMARC and/or ARC, and can set whatever policy they want (reject, quarantine, flag, etc) based on each of them. Also, as you have seen, when comparing domains, there is leeway in from where the SMTP service grabs the domain in the headers.  As a resender, like our mailing lists, or LIGOs vanity email service which forwards email sent to personal ligo.org addresses, you have to care about what every SMTP endpoint to which you send does and set a policy that tries to satisfy all of them. I do not think there is guaranteed to be such a policy.

It's a mess.

Warren

Warren G Anderson, Ph.D.

Leonard E Parker Center for Gravitation, Cosmology and Astrophysics

---

From: <> on behalf of Duncan Brown <>
Sent: Wednesday, February 9, 2022 17:18
To: Duncan Brown <>
Cc: Duncan Brown <>
Subject: Re: [comanage-users] Comanage, Mailman, DMARC, DKIM, and SPF

 

One other thing I tried that didn't work: I tried setting up ARC signing in mailman following the docs:

<https://nam02.safelinks.protection.outlook.com/?url="https%3A%2F%2Fdocs.mailman3.org%2Fprojects%2Fmailman%2Fen%2Flatest%2Fsrc%2Fmailman%2Fhandlers%2Fdocs%2Farc_sign.html&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=HA6yyM0Qy4ayMaGMu81sSXlDGU64k1HSIDvK03sriL0%3D&amp;reserved=0>

No matter what I did in the configuration, mailman wouldn't ARC sign the messages. One thing I did *not* try was to strip any incoming ARC headers before mailman and see if that allowed ARC signing. The DMARC/SPF/DKIM solution seemed to work, so I just left ARC off and ignored it. It's also not completely clear to me how ARC fits into the DMARC ecosystem anyway...

Cheers,
Duncan.

> On Feb 9, 2022, at 6:13 PM, Duncan Brown <> wrote:
>
> Hi Scott, Jim, Warren,
>
> I've been having issues with spam filters junking mail from my comanage+mailman instances on cosmicexplorer.org and np3m.org. I spent some time digging into this and I thought that I'd share what I discovered, incase it is useful to others or if you spot something that I've done that seems bad. I'll use np3m.org as the example here, but Cosmic Explorer sees the same thing.
>
> NP3M runs a comanage instance on roster.np3m.org (really the docker container np3m-roster.phy.syr.edu running on the host np3m-services.phy.syr.edu) and a mailman instance on mail.np3m.org (really the docker comanage-registry-docker containers running on np3m-services) The MX record for np3m.org points to smtp-ext.syr.edu and Rich routes mail to port 25 on np3m-mail.phy.syr.edu which routes to the container running postfix. Outgoing mail from mailman is routed via the postfix container to port 25 on smtp-host.syr.edu which routes to the outside world. comanage itself sends also mail to port 25 on smtp-host.syr.edu.
>
> The two main problems are:
>
> 1. One class of users has problems completing enrollment flows as the confirmation emails (and other emails) from and sent by np3m-roster.phy.syr.edu get junked. There are some universities (e.g. msu.edu) that will junk and reject the email even is the user tries to whitelist the np3m.org in outlook.
>
> 2. Mailman. There's a world of pain with mailman and DMARC with lots of tales of woe on the internets of mail servers servers junking mail from mailman, but not a lot of good recipes on how to solve it.
>
> Digging into how DMARC works, I discovered the following: to pass the DMARC spam test, a mail must
>
> (EITHER: pass the SPF check, which checks that the message comes from an ip address that the domain claims that it sends from in a DNS record; OR: pass the DKIM check which signs the message with a private key whose public key is published in the domain's DNS record) AND (has From field in the mail header is the same as to the MSG FROM sender domain in the SMTP envelope).
>
> The clause after the AND is critical and will cause a DMARC rejection, even if SPF and/or DKIM pass.
>
> I solved problem 1 by setting up a DMARC record and and SPF record in the DNS for np3m.org. I created a txt record in the DNS with the name _dmarc that contains the string
>
> v=DMARC1; p=reject; sp=reject; rua=mailto:; ruf=mailto:; fo=1; rf=afrf; pct=100; ri=86400
>
> This is basically the Syracuse DMARC record and I'm using the SU URIs for XML feedback (rua) an forensic reports (ruf). Then I created a txt record for the top-level domain (@ in GoDaddy) that contains the string:
>
> v=spf1 ip4:128.230.21.177 ip4:128.230.21.178 ip4:128.230.21.179 ip6:fe80::250:56ff:fead:e75b ip6:fe80::250:56ff:fead:805a ip6:fe80::250:56ff:fead:b06f include:syr.edu -all
>
> This includes the ip4 and ip6 addresses of the machines that can send email from np3m.org and includes the syr.edu SPF record, as we relay though smtp-host.syr.edu.
>
> That seemed to fix the problem where e.g. MSU would bounce enrollment flow emails from comanage.
>
> Next I tried to fix mailman. Oh boy, as Sam Beckett might say.
>
> SPF is supposed to compare the domain in the email's Envelope From with the record in the DNS. I configured my mailman lists to turn on the DMARC mitigation option "Replace From: with list address" and mitigate unconditionally so that all mails come from the mailing list e.g. .
>
> However, this did not fix mailman delivery for everything. For reasons I don't understand, SPF validation sometimes seems to be done on the hostname of the first IP address found in the Received: from headers. To get around this, I configured postfix to strip all the Received: from headers from the incoming mailing list mail before delivering it to mailman. This ensures that the first Received: from header in the outgoing mail, as well last the X-Originating-IP header:
>
> <https://nam02.safelinks.protection.outlook.com/?url="https%3A%2F%2Fgithub.com%2Fcosmic-explorer%2Fcomanage-registry-docker%2Fblob%2Fmaster%2Fcomanage-registry-mailman%2Fpostfix%2Fmain.cf%23L81&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=%2BppBpuMdDfdETCAzT7YXS8kVY5xT%2B8uSrYJx86E15Ns%3D&amp;reserved=0>
>
> <https://nam02.safelinks.protection.outlook.com/?url="https%3A%2F%2Fgithub.com%2Fcosmic-explorer%2Fcomanage-registry-docker%2Fblob%2Fmaster%2Fcomanage-registry-mailman%2Fpostfix%2Fheader_checks&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=a14kFW3USclWCp%2BalJodU9JAaZPzJuT5An%2BVzppFyxI%3D&amp;reserved=0>
>
> This was sufficient to get SPF to pass on a bunch of different hosts. However, some hosts also seemed to want DKIM to keep the mail out of spam, even though SPF is supposed to be enough...
>
> To get around this, I created a public/private key pair for DKIM
>
> <https://nam02.safelinks.protection.outlook.com/?url="https%3A%2F%2Fgithub.com%2Fnp3m%2Fce-it-infrastructure%2Fblob%2Fmaster%2Fmail%2Fbuild-mailman.sh%23L80&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=VTTaSk7glDL80znalF7TfPYNTKi0usts7%2Bc2lrtUkZI%3D&amp;reserved=0>
>
> I used the selector mailman022022 to name the key (this is an arbitrary string, just has to be a valid in a domain name) and published it into GoDaddy as a txt record for the host mailman022022._domainkey
>
> v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArZ7zl5yRwK3pBuXxxWunkwd8dX+EqA310shWZ49qLbr5FmzELUD/edaqmKuvY4lmPPE2eysWN9imWMByM0d6LeWwxpOt9G/5NJViZUKeRMc13hfvlB2c6L0b7q774p9BGGAGIailAFb0alk+3hyRaxRJAJ/+bGrCdiz6U+DHUqJBrmxrWOMFDylnO8e49H/8G56erpz1P2Zj5wXubKWnXQTE73Ns51yM6ZfyeEesPMZ0LlpNpJirUouusUlPh5SMIzIn+UrxZMs/z9+UgWzq+g1UHnefU3vyYMY6xxrp3aCE/H/XUSOq595mY8i/IiA1mO8/2dtBxmZLBXiWbd5lwQIDAQAB
>
> I had to configure mailman to strip the DKIM headers from inbound messages, as apparently some servers don't like it if there is more than one DKIM signature in the headers of a message:
>
> <https://nam02.safelinks.protection.outlook.com/?url="https%3A%2F%2Fgithub.com%2Fcosmic-explorer%2Fcomanage-registry-docker%2Fblob%2Fmaster%2Fcomanage-registry-mailman%2Fcore%2Fdocker-entrypoint.sh%23L163&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=eFRBwbycZIlGrUseKONIB7wQn33POVwSsTDMVTMqFUg%3D&amp;reserved=0>
>
> I then installed and configured OpenDKIM in the postfix docker container. Take a look at
>
> <https://nam02.safelinks.protection.outlook.com/?url="https%3A%2F%2Fgithub.com%2Fcosmic-explorer%2Fcomanage-registry-docker%2Ftree%2Fmaster%2Fcomanage-registry-mailman%2Fpostfix&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=8q0uRUrOBXWofYeHJ%2Fcj02BWU9beAWTliEsQqKrGha0%3D&amp;reserved=0>
>
> for changes to the Dockerfile, supervisord.conf and OpenDKIM config files. The file TrustedHosts has to contain the IP of the internal address of the mailman container (for me, this is 172.30.100.7)
>
> <https://nam02.safelinks.protection.outlook.com/?url="https%3A%2F%2Fgithub.com%2Fcosmic-explorer%2Fcomanage-registry-docker%2Fblob%2Fmaster%2Fcomanage-registry-mailman%2Fpostfix%2FTrustedHosts&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=X9fsyD93FLvelR4XTM03ehurtkDy99h4xypMyinJB9A%3D&amp;reserved=0>
>
> and the SigningTable configures OpenDKIM to sign all messages sent by this host that match From: *@np3m.org with the key I created:
>
> <https://nam02.safelinks.protection.outlook.com/?url="https%3A%2F%2Fgithub.com%2Fcosmic-explorer%2Fcomanage-registry-docker%2Fblob%2Fmaster%2Fcomanage-registry-mailman%2Fpostfix%2FSigningTable&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=6LTUNksjCgSG6NU09pZ2e6TzIeEohiI7Lwnusa5rAbc%3D&amp;reserved=0>
> <https://nam02.safelinks.protection.outlook.com/?url="https%3A%2F%2Fgithub.com%2Fcosmic-explorer%2Fcomanage-registry-docker%2Fblob%2Fmaster%2Fcomanage-registry-mailman%2Fpostfix%2FKeyTable&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=YXrd9tR%2BBAnB3OoXAcA5Y1zSf1nttUEjSXJOUYvO%2B2o%3D&amp;reserved=0>
>
> Finally, postfix is considered to used OpenDKIM as a milter to sign mail that passes through it:
>
> <https://nam02.safelinks.protection.outlook.com/?url="https%3A%2F%2Fgithub.com%2Fcosmic-explorer%2Fcomanage-registry-docker%2Fblob%2Fmaster%2Fcomanage-registry-mailman%2Fpostfix%2Fmain.cf%23L84&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=0JDtPqdtae7er8POjkR2NznTcH28zEGpI%2BpZjY54kRU%3D&amp;reserved=0>
>
> Since I used mailman's "Replace From: with list address," all mail coming from mailman comes from , so OpenDKIM signs all list emails on their way out to smtp-host.syr.edu. This happens after mailman munges with the message and headers, so
>
> That seems to be the magic needed to minimize DMARC rejections. There might be an easier way of doing this, but this works...
>
> Caveats:
>
> 1. You have to use mailman's "Replace From: with list address" feature for all messages. If you don't then envelope from doesn't match the header from and DMARC will fail even if SPF and DKIM pass.
>
> 2. Some users who forward their institutional mail to gmail are screwed whatever you do. gmail won't let users specify trusted domains, so if your institution changes the envelope from when it forwards to gmail, DMARC will fail. This is a widely known problem with the solution "don't forward your mail to gmail."
>
> 3. Apple mail has a nasty feature where it caches the From: and Reply-To: fields of mailman mailing lists in its previous recipients tab complete. This means that if you start typing
>
> Duncan....
>
> it might complete to
>
> Duncan Brown via PIs <>
>
> and go to the list rather than
>
> Duncan Brown <>
>
> which would just go to me. Because Apple Mail hides the real email in the blue box, you need to watch for the "via." There's no way to disable this cacheing in Apple Mail, unfortunately. This could result in embarrassment.
>
> Hope this is useful to others. Happy to corrected if I did something crazy.
>
> Cheers,
> Duncan.
>
> p.s. hello to me when I find this email googling some related problem six months from now...
>
> --
>
> Duncan Brown                              Room 263-1, Physics Department
> Charles Brightman Professor of Physics     Syracuse University, NY 13244
> https://nam02.safelinks.protection.outlook.com/?url="http%3A%2F%2Fdabrown.expressions.syr.edu%2F&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=bM1N3PdeUpqjoTTG6ovmJKtVgtbIiNxIDn927lRtCv8%3D&amp;reserved=0                     (+1) 315 443 5993
>
>

--

Duncan Brown                              Room 263-1, Physics Department
Charles Brightman Professor of Physics     Syracuse University, NY 13244
https://nam02.safelinks.protection.outlook.com/?url="http%3A%2F%2Fdabrown.expressions.syr.edu%2F&amp;data=04%7C01%7Canders15%40uwm.edu%7Ced425826923147ebb1f208d9ec228cef%7C0bca7ac3fcb64efd89eb6de97603cf21%7C0%7C0%7C637800455534990977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=bM1N3PdeUpqjoTTG6ovmJKtVgtbIiNxIDn927lRtCv8%3D&amp;reserved=0                     (+1) 315 443 5993