My server keeps sending emails to unknown destinations

IyadKh · October 16, 2021, 4:34pm

Hello,

I have successfully installed MIAB on my server a time ago. Yesterday, I noticed a strange behavior on the server control panel: the CPU usage is very high (>90%), high traffic volume and high consumption of hard disk storage.
The mail.log file was growing very fast. Decrypting the log data, I realized that one of my email addresses on the server is sending a lot of emails, most of them are returned to that same email address with a ‘Undelivered Mail’ notification message (41431 emails in the inbox and 16123 in the spam). I archived that user and waited for a while hoping the problem will be solved and MIAB will stop sending these messages. However, the server measurements are still at the peak values above as before and the mail log files shows entries I do not understand anymore. A sample of this log is below:

Oct 16 17:11:58 box postfix/smtp[12201]: mx-eu. mail .am0. yahoodns .net[188.125.72.73]:25: subject_CN=*.am0.yahoodns.net, issuer_CN=DigiCert SHA2 High Assurance Server CA, fingerprint=11:DE:ED:CE:B2:D8:24:9A:A1:D0:B4:0F:69:B5:5C:D5, pkey_fingerprint=48:21:A3:FB:D9:2C:A3:30:4C:62:E0:58:6A:8A:B2:9B
Oct 16 17:11:58 box postfix/smtp[12217]: mta6 .am0 .yahoodns .net[98.136.96.74]:25: subject_CN=*.am0.yahoodns.net, issuer_CN=DigiCert SHA2 High Assurance Server CA, fingerprint=11:DE:ED:CE:B2:D8:24:9A:A1:D0:B4:0F:69:B5:5C:D5, pkey_fingerprint=48:21:A3:FB:D9:2C:A3:30:4C:62:E0:58:6A:8A:B2:9B
Oct 16 17:11:58 box postfix/smtp[12064]: mta5. am0. yahoodns. net[67.195.204.77]:25: subject_CN=*.am0.yahoodns.net, issuer_CN=DigiCert SHA2 High Assurance Server CA, fingerprint=11:DE:ED:CE:B2:D8:24:9A:A1:D0:B4:0F:69:B5:5C:D5, pkey_fingerprint=48:21:A3:FB:D9:2C:A3:30:4C:62:E0:58:6A:8A:B2:9B
Oct 16 17:11:58 box postfix/smtp[12201]: Trusted TLS connection established to mx-eu. mail. am0. yahoodns. net[188.125.72.73]:25: TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
Oct 16 17:11:58 box postfix/smtp[12217]: Trusted TLS connection established to mta6. am0. yahoodns. net[98.136.96.74]:25: TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
Oct 16 17:11:58 box postfix/smtp[12064]: Trusted TLS connection established to mta5. am0. yahoodns. net[67.195.204.77]:25:
TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
Oct 16 17:11:58 box postfix/smtp[12209]: SSL_connect:SSLv3/TLS write client hello
Oct 16 17:11:58 box postfix/smtp[12209]: SSL_connect:SSLv3/TLS read server hello
Oct 16 17:11:58 box postfix/smtp[12669]: SSL_connect:SSLv3/TLS write client hello
Oct 16 17:11:58 box postfix/smtp[12669]: SSL_connect:SSLv3/TLS read server hello
Oct 16 17:11:58 box postfix/smtp[12669]: SSL_connect:TLSv1.3 read encrypted extensions
Oct 16 17:11:58 box postfix/smtp[12669]: mta5. am0. yahoodns. net[98.136.96.75]:25: depth=2 verify=1 subject=/C=US/O=DigiCert Inc/OU=www. digicert. com/CN=DigiCert High Assurance EV Root CA
Oct 16 17:11:58 box postfix/smtp[12209]: SSL_connect:TLSv1.3 read encrypted extensions
Oct 16 17:11:58 box postfix/smtp[12192]: 93B894811C: to=<mgrbasketball @yahoo. com>, relay=mta7. am0. yahoodns. net[67.195.228.110]:25, delay=81787, delays=81784/0.13/2.4/0.15, dsn=4.7.0, status=deferred (host mta7. am0. yahoodns. net[67.195.228.110] said: 421 4.7.0 [TSS04] Messages from 172.105.69.252 temporarily deferred due to unexpected volume or user complaints - 4.16.55.1; see https :// postmaster. yahooinc. com /error-codes (in reply to MAIL FROM command))
Oct 16 17:11:58 box postfix/smtp[12215]: setting up TLS connection to mx-aol. mail. gm0. yahoodns. net[67.195.204.80]:25
Oct 16 17:11:58 box postfix/smtp[12215]: mx-aol. mail. gm0. yahoodns. net[67.195.204.80]:25: TLS cipher list "aNULL:-aNULL:HIGH:MEDIUM:+RC4:@STRENGTH:!aNULL:!RC4"
Oct 16 17:11:58 box postfix/smtp[12215]: looking for session smtp&aol.com&mx-aol.mail.gm0.yahoodns.net&67.195.204.80&&BEE06FEFE00B01FA1093F783A939C820ADA473766EFE761C9780DE55587AA8FB in smtp cache
...

(I have added spaces to the urls in log records above to be able to post this topic)

After archiving the faulty email address, it no more shows in the log any more. But the problem persists. Could someone please help me to figure out what is going on? and how to inform MIAB to stop re-sending these messages?

Than you a lot in advance,
IK

openletter · October 16, 2021, 4:43pm

I haven’t experienced this type of problem before, but my guess is a compromised client or leaked credentials.

I think you need to check your mail queue by running the command mailq.

To delete mails in the queue, use postsuper command and you can review the options in the man page or delete absolutely all mails in the queue (including from all users) with postsuper -d ALL or all deferred mail with postsuper -d ALL deferred.

IyadKh · October 16, 2021, 4:58pm

Thank you very much @openletter for your reply.

I deleted all the messages in the queue (~100k messages) using the postsuper command. The CPU usages has fallen directly to less that 1%. I will keep an eye on the server to check that every thing is as smooth as before.

openletter · October 16, 2021, 5:02pm

You should also look at messages from receiving servers in the logs, like how Yahoo! is informing you that your IP address is flagged and follow any instructions provided. Some will have their own policies for their own blacklists.

And check https://www.dnsbl.info to see if you are on any public blacklists.

alento · October 20, 2021, 2:21pm

@IyadKh

I suspect that you already realize what happened here, correct?

The user which you suspended was compromised. Most likely by using a weak password.

IyadKh · October 20, 2021, 2:41pm

Correct!

Now my server is banned on gmail, Microsoft among others… Hard moments for the time

alento · October 20, 2021, 2:44pm

Ouch! How plausible is it to get a new IP address from your VPS provider?

IyadKh · October 20, 2021, 2:49pm

I did not ask yet. I am worried that chaning the IP may impact MIAB. Reading posts and replies here, I got the feeling that it may not be a good idea to change the MIAB IP, so I gave up the idea for the moment…

alento · October 20, 2021, 2:51pm

The impact will certainly be less than the impact of not being able to deliver emails to the major providers. Yes, it will entail a small bit of work on your part, but …

IyadKh · October 20, 2021, 3:02pm

@alento
Thank you very much for your replies and encouragement. I will give it a try today.
Would you kindly point me to some good reference on how to change the IP addresse of a MIAB?

Thank you a lot in advance.

alento · October 20, 2021, 3:05pm

First is to obtain a new IP address from your VPS provider.

Second is to update your Glue Records with that IP address.

Third is to rerun the command sudo mailinabox and confirm that it auto-detects the new IP address.

This all assumes that you use MiaB for your DNS for your domain(s). The work is more involved if you use External DNS at any point along the way.

It is best to look at each situation individually, so if you do not have a standard installation of MiaB we can then address the extra steps.

IyadKh · October 20, 2021, 3:08pm

It is my lucky day because MIAB is my DNS server
In my case, it does not look difficult after all.

Thank you for the steps. I will start applying them right now.

alento · October 20, 2021, 3:12pm

Reach out if you need any help or run in to any issues …

IyadKh · October 20, 2021, 5:44pm

@alento My MIAB is as clean as before
My litte advice to those who would follow the steps indicated by alento to change the MIAB IP: wait for for the dns changes to propagate before running the ‘sudo mailinabox’ command. I got a warning message that the installer could not find the box domain name. As skeptic as I am, if it could cause a problem, I re-run the command after a while.

andrew · October 21, 2021, 4:38am

Now it’s working again - time for the fun bits…

Have a quiet word to the offending account holder. Find out how the breach started (was their password “password1”?) and make sure it doesn’t happen again. (Some people seem like magnets for these kind of problems.)
Be sure you monitor Munin regularly - it shows msg queue size, etc and will indicate quickly when there is a problem. You can setup (on another box) a munin to monitor all your servers, and send out automated alerts.
Consider email rate limits. This out of MIAB scope but easy enough to do - just keep a record of the config changes you make, because you may have to reapply them after each MIAB update. You might start with a look at http://www.postfix.org/TUNING_README.html#conn_limit.

Fun, fun, fun

system · November 30, 2021, 4:39am

This topic was automatically closed 40 days after the last reply. New replies are no longer allowed.