I am finding myself that the spam emails that i am getting are coming from hotmail or gmail or some other legitimate sending server²
I’m assuming you receive weekly smtp-tls-reporting by email from Google, Microsoft and any other good citizen that sent email to your domain. Do these spam emails show up in those stats? If they are there it’s my understanding that the idea would be to report to the sending email provider which addresses they host have been seen spamming you and they would be obliged to take action to curb that. If they don’t appear in the report it is likely that the sender is merely pretending to be sending from Hotmail or Gmail (spoofing) and could probably be caught another way. My guess would be that those emails would be missing SPF or other DMARC headers since I often see SpamAssassin failing add any spam score for such obvious transgressions.
they are the kind we have looked at your site and it isn’t ranking in the first page of google or other search engines trying to sell seo services
okay looks like they are spoofing the DMARC and SPF somehow
Thanks for posting on this. It’s a subject that I find surprisingly fascinating (so apologies in advance for the longwinded post). Speaking as novice user (I certainly wouldn’t be able to help with any development anyways), I too am finding myself questioning the effectiveness of greylisting (though I appreciate it still may be worth it, and I cannot claim to have done even an exhaustive analysis of our own logs, let alone investigated it more generally).
I suspect that its usefulness is being eroded on both sides. Ie, on the one side, because spammers are using lists of known email addresses, the cost of defeating the greylist by resending is now well worth it for them. And then on the other side, as more and more websites are using email addresses as a layer of authentication, it’s getting more annoying to have to wait for your one-time password every time you sign up to a new service, or when an old service changes servers. In combination, those two factors seem to be spelling (at least according to my crystal ball) the end of the widespread use of greylisting. That being said, I certainly don’t have the desire or ability to implement changing it myself, and am not about to start tapping my foot either. I think MiaB is fantastic how it is, and I am very grateful it’s maintained, and security fixes etc are so promptly rolled out, which is a much higher priority to me.
10-15 years ago, it might have made sense to get a collective of servers/admins sharing their whitelists to address the second of those above problems, but now I am not sure the juice is worth the squeeze. If the project was successful, it would become well known, and that would have the unfortunate consequence of undermining itself, because spammers would learn that they should resend emails that are greylisted by those servers, which would result in the rest of their emails getting through to all the other servers, without having to pay the cost of resending them.
As for a shared whitelist/ blacklist within the MiaB community in general, I am suspicious of this too. Speaking only from my personal experience, the effort needed to fight spam in advance quickly exceeds the benefit of no longer having to delete useless emails from my inbox. Coordinating the fighting of spam with other people would potentially increase the effectiveness, but also the effort.
I also have a different experience when it comes to the “mythical” useful email that gets flagged as spam that you mentioned above. I find I do need to glance through my junk folder every few days to make sure there isn’t something I want in there. This is something that is very likely to vary depending on the user. We are a company, but many other users of MiaB (most I assume –except the spammers) are individuals that don’t actually get many legitimate businesses wanting to buy their products emailing them. I think this means that it will be very hard to create a one size fits all whitelist/ spam fighting approach, because we are all going to have different thresholds of what we want going to our junk folders.
Further, even if we had a way to address the above (maybe different whitelists, or customizable thresholds for spam), most bulk senders whose emails actually make it into inboxes have both good and bad customers (the bad customers bring their own lists with people who have tried to opt out already). Thus, when a good customer (of the bulk sender) sends me a newsletter that I want to receive, I am going to add it to the whitelist. But then, when a bad customer of the same sender sends you an email that you clearly didn’t sign up for, you are going to add it to the blacklist. I am not sure there is going to be an effective solution to this problem. I hope I am not misunderstanding your suggestion.
Wiping off my crystal ball again, I do actually think the best solution that will arise for fighting spam is going to be some sort of domain and reputation based whitelist, but one that is going to have to be done on a very large scale, and one which is unfortunately always going to seem to be far from satisfactory.
Yes, one way or another that is what happened - the spammers evolved but postgrey didn’t.
As you describe it I would agree, but your words “to fight spam in advance” still suggests some kind of policy that needs to be put in place before the emails start streaming in. That’s what we’ve been having all along and from having to still be having this discussion we know it didn’t remain effective for very long.
What I’m having in mind is rather different. The key is that a single server has to treat every email it gets as if it might be from a legitimate source simply because it has no reliable way of knowing that it’s coming from a bulk mail source. When we allow our servers to cooperate and exchange key information about the emails flowing through the collective of all participating servers, we can identify bulk sources with eye-watering accuracy provided enough of us participate. The real beauty of it is that we wouldn’t need to do anything to let those genuinely innocent emails come through unharmed from old and new servers alike. We could mark them as irregular if we want to and/or in principle let the people still running their '90s email server with none of the modern verification methods in place know that their systems can do with an update but that’s another day’s enhancement. Our new filter mechanism would only jump into action when collectively we identify bulk email from the same source based on any one of a number of criteria. As soon as we pick up on bulk email, we mark all email from their as spam to send it to junk mail folders everywhere. What people then do determines the fate of that bulk service. If they find legitimate email in their junk mail folders, they’d jump onto the web interface of our distributed system, press a few buttons and vouch for the source as being legitimate. If they identify something in the spam folders that confirms that it’s garbage, they’d also jump onto the web interface and condemn the source as being involved in sending spam. Of course there will be disagreements and there will be agents of spam enabling operators posing as users to vouch for their servers to be whitelisted, but when they do, it will invariably catch the attention of other users wanting to blacklist the source. The community can then decide to discredit the person(s) who vouched for a spam enabler by overturning their votes and banning them from voting again.
Since this is an arms race at best, over time there will no doubt be more and more agents infiltrating the community and more and more servers spinning up for individual campaigns. But there are a lot more we can do to stop the loopholes that would make it harder and harder for new email servers to start up and start sending bulk email without announcing themselves and making sure that the people who run them are known, contactable and under a jurisdiction that is serious about taking action against spam operators.
The majority of email users, even rather industrous workers sending out many emails to many people every day would be largely unaffected by this proposed solution. If the next generation of spam is so personalised with safe and useful content that people don’t mind it, we couldn’t rightfully call it spam anymore and we’d still consider the battle won.
This is how Proofpoint works + AI.
It’s only logical and confirms that it would be a workable model. But Proofpoint is (a massive (thousands of employees, bought for $12.3bn in 2021) entrerprise going about its business including this aspect again using a centralized approach using policies. What I’m proposing is quite different in that it’s every bit as distributed/federated as email itself, meaning that it lets each server monitor only its own traffic which it would do anyway, report key parameters of what they see (scrubbed for privacy purposes as well) in a highly efficient format to the others in the network where it aggregates to the factual data we need. At least that’s how I’d design it to run because it makes sense and has become viable through recent advances in technology.
Whether or not Proofpoint and its new owners will be keen on us succeeding without them is a very different question.