After spending months trying to troubleshoot intermittent problems with my self-hosted MIAB server, I tracked the problem down to Comcast. The installation was completely by-the-book: vanilla Ubuntu 22.04 x64, then MIAB setup script. It would work initially, but occasionally would throw errors when trying to perform certain actions. The problem presented as the box losing the ability to resolve names (“temporary failure in name resolution”), so I thought at first it must be a configuration or software problem. Unfortunately, the error also prevented me from reinstalling/upgrading MIAB and other software. Finally, I came across the following reddit post:
https://www.reddit.com/r/msp/comments/1c4nrbk/comcast_poisoning_dns_lookups_wtf/
It turns out Comcast and Xfinity run security “features” off-premise on their edge routers (even for commercial customers with static IP blocks). Comcast SecurityEdge and xFi Advanced Security come bundled by default and hijack non-encrypted DNS requests among other things. I called Comcast and had them turn off SecurityEdge (they called it “web filtering”) on my account, and immediately my MIAB started working flawlessly. I was able to upgrade MIAB, I received no more DNS errors, and my Status page went from many errors (most of them due to not being able to verify forward/reverse DNS) to no errors. It’s unbelievable they think this is an acceptable practice to enable by default.
Original reddit post follows:
r/msp icon
Go to msp
r/msp
•
1 yr. ago
Early-Ad-2541
Comcast poisoning DNS lookups? WTF??!?!
We’ve been having all sorts of DNS issues from behind Comcast connections. Certain SRV record lookups simply fail. Our DNS filtering no longer works. This happens no matter how we set our DNS settings. Pointing DNS to Google DNS or any other provider makes no difference. When we point DNS to our DNSFilter addresses, the lookups still fail and the filtering does not work.
It appears Comcast is intercepting ALL DNS LOOKUPS and preventing us from filtering. This is also breaking SRV lookups for our VOIP services, causing provisioning of phones and updates to phone settings to fail.
If we disconnect our Comcast and allow our firewall to fail over to our bacup T-Mobile 5G, everything works as expected.
Anyone else having these issues?
This is impacting our office and several customers.
Upvote
46
Downvote
47
Go to comments
Share
Share
Add a comment
Sort by:
Best
Search Comments
Expand comment search
Comments Section
[deleted]
•
1y ago
u/SWITmsp avatar
SWITmsp
•
1y ago
If you try to cancel SecurityEdge, make sure you confirm with the billing department that it does NOT cancel any bundle discounts you have. I’ve heard stories of them canceling securityedge and that kills off the whole discounted bundle, making the monthly bill go way up.
Upvote
34
Downvote
Reply
reply
Award
Share
Share
u/tfox-mi avatar
tfox-mi
•
1y ago
MSP - US (Detroit)
This… You’ll need to have them “turn off” SecurityEdge every 3 months or so, if you cancel it complete, it cancels your bundle and you end up at rack rate for your Internet service. We just have a recurring monthly task to check the status and call to disable it - for some reason, doing it in their portal doesn’t work for us.
I don’t know it as a fact, but I’m pretty sure they’re selling the Security Edge data. Why else would they offer this “service” for “free?”
Upvote
22
Downvote
Reply
reply
Award
Share
Share
Amorhan
•
1y ago
Not just free, they’re giving huge discounts if you bundle it in. Definitely selling data.
Upvote
10
Downvote
Reply
reply
Award
Share
Share
u/team_jj avatar
team_jj
•
1y ago
MSP - US
Turn of SecurityEdge as already mentioned, or use DNS over HTTPS so they can’t intercept it.
Upvote
17
Downvote
Reply
reply
Award
Share
Share
[deleted]
•
1y ago
[deleted]
•
9mo ago
They can still see SNI from DoH tho, so weird ISPs fetishize this data for business customers of all people
Upvote
1
Downvote
Reply
reply
Award
Share
Share
Newtronic
•
1y ago
Time to checkout encrypted DNS. DNS Encryption Explained
Upvote
15
Downvote
Reply
reply
Award
Share
Share
u/mnITd00d avatar
mnITd00d
•
1y ago
To echo this and what others have said, the issue the OP describes is indeed Comcast SecurityEdge. They will turn it off (reluctantly) upon request, but eventually it will get turned back on without telling you.
To work around this, we have moved many of our Comcast customers to encrypted DNS to bypass Comcast completely and prevent them from DNS hijacking, snooping, and poisoning.
Upvote
3
Downvote
Reply
reply
Award
Share
Share
CrafTech-Stephane
•
1y ago
Make sure you have their Security Edge service turned off, that’s usually the culprit.
Upvote
14
Downvote
Reply
reply
Award
Share
Share
u/Early-Ad-2541 avatar
Early-Ad-2541
OP
•
1y ago
This was the issue, it just started causing this specific issue though.
Upvote
4
Downvote
Reply
reply
Award
Share
Share
BobRepairSvc1945
•
1y ago
If you put the Comcast router into Bridge Mode that will disable SecurityEdge too.
Upvote
6
Downvote
Reply
reply
Award
Share
Share
u/Early-Ad-2541 avatar
Early-Ad-2541
OP
•
1y ago
Problem is these locations require a static IP.
Upvote
0
Downvote
Reply
reply
Award
Share
Share
u/q547 avatar
q547
•
1y ago
Why would bridge mode impact a static IP?
Upvote
9
Downvote
Reply
reply
Award
Share
Share
Belgarion30
•
1y ago
Put in passthrough, problem solved.
Upvote
4
Downvote
Reply
reply
Award
Share
Share
u/myrianthi avatar
myrianthi
•
1y ago
Ah, Security Edge. Don’t forget to call Comcast every now and then to verify it’s disabled since it seems to magically re-enable itself.
Upvote
4
Downvote
Reply
reply
Award
Share
Share
u/Early-Ad-2541 avatar
Early-Ad-2541
OP
•
1y ago
This is going to be fun with all my fucking customers this is impacting. It used to not happen when I was using a static IP with a customer owned firewall. I’m absolutely livid at Comcast.
Upvote
2
Downvote
Reply
reply
Award
Share
Share
u/myrianthi avatar
myrianthi
•
1y ago
It’s got nothing to do with the firewall appliances or the static/dynamic IP. If you read the Comcast invoice you will see Security Edge is included, which comes bundled. It’s a firewall feature they run on their end. There’s a residential version of the same thing called xFi Advanced Security which is one of the first things I check on (and disable) when troubleshooting home connections.
Upvote
2
Downvote
Reply
reply
Award
Share
Share
u/Early-Ad-2541 avatar
Early-Ad-2541
OP
•
1y ago
It used to. Whenever we would install a firewall for a customer and put a static on it, they would eventually get an email saying SecurityEdge wasn’t working. We could also still do DNS filtering until just a couple weeks ago.
Upvote
1
Downvote
Reply
reply
Award
Share
Share
GlowGreen1835
•
1y ago
DNS filter is great for anything browser based, but be careful with users using any software that requires heavy cloud sync. They have the big ones down, but there was a software a user was using that interfaced with a cloud DB and it would refuse to authenticate if DNS filter was enabled, ended up having to uninstall it for that user and anyone else using that connection. I wish I could look up what software that was, but I left that MSP months ago.
Upvote
3
Downvote
Reply
reply
Award
Share
Share
u/marklein avatar
marklein
•
1y ago
For some reason at my home my wife’s iPhone won’t download images unless I disable DNSFilter on the edge firewall. That’s the only glitch I’ve noticed so far and I don’t care enough to fix it, but I assume her iphone is trying to force using Apple DNS services somewhere.
Upvote
1
Downvote
Reply
reply
Award
Share
Share
88lbody
•
1y ago
Allow the iDevices to do whatever they want to Apple servers. They get super finicky if they can’t DNS their way. I always come across this on captive portal deployments or DNSFilter and similar.
Upvote
1
Downvote
Reply
reply
Award
Share
Share
u/Kiernian avatar
Kiernian
•
1y ago
Allow the iDevices to do whatever they want to Apple servers. They get super finicky if they can’t DNS their way.
Seriously.
They’re the only thing I’ve found that’s WORSE than samsung smart tv’s with regards to random inexplicable dns issues if you try to exert any control over their traffic whatsoever.
I can whitelist the whole damn apple /8 and the iDevices will still just randomly throw a fit if I so much as touch their outbound port 53 traffic.
It’s not even consistent, either.
Sure, MOSTLY it’s update-related, but sometimes it’ll jig over to the sign-in process and puke there instead.
Mind-boggling levels of needroot.
Upvote
2
Downvote
Reply
reply
Award
Share
Share
88lbody
•
1y ago
Samsung is pretty nasty for sure. I’ve been noticing more and more issues with Android TV/Chromecast devices in general too. Before that it was just the individual apps and was whatever.
Slowly, one by one my Chromecasts at home just disappear and tell me they’re offline, bypass pihole, come backs like nothing happened… Only one is left behind the pihole now and I’m sure it’s days are numbered.
The only consistency we get is inconsistency. I remember discovering this around 2016 in my first UniFi captive portal deployment. It was such an experience I’ll never forget to just iStuff do what it wants. 
Upvote
2
Downvote
Reply
reply
Award
Share
Share
u/Kiernian avatar
Kiernian
•
1y ago
in my first UniFi captive portal deployment
Ahh, UniFi IDS/IPS. “We won’t tell you what we’re blocking in those categories but we promise you it’s for your own good.”
I love their stuff for a lot of types of deployments, but they suffer from a whole bunch of assumed “noone will ever need or want to fine tune this” in their software.
Upvote
1
Downvote
Reply
reply
Award
Share
Share
OverwatchIT
•
1y ago
That’s apples way of saying “If we can’t track you, the fuck you!” There’s not enough time in the day to sort through apples bullshit, aggregate, and whitelist their never ending domain collection anyway. I farmed it out to a team of Indians on Fiverr. Best $25 I ever spent.
Upvote
2
Downvote
Reply
reply
Award
Share
Share
u/ramblingnonsense avatar
ramblingnonsense
•
1y ago
Yes, Comcast intercepts all outbound DNS traffic and forges the replies to make it appear as though they’re coming from the original server. Mediacom and Cox cable also do this on residential accounts, though I’ve never seen them try it on a business account.
Recently dealt with this same issue and DNSFilter specifically on Comcast. Even after turning off SecurityEdge, they continued to intercept and hijack DNS lookups. We got around it by setting up a pair of DNSFilter relays (their documentation kind of sucks for this but it works great once it’s up) that ONLY use DoTLS for lookups. Pointed everything on the network at them with internal domains pointed to the DCs, and it worked great. Only remaining “hole” is the DCs themselves, which we can’t point at the DNSFilter relays for forwarding due to a nasty lookup failure loop they get in. Fortunately, only processes running locally on the DCs that require a recursive lookup require this, so not much gets missed.
Upvote
3
Downvote
Reply
reply
Award
Share
Share
zer04ll
•
1y ago
The only way a state can block porn sites is with DNS hijaking, comcast uses transparent proxies to achieve this and its starting to break VPN, you have to go through steps to disable “security edge” which is essential a man in the middle attack by your ISP
Upvote
2
Downvote
Reply
reply
Award
Share
Share
dfwtim
•
1y ago
Vendor - ScoutDNS
Honestly it should be criminal. You should never have to “opt out” of having your network traffic hijacked. If this was a valid service, they wouldn’t need to SNEAK it on their customers in fine print, and any opt out would be permanent, which we all know is not.
For our customers we recommend our DNS-over-HTTPS roaming client where possible, and they use our Network Relay, which also uses DoH for headless devices, BYOD, servers, or anything else on the network you don’t load an agent on. Both of these will take this hassle away.
Upvote
2
Downvote
Reply
reply
Award
Share
Share
u/KenAdams02 avatar
KenAdams02
•
1y ago
Something has definitely been going on as of late; the amount of times I have had to power cycle the cable modem and my access points has increased. I don’t let Comcast’s DNS pass through to my AP’s, rather I have set Cisco Open DNS addresses as static on said AP’s.
I always think it’s a hoot everytime Comcast makes a remark “we see the traffic, but not the devices beyond a firewall…” to which I always respond GOOD. Looks like Comcast is flat-out disrupting service now because they don’t get their way…
If they were not the only high speed option, I would have canned their services long ago…
Edit: to answer your question, power cycling the equipment seems to work for now. The fact it’s been more frequent has me a bit concerned that one day it won’t make the difference.
Upvote
1
Downvote
Reply
reply
Award
Share
Share
krisleslie
•
1y ago
Better question is why use theirs lol
Upvote
1
Downvote
Reply
reply
Award
Share
Share
Assumeweknow
•
1y ago
Honestly, this is why I have a partner ISP that sells cable services. Comcast might provide the last mile. But there are no special deals etc. involved and I never have to worry about security edge. The only limitation is 500mb service. But beyond that, it’s been near perfect install experience every time.
Upvote
1
Downvote
Reply
reply
Award
Share
Share
DimitriElephant
•
1y ago
I have a vendor event with Comcast tomorrow, going to ask my rep about this and see what kind of world salad they give me in return.
How do you find out if your clients have this turned on?
Upvote
1
Downvote
Reply
reply
Award
Share
Share
dfwtim
•
1y ago
Vendor - ScoutDNS
You can easily test if this is on:
www.dnsleaktest.com to confirm what actual DNS resolvers are being used by clients. If Comcast Edge is active, you will see something from Net Actuate in the response. I have heard they also use the OpenDNS network in some locations, but we have not seen this.
Upvote
1
Downvote
Reply
reply
Award
Share
Share
Zanthexter
•
1y ago
Sort of related:
In my area, T-Mobile uses Comcast to provide data to their towers.
When there’s an area wide Comcast outage, you lose T-Mobile as well.
So we either use Comcast’s Convection Pro (which uses Verizon with AT&T as an alternate around here.) if we can get by with 1-2 Mb, or direct with AT&T when we need 5G.
Area outages are much less common than site outages, but they do happen.
Just something to keep in mind.
Upvote
1
Downvote
Reply
reply
Award
Share
Share
u/Early-Ad-2541 avatar
Early-Ad-2541
OP
•
1y ago
We’ve been using T-Mobile as a backup to Comcast at our office for a few years and so far it has never gone down when our Comcast did.
Upvote
1
Downvote
Reply
reply
Award
Share
Share
Zanthexter
•
1y ago
Well, as I said, “my area”, as in dozens of locations in the Houston region.
Your area could be set up differently.
Or you could just not have gotten unlucky yet.
Unfortunately I don’t know of a way to check this beyond a Comcast area wide outage also including T-Mobile data.
Upvote
2
Downvote
Reply
reply
Award
Share
Share
u/Early-Ad-2541 avatar
Early-Ad-2541
OP
•
1y ago
I’ll definitely be alert for this in case it happens.
Upvote
1
Downvote
Reply
reply
Award
Share
Share
dfwtim
•
1y ago
Vendor - ScoutDNS
You can easily test if this is on:
www.dnsleaktest.com to confirm what actual DNS resolvers are being used by clients. If Comcast Edge is active, you will see something from Net Actuate in the response. I have heard they also use the OpenDNS network in some locations, but we have not seen this.
Upvote
1
Downvote
Reply
reply
Award
Share
Share
[deleted]
•
1y ago