Issues with Outlook and autodiscover

jfeazell · November 3, 2023, 2:58pm

I am having some peculiar issues related to Outlook. This is technically more of a (possibly) Z-Push thing and also a (I guess?) Microsoft thing, not so much MIAB, but perhaps someone here has some thoughts, because I’m my witts end after screwing around with this for a week. Please consider the details below, because I’ve done quite a bit of troubleshooting, all to no avail. TL;DR: Outlook (on any PC or phone), when adding email for one of my domains, always autodiscover the wrong mail server. The server is a (VERY) old server from way back when we hosted our email with Godaddy years ago.

I have a single instance of MIAB running on a Linode at mail.feazell.net. This MIAB actually handles the email for several domains, including islandiaproperties.com and iflyfla.com (but interestingly enough, not for the feazell.net domain, as that domain’s MX is Proton Mail. Just pointing that out in case anyone has a look as the DNS settings for these domains).

islandiaproperties.com is an older domain that, once upon a time, was registered and hosted with godaddy, where we had their unlimited business email. Back then, our IMAP and SMTP servers were imap.secureserver.net and smtpout.secureserver.net, respectively. We eventually moved our registration and hosting to HostGator, and then eventually moved away from them to register the domain and its DSN on Amazon Route 53 (AWS). All the content for the domain is hosted in different Linode instances.

feazell.net is also a domain that I’ve owned for several years. It has changed registrars a few times but has generally hosted little content. I primarily use it for email and personal IoT applications. Its DNS is managed by ZoneEdit and the MX server, mail.feazell.net, is hosted on a Linode. As previously mentioned, this MX server is the MX for islandiaproperties.com and iflyfla.com

iflyfla.com is a relatively new domain that currently hosts no content, and is only used for email. It is registered with AWS and its DNS is AWS Route 53 (just like islandiaproperties.com). The MX for this domain, again like islandiaproperties.com, is mail.feazell.net

At first, I thought my issue might be due to something cached in a local Outlook profile, or possibly even a cloud-based profile somehow associated with my Outlook identity. To test that theory, I performed the same operations on separate desktop PCs where I have Outlook installed, on user accounts that belong to me, and user accounts that belong to other users. When the behavior was the same for all cases, I also tried on my Android phone’s Outlook application, where the same problem exists. Finally, I tried on my wife’s Android phone with Outlook installed, which should have absolutely no knowledge of me any cached profile data, locally or otherwise. Nonetheless, the same problem continues to occur!

What is supposed to happen: In Outlook, when I add a new email account (james @ iflyfla.com), Outlook performs and attempt to autodiscover the MX settings (which would lead it to mail.feazell.net, the MX for the iflyfla.com domain), and then the autodiscover service running there (presumably Z-Push on my MIAB install), and that is precisely what happens; the IMAP and SMTP servers for that email address are correctly identified as mail.feazell.net, with the correct secure points and parameters. All I have to do is enter my email address and password and the account adds successfully.

The problem: When I try to add an email address for the islandiaproperties.com it performs the autodiscovery, but it always resolves imap.secureserver.net and smtpout.secureserver.net as the IMAP and SMTP servers, respectively. I cannot find any references to these (GoDaddy) servers anywhere in my DNS records for any of these domains. Autodiscover was set up by MIAB, and if it works for one domain, it certainly ought to work for any other. My theory, which has absolutely zero quantifiable evidence at this point, is that somehow, Outlook is querying some Microsoft repository and getting back some VERY old mail server. I spoke with GoDaddy today and they have no record of anything related to email being published over there.

Stuff I’ve tried: There are four tools I’ve tried to use to address this, all to no avail.

First, I tried the tool in Outlook itself where you can test autodiscover. From what I can see in the logs, this “sort of” works, in that it does end up resolving the correct server to go to (it tries all kinds of variants of the email’s domain until it gets one that works). However, it always gets two responses. First a 400 (Bad Request), and then a 401 (Not Authorized).
Next, I simply tried browsing to the URL of the autodiscover XML file. That worked, and I was prompted for the username and password. When I provided the email and password, I got an HTML page with some info about Z-Push, that said “Get is not supported”. My guess (largely corroborated by a subsequent test) is that this is nature of the 401 (Not Authorized) error I was getting with the first test.
Next, I tried the XML Discovery tool (free from Priasoft, a pretty robust tool). That also failed, but at least it gave some more verbose logging to give me an idea of what it was doing and zero in more clearly on what was failing. See the log below.
Going off of the previous two tests, I decided to fire up Postman and query the autodiscover myself and see if I could get it to work. Z-Push doesn’t seem to like this being a “Get”, based on the error I was seeing from Priasoft and my own https browser test. When I entered everything into Postman and did a GET, I got back a text/html response 200 with the same test that I got when using the web browser, saying that GET is not supported. I changed the method instead to POST, but all I would get back then (and with every other method for that matter) was a 500 (Internal Server Error). Strangely enough, if I change the URL in the request from https to http, I get back the HTML page saying GET is not supported, even though its still a POST. Maybe that has something to do with the redirect to https that’s built in to the autodiscover? I dunno. What is further interesting is that doing all these tests with the address james @ iflyfla.com don’t work either; I get all the same errors with the test tools, but Outlook will nonetheless autodiscover the MX correctly for this domain.

Again, it’s like Outlook is going out somewhere else to query first, and in one case (islandiaproperties.com) it’s getting back some very old cached data from someone, but for any other domain, whatever repository that is, its data is apparently up-to-date. As for the autodiscover (e.g. Z-Push) service on MIAB, that doesn’t seem to be working, at least not as far as I can tell based on all these tests. I did follow the links from that HTTP message from Z-Push. The Wiki link is dead, and it looks like there hasn’t been an update to Z-Push in a couple of years, so I’m thinking that project might be dead or badly stagnating. The documentation on the site is pretty lean, tool, so I’m not even 100% confident that my attempts to test it in Postman are correct. It does seem peculiar that it doesn’t respect and HTTP GET, since that’s pretty much what such a query is, although the body having XML content you would think a POST would work, but like I said, HTTP 500 when I try it with Postman.

That’s about all I have. Sorry for talking a bit in circles, but I’ve clearly just been chasing things with this, trying to get a consistent result. Please bear in mind that I’ve tested this across multiple devices, platforms, and people, so it’s not like this is something specifically related to my Outlook install or even my own user profile.

Logs - Priasoft

2023-11-03T08:00:08 :: Getting a list of AutoDiscover urls to use...
2023-11-03T08:00:08 :: Getting currently configured DNS servers from host...
2023-11-03T08:00:09 ::     DNS IP: 192.168.1.1
2023-11-03T08:00:09 :: Attempting to determine the AutoDiscover endpoint to use.
2023-11-03T08:00:09 :: Checking to see of mailbox ID is Office 365
2023-11-03T08:00:09 :: Checking MX record for 'islandiaproperties.com' to determine if it is Office 365.
2023-11-03T08:00:09 :: MX query returned 1 records
2023-11-03T08:00:09 ::     mail.feazell.net.
2023-11-03T08:00:09 :: Mail domain does not appear to be Office 365.
2023-11-03T08:00:09 :: Checking local domain SCP records, if host is domain joined.
2023-11-03T08:00:09 :: No SCP information found.
2023-11-03T08:00:09 :: Search for locally cached AutoDiscover.xml files.
2023-11-03T08:00:09 ::     Found 3 files
2023-11-03T08:00:09 :: End of cached files analysis
2023-11-03T08:00:09 :: Adding http://islandiaproperties.com/autodiscover/autodiscover.xml to list of redirection test hosts
2023-11-03T08:00:09 :: Adding https://islandiaproperties.com/autodiscover/autodiscover.xml to list of redirection test hosts
2023-11-03T08:00:09 :: SRV query returned 0 records
2023-11-03T08:00:09 :: Calculated the following 6 urls for AutoDiscover lookup, and will be queried in the order listed...
2023-11-03T08:00:09 ::      0. ROOT: https://islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:09 ::      1. AUTOD: https://autodiscover.islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:09 ::      2. REDIRECT: http://islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:09 ::      3. REDIRECT: http://autodiscover.islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:09 ::      4. REDIRECT: https://islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:09 ::      5. REDIRECT: https://autodiscover.islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:09 :: 
2023-11-03T08:00:09 :: 
2023-11-03T08:00:09 :: Starting XML requests...
2023-11-03T08:00:09 :: Attempting Root Domain AutoDiscover query using https://islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:09 :: Setting request type to EmailAddress
2023-11-03T08:00:09 :: Request XML will be:
2023-11-03T08:00:09 :: 
			<Autodiscover xmlns="http://schemas.microsoft.com/exchange/autodiscover/outlook/requestschema/2006">
			    <Request>
			        <EMailAddress>apply@islandiaproperties.com</EMailAddress>
			        <AcceptableResponseSchema>http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a</AcceptableResponseSchema>
			    </Request>
			</Autodiscover>
			
2023-11-03T08:00:09 :: Request will be sent to: https://islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:09 :: HttpWebRequest object created
2023-11-03T08:00:09 :: HttpWebRequest set to allow auto redirect
2023-11-03T08:00:09 :: HttpWebRequest set to POST method
2023-11-03T08:00:09 :: HttpWebRequest ContentType set to 'text/xml'
2023-11-03T08:00:09 :: HttpWebRequest UserAgent set to Microsoft Office/16.0 (Windows NT 6.2; Microsoft Outlook 16.0.16731; Pro)
2023-11-03T08:00:09 :: HttpWebRequest Headers set to the following to request mapiHTTP details:
2023-11-03T08:00:09 ::     X-MapiHTTPCapability=1
2023-11-03T08:00:09 :: Set integrated authentication using: apply@islandiaproperties.com/••••••••••••
2023-11-03T08:00:09 :: Communicating with remote host...this can take a few seconds to a minute or more...please wait..
2023-11-03T08:00:09 :: Wrote xml request data to the http request object.
2023-11-03T08:00:09 :: Sending request to the server...please wait...
2023-11-03T08:00:09 :: Error when sending request to server:
			System.Net.WebException: The remote server returned an error: (400) Bad Request.
			   at System.Net.HttpWebRequest.GetResponse()
			   at AutoDiscoverXMLTool.frmMain.GetAutodiscoverXML(String user, String url, NetworkCredential creds, Boolean RequestMapiHttp, String UserAgent, Dictionary`2 AdditionalHeaders, Boolean UseBasicAuth)
2023-11-03T08:00:09 :: 
2023-11-03T08:00:09 :: Attempting AutoDiscover Prefixed Domain AutoDiscover query using https://autodiscover.islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:09 :: Setting request type to EmailAddress
2023-11-03T08:00:09 :: Request XML will be:
2023-11-03T08:00:09 :: 
			<Autodiscover xmlns="http://schemas.microsoft.com/exchange/autodiscover/outlook/requestschema/2006">
			    <Request>
			        <EMailAddress>apply@islandiaproperties.com</EMailAddress>
			        <AcceptableResponseSchema>http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a</AcceptableResponseSchema>
			    </Request>
			</Autodiscover>
			
2023-11-03T08:00:09 :: Request will be sent to: https://autodiscover.islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:09 :: HttpWebRequest object created
2023-11-03T08:00:09 :: HttpWebRequest set to allow auto redirect
2023-11-03T08:00:09 :: HttpWebRequest set to POST method
2023-11-03T08:00:09 :: HttpWebRequest ContentType set to 'text/xml'
2023-11-03T08:00:09 :: HttpWebRequest UserAgent set to Microsoft Office/16.0 (Windows NT 6.2; Microsoft Outlook 16.0.16731; Pro)
2023-11-03T08:00:09 :: HttpWebRequest Headers set to the following to request mapiHTTP details:
2023-11-03T08:00:09 ::     X-MapiHTTPCapability=1
2023-11-03T08:00:09 :: Set integrated authentication using: apply@islandiaproperties.com/••••••••••••
2023-11-03T08:00:09 :: Communicating with remote host...this can take a few seconds to a minute or more...please wait..
2023-11-03T08:00:09 :: Wrote xml request data to the http request object.
2023-11-03T08:00:09 :: Sending request to the server...please wait...
2023-11-03T08:00:09 :: Error when sending request to server:
			System.Net.WebException: The remote server returned an error: (400) Bad Request.
			   at System.Net.HttpWebRequest.GetResponse()
			   at AutoDiscoverXMLTool.frmMain.GetAutodiscoverXML(String user, String url, NetworkCredential creds, Boolean RequestMapiHttp, String UserAgent, Dictionary`2 AdditionalHeaders, Boolean UseBasicAuth)
2023-11-03T08:00:09 :: 
2023-11-03T08:00:09 :: Request will be sent to: http://islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:10 :: HttpWebRequest object created
2023-11-03T08:00:10 :: HttpWebRequest set to disallow auto redirect
2023-11-03T08:00:10 :: HttpWebRequest set to GET method
2023-11-03T08:00:10 :: HttpWebRequest ContentType set to 'text/xml'
2023-11-03T08:00:10 :: Sending request to the server...please wait...
2023-11-03T08:00:10 :: Response received from server...
2023-11-03T08:00:10 :: Server responded with a status code of 301 [MovedPermanently] with a Location Header of: https://islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:10 :: Response from url provided a new url for AutoDiscover queries: https://islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:10 :: Attempting Redirected URL AutoDiscover query using https://islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:10 :: Setting request type to EmailAddress
2023-11-03T08:00:10 :: Request XML will be:
2023-11-03T08:00:10 :: 
			<Autodiscover xmlns="http://schemas.microsoft.com/exchange/autodiscover/outlook/requestschema/2006">
			    <Request>
			        <EMailAddress>apply@islandiaproperties.com</EMailAddress>
			        <AcceptableResponseSchema>http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a</AcceptableResponseSchema>
			    </Request>
			</Autodiscover>
			
2023-11-03T08:00:10 :: Request will be sent to: https://islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:10 :: HttpWebRequest object created
2023-11-03T08:00:10 :: HttpWebRequest set to allow auto redirect
2023-11-03T08:00:10 :: HttpWebRequest set to POST method
2023-11-03T08:00:10 :: HttpWebRequest ContentType set to 'text/xml'
2023-11-03T08:00:10 :: HttpWebRequest UserAgent set to Microsoft Office/16.0 (Windows NT 6.2; Microsoft Outlook 16.0.16731; Pro)
2023-11-03T08:00:10 :: HttpWebRequest Headers set to the following to request mapiHTTP details:
2023-11-03T08:00:10 ::     X-MapiHTTPCapability=1
2023-11-03T08:00:10 :: Set integrated authentication using: apply@islandiaproperties.com/••••••••••••
2023-11-03T08:00:10 :: Communicating with remote host...this can take a few seconds to a minute or more...please wait..
2023-11-03T08:00:10 :: Wrote xml request data to the http request object.
2023-11-03T08:00:10 :: Sending request to the server...please wait...
2023-11-03T08:00:10 :: Error when sending request to server:
			System.Net.WebException: The remote server returned an error: (400) Bad Request.
			   at System.Net.HttpWebRequest.GetResponse()
			   at AutoDiscoverXMLTool.frmMain.GetAutodiscoverXML(String user, String url, NetworkCredential creds, Boolean RequestMapiHttp, String UserAgent, Dictionary`2 AdditionalHeaders, Boolean UseBasicAuth)
2023-11-03T08:00:10 :: 
2023-11-03T08:00:10 :: Request will be sent to: http://autodiscover.islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:10 :: HttpWebRequest object created
2023-11-03T08:00:10 :: HttpWebRequest set to disallow auto redirect
2023-11-03T08:00:10 :: HttpWebRequest set to GET method
2023-11-03T08:00:10 :: HttpWebRequest ContentType set to 'text/xml'
2023-11-03T08:00:10 :: Sending request to the server...please wait...
2023-11-03T08:00:10 :: Response received from server...
2023-11-03T08:00:10 :: Server responded with a status code of 301 [MovedPermanently] with a Location Header of: https://autodiscover.islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:10 :: Response from url provided a new url for AutoDiscover queries: https://autodiscover.islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:10 :: Attempting Redirected URL AutoDiscover query using https://autodiscover.islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:10 :: Setting request type to EmailAddress
2023-11-03T08:00:10 :: Request XML will be:
2023-11-03T08:00:10 :: 
			<Autodiscover xmlns="http://schemas.microsoft.com/exchange/autodiscover/outlook/requestschema/2006">
			    <Request>
			        <EMailAddress>apply@islandiaproperties.com</EMailAddress>
			        <AcceptableResponseSchema>http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a</AcceptableResponseSchema>
			    </Request>
			</Autodiscover>
			
2023-11-03T08:00:10 :: Request will be sent to: https://autodiscover.islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:10 :: HttpWebRequest object created
2023-11-03T08:00:10 :: HttpWebRequest set to allow auto redirect
2023-11-03T08:00:10 :: HttpWebRequest set to POST method
2023-11-03T08:00:10 :: HttpWebRequest ContentType set to 'text/xml'
2023-11-03T08:00:10 :: HttpWebRequest UserAgent set to Microsoft Office/16.0 (Windows NT 6.2; Microsoft Outlook 16.0.16731; Pro)
2023-11-03T08:00:10 :: HttpWebRequest Headers set to the following to request mapiHTTP details:
2023-11-03T08:00:10 ::     X-MapiHTTPCapability=1
2023-11-03T08:00:10 :: Set integrated authentication using: apply@islandiaproperties.com/••••••••••••
2023-11-03T08:00:10 :: Communicating with remote host...this can take a few seconds to a minute or more...please wait..
2023-11-03T08:00:10 :: Wrote xml request data to the http request object.
2023-11-03T08:00:10 :: Sending request to the server...please wait...
2023-11-03T08:00:10 :: Error when sending request to server:
			System.Net.WebException: The remote server returned an error: (400) Bad Request.
			   at System.Net.HttpWebRequest.GetResponse()
			   at AutoDiscoverXMLTool.frmMain.GetAutodiscoverXML(String user, String url, NetworkCredential creds, Boolean RequestMapiHttp, String UserAgent, Dictionary`2 AdditionalHeaders, Boolean UseBasicAuth)
2023-11-03T08:00:10 :: 
2023-11-03T08:00:10 :: Request will be sent to: https://islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:10 :: HttpWebRequest object created
2023-11-03T08:00:10 :: HttpWebRequest set to disallow auto redirect
2023-11-03T08:00:10 :: HttpWebRequest set to GET method
2023-11-03T08:00:10 :: HttpWebRequest ContentType set to 'text/xml'
2023-11-03T08:00:10 :: Sending request to the server...please wait...
2023-11-03T08:00:10 :: The remote server returned an error: (401) Unauthorized.
2023-11-03T08:00:10 :: 
2023-11-03T08:00:10 :: Request will be sent to: https://autodiscover.islandiaproperties.com/autodiscover/autodiscover.xml
2023-11-03T08:00:10 :: HttpWebRequest object created
2023-11-03T08:00:10 :: HttpWebRequest set to disallow auto redirect
2023-11-03T08:00:10 :: HttpWebRequest set to GET method
2023-11-03T08:00:10 :: HttpWebRequest ContentType set to 'text/xml'
2023-11-03T08:00:10 :: Sending request to the server...please wait...
2023-11-03T08:00:10 :: The remote server returned an error: (401) Unauthorized.
2023-11-03T08:00:10 ::

dms · November 3, 2023, 8:40pm

Have you looked at the z-push logs on the MiaB server (/var/log/z-push)? Maybe you could try comparing the logs between one that works correctly and one that doesn’t.

stylnchris · November 4, 2023, 1:19am

Are you using MIAB for dns or external?

I’m on a mobile phone and can’t check your dns records right now but I suspect something is off with your autodiscover records.

matidau · November 5, 2023, 2:03am

The host that is returned from autodiscover is set in the location /usr/local/lib/z-push/autodiscover/config.php with the line

define('ZPUSH_HOST', 'PRIMARY_HOSTNAME');

where PRIMARY_HOSTNAME should be set by the MiaB setup.

Have you tested adding an account on Android with Gmail using the Exchange option?

To test the autodiscover post request you will need to submit the expected xml as a payload, I don’t have an example unfortunately, but I think the only required element is Request->EmailAddress (I haven’t tested this).

Z-Push autodiscover doesn’t provide IMAP details back but ActiveSync details, the type of ActiveSync it provides in the Autodiscover is MobileSync. Z-Push is not compatible with the Outlook mobile app as it makes use of EWS as well as ActiveSync, Z-Push only provides ActiveSync.

The project is still active but the links have all moved (and the documentation is still out of date). I haven’t updated the links in the source yet but I’m aware of it, if you feel like it you can raise it as an issue Issues · Z-Hub/Z-Push · GitHub or if your really keen submit a PR.

Outlook should be falling back on to IMAP, POP and SMTP detection after autodiscover fails. Outlook has a few steps in the autodiscover process, including ones that might be affected by AD and 365. Outlook 2016 implementation of Autodiscover - Microsoft Support

There are some registry settings in this that might help you in tracking this down through Outlook desktop.

It is also interesting that GoDaddy’s current email service is 365 based. This is only me thinking aloud, perhaps during their migration to 365 they had their IMAP and SMTP servers populated in 365.

Hope you get to the bottom of this.

sesser · January 18, 2024, 2:30pm

I think this is an Outlook/Microsoft thing. I noticed the same recently when adding an account to the “new Outlook” on Win 11… it used Fastmail imap/smtp host names which I was on years ago. Every other email client I’ve tried either pulls in the proper settings from MiaB (Thunderbird) or leaves the fields blank.