Microsoft Exchange 2016 and 2010 coexistence – Outook shows login promt

During a migration from Exchange 2010 to Exchange 2016 I was facing a strange problem. Users with a Mailbox on 2016 always received a login prompt when they started Outlook.

The problem only happened for users that had access to public folders (still hosted on Exchange 2010) or mailboxes that also had not been migrated. If you start to google for the problem, the first hint you get is to change the Outlook Anywhere Authentication on Exchange 2016 (the default now is Negotiate) to NTLM. Ok – let’s check the current settings with PowerShell (Exchange Management Shell) for each CAS-Server:

Get-OutlookAnywhere –server SERVERNAME

Exchange 2016:

Already Ntlm – so let’s check 2010:

As you can see this was still Basic.

Interestingly the Exchange 2010 Shell shows the option ClientAuthenticationMethod – while the 2016 Shell shows InternalClientAuthenticationMethod AND ExternalClientAuthenticationMethod.

Time to change the Exchange 2010 CAS to NTLM Authentication:

Set-OutlookAnywhere -Identity "SERVERNAME\Rpc (Default Web Site)" -ClientAuthenticationMethod ntlm

Restart IIS to make sure the setting is applied (CMD – IISRESET):

Unfortunately this didn’t fix the problem So time to look for other solutions. I then found Hotfix 2990117 from Microsoft which exactly described our problem. I downloaded the hotfix – but the installation “failed”. There was no error message nor anything else – the installation just started fine but never finished. Luckily the Hotfix also describes a workaround:

Open the IIS Manager on the 2010 CAS and go to Application Pools => DefaultAppPool and choose Advanced Settings on the right side.

Go to Identity and press “…”.

Now switch the setting to NetworkService.

After this confirm the settings with OK (twice) and press Recycle on the right side in the IIS Console (when the Default-App-Pool is selected). To make sure everything is applied correctly I also did an IISRESET. First I only changed the setting on the 2010 CAS-Servers – than also on 2016. But guess – still the same problem

Time to check the information’s that I had until now. All information’s that are available point to an Authentication Problem. Mainly Exchange 2016 uses different methods compared to 2010. So it’s often suggested to change the settings to NTLM. What happens if Outlook uses Outlook Anywhere? It “opens” the RPC Website. Let’s check the Authentication settings for the RPC Website:

Open IIS Manager and go to  Sites => Default Web Site => RPC => Authentication

Now select Windows Authentication =>  Providers.

We can now see that Negotiate is the first configured provider. If we now remember that we had to switch our Outlook Anywhere Settings for Exchange 2016 to NTLM to make it compatible with 2010 this doesn’t sound correct.

So I moved NTLM to the top and restarted the IIS (IISRESET).

I repeated this on every other Exchange 2010 CAS-Server and 2016 Server – and after that the login prompt didn’t occur any longer. To make this a permanent change (and remove Negotiate until all Exchange 2010 Servers are removed) enter the following command for every Exchange-Server:

Get-OutlookAnywhere -Identity "SERVER-FQDN\rpc (Default Web Site)" | Set-OutlookAnywhere -IISAuthenticationMethods "Basic, ntlm"

PS:
After switching back the App-Pool-Identity to AppliactionPoolIdentity some clients (especially Outlook 2016) again showed the login prompt – so I think it’s necessary to change both settings.