CCE unable to report into Office 365.

2x CCE Appliances. Both using the same username and password under tenant admin (checked using Get-CcCredential).

Auto Upgrade kicks in and both successfully upgrade to v2.0.0:


However looking in Office 365 Admin Portal we can see that one never checked back in with the mothership to say its upgraded:


Looking in the CceManagementService.log we saw the following:

"CceService Warning: 0 : Appliance Manager: Failed to Load or update tenant configuration. Exception: System.Management.Automation.CmdletInvocationException: Failed to logon with given credentials. Make sure correct user name and password provided. ---> Microsoft.Rtc.Admin.Authentication.CommonAuthException: Failed to logon with given credentials. Make sure correct user name and password provided. ---> Microsoft.Rtc.Admin.Authentication.IdcrlExtendedException: 
AuthState=0x80048800RequestStatus=0x80048820"

All checks that we could think of were performed (re-entering credentials, reboots, checking networking, proxies) but to no avail.

Opened a ticket with Office 365 support who at the outset seemed equally as confused. It should be noted that all through this the VM's continued to work fine and calls were traversing the CCE's - it was "only" the management service that was failing to login.

While this was happening, version 2.0.1 of CCE was released and BOTH hosts upgraded themselves:


even though still in the portal only one was reporting back in:


Eventually a reply came back to try to upgrade the Skype for Business Online, Windows PowerShell Module to a newer version, taking it from:


to:


And after restarting the Management Service we have success:



The explanation I had back from Microsoft Support was the tenant was enabled for ADAL and the latest PowerShell supports MFA enabled accounts. They were unable to explain why this was only affecting one CCE though.......