Presence Unknown..... BUT WHY!

Have had a long running issue with a single user at a customer where I was unable to IM them or see their presence:


However they could IM me and see my presence fine.

The customer has on-prem Lync servers. I am on Office 365 which is setup in a Hybrid with our on-prem servers. Other people in Modality who are on-prem (Response Group users) could see this person fine (you want a name - okay, it's Leon).

It should be noted that I used to be able to see the presence and IM with no issue, also, after I moved to O365 I could. Something changed later* Anyway - back to the story....

When Leon IM'd me I would get errors like this in my event logs:

504  Server Time-Out
ms-diagnostics:  27002;reason="From-Uri Domain is not in the receiver-tenant allow list";source="Office365ServerName.INFRA.LYNC.COM";appName="IncomingFederation";OriginalPresenceState="0";CurrentPresenceState="0";MeInsideUser="No";ConversationInitiatedBy="6";SourceNetwork="5";RemotePartyCanDoIM="Yes"


A search on that error didn't really bring anything up of value as it was talking about the whole domain needing white listing and that couldn't be correct as it was a single user issue. We tried moving to different PC's, different networks, investigated policies that Leon had but all came up nil.

The issue wasn't a big enough pain for Leon to want to spend too much time troubleshooting but eventually while discussing about their customers Office 365 plans a light bulb went off.

"Leon, have you got your user account in Office 365 as well"

After confirming he had it was as simple as turning off Skype for Business for his user account in the customers O365 tenant:


and we were back in business:



So what happened?

My account is in Office 365. Leon's account was on-prem. He also had an account in Office 365 but Hybrid was not setup.

Therefore, when Leon IM'd me, his client talked to his on-prem Edge, resolved the DNS for Modality Edge, and got proxied to me in O365.

However, when I attempted to IM Leon, my client talked to O365, who saw that there was a matching O365 tenant for the domain and sent the IM there. Simply turning off Leon from having an Office 365 Skype for Business account allowed the Modality Office 365 tenant to ignore looking up his details in the cloud, I found the customers Edge server and all was well in the world.

Simple when you know how!

*what changed? The customer got Office 365 but had not setup all the hybrid integration (as they didn't want to use it all at that time).

CCE 2.1.0 - Draining Calls

The Cloud Connector Edition system for Skype for Business online is a very impressive collection of scripts and glue and luck that has the ability to build a complete SfB voice infrastructure and patch it automatically. One of the parts of this is not as good as it could be, read on if you get call drops when updating :-)

Environment is 2x CCE hosts in the same site. For the test below I first put the CCE2 into maintenance and make calls. Therefore forcing all calls through CCE1.

I make two calls:

Ensure that both calls are up and running and show on the Sonus:


Logging into the Mediation Server on the CCE1 you can see the calls are running through it:



Take CCE2 out of maintenance so that both CCE's take service calls (both calls are still running on CCE1).

On CCE1 run the command Enter-CcUpdate on an elevated PowerShell session

Based on the documentation (https://technet.microsoft.com/en-us/library/mt492520.aspx) I would expect this to drain the mediation server, waiting for the two nailed up calls to complete and to then gracefully stop the services.

Instead I get this:



Services are stopped and both calls are dropped. Bit of a fail when the documentation says:

"The appliance is “drained”—that is, all existing calls will complete, but new calls are rejected."

and

"The Enter-CcUpdate cmdlet will ensure that all running calls on a Cloud Connector appliance will complete, but the appliance will reject any new calls, which are transferred to other production appliances. This cmdlet enables you to update an appliance without affecting end users calls." (my emphasis!)

(Bonus points for the spelling of "Drainning" and "Forceing")

Its now logged as a ticket with Microsoft and I'll update as and when I have a resolution.








Unable to login to Skype for Business Online with BT Home Hub 6

This is an embarrassing post to write but I hope it will help someone out there in troubleshooting! Anyway, on with the story:

Working for Modality Systems is great, but when I joined I pointed out that the daily commute was not something I fancied:

(side note - 2 hour 11 minutes - in which universe??)

Luckily working from home is one of those things you can do with the magic of Unified Communications Intelligent Communications as work really is that thing you do and not the place you go.

And that's fine until it stops working, lets set the scene.....

Its the Thursday after patch Tuesday. The previous day (Wednesday) I was in the Modality Systems main off in St Albans. Laptop had updated and working fine, Office install is Click-to-run. My user account is sync'ed from our internal domain to Office 365 and my Skype for Business account is homed in Office 365 with our setup being hybrid.

I boot my laptop and login. Direct access does its stuff, Group Policies apply, Outlook, Teams, OneNote, and SfB all load up and I start on some emails. I need to reach out to a colleague so switch to SfB and am presented by this:

"interesting" I think, I wonder why Skype is not signing in. Its been 15 minutes so since I logged onto the laptop so its really stuck, click cancel and try again but no joy..... 

Must be an O365 outage on Skype only, Teams and Outlook are fine and I can browse the internet with no issues so I plod over to the portal to check service status....



.....okay, so something about my account then, I check the internal AD and Azure AD, nothing looks out of place. I clear out the certificates from my local store....


....delete the contents of C:\Users\tobie.fysh\AppData\Local\Microsoft\Office\16.0\Lync and reboot, still no joy!

Boot up a laptop that has not been patched (and as a separate change, runs Office MSI). Still no joy. Leave that updating to latest versions and try https://testconnectivity.microsoft.com/ and see this:


Ohhhh, so something is broken on our tenant then! I click the to “tell me more” and it takes me to:




Post on our internal Teams chat and Email some of the guys in the office (feeling like a failure at this point). No-one else in the org appears to be having issues so seems to be local to me.

While awaiting a reply go think maybe Edge related (as we are hybrid the DNS records point to our on-prem Edge server so prior to the endpointcache being updated I'm going to be hitting that). I RDPto a VM inside the network (over the magic that is Direct Access and IP4 to IP6 natting) and load Lync 2013 (its a test box) and I can login to my account!

So if I'm external to our network I can login to SfB but inside the network I'm fine?? Can't be tenant related. Maybe something about our Azure AD boxes, scratching head here.....

Suggestion comes back from colleague to try my mobile app:

wait... wait... wait...


No joy.

Okay, so looks like my account, for fun I switch to bounSky and just check that I can login using my client to another estate and bang, I'm in. I test a few (Lync Server 2010, 2013 and a SfB 2015 server) all okay.

I then try a pure cloud customer.

And can't login..... I try a second customer who is hybrid and I can't login to a O365 user. On that same customer I then try a user account that is homed on-prem and CAN login (and these tenants are all hosted in different regions to the Modality tenant).

Brain is tied in knots now. About to log a ticket with O365 support but for fun think that there might be something about my home network. I turn on the hotspot fucntion of my phone connect the laptop to it and bang, the lovely 4G network allows me to login straight away.

Have a think about my network. What's changed recently. My Router. My lovely new Home Hub 6. Thanks BT, maybe its you! I reboot the router.

No joy, I then remember that I commented while doing firewall traces last week that I appeared to have an IP6 address with my new router:




I dig out the old Home Hub 5 (it was in the returns box awaiting to go back to BT) and plumb it in...... I now have an IP4 address:


And immediately desktop Skype for Business signs in.

My phone, signs in (it was on the WIFI before, go check the screenshot!)

Obviously Mr Cropley has already tweeted a reply to me:


And directs me to the source:



So - I need to support customers who wont have enabled IP6 in their tenants so as a consequence the Home Hub 5 is back in pride of place beside the fish tank and the HH6 is being say in the corner like a naughty child

I've tweeted BT to see if they can help disable IP6 on my account:


and will update if I get a response.

PEM certificate files on Windows

While doing certificate renewals for a client recently I was given PEM format files which I needed to convert into a certificate that Windows can consume.


  1. Download a copy of OpenSSL which has been complied for Windows (https://wiki.openssl.org/index.php/Binaries).
    Note: You can do this on your workstation, it does not need to be done on the same machine that created the certificate request.

  2. Extract to a temporary directory:











  3. Extract the files you got from the Public CA into the same directory



  4. From an elevated command prompt Change Directory into your folder and type the following command:

    openssl.exe pkcs12 -export -out server.p12 -inkey PrivateKey.txt -in SSLCert.txt

  5. OpenSSL will ask you for a Password and then ask you to confirm:



  6. And a portable certificate file will be created:



  7. Now simply import into the Windows certificate store and you're good to go

CCE unable to report into Office 365.

2x CCE Appliances. Both using the same username and password under tenant admin (checked using Get-CcCredential).

Auto Upgrade kicks in and both successfully upgrade to v2.0.0:


However looking in Office 365 Admin Portal we can see that one never checked back in with the mothership to say its upgraded:


Looking in the CceManagementService.log we saw the following:

"CceService Warning: 0 : Appliance Manager: Failed to Load or update tenant configuration. Exception: System.Management.Automation.CmdletInvocationException: Failed to logon with given credentials. Make sure correct user name and password provided. ---> Microsoft.Rtc.Admin.Authentication.CommonAuthException: Failed to logon with given credentials. Make sure correct user name and password provided. ---> Microsoft.Rtc.Admin.Authentication.IdcrlExtendedException: 
AuthState=0x80048800RequestStatus=0x80048820"

All checks that we could think of were performed (re-entering credentials, reboots, checking networking, proxies) but to no avail.

Opened a ticket with Office 365 support who at the outset seemed equally as confused. It should be noted that all through this the VM's continued to work fine and calls were traversing the CCE's - it was "only" the management service that was failing to login.

While this was happening, version 2.0.1 of CCE was released and BOTH hosts upgraded themselves:


even though still in the portal only one was reporting back in:


Eventually a reply came back to try to upgrade the Skype for Business Online, Windows PowerShell Module to a newer version, taking it from:


to:


And after restarting the Management Service we have success:



The explanation I had back from Microsoft Support was the tenant was enabled for ADAL and the latest PowerShell supports MFA enabled accounts. They were unable to explain why this was only affecting one CCE though.......

Skype Room System v2 custom image guide

Working at Modality Systems means that we get access to future tech by being part of the TAP programs for Skype for Business, usually that’s access to early versions of software (such as the redesigned Mac client, Teams, or pre-release Cumulative Updates) or new features being enabled on our Office 365 tenant (such as Auto Attendant, and Call Queues).

Iain Smith managed to get us onto the Rigel TAP program and have been using beta hardware devices in both of our main meeting rooms in our St Albans office for a fair few months going through different iterations of the software. Recently Logitech gave us a one of their Smart Docks which we have put to good use in our boardroom. The extender cable system means we can easily have the console of the Skype Room System v2 in the centre of the room driving the two Front of Room displays.

Recently Microsoft released the documentation for how to put a custom image onto the device but the documentation is clearly a v1 as it doesn’t give the full information about how the image will work over the screens.

I’ve spent a little while playing with the images and have found the following:

The SRSv2 can be deployed with either a single Front of Room display or dual displays, but for both you need to create an image that is 3840X1080 pixels. This is 2x Full HD screens stitched together. So even if your SRS has a single screen then you are require to create the same size file but it will use the right hand side of the image only.

However there is a complication in that the console will also use this image, however it is not a Full HD screen. As such I've produced the following image that you can use as a template for your SRSv2's


This image and the associated SkypeSettings.xml can be found here): 

A device with dual Front of Room displays will show the red rectangle on the left screen, the yellow and blue rectangle on the right hand screen. The console will only show the yellow rectangle (which makes the console screen resolution 1620X1080 pixels).

If you device only has a single Front of Room display then you still need to create an image that 3840X1080 pixels however only the right hand side is used.

To demonstrate this here are some images of the two meeting rooms in the Modality Systems St Albans office with the above image on both:

Single Front of Room screen system:





Dual Front of Room screen system:



If you have any questions about the SRSv2 please let me know in the comments and I'll do my best to help.

LS Data MCU error on Lync 201x & SfB 2015 after May 2017 OS patching

Update 12/12/2017 12:05 - SfB CU out taking the version to 6.0.9319.510 has a fix for this. Nothing expected for Lync Server 2013 or 2010.

Update 23/05/2017 23:12 - Official confirmation should appear under https://support.microsoft.com/en-gb/help/4023993 within 24 hours. Product Group have Development Resource assigned so looks like a CU will be coming to fix this.

Seeing multiple customers on Lync 2010, Lync 2013 and Skype for Business server 2015 front ends:

Front End event log every minute, Event ID 41026 followed by 41025:




"No connectivity with any of Web Conferencing Edge Server, External Skype for Business clients cannot use Web Conferencing modality

On the Edge server seeing the following:


"Web Conferencing Server connection failed to establish

Over the past 3 minutes Skype for Business Server has experienced incoming TLS connection failures 1 times(s). The error code of the last failure is 0x80072746 and the last connection was from the host ""."

After trying disabling IP 6 on FE and Edge:

and “On FE you can change IIS Web sites bindings to IPv4 IP address instead of all unassigned.”


The fix so far was to uninstall the May Security and Quality rollup for the .Net Framework 4.5.2, reading the release notes this hardens TLS communications for EKU so seems to fit with the error messages being shown

Server 2012: https://support.microsoft.com/en-gb/help/4014513

Server 2012 r2: https://support.microsoft.com/en-gb/help/4014597

Logged with Microsoft as ticket 117051115723411

Update 21:54 (changed title as well):

Confirmed by Microsoft as known issue and public KB is being prepared:

"This update adds an additional check on Enhanced Key Usage (EKU), since all Lync/ SfB Server usually use the Web Server template they will only have the Server Authentication in the EKU."

Issue has been reproduced on Lync 2010, Lync 2013 and Skype for Business 2015 on all supported server versions (2008r2, 2012, 2012r2).

Current Workarounds:

1 - Request new Edge Internal certificate with the Client and Server Authentication.

OR

2 - On the Front Ends disable the check for the Web Conferencing Service. Please note that these registry keys are for the default install locations.

Lync Server 2010:

reg add HKLM\SOFTWARE\Microsoft\.NETFramework\v2.0.50727\System.Net.ServicePointManager.RequireCertificateEKUs /v "C:\Program Files\Microsoft Lync Server 2010\Web Conferencing\DataMCUSvc.exe" /t REG_DWORD /d 0 /f

Note: Lync Server 2010 still uses the .NET 3.5 this is why we have the v2.0.50727.

Lync Server 2013:

reg add HKLM\SOFTWARE\Microsoft\.NETFramework\v4.0.30319\System.Net.ServicePointManager.RequireCertificateEKUs /v "C:\Program Files\Microsoft Lync Server 2013\Web Conferencing\DataMCUSvc.exe" /t REG_DWORD /d 0 /f

Skype for Business Server 2015:

reg add HKLM\SOFTWARE\Microsoft\.NETFramework\v4.0.30319\System.Net.ServicePointManager.RequireCertificateEKUs /v "C:\Program Files\Skype for Business Server 2015\Web Conferencing\DataMCUSvc.exe" /t REG_DWORD /d 0 /f

After adding the registry key simply restart the Web Conferencing service

Thanks to David Paulino (Twitter) at Microsoft for the update.

Update 22nd May 2017 11:07
Seeing different items broken in different environments from the following list: Q and A, Screen Share, Whiteboard, PowerPoint sharing via OWAS/WAK/OOS (Thanks Py7h0n and others for reporting).