CCE and Speculative Execution

Just a heads up that by default Cloud Connector Edition for Skype for Business Online will not automatically download and install the January or February 2018 Windows Updates as they don't have Anti-Virus installed and therefore do not get the QualityCompat reg key as detailed here:

Documentation for AV on CCE only mentions the Host and not the Guest VMs: (so therefore you can add the key to the Host machine if no AV is present)

I suppose I could create a new base VHDX that has the QualityCompat key set (the same way that you can pause the update to add in a Proxy), but seems a bit pointless when this is supposed to be a managed update service (fire and forget).

I can see three (official) solutions that Microsoft might go with:
  1. Roll out a new version of CCE that will add the key during the build process (hopefully in a new build they will fix my Draining Calls issue:
  2. Wait for the key to no longer be required (leaves CCE guests vulnerable until then). This is a scenario that will happen eventually: "
    • "Q3: How long will Microsoft require setting a registry key to receive the Windows  security updates?
    • A3: Microsoft added this requirement to ensure customers can successfully install the January and February 2018 security updates. Microsoft will continue to enforce this requirement until there is high confidence that the majority of customers will not encounter device crashes after installing the Windows security updates."
  3. Tell customers that they need to manually add the key in (this is the worst option as the idea is that these VMs don't need feeding and watering like "regular" Windows servers).
Once I have an official response back from Microsoft I'll update this post.

Update 1st March 2018
After logging this via O365 Support and also via Partner Advisory support I've had back an update that this should be logged on the Skype for Business Feedback forum as it is a design change.....

I've logged it here: I'd appreciate votes.

And yes - I'm aware I called it Speculative Update (not Speculative Execution) on the feedback  

We discussed this internally and from the great contacts we have at Modality Systems we have now got an escalation into the Product Group. One late night phone call later and the problem is understood internally at Microsoft. Looking forward to getting some traction on this now. :-)

Skype for Business server 2015 CU appearing in Windows Update again

Looks like Microsoft have started pushing the latest Skype for Business 2015 CU via Automatic Updates:

Even thought the master KB for updates still says that they wont do this:

Seen on Edge, Front End, stand alone Mediation and PChat servers.

A change in policy at Microsoft or someone messing up?

If you do try to install this way then you're going to get a nice error as the CU (as usual) requires that the SfB Services are stopped:

If you do stop the services (Stop-CsWindowsService) prior to running Windows Update, then the update will pop the installer window as if you had manually downloaded the update:

As there is no database update since .281 maybe this is an okay way to install the updates, but just remember to restart the services afterwards if you are not going to be restarting the server!

To be honest anyone who wants to have control over the deployment of the CU wont be allowing this anyway as they would control via WSUS/SCCM etc.

Unable to login to Skype for Business Online with BT Home Hub 6 - part 2

My frustrations with using the BT Home Hub 6 and Skype for Business Online are documented here:

Here's my write up on how I've fixed it:

First thing I tired was contacting BT. My first call was not great, eventually I got through to a team who I was told would be happy to talk to me about the issues but they would want a credit card number..... I made my excuses and left....   ;-)

I tried again and got through to a grumpy lady who (after I asked if she could disable IP6 on the Home Hub 6) literally said:

and said I should send the Home Hub 6 back < sigh > 

I went digging into the Home Hub 6 and found that I have both IP6 and IP4 public IP addresses, meaning things like my Tado which don't support IP6 can continue to work:

However my work laptop has an IP4 and IP6 address:

So the easiest thing to force my SfB client to talk to the O365 homed servers would be to disable IP6 on my laptop. The correct way of doing that is documented here:

But doing so would break Direct Access on my laptop, which would be a Bad Thing (TM)

Instead I forced the laptop to prefer IP4 over IP6 by making the following registry change:

(reg change file here:!Arx7Ss1l4DQIgZSrJsx7M0EtARKBXuI)

After a reboot I'm in business!

Hope that this helps someone out there.

Presence Unknown..... BUT WHY!

Have had a long running issue with a single user at a customer where I was unable to IM them or see their presence:

However they could IM me and see my presence fine.

The customer has on-prem Lync servers. I am on Office 365 which is setup in a Hybrid with our on-prem servers. Other people in Modality who are on-prem (Response Group users) could see this person fine (you want a name - okay, it's Leon).

It should be noted that I used to be able to see the presence and IM with no issue, also, after I moved to O365 I could. Something changed later* Anyway - back to the story....

When Leon IM'd me I would get errors like this in my event logs:

504  Server Time-Out
ms-diagnostics:  27002;reason="From-Uri Domain is not in the receiver-tenant allow list";source="Office365ServerName.INFRA.LYNC.COM";appName="IncomingFederation";OriginalPresenceState="0";CurrentPresenceState="0";MeInsideUser="No";ConversationInitiatedBy="6";SourceNetwork="5";RemotePartyCanDoIM="Yes"

A search on that error didn't really bring anything up of value as it was talking about the whole domain needing white listing and that couldn't be correct as it was a single user issue. We tried moving to different PC's, different networks, investigated policies that Leon had but all came up nil.

The issue wasn't a big enough pain for Leon to want to spend too much time troubleshooting but eventually while discussing about their customers Office 365 plans a light bulb went off.

"Leon, have you got your user account in Office 365 as well"

After confirming he had it was as simple as turning off Skype for Business for his user account in the customers O365 tenant:

and we were back in business:

So what happened?

My account is in Office 365. Leon's account was on-prem. He also had an account in Office 365 but Hybrid was not setup.

Therefore, when Leon IM'd me, his client talked to his on-prem Edge, resolved the DNS for Modality Edge, and got proxied to me in O365.

However, when I attempted to IM Leon, my client talked to O365, who saw that there was a matching O365 tenant for the domain and sent the IM there. Simply turning off Leon from having an Office 365 Skype for Business account allowed the Modality Office 365 tenant to ignore looking up his details in the cloud, I found the customers Edge server and all was well in the world.

Simple when you know how!

*what changed? The customer got Office 365 but had not setup all the hybrid integration (as they didn't want to use it all at that time).

CCE 2.1.0 - Draining Calls

The Cloud Connector Edition system for Skype for Business online is a very impressive collection of scripts and glue and luck that has the ability to build a complete SfB voice infrastructure and patch it automatically. One of the parts of this is not as good as it could be, read on if you get call drops when updating :-)

Environment is 2x CCE hosts in the same site. For the test below I first put the CCE2 into maintenance and make calls. Therefore forcing all calls through CCE1.

I make two calls:

Ensure that both calls are up and running and show on the Sonus:

Logging into the Mediation Server on the CCE1 you can see the calls are running through it:

Take CCE2 out of maintenance so that both CCE's take service calls (both calls are still running on CCE1).

On CCE1 run the command Enter-CcUpdate on an elevated PowerShell session

Based on the documentation ( I would expect this to drain the mediation server, waiting for the two nailed up calls to complete and to then gracefully stop the services.

Instead I get this:

Services are stopped and both calls are dropped. Bit of a fail when the documentation says:

"The appliance is “drained”—that is, all existing calls will complete, but new calls are rejected."


"The Enter-CcUpdate cmdlet will ensure that all running calls on a Cloud Connector appliance will complete, but the appliance will reject any new calls, which are transferred to other production appliances. This cmdlet enables you to update an appliance without affecting end users calls." (my emphasis!)

(Bonus points for the spelling of "Drainning" and "Forceing")

Its now logged as a ticket with Microsoft and I'll update as and when I have a resolution.

Update: 22nd January 2018
Confirmed as a bug and passed to Product Group to address. Workaround is to connect to Mediation Server and perform a Stop-CSWindowsService -Graceful command

Unable to login to Skype for Business Online with BT Home Hub 6

This is an embarrassing post to write but I hope it will help someone out there in troubleshooting! Anyway, on with the story:

Working for Modality Systems is great, but when I joined I pointed out that the daily commute was not something I fancied:

(side note - 2 hour 11 minutes - in which universe??)

Luckily working from home is one of those things you can do with the magic of Unified Communications Intelligent Communications as work really is that thing you do and not the place you go.

And that's fine until it stops working, lets set the scene.....

Its the Thursday after patch Tuesday. The previous day (Wednesday) I was in the Modality Systems main off in St Albans. Laptop had updated and working fine, Office install is Click-to-run. My user account is sync'ed from our internal domain to Office 365 and my Skype for Business account is homed in Office 365 with our setup being hybrid.

I boot my laptop and login. Direct access does its stuff, Group Policies apply, Outlook, Teams, OneNote, and SfB all load up and I start on some emails. I need to reach out to a colleague so switch to SfB and am presented by this:

"interesting" I think, I wonder why Skype is not signing in. Its been 15 minutes so since I logged onto the laptop so its really stuck, click cancel and try again but no joy..... 

Must be an O365 outage on Skype only, Teams and Outlook are fine and I can browse the internet with no issues so I plod over to the portal to check service status....

.....okay, so something about my account then, I check the internal AD and Azure AD, nothing looks out of place. I clear out the certificates from my local store....

....delete the contents of C:\Users\tobie.fysh\AppData\Local\Microsoft\Office\16.0\Lync and reboot, still no joy!

Boot up a laptop that has not been patched (and as a separate change, runs Office MSI). Still no joy. Leave that updating to latest versions and try and see this:

Ohhhh, so something is broken on our tenant then! I click the to “tell me more” and it takes me to:

Post on our internal Teams chat and Email some of the guys in the office (feeling like a failure at this point). No-one else in the org appears to be having issues so seems to be local to me.

While awaiting a reply go think maybe Edge related (as we are hybrid the DNS records point to our on-prem Edge server so prior to the endpointcache being updated I'm going to be hitting that). I RDPto a VM inside the network (over the magic that is Direct Access and IP4 to IP6 natting) and load Lync 2013 (its a test box) and I can login to my account!

So if I'm external to our network I can login to SfB but inside the network I'm fine?? Can't be tenant related. Maybe something about our Azure AD boxes, scratching head here.....

Suggestion comes back from colleague to try my mobile app:

wait... wait... wait...

No joy.

Okay, so looks like my account, for fun I switch to bounSky and just check that I can login using my client to another estate and bang, I'm in. I test a few (Lync Server 2010, 2013 and a SfB 2015 server) all okay.

I then try a pure cloud customer.

And can't login..... I try a second customer who is hybrid and I can't login to a O365 user. On that same customer I then try a user account that is homed on-prem and CAN login (and these tenants are all hosted in different regions to the Modality tenant).

Brain is tied in knots now. About to log a ticket with O365 support but for fun think that there might be something about my home network. I turn on the hotspot fucntion of my phone connect the laptop to it and bang, the lovely 4G network allows me to login straight away.

Have a think about my network. What's changed recently. My Router. My lovely new Home Hub 6. Thanks BT, maybe its you! I reboot the router.

No joy, I then remember that I commented while doing firewall traces last week that I appeared to have an IP6 address with my new router:

I dig out the old Home Hub 5 (it was in the returns box awaiting to go back to BT) and plumb it in...... I now have an IP4 address:

And immediately desktop Skype for Business signs in.

My phone, signs in (it was on the WIFI before, go check the screenshot!)

Obviously Mr Cropley has already tweeted a reply to me:

And directs me to the source:

So - I need to support customers who wont have enabled IP6 in their tenants so as a consequence the Home Hub 5 is back in pride of place beside the fish tank and the HH6 is being say in the corner like a naughty child

I've tweeted BT to see if they can help disable IP6 on my account:

and will update if I get a response.

Update 14th December: Here's the fix!

PEM certificate files on Windows

While doing certificate renewals for a client recently I was given PEM format files which I needed to convert into a certificate that Windows can consume.

  1. Download a copy of OpenSSL which has been complied for Windows (
    Note: You can do this on your workstation, it does not need to be done on the same machine that created the certificate request.

  2. Extract to a temporary directory:

  3. Extract the files you got from the Public CA into the same directory

  4. From an elevated command prompt Change Directory into your folder and type the following command:

    openssl.exe pkcs12 -export -out server.p12 -inkey PrivateKey.txt -in SSLCert.txt

  5. OpenSSL will ask you for a Password and then ask you to confirm:

  6. And a portable certificate file will be created:

  7. Now simply import into the Windows certificate store and you're good to go