Why is my Ribbon SBA showing as unpatched in vulnerability reports?


Survivable Branch Appliances are a pretty common device in multi-site companies that need local breakout for calls or need to have the ability to make calls when the connection to the Front End pool of Skype for Business servers is unavailable (WAN outage, etc).

In patching of these this question comes up at least every 3 months:
“I’ve applied all the Windows Updates that Sonus say my SBA needs however it’s still showing as needing updates in WSUS”
Or
“We’ve had a vulnerability scan and loads of updates are missing from all of our SBAs”

So what gives?

This comes down to past decisions from Microsoft and Sonus…..
In the dim and distant Microsoft used to release multiple updates every Patch Tuesday. You could choose if you wanted to install update X for GDI+ but not update Y for TrueType fonts. That meant that other software companies could say:
“Yea – we found an issue with Contoso Magic Application and KB938464 so if you want to use our software to do your business critical function don’t install that update…..
……honest – we’ll release an update to fix this in our application at some point”
And you as an administrator would be told from the business that you can’t install that update as they need Contoso Magic Application to just work.

So how does this relate to SBA’s

Each month Sonus looked at the updates available and then at the profile of the SBA with the Sonus hardened config on there and said:
“okay, so update X and Y we need, but update Z is for part of Windows that is not exposed to the network with our hardening so therefore no need to install it”
Thus inside the PKG file you download from Sonus you have a list of updates that are allowed and only those would get installed.

Then things changed

Microsoft got bored of having to support a Swiss Cheese deployment of Windows so they started in October 2016 to release single monthly updates for OS. I believe that this is in part to do with this statistic I picked up from Henk van Roest who stated “30% of support calls to Microsoft are fixed by applying updates that are already available”. You could no longer pick and choose which updates to install, you either installed this months updates or you didn’t.

However Sonus still release an update each month that contains this update. However they have not gone back and subsequently authorised the updates that came out prior to October 2016.

Where does this leave you as an Administrator?

The reason for applying updates from the Sonus PKG file is to ensure that the SBA stays in Appliance Mode. Appliance Mode means that the SBA call paths are supported by Sonus. However – you will not be able to install all Windows Updates that a offered from WSUS and thus may fall down on an audit. So, you have a choice:
  • Stay in Appliance Mode - only apply PKG files from Sonus
  • Apply all updates - keep off those audit reports
The choice is yours!


Getting a Topology when Topology Builder is not available

Quick and dirty post about how I got around an issue with an environment today.

Environment is Lync 2010 with multiple Enterprise Edition pools. A single Skype for Business 2015 server exists in the environment as a proof of concept server which means that the Lync 2010 topology builder can't deal with to download the topology any more. Unfortunately the SfB server was inaccessible and I've not been given rights to be able to help fix that, as such I needed to get the topology out of the environment using the Lync 2010 servers (note, I also didn't have any machines that I could install the SfB tools onto either).

Loading and downloading Lync 2010 Topology Builder works.....



And I get the option to choose a filename for the download, but then the tool crashes:


And no file exists on disk.

Hum.... So looking at the 2010 progress report I have the commands that I need now:


I've added pipe clip to throw the result onto the clipboard which I can then paste into notepad.

On my scratch Virtual Machine I load the 2015 Topology Builder and load the notepad file, and:


I've got an error as my XML only has the topology, I don't really care about the contents of Get-CsSimpleUrlConfiguration so that's the issue here (I suppose I could output them and stitch it together in the text file but anyway), boom:



And why did I spin up my scratch VM.... Well, my full fat machine has SfB 2019 tools installed which does not allow 2010 environments to be loaded:









CCE and Speculative Execution

Update 11th April 2018: The "A/V Gate" (registry requirement) has now been removed. This means that CCE guest VM's will now have the updates installed. Panic over :-)




Just a heads up that by default Cloud Connector Edition for Skype for Business Online will not automatically download and install the January or February 2018 Windows Updates as they don't have Anti-Virus installed and therefore do not get the QualityCompat reg key as detailed here: https://support.microsoft.com/en-us/help/4072699/

Documentation for AV on CCE only mentions the Host and not the Guest VMs: https://technet.microsoft.com/en-us/library/mt740658.aspx (so therefore you can add the key to the Host machine if no AV is present)

I suppose I could create a new base VHDX that has the QualityCompat key set (the same way that you can pause the update to add in a Proxy), but seems a bit pointless when this is supposed to be a managed update service (fire and forget).

I can see three (official) solutions that Microsoft might go with:
  1. Roll out a new version of CCE that will add the key during the build process (hopefully in a new build they will fix my Draining Calls issue: www.tobiefysh.co.uk/2017/12/cce-210-draining-calls.html)
  2. Wait for the key to no longer be required (leaves CCE guests vulnerable until then). This is a scenario that will happen eventually: "
    • "Q3: How long will Microsoft require setting a registry key to receive the Windows  security updates?
    • A3: Microsoft added this requirement to ensure customers can successfully install the January and February 2018 security updates. Microsoft will continue to enforce this requirement until there is high confidence that the majority of customers will not encounter device crashes after installing the Windows security updates."
      Source: https://support.microsoft.com/en-us/help/4072699/
  3. Tell customers that they need to manually add the key in (this is the worst option as the idea is that these VMs don't need feeding and watering like "regular" Windows servers).
Once I have an official response back from Microsoft I'll update this post.

Update 1st March 2018
After logging this via O365 Support and also via Partner Advisory support I've had back an update that this should be logged on the Skype for Business Feedback forum as it is a design change.....

I've logged it here: https://www.skypefeedback.com/forums/299913-generally-available/suggestions/33492559-cce-and-speculative-update I'd appreciate votes.

And yes - I'm aware I called it Speculative Update (not Speculative Execution) on the feedback  

We discussed this internally and from the great contacts we have at Modality Systems we have now got an escalation into the Product Group. One late night phone call later and the problem is understood internally at Microsoft. Looking forward to getting some traction on this now. :-)

Skype for Business server 2015 CU appearing in Windows Update again

Looks like Microsoft have started pushing the latest Skype for Business 2015 CU via Automatic Updates:






Even thought the master KB for updates still says that they wont do this:





Seen on Edge, Front End, stand alone Mediation and PChat servers.

A change in policy at Microsoft or someone messing up?

If you do try to install this way then you're going to get a nice error as the CU (as usual) requires that the SfB Services are stopped:




If you do stop the services (Stop-CsWindowsService) prior to running Windows Update, then the update will pop the installer window as if you had manually downloaded the update:



As there is no database update since .281 maybe this is an okay way to install the updates, but just remember to restart the services afterwards if you are not going to be restarting the server!

To be honest anyone who wants to have control over the deployment of the CU wont be allowing this anyway as they would control via WSUS/SCCM etc.

Unable to login to Skype for Business Online with BT Home Hub 6 - part 2

My frustrations with using the BT Home Hub 6 and Skype for Business Online are documented here:
www.tobiefysh.co.uk/2017/11/unable-to-login-to-skype-for-business.html

Here's my write up on how I've fixed it:

First thing I tired was contacting BT. My first call was not great, eventually I got through to a team who I was told would be happy to talk to me about the issues but they would want a credit card number..... I made my excuses and left....   ;-)

I tried again and got through to a grumpy lady who (after I asked if she could disable IP6 on the Home Hub 6) literally said:


and said I should send the Home Hub 6 back < sigh > 

I went digging into the Home Hub 6 and found that I have both IP6 and IP4 public IP addresses, meaning things like my Tado which don't support IP6 can continue to work:


However my work laptop has an IP4 and IP6 address:


So the easiest thing to force my SfB client to talk to the O365 homed servers would be to disable IP6 on my laptop. The correct way of doing that is documented here:

https://support.microsoft.com/en-us/help/929852/how-to-disable-ipv6-or-its-components-in-windows

But doing so would break Direct Access on my laptop, which would be a Bad Thing (TM)

Instead I forced the laptop to prefer IP4 over IP6 by making the following registry change:


(reg change file here: https://1drv.ms/u/s!Arx7Ss1l4DQIgZSrJsx7M0EtARKBXuI)

After a reboot I'm in business!

Hope that this helps someone out there.






Presence Unknown..... BUT WHY!

Have had a long running issue with a single user at a customer where I was unable to IM them or see their presence:


However they could IM me and see my presence fine.

The customer has on-prem Lync servers. I am on Office 365 which is setup in a Hybrid with our on-prem servers. Other people in Modality who are on-prem (Response Group users) could see this person fine (you want a name - okay, it's Leon).

It should be noted that I used to be able to see the presence and IM with no issue, also, after I moved to O365 I could. Something changed later* Anyway - back to the story....

When Leon IM'd me I would get errors like this in my event logs:

504  Server Time-Out
ms-diagnostics:  27002;reason="From-Uri Domain is not in the receiver-tenant allow list";source="Office365ServerName.INFRA.LYNC.COM";appName="IncomingFederation";OriginalPresenceState="0";CurrentPresenceState="0";MeInsideUser="No";ConversationInitiatedBy="6";SourceNetwork="5";RemotePartyCanDoIM="Yes"


A search on that error didn't really bring anything up of value as it was talking about the whole domain needing white listing and that couldn't be correct as it was a single user issue. We tried moving to different PC's, different networks, investigated policies that Leon had but all came up nil.

The issue wasn't a big enough pain for Leon to want to spend too much time troubleshooting but eventually while discussing about their customers Office 365 plans a light bulb went off.

"Leon, have you got your user account in Office 365 as well"

After confirming he had it was as simple as turning off Skype for Business for his user account in the customers O365 tenant:


and we were back in business:



So what happened?

My account is in Office 365. Leon's account was on-prem. He also had an account in Office 365 but Hybrid was not setup.

Therefore, when Leon IM'd me, his client talked to his on-prem Edge, resolved the DNS for Modality Edge, and got proxied to me in O365.

However, when I attempted to IM Leon, my client talked to O365, who saw that there was a matching O365 tenant for the domain and sent the IM there. Simply turning off Leon from having an Office 365 Skype for Business account allowed the Modality Office 365 tenant to ignore looking up his details in the cloud, I found the customers Edge server and all was well in the world.

Simple when you know how!

*what changed? The customer got Office 365 but had not setup all the hybrid integration (as they didn't want to use it all at that time).

CCE 2.1.0 - Draining Calls

The Cloud Connector Edition system for Skype for Business online is a very impressive collection of scripts and glue and luck that has the ability to build a complete SfB voice infrastructure and patch it automatically. One of the parts of this is not as good as it could be, read on if you get call drops when updating :-)

Environment is 2x CCE hosts in the same site. For the test below I first put the CCE2 into maintenance and make calls. Therefore forcing all calls through CCE1.

I make two calls:

Ensure that both calls are up and running and show on the Sonus:


Logging into the Mediation Server on the CCE1 you can see the calls are running through it:



Take CCE2 out of maintenance so that both CCE's take service calls (both calls are still running on CCE1).

On CCE1 run the command Enter-CcUpdate on an elevated PowerShell session

Based on the documentation (https://technet.microsoft.com/en-us/library/mt492520.aspx) I would expect this to drain the mediation server, waiting for the two nailed up calls to complete and to then gracefully stop the services.

Instead I get this:



Services are stopped and both calls are dropped. Bit of a fail when the documentation says:

"The appliance is “drained”—that is, all existing calls will complete, but new calls are rejected."

and

"The Enter-CcUpdate cmdlet will ensure that all running calls on a Cloud Connector appliance will complete, but the appliance will reject any new calls, which are transferred to other production appliances. This cmdlet enables you to update an appliance without affecting end users calls." (my emphasis!)

(Bonus points for the spelling of "Drainning" and "Forceing")

Its now logged as a ticket with Microsoft and I'll update as and when I have a resolution.


Update: 22nd January 2018
Confirmed as a bug and passed to Product Group to address. Workaround is to connect to Mediation Server and perform a Stop-CSWindowsService -Graceful command