Response Group Service bug in SfB Aug 2021 CU (6.0.9319.619)

TLDR: Response Groups might not work same as expected after installing CU 6.0.9319.619. to fix install CU 6.0.9319.623

SfB 2019 New Cmdlets (and removed 2015 ones)

For a little project at work I wanted to find all the new Skype for Business Server 2019 Cmdlets that had been added compared to 2015. Google failed me so did it myself.

First PowerShell to grab all from a 2015 and 2019 server:


Get-Command -Module SkypeForBusiness

Then copy the output into two files and run them through WinMerge to find the differences and:

Introduced in 2019 (found on CU 7.0.2046.244) compared to 2015 (CU 6.0.9319.591)

  • Debug-CsStorageConversationHistory
  • Debug-CsUserDelegation
  • Get-CsCloudCallDataConnector
  • Get-CsCloudCallDataConnectorConfiguration
  • Get-CsEventServiceSettings
  • Get-CsHybridConfiguration
  • Get-CsIPPhonePolicy
  • Get-CsPlatformServiceNGCSettings
  • Get-CsRecordingServiceConfiguration
  • Get-CsTenantHostingProvider
  • Get-CsTenantHybridConfiguration
  • Get-CsUpgradeDomainInfo
  • Get-CsUserCallForwardingSettings
  • Get-CsUserDelegates
  • Get-CsUserMobilityData
  • Get-CsUserSettingsPageConfiguration
  • Get-CsUserTeamMembers
  • Grant-CsDialoutPolicy
  • Grant-CsIPPhonePolicy
  • Invoke-CsRgsStoreReplicateData
  • New-CsCloudCallDataConnectorConfiguration
  • New-CsEventServiceSettings
  • New-CsIPPhonePolicy
  • New-CsNetworkConfiguration
  • New-CsPlatformServiceNGCSettings
  • New-CsPlatformServiceSettingsThrottlingConfigur...
  • New-CsRecordingServiceConfiguration
  • Remove-CsCloudCallDataConnectorConfiguration
  • Remove-CsEventServiceSettings
  • Remove-CsIPPhonePolicy
  • Remove-CsPlatformServiceNGCSettings
  • Remove-CsRecordingServiceConfiguration
  • Remove-CsRgsStoreBackupData
  • Set-CsCloudCallDataConnector
  • Set-CsCloudCallDataConnectorConfiguration
  • Set-CsEventServiceSettings
  • Set-CsHybridConfiguration
  • Set-CsIPPhonePolicy
  • Set-CsPlatformServiceNGCSettings
  • Set-CsRecordingServiceConfiguration
  • Set-CsTenantHybridConfiguration
  • Set-CsUserCallForwardingSettings
  • Set-CsUserDelegates
  • Set-CsUserSettingsPageConfiguration
  • Set-CsUserTeamMembers
  • Test-CsConferenceGateway
  • Test-CsJoinConferencing
  • Test-CsMeetingsPool
  • Test-CsOnlineMeetings
  • Test-CsPlatformService

The following have been removed in 2019 compared to 2015:

  • Get-CsPoolUpgradeReadinessState
  • Test-CsMcxConference
  • Test-CsMcxP2PIM
  • Test-CsMcxPushNotification
  • Test-CsPersistentChatMessage





Setting up a brand new MECM ADR pain

In %dayjob% I’m now starting to use Microsoft Endpoint Configuration Manager (nee System Centre [Center] Configuration Manager, nee System Management Server) a lot more.

My area I’ve been looking at is Windows Updates. It’s not a sexy area (I’ll leave that to Leon and Ben) but it pays the mortgage. I’m cleaning things up as I go, getting rid of old Software Update Groups and moving to Automatic Deployment Rules. This has led me to blasting away old downloaded updates and starting again.

However, if you’ve had an install that has been upgraded many times you might find that some of that old content won’t download again and you disappear down a rabbit hole.

A couple of hours later I have the answer to why my ADR would never complete, come with me on a journey!

First thing. My ADR is having EVERY update that has not been superseded for Windows Server 2012R2 and Server 2016, this is because as new customers come on board we don’t know the state of their OS patching, as such we want every update to be evaluated and pushed out to their servers.

Creating my ADR (and apologies, I have had to fabricate some of these screenshots as I didn’t have them while making the initial ADR). Selected Updates for Server 2012R2 and 2016


Selected the classification of updates

 

 Sit back and wait for things to download

~~~~~~~~~~~~~~Wavy lines of time.~~~~~~~~~~~~~~

Come back to ADR and…. Hum…..

 Okay, lets dig into the patchdownloader.log file to see what happened.


Strange, looks like a certificate thing…..

Humm, okay, let’s try downloading the update from All Software Updates and putting into the Deployment Package manually


I got back to the logs and find the list of files that fail to download, then I go and download them manually.

  • kb3172989
  • kb3207296
  • kb3198389
  • kb3173423
  • kb3172729
  • kb3209498

Try downloading the files manually and all download fine:

 


Okay, how about I import it manually, different error but same problem in that it didn’t import.


Okay…. Let’s go and interrogate the files and see what’s the same between them all….. hummm, all are for Server 2016 Tech Preview.

kb3172989

Security Update for Adobe Flash Player for Windows Server 2016 Technical Preview 5 (for x64-based Systems) (KB3209498)

kb3207296

Cumulative Update for Windows Server 2016 Technical Preview 5 for x64-based Systems (KB3207296)

kb3198389

Security Update for Windows Server 2016 Technical Preview 5 (KB3198389)

kb3173423

Update for Windows Server 2016 Technical Preview 5 (KB3173423)

kb3172729

Security Update for Windows Server 2016 Technical Preview 5 (KB3172729)

kb3209498

Security Update for Adobe Flash Player for Windows Server 2016 Technical Preview 5 (for x64-based Systems) (KB3209498)

 Lets go look at the files…..


 

Ohhh, so SHA256 cert on all has expired. And since August SHA1 updates are no longer allowed:
https://support.microsoft.com/en-us/help/4472027/2019-sha-2-code-signing-support-requirement-for-windows-and-wsus

My thoughts are if I had done this work in July then the updates would have downloaded fine!

So….. how to get around it. I simply excluded these updates from the ADR





And now my ADR is fully up to date!

 


Why is my Ribbon SBA showing as unpatched in vulnerability reports?


Survivable Branch Appliances are a pretty common device in multi-site companies that need local breakout for calls or need to have the ability to make calls when the connection to the Front End pool of Skype for Business servers is unavailable (WAN outage, etc).

In patching of these this question comes up at least every 3 months:
“I’ve applied all the Windows Updates that Sonus say my SBA needs however it’s still showing as needing updates in WSUS”
Or
“We’ve had a vulnerability scan and loads of updates are missing from all of our SBAs”

So what gives?

This comes down to past decisions from Microsoft and Sonus…..
In the dim and distant Microsoft used to release multiple updates every Patch Tuesday. You could choose if you wanted to install update X for GDI+ but not update Y for TrueType fonts. That meant that other software companies could say:
“Yea – we found an issue with Contoso Magic Application and KB938464 so if you want to use our software to do your business critical function don’t install that update…..
……honest – we’ll release an update to fix this in our application at some point”
And you as an administrator would be told from the business that you can’t install that update as they need Contoso Magic Application to just work.

So how does this relate to SBA’s

Each month Sonus looked at the updates available and then at the profile of the SBA with the Sonus hardened config on there and said:
“okay, so update X and Y we need, but update Z is for part of Windows that is not exposed to the network with our hardening so therefore no need to install it”
Thus inside the PKG file you download from Sonus you have a list of updates that are allowed and only those would get installed.

Then things changed

Microsoft got bored of having to support a Swiss Cheese deployment of Windows so they started in October 2016 to release single monthly updates for OS. I believe that this is in part to do with this statistic I picked up from Henk van Roest who stated “30% of support calls to Microsoft are fixed by applying updates that are already available”. You could no longer pick and choose which updates to install, you either installed this months updates or you didn’t.

However Sonus still release an update each month that contains this update. However they have not gone back and subsequently authorised the updates that came out prior to October 2016.

Where does this leave you as an Administrator?

The reason for applying updates from the Sonus PKG file is to ensure that the SBA stays in Appliance Mode. Appliance Mode means that the SBA call paths are supported by Sonus. However – you will not be able to install all Windows Updates that a offered from WSUS and thus may fall down on an audit. So, you have a choice:
  • Stay in Appliance Mode - only apply PKG files from Sonus
  • Apply all updates - keep off those audit reports
The choice is yours!


Getting a Topology when Topology Builder is not available

Quick and dirty post about how I got around an issue with an environment today.

Environment is Lync 2010 with multiple Enterprise Edition pools. A single Skype for Business 2015 server exists in the environment as a proof of concept server which means that the Lync 2010 topology builder can't deal with to download the topology any more. Unfortunately the SfB server was inaccessible and I've not been given rights to be able to help fix that, as such I needed to get the topology out of the environment using the Lync 2010 servers (note, I also didn't have any machines that I could install the SfB tools onto either).

Loading and downloading Lync 2010 Topology Builder works.....



And I get the option to choose a filename for the download, but then the tool crashes:


And no file exists on disk.

Hum.... So looking at the 2010 progress report I have the commands that I need now:


I've added pipe clip to throw the result onto the clipboard which I can then paste into notepad.

On my scratch Virtual Machine I load the 2015 Topology Builder and load the notepad file, and:


I've got an error as my XML only has the topology, I don't really care about the contents of Get-CsSimpleUrlConfiguration so that's the issue here (I suppose I could output them and stitch it together in the text file but anyway), boom:



And why did I spin up my scratch VM.... Well, my full fat machine has SfB 2019 tools installed which does not allow 2010 environments to be loaded:









CCE and Speculative Execution

Update 11th April 2018: The "A/V Gate" (registry requirement) has now been removed. This means that CCE guest VM's will now have the updates installed. Panic over :-)




Just a heads up that by default Cloud Connector Edition for Skype for Business Online will not automatically download and install the January or February 2018 Windows Updates as they don't have Anti-Virus installed and therefore do not get the QualityCompat reg key as detailed here: https://support.microsoft.com/en-us/help/4072699/

Documentation for AV on CCE only mentions the Host and not the Guest VMs: https://technet.microsoft.com/en-us/library/mt740658.aspx (so therefore you can add the key to the Host machine if no AV is present)

I suppose I could create a new base VHDX that has the QualityCompat key set (the same way that you can pause the update to add in a Proxy), but seems a bit pointless when this is supposed to be a managed update service (fire and forget).

I can see three (official) solutions that Microsoft might go with:
  1. Roll out a new version of CCE that will add the key during the build process (hopefully in a new build they will fix my Draining Calls issue: www.tobiefysh.co.uk/2017/12/cce-210-draining-calls.html)
  2. Wait for the key to no longer be required (leaves CCE guests vulnerable until then). This is a scenario that will happen eventually: "
    • "Q3: How long will Microsoft require setting a registry key to receive the Windows  security updates?
    • A3: Microsoft added this requirement to ensure customers can successfully install the January and February 2018 security updates. Microsoft will continue to enforce this requirement until there is high confidence that the majority of customers will not encounter device crashes after installing the Windows security updates."
      Source: https://support.microsoft.com/en-us/help/4072699/
  3. Tell customers that they need to manually add the key in (this is the worst option as the idea is that these VMs don't need feeding and watering like "regular" Windows servers).
Once I have an official response back from Microsoft I'll update this post.

Update 1st March 2018
After logging this via O365 Support and also via Partner Advisory support I've had back an update that this should be logged on the Skype for Business Feedback forum as it is a design change.....

I've logged it here: https://www.skypefeedback.com/forums/299913-generally-available/suggestions/33492559-cce-and-speculative-update I'd appreciate votes.

And yes - I'm aware I called it Speculative Update (not Speculative Execution) on the feedback  

We discussed this internally and from the great contacts we have at Modality Systems we have now got an escalation into the Product Group. One late night phone call later and the problem is understood internally at Microsoft. Looking forward to getting some traction on this now. :-)

Skype for Business server 2015 CU appearing in Windows Update again

Looks like Microsoft have started pushing the latest Skype for Business 2015 CU via Automatic Updates:






Even thought the master KB for updates still says that they wont do this:





Seen on Edge, Front End, stand alone Mediation and PChat servers.

A change in policy at Microsoft or someone messing up?

If you do try to install this way then you're going to get a nice error as the CU (as usual) requires that the SfB Services are stopped:




If you do stop the services (Stop-CsWindowsService) prior to running Windows Update, then the update will pop the installer window as if you had manually downloaded the update:



As there is no database update since .281 maybe this is an okay way to install the updates, but just remember to restart the services afterwards if you are not going to be restarting the server!

To be honest anyone who wants to have control over the deployment of the CU wont be allowing this anyway as they would control via WSUS/SCCM etc.