Setting up a brand new MECM ADR pain

In %dayjob% I’m now starting to use Microsoft Endpoint Configuration Manager (nee System Centre [Center] Configuration Manager, nee System Management Server) a lot more.

My area I’ve been looking at is Windows Updates. It’s not a sexy area (I’ll leave that to Leon and Ben) but it pays the mortgage. I’m cleaning things up as I go, getting rid of old Software Update Groups and moving to Automatic Deployment Rules. This has led me to blasting away old downloaded updates and starting again.

However, if you’ve had an install that has been upgraded many times you might find that some of that old content won’t download again and you disappear down a rabbit hole.

A couple of hours later I have the answer to why my ADR would never complete, come with me on a journey!

First thing. My ADR is having EVERY update that has not been superseded for Windows Server 2012R2 and Server 2016, this is because as new customers come on board we don’t know the state of their OS patching, as such we want every update to be evaluated and pushed out to their servers.

Creating my ADR (and apologies, I have had to fabricate some of these screenshots as I didn’t have them while making the initial ADR). Selected Updates for Server 2012R2 and 2016


Selected the classification of updates

 

 Sit back and wait for things to download

~~~~~~~~~~~~~~Wavy lines of time.~~~~~~~~~~~~~~

Come back to ADR and…. Hum…..

 Okay, lets dig into the patchdownloader.log file to see what happened.


Strange, looks like a certificate thing…..

Humm, okay, let’s try downloading the update from All Software Updates and putting into the Deployment Package manually


I got back to the logs and find the list of files that fail to download, then I go and download them manually.

  • kb3172989
  • kb3207296
  • kb3198389
  • kb3173423
  • kb3172729
  • kb3209498

Try downloading the files manually and all download fine:

 


Okay, how about I import it manually, different error but same problem in that it didn’t import.


Okay…. Let’s go and interrogate the files and see what’s the same between them all….. hummm, all are for Server 2016 Tech Preview.

kb3172989

Security Update for Adobe Flash Player for Windows Server 2016 Technical Preview 5 (for x64-based Systems) (KB3209498)

kb3207296

Cumulative Update for Windows Server 2016 Technical Preview 5 for x64-based Systems (KB3207296)

kb3198389

Security Update for Windows Server 2016 Technical Preview 5 (KB3198389)

kb3173423

Update for Windows Server 2016 Technical Preview 5 (KB3173423)

kb3172729

Security Update for Windows Server 2016 Technical Preview 5 (KB3172729)

kb3209498

Security Update for Adobe Flash Player for Windows Server 2016 Technical Preview 5 (for x64-based Systems) (KB3209498)

 Lets go look at the files…..


 

Ohhh, so SHA256 cert on all has expired. And since August SHA1 updates are no longer allowed:
https://support.microsoft.com/en-us/help/4472027/2019-sha-2-code-signing-support-requirement-for-windows-and-wsus

My thoughts are if I had done this work in July then the updates would have downloaded fine!

So….. how to get around it. I simply excluded these updates from the ADR





And now my ADR is fully up to date!

 


Why is my Ribbon SBA showing as unpatched in vulnerability reports?


Survivable Branch Appliances are a pretty common device in multi-site companies that need local breakout for calls or need to have the ability to make calls when the connection to the Front End pool of Skype for Business servers is unavailable (WAN outage, etc).

In patching of these this question comes up at least every 3 months:
“I’ve applied all the Windows Updates that Sonus say my SBA needs however it’s still showing as needing updates in WSUS”
Or
“We’ve had a vulnerability scan and loads of updates are missing from all of our SBAs”

So what gives?

This comes down to past decisions from Microsoft and Sonus…..
In the dim and distant Microsoft used to release multiple updates every Patch Tuesday. You could choose if you wanted to install update X for GDI+ but not update Y for TrueType fonts. That meant that other software companies could say:
“Yea – we found an issue with Contoso Magic Application and KB938464 so if you want to use our software to do your business critical function don’t install that update…..
……honest – we’ll release an update to fix this in our application at some point”
And you as an administrator would be told from the business that you can’t install that update as they need Contoso Magic Application to just work.

So how does this relate to SBA’s

Each month Sonus looked at the updates available and then at the profile of the SBA with the Sonus hardened config on there and said:
“okay, so update X and Y we need, but update Z is for part of Windows that is not exposed to the network with our hardening so therefore no need to install it”
Thus inside the PKG file you download from Sonus you have a list of updates that are allowed and only those would get installed.

Then things changed

Microsoft got bored of having to support a Swiss Cheese deployment of Windows so they started in October 2016 to release single monthly updates for OS. I believe that this is in part to do with this statistic I picked up from Henk van Roest who stated “30% of support calls to Microsoft are fixed by applying updates that are already available”. You could no longer pick and choose which updates to install, you either installed this months updates or you didn’t.

However Sonus still release an update each month that contains this update. However they have not gone back and subsequently authorised the updates that came out prior to October 2016.

Where does this leave you as an Administrator?

The reason for applying updates from the Sonus PKG file is to ensure that the SBA stays in Appliance Mode. Appliance Mode means that the SBA call paths are supported by Sonus. However – you will not be able to install all Windows Updates that a offered from WSUS and thus may fall down on an audit. So, you have a choice:
  • Stay in Appliance Mode - only apply PKG files from Sonus
  • Apply all updates - keep off those audit reports
The choice is yours!


Getting a Topology when Topology Builder is not available

Quick and dirty post about how I got around an issue with an environment today.

Environment is Lync 2010 with multiple Enterprise Edition pools. A single Skype for Business 2015 server exists in the environment as a proof of concept server which means that the Lync 2010 topology builder can't deal with to download the topology any more. Unfortunately the SfB server was inaccessible and I've not been given rights to be able to help fix that, as such I needed to get the topology out of the environment using the Lync 2010 servers (note, I also didn't have any machines that I could install the SfB tools onto either).

Loading and downloading Lync 2010 Topology Builder works.....



And I get the option to choose a filename for the download, but then the tool crashes:


And no file exists on disk.

Hum.... So looking at the 2010 progress report I have the commands that I need now:


I've added pipe clip to throw the result onto the clipboard which I can then paste into notepad.

On my scratch Virtual Machine I load the 2015 Topology Builder and load the notepad file, and:


I've got an error as my XML only has the topology, I don't really care about the contents of Get-CsSimpleUrlConfiguration so that's the issue here (I suppose I could output them and stitch it together in the text file but anyway), boom:



And why did I spin up my scratch VM.... Well, my full fat machine has SfB 2019 tools installed which does not allow 2010 environments to be loaded:









CCE and Speculative Execution

Update 11th April 2018: The "A/V Gate" (registry requirement) has now been removed. This means that CCE guest VM's will now have the updates installed. Panic over :-)




Just a heads up that by default Cloud Connector Edition for Skype for Business Online will not automatically download and install the January or February 2018 Windows Updates as they don't have Anti-Virus installed and therefore do not get the QualityCompat reg key as detailed here: https://support.microsoft.com/en-us/help/4072699/

Documentation for AV on CCE only mentions the Host and not the Guest VMs: https://technet.microsoft.com/en-us/library/mt740658.aspx (so therefore you can add the key to the Host machine if no AV is present)

I suppose I could create a new base VHDX that has the QualityCompat key set (the same way that you can pause the update to add in a Proxy), but seems a bit pointless when this is supposed to be a managed update service (fire and forget).

I can see three (official) solutions that Microsoft might go with:
  1. Roll out a new version of CCE that will add the key during the build process (hopefully in a new build they will fix my Draining Calls issue: www.tobiefysh.co.uk/2017/12/cce-210-draining-calls.html)
  2. Wait for the key to no longer be required (leaves CCE guests vulnerable until then). This is a scenario that will happen eventually: "
    • "Q3: How long will Microsoft require setting a registry key to receive the Windows  security updates?
    • A3: Microsoft added this requirement to ensure customers can successfully install the January and February 2018 security updates. Microsoft will continue to enforce this requirement until there is high confidence that the majority of customers will not encounter device crashes after installing the Windows security updates."
      Source: https://support.microsoft.com/en-us/help/4072699/
  3. Tell customers that they need to manually add the key in (this is the worst option as the idea is that these VMs don't need feeding and watering like "regular" Windows servers).
Once I have an official response back from Microsoft I'll update this post.

Update 1st March 2018
After logging this via O365 Support and also via Partner Advisory support I've had back an update that this should be logged on the Skype for Business Feedback forum as it is a design change.....

I've logged it here: https://www.skypefeedback.com/forums/299913-generally-available/suggestions/33492559-cce-and-speculative-update I'd appreciate votes.

And yes - I'm aware I called it Speculative Update (not Speculative Execution) on the feedback  

We discussed this internally and from the great contacts we have at Modality Systems we have now got an escalation into the Product Group. One late night phone call later and the problem is understood internally at Microsoft. Looking forward to getting some traction on this now. :-)

Skype for Business server 2015 CU appearing in Windows Update again

Looks like Microsoft have started pushing the latest Skype for Business 2015 CU via Automatic Updates:






Even thought the master KB for updates still says that they wont do this:





Seen on Edge, Front End, stand alone Mediation and PChat servers.

A change in policy at Microsoft or someone messing up?

If you do try to install this way then you're going to get a nice error as the CU (as usual) requires that the SfB Services are stopped:




If you do stop the services (Stop-CsWindowsService) prior to running Windows Update, then the update will pop the installer window as if you had manually downloaded the update:



As there is no database update since .281 maybe this is an okay way to install the updates, but just remember to restart the services afterwards if you are not going to be restarting the server!

To be honest anyone who wants to have control over the deployment of the CU wont be allowing this anyway as they would control via WSUS/SCCM etc.

Unable to login to Skype for Business Online with BT Home Hub 6 - part 2

My frustrations with using the BT Home Hub 6 and Skype for Business Online are documented here:
www.tobiefysh.co.uk/2017/11/unable-to-login-to-skype-for-business.html

Here's my write up on how I've fixed it:

First thing I tired was contacting BT. My first call was not great, eventually I got through to a team who I was told would be happy to talk to me about the issues but they would want a credit card number..... I made my excuses and left....   ;-)

I tried again and got through to a grumpy lady who (after I asked if she could disable IP6 on the Home Hub 6) literally said:


and said I should send the Home Hub 6 back < sigh > 

I went digging into the Home Hub 6 and found that I have both IP6 and IP4 public IP addresses, meaning things like my Tado which don't support IP6 can continue to work:


However my work laptop has an IP4 and IP6 address:


So the easiest thing to force my SfB client to talk to the O365 homed servers would be to disable IP6 on my laptop. The correct way of doing that is documented here:

https://support.microsoft.com/en-us/help/929852/how-to-disable-ipv6-or-its-components-in-windows

But doing so would break Direct Access on my laptop, which would be a Bad Thing (TM)

Instead I forced the laptop to prefer IP4 over IP6 by making the following registry change:


(reg change file here: https://1drv.ms/u/s!Arx7Ss1l4DQIgZSrJsx7M0EtARKBXuI)

After a reboot I'm in business!

Hope that this helps someone out there.






Presence Unknown..... BUT WHY!

Have had a long running issue with a single user at a customer where I was unable to IM them or see their presence:


However they could IM me and see my presence fine.

The customer has on-prem Lync servers. I am on Office 365 which is setup in a Hybrid with our on-prem servers. Other people in Modality who are on-prem (Response Group users) could see this person fine (you want a name - okay, it's Leon).

It should be noted that I used to be able to see the presence and IM with no issue, also, after I moved to O365 I could. Something changed later* Anyway - back to the story....

When Leon IM'd me I would get errors like this in my event logs:

504  Server Time-Out
ms-diagnostics:  27002;reason="From-Uri Domain is not in the receiver-tenant allow list";source="Office365ServerName.INFRA.LYNC.COM";appName="IncomingFederation";OriginalPresenceState="0";CurrentPresenceState="0";MeInsideUser="No";ConversationInitiatedBy="6";SourceNetwork="5";RemotePartyCanDoIM="Yes"


A search on that error didn't really bring anything up of value as it was talking about the whole domain needing white listing and that couldn't be correct as it was a single user issue. We tried moving to different PC's, different networks, investigated policies that Leon had but all came up nil.

The issue wasn't a big enough pain for Leon to want to spend too much time troubleshooting but eventually while discussing about their customers Office 365 plans a light bulb went off.

"Leon, have you got your user account in Office 365 as well"

After confirming he had it was as simple as turning off Skype for Business for his user account in the customers O365 tenant:


and we were back in business:



So what happened?

My account is in Office 365. Leon's account was on-prem. He also had an account in Office 365 but Hybrid was not setup.

Therefore, when Leon IM'd me, his client talked to his on-prem Edge, resolved the DNS for Modality Edge, and got proxied to me in O365.

However, when I attempted to IM Leon, my client talked to O365, who saw that there was a matching O365 tenant for the domain and sent the IM there. Simply turning off Leon from having an Office 365 Skype for Business account allowed the Modality Office 365 tenant to ignore looking up his details in the cloud, I found the customers Edge server and all was well in the world.

Simple when you know how!

*what changed? The customer got Office 365 but had not setup all the hybrid integration (as they didn't want to use it all at that time).