Sophos UTM when running on clustered Hyper-V 2012 R2 (or why is Linux is different?)

Originally my post was going to be about how painful it was working on a bug in the Sophos UTM virtual appliance. But after doing some digging I’ve realised I need to point the fingers in a different way!

What caused me pain was this:

Build a Highly Available VM using latest UTM download from Sophos (4x network cards and shared SMB storage in our case).

After built go into the console (loginuser and then su to root):
ifconfig | less

Note the eth network cards that are available
Live migrate the UTM to another host in the cluster
ifconfig | less

Note all eth network cards have now disappeared

Live migrate back to initial hosts
ifconfig | less

Note eth network cards have not come back

The system will continue to run and can be moved around hosts with no problems. The issue will only manifest itself once the UTM gets rebooted (in my case for an Up2date firmware fix).

After building a new UTM VM I did some testing to replicate the problem:

First thing - lets shut the VM down, move to the other host and bring it up, maybe the bug is in the Live migration.

Unfortunately, no, exactly the same problem, no eth cards.

A-ha I thought, so lets shut the VM down and move back to the initial host and bring it up, maybe it’s a bug in the network card stack so the cards will magically come back.

No, cards still missing

So at this point I was speaking to Donald at Sophos technical support (top guy, many thanks to him) and we had come to the conclusion it was a bug in the Linux kernel that the UTM uses.

He was going to add a bug number in and I was going to leave the UTM sitting on a single host with instructions to never move it!

And there might have ended the story and I would have gone away thinking that I’ve found a bug but I started to do some Googling Binging and hit the answer……

Those of you who have been using Linux on Hyper-V will have seen the problem straight away.
(for this part please forgive my lack of Linux knowledge, this is how I see the problem – but I really don’t want to start a flame war!)

And it’s all down to how Linux binds its network cards to the MAC address of the virtual machine. Windows boxes will happily use any MAC address it is given on boot whereas the first MAC address that is given to a Linux box will get hard coded into the config files and will always be used. This means when the virtual appliance (because to me that what it is) migrates between hosts the MAC address changes and Linux gets unhappy.

I’m sure that there are some people who will read this and say “surely you should have read up on the Microsoft documentation for running Linux in Hyper-V”

I’ll counter with, “nope, I was running an appliance. Not a Linux box so had not thought of even reading up on this, and don’t call me Shirley”

My plea to vendors who say that they have an appliance that will run in your virtual environment is that they make it clear in their documentation that this could happen so other hapless “Windoze” geeks like myself are not caught out in this way in the future.

And me – I destroyed my VM (for the 5 time!) and rebuilt it – this time with the MAC address set to a static address (side note: if you do not type a MAC address after setting a static one then Hyper-V will assign you one automatically) and live migrated my Linux VM Appliance to my hearts content.......

……..or so I thought: By setting the MAC to a static MAC on our WAN interface stopped all traffic over the WAN, so with time against us we had to set the WAN back to being dynamic, this means if the UTM moves hosts it’ll need a reboot and config changes to get the WAN interface back onto a the virtual NIC……

The story will continue if I get an answer back from Sophos prior to me moving to my new job!

Cochlear Aqua+ Review

Having a blog and being active on Twitter does occasionally pay off, as I had shown interest in Cochlear’s previous waterproofing solution (do not call it a bag) they contacted me asking if we would like to trial their new Aqua+ accessory for the N5 and N6 range of processors.

We jumped at the chance and after supplying details (coil magnet strength and coil cable length) a massive package turned up:
 

On opening it we got a surprise that there were two kits enclosed (makes sense I suppose as Josef is bilaterally implanted but I thought we would have only got one as a test):

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

It turns out that the reason it is so large is that you get everything you need for a few weeks:

You can see pictured, the guide (multiple languages, only the first 17 pages in English so not as daunting as it first looks!), The Aqua+ (part number Z463273), Nice carry case, Nucleus Aqua+ Coil (part numbers Z463263 – 6 cm length & Z463270 – 8 cm length), Nucleus Safety Line (part number Z467062), Mic Lock-Stirrup (part number Z368868) and a CP900 series magnet (too many part numbers to list!).

 

As a side note: this appears to show that the coil cable and coil are compatible between the N5 and N6 processors.

 

My initial comments on opening the kit are positive:

• Very nicely packaged (positive impression),
• In a single kit you get 2x of the Aqua+ sleeve’s and 4x of the Mic Lock- Stirrup’s (extras), 
• The carry case is very useful for putting all the bits in that you need when going swimming

In the accompanying documentation it says that the sleeve is usable for around 50 sessions so each pack of 2 could do you for up to 100 swims. As a pack of sleeves retails at around £34 (direct from Cochlear) I think this works out very economically. But a note here about the original Aqua Accessory

 

Josef with his @Cochlear_Europe Aqua Accessory on. First bath where he could hear us! Technology rules. http://t.co/WyMqOoLc
— Tobie Fysh (@tobiefysh) February 19, 2013

The single use version launched in 2013 and picture above has now been withdrawn from sale and replaced with the new reusable Aqua Accessory; Connevans these new ones on sale at C£39 at the moment for a box of five, each of which can be used up to 20 times.

 

This brings me to the first difference between the Aqua+ and the Aqua Accessory. With the previous version (non-reusable) we had no issues other than the air in the bag would cause the bag to float away if Josef tried to put his head underwater. As the bag was also bulkier and made the processor less movable around the cable it was harder for Josef to put back on himself so what would usually happen was after he had dived under water a few times the “ears” would be off, passed to us parents and he would happily play without sound.

 

With the Aqua+ we have the opposite problem. Josef jumps underwater and or moves quickly with his head under the water and the force of the water against the processor causes it to detach from his head, it then sinks (no air in the sleeve). The first time this happened Josef was a bit worried (he was used to looking for the floating ear, the second time it turned into a game, let the ear sink to the bottom and then try to swim down to it!

 

Thankfully Cochlear have supplied two bits of kit in the pack that can help with this. The first being the Mic Lock-Stirrup:

 

 

This helps to keep the processor attached to the ear – as yet we have not had chance to use it with Josef as he complains that it squashes his ear, it’s very difficult to disagree as that is what its intended to do! I think with an older child you should be able to reason with them to say this will help to keep it on your head but not yet for Josef!

 

The second part included by Cochlear is the Safety Line – this is for making sure that when the processor does come off that it stays attached to clothing. For that reason you need to be wearing something on the top half of your body to attach the line too. Females will be fine with swimming costumes/bikinis tops to clip to in a swimming pool but males will struggle to find somewhere near the neck line to clip it onto - maybe a fashion necklace?

 

I can see that the main reason for the safety line is for when using the Aqua+ in the sea or open water when you would probably be wearing a wetsuit/top of some kind and in that scenario it will certainly save you expensive processor from sinking from sight! My only complaint is that the line is probably 2cm too short for an adult and wearing it and turning your head may pull the processor off unintentionally.
 

There is one bit of preparation that you will need to do prior to using the Aqua+ for the first time and that is to make sure you have some spare earhooks to take with you. This is because you need to remove the earhook from the processor to insert it into the sleeve. In my experience the ear hooks get brittle over time so ensure that you take some spares with you in case removing them causes them to break. The manual again shows you what to do here:

 

 

(Another note on the earhooks – if you have the small tamper resistant ones make sure you take the tool to remove them!)

 

Now a couple of photos of Josef in the swimming pool actually using the Aqua+ (note, this is right at the beginning of the swimming session and he has not put his head under that water at this point. I’m taking photos at the side and getting nagged by both boys to get in and play, hopefully next time I’ll be able to get some better photos of them in-use!) 

 

 

Finally – is it worth it?

 

I would say whole heartedly: Yes. The sleeves are very thin and do not make the processor look any bulkier on the ear. It actually works and pricing is, IMHO, fair: Cochlear sell you the initial pack for either £160+VAT (mono) or £289+VAT (bilateral). This gets you the two sets as shown above. I do think that Cochlear could make a better bilateral kit which only has one of the carry cases and instruction manuals and maybe drop the bilateral price by £20 but understand carrying more product lines increases their costs. That being said each set is made up from the component parts (as you pick the magnet and coil cable length) so this would be an easy change to make.

 

If you would like to find out more please visit the Cochlear website or call (0800 035 6317) or email them.

I'm happy to take questions in the comments or via Twitter.

Thanks to Cochlear for allowing us to trial their new kit.

Exchange 2013 – The things they don’t tell you

(This post is a work in progress, at the moment I’m still migrating things so I have the joy of moving to the new public folder architecture in a few weeks, I’m sure this post will grow after that!!)

So the migration is all but done:

Bye Bye Exchange 2010 Voicemail....... <sniff> pic.twitter.com/vSk1BuPz0t
— Tobie Fysh (@tobiefysh) January 26, 2015

One final server to remove (once we track down all the services using it for SMTP relay) but that has been a mostly painless experience. 

• If your Exchange Server is virtual and it’s on Hyper-V then you’ll want to do this on the Hyper-V hosts if you want to use DAG’s:
  http://support.microsoft.com/kb/2872325 

• Your Content Index will fail and you’ll fix every-so-often by deleting the index and restarting the search services, but there is a proper fix:
  http://support.microsoft.com/kb/2807668
• Don't remove the Self Signed Certificate, it's used by the Back End services. After I had everything running I was getting a bit annoyed that I still had the Self Signed cert sitting there, it is still bound to IIS and SMTP. After a lot of reading I found the following KB article which basically says "leave it alone":
  http://support.microsoft.com/kb/2779694
• Make sure you receive connectors are scoped so voicemail does not start failing with "Event ID 1446, MSExchange Unified Messaging,
  The Microsoft Exchange Unified Messaging service on the Mailbox server failed to process the message with header file "FILENAME" within "11" minutes. The server will continue to process and deliver the message, but the "MSExchangeUMAvailability: % of Messages Successfully Processed Over the Last Hour" performance counter will be decreased.  http://exchangepro.dk/2014/05/06/voicemail-not-delivered-to-mailbox/
• Play on phone from the full fat Outlook 2010 client was failing for users still homed on the Exchange 2010 servers but working fine for users homed on Exchange 2013. Play on phone comes up with the following error:
  
  
  
  Strangely playing the same voicemail via OWA works fine
  I've done no further trouble shooting as the users will all be moved to 2010 within the week and we have the above work around in place. If anyone wants to jump in with a suggestion in the next few days I'll be glad to hear it.
• Running the Public Folder upgrade script (Export-MailPublicFoldersForMigration.ps1) gives warnings about Property errors:
  
  "WARNING: The object Domain.local/Microsoft Exchange System Objects/Meeting Room 5 (10) has been corrupted, and it's in an inconsistent state. The following validation errors happened:
  WARNING: Property expression "Board Room" isn't valid. Valid values are: Strings formed with characters from A to Z (uppercase or lowercase), digits from 0 to 9, !, #, $, %, &, ', *, +, -, /, =, ?, ^, _, `, {, |, } or ~. One or more periods may be embedded in an alias, but each period should be preceded and followed by at least one of the other characters. Unicode characters from U+00A1 to U+00FF are also valid in an alias, but they will be mapped to a best-fitUS-ASCII string in the e-mail address, which is generated from such an alias."
  
  For me this was some of the Public Folders not having valid alias entries. It didn't cause a problem (as far as I can see) in all the time we have been running Exchange 2010 but needed to be fixed while moving to 2013 and its new PF architecture.
  
  I had an additional problem that I couldn't get into the item to edit it in the Public Folder Management Console without applying the following fix (the error was about "no existing publicfolderproxyinformation" sorry, didn't screenprint before fixing!): https://workinghardinit.wordpress.com/2010/11/04/exchange-2010-public-folder-worries-at-customer-no-existing-publicfolderproxyinformation-matches-the-following-identity/ which turns out to be a problem with the homeMDB property of the Microsoft System Attendant!
  
  Then editing the item to remove the space fixed the issue:

• While migrating the Public Folders we got to one of the final stages and the process appeared to hang.
  Running Get-PublicFolderMigrationRequest | Get-PublicFolderMigrationRequestStatistics gave the response "StalledDueToMailboxLock" and :
  Informational: The request has been temporarily postponed because the mailbox is locked. The Microsoft Exchange Mailbox Replication service will attempt to continue processing the request after 26/01/2015 17:21:07.
  
  The solution was to restart the Information Store Service on the Legacy Mailbox server and wait 10 minutes for the process to continue.

And as a bonus feature - TLDR: Exchange Unified Messaging Certificates need simply be the server Fully Qualified Domain Name with no additional SAN names.

For my pain simply read on...

I have three new Exchange 2013 servers:

• FCH-XS13-01.fch.local
• FCH-XS13-02.fch.local
• FCH-XS13-03.fch.local

 

(and before someone complains that I’ve given away the names of the Exchange server in my organisation I counter with – “you’d get that simply by receiving an email from anyone in the organisation and looking at the headers” <sigh>)

 

When I created the certificates I used the brilliant Digicert Tool and bashed the FQDN in:

 

fch-xs13-01.fch.local

 

threw in the NetBIOS name for kicks

 

fch-xs13-01

 

sent the response off to my local CA, downloaded the response. Imported it and applied in the ECP (I know, no PowerShell, naughty me).

 

After restarting the two UM services:

 

 

 

No Voicemail – calls just hung up. And not simply for my test mailbox on the Exchange 2013 environment – this was effecting everyone as the Exchange 2013 servers proxy the 2010 traffic

 

Looking in the event log we see Event ID 36884:

 

 

 

Event ID 1649:

 

 

 

Event ID 1113:

 

 

 

 

On the Lync 2010 servers we see the following:

 

Event ID 14366

 

 

So, I’m off on a hunt. And it take me hours. Finally after comparing everything I can think of I decide to recreate the certificates (for what feels like the 1 zillionth time) and while I’m doing so this comes to me:

 

The server name we were expecting is fch-xs13-03.fch.local

 

And the certificate I was giving out had two names, the Common Name and the Subject Alternative Name….. so sometimes the service was seeing the NetBIOS SAN name - BINGO. I guess I’ve been spoiled by SAN names on Lync and Exchange so just through “throw it in, what's the worse that’ll happen" - It turns out, a lot!

When the Internet of Things doesn't play nice, who can help? Tado Home Heating?

I’ve got an ongoing problem with the tado bridge not appearing to obtain an IP address from a BT Home Hub 5 or the BT Home Hub 2.

The reason I choose to go with tado over, Hive or Nest is the cross platform control - iPhone, Android and Windows Phone.

On a Home Hub 5 with the Home Hub reset to factory new we see the following:

Showing the laptop called SCRATCH plugged into LAN1 and something plugged into LAN4 (that something is the tado bridge).
The DHCP table does not show the tado:

But the event log does have mention of it:

So it appears that all of my other devices (Xbox One, Xbox 360, PlayStation 3, Wii, Sky TV, Surface Pro 2, Surface Pro 3, iPhone 5s, Windows Phone 1020, Dell Latitude D630, Home built PC) all connect quite happily onto the Home Hub 5 and can route to the internet.

But not the tado bridge.

As I have an old Home Hub 2 (HH2) I thought I would try that to see if the tado bridge could get a DHCP lease.

The Home Hub 2 cannot go onto my home broadband (I have fibre instead of DSL so the modem in the HH2 is not compatible) as such I rigged up a simple network of the HH2 with two Ethernet connections: 1 – tado, 2 – laptop (called “SCRATCH”)

In the HH2 control panel (which is quite basic) all I can see is the following:

This shows that the tado bridge has not connected. Looking in the DHCP table on the HH2 we see this:

(the event log has nothing for DHCP)

Short story long: the tado has not attempted to or got an IP address on either of the Home Hubs.

So far tado support have said that it must be my router as I took the bridge to work and it did get an IP address on a different model of router (NetGear). I agree with them, it is a problem on the BT Home Hub 5 (and maybe HH2 as well) connecting to the tado bridge. But as all these other devices connect correctly to the Home Hubs I believe that the problem must lie within the implementation of the IP stack on the tado.

I would like to work with tado to get this fault resolved. Looking at the above errors it seems to be that either the tado bridge in general (all of them) or my specific one does not like the BT Home Hub series of routers (as a final test I took the tado bridge to my fathers house who also has a Home Hub 5 and it also could not get an IP address there).

Ideally I would like some way of setting a static IP on the tado, that would take DHCP out of the equation, if that is not possible maybe a way to have the tado log so that we could see what the tado is attempting to do (but as there does not appear to be an interface on the tado to look at I don’t know how this would be possible).

If tado can’t work with me to resolve I’ll have to take advantage of their no quibble 30 day money back guarantee and be left with old heating controls…… 

Lync 2010 Edge Certificates expiring. What does it look like?

I was at home and logging into my Surface Pro 2 and my desktop Lync client was not signing in - but also not coming back with any errors. On further investigation the event log had the following error:

We are running Lync Sever 2010 but I am running Lync 2013 (Office Pro Plus) as my client. Lync Server 2010 has never had that DNS name so I knew that the problem was different to what the event log was showing. As I was external (Direct Access on the laptop but as per best practice Lync runs outside the VPN) the first place to look was the Lync Edge server.

Logging onto the Edge server the first place to look is the event log and the following was a glaring problem:

As there had been no DNS, Firewall or Proxy changes that only left credentials. I connected into the Front End server and the following two errors give some big clues:

 

So it appears that a certificate has expired (as a side note we use the DigiCert Discovery Tool - you need a DigiCert account - to check for any certificates on the estate that are going to expire, the reason that this one was not picked up was because even thought we where scanning the Edge server we were not checking port 4443 this has now been added as a change) Anyway, checking the certs on the Edge Server with the command Get-CSCertitificate gave the following:

Both Internal and AudioVideoAuthentication have expired, next checking the certificates in the computer personal store we can see the following:

 

This is showing two certificates that have expired and ties into what PowerShell is telling us (for a good pointer of what you need on the Edge check Jeff Schertz's Blog: http://blog.schertz.name/2012/07/lync-edge-server-best-practices/).

So, simple fix. Renew the certificates using the Enterprise CA and then assign (I’m not going to document how as again Jeff has done a great job of this here: http://blog.schertz.name/2012/01/simple-certificate-requests-in-lync/).

While I was here I thought I might as well tidy up the old Root CA that the Edge Server had imported so deleted that - the new certificates don’t use it so what’s the harm........

.........This proved to be a mistake, even though the Edge server didn’t host any certificates that needed the old Root CA there were some certificates on the Front End servers that couldn’t be verified as they had been signed by the previous Root CA certificate, this can be seen here:

Simply downloading the old Root CA Cert from the Enterprise Root CA (https://<certsrv>/certcarc.asp) and importing it into the Edge server made the certificates being presented by the Front End servers immediately valid and my remote Lync Client could finally login!

My lesson learned, don’t “tidy” until you have fixed the underlying problem!

Lync 2010 user getting locked out

Many posts about this on the internet but having been through this today with a customer think I have it cracked.

1. Exit Lync on users desktop
2. Unlock user in AD
3. Remove user certificate in Lync Control Panel
4. Remove local machine Lync user certificate
5. Remove cached credentials from users PC
6. Delete contents of C:\Users\%username%\AppData\Local\Microsoft\Communicator
7. Delete reg key HKEY_CURRENT_USER\Software\Microsoft\Communicator
8. Load Lync

If Lync prompts for a username and password then you still have the problem and at this point dissolve in tears.

Lync 2013 Client - Echo... echo... echo...

Looks like someone in the client team at Microsoft has a sense of humour (or humor for our American friends):