Lync 2010 - My pain finding a call quality issue.

On migrating one of the final teams (Finance) at %dayjob% to Lync Enterprise Voice I hit an interesting frustrating problem that I thought I would share.

TL:DR at the bottom but I would love you to read it all to feel my pain!

Background: User has brand new Windows 7 Dell Optiplex 7010 full patched. Lync Servers are 2010 CU6. Endpoint is Polycom CX300.
Our machines are all built from a deployment server so standard software, but this also had some additional software which is not pushed out by System Centre 2012.
The way a migration is done is user at a time, remove number from Mitel and migrate it to Lync and then sit with them for a 15 minute orientation session. Then after they have gone home for the evening rip out the old Mitel phone (the PC is networked into the Mitel 5212 which is plugged into the floor socket).

Issue: Call quality was crap on a single PC. Looking in the monitoring server the following was being logged:
Average difference between the OverallAvgNetworkMOS and the maximum possible OverallAvgNetworkMOS for the codec used. High values can be the cause of congestion or interference, or an overloaded media server or endpoint, and results in distorted or lost audio.
Average difference between the OverallAvgNetworkMOS and the maximum possible OverallAvgNetworkMOS for the codec used. High values can be the cause of congestion or interference, or an overloaded media server or endpoint, and results in distorted or lost audio.

So this for me pointed to the network as the first thing. As I had migrated the users number from the old Mitel phone system we still had the PC having its network going into the Mitel 5212 phone and then into the floor box. I scheduled some downtime with the user over lunch so I could patch with brand new CAT6 and remove the Mitel.

Call quality still crap. Monitoring server – ditto.

Switched where the PC was patched in with known good PC. Problem followed suspect PC so we have ruled out the network.

Rebooting and logging on with other users and call quality was still terrible so that ruled out config issues.

A quick look through appwiz.cpl showed an additional USB driver for a device (smart card) which was no longer used so uninstalled and rebooted. Also uninstall and reinstall Lync client.

Call quality still crap.

As the endpoint being used is USB next had a look at the other devices on the PC. The end user had a anon-standard keyboard and mouse due to RSI so after logging in these where removed and replaced with known working input devices.

Call quality still crap

Move all USB devices around so they are in different ports:

Call quality still crap.

At this stage I had a colleague (@spongmokey) come and sanity check my work and together we decided to swap the hard drive into a machine from another users desk to rule out hardware. I go to another floor where I know a machine from the same batch is sitting unused due to holiday and bring it back to the desk, rip out the hard drive, plug all peripherals back in, boot up.

Call quality is still crap.

Replace CX300

Call quality is still crap.

Hurray – so that’s proved it’s a software problem, I pop back to IT to see if the guys who can install the custom software can schedule it in. Just for laughs he (@dzmuk) asks have we tried a known working hard drive in the faulty PC. I scoff at this – we’ve already proved its software but as I want to prove him wrong, rule this out I go back to the desk, slap the working hard drive into the original chassis, boot up.

Call quality is crap

HUH?

So both the hard drive and the chassis have the problem? Switch the hard drives back to their original chassis, plug in original PC:

Call quality is crap

Plug in known good PC:

Call quality is crap

DOUBLE HUH

So maybe the problem is not software, or is it the hardware of the original PC has a different software level and installs a newer/different driver which causes the problem.
Hunt through device manager but all drivers are the same version. Uninstall the USB, network and sound drivers, reboot:

Call quality is crap

Cue head scratching, maybe it’s something to do with the USB being overloaded. Boot machine with all USB device connected and then remove then after logged in leaving just the CX300 connected.

Call quality is still crap.

Okay – now very confused. Have been trying to track this down all afternoon with no joy, one of the other people in the department has shut down their PC to go home so we move the “faulty” machine to that desk…. And the problem stops……

So it must be something on the desk which is causing the problem. Only things can be the monitors (duel head graphics card) and power cables.

Power up machine with monitors attached and then remove them

Call quality is still crap.

Replace all power cables

Call quality is still crap.

Grab an extension lead and plug PC into another floor box.

Call quality is still crap.

We focus on the USB again, remove all device and power the machine up, plug everything back in:

Call quality is good

WHAT?

Reboot machine

Call quality is still crap.

So if the USB devices are plugged in at boot we have an issue. If we load Windows and then plug them in it’s fine.

Unplug all devices (other than CX300 and keyboard), reboot:

Call quality is still good.

Plug in mouse, reboot:

Call quality is still good.

Plug in webcam, reboot:

Call quality is still good.

Plug in the final device – the USB hub that is built into the screen of one of the monitors (unused but there if needed), reboot:

Call quality is crap.

HURRAY, found it.

Move to using USB hub in second monitor, reboot:

Call quality is still good.

A bloody USB hub – WHICH WASN’T EVEN BEING USED.

TL:DR – check all USB device first with reboots between as they react differently when plugged in on boot.

No comments:

Post a comment