A long and sad story about trouble upgrading the firmware in Office Communication Server enabled phones, specifically the Polycom CX700. This particular problem manifests itself as follows: The phone logs on to the server successfully and gets almost to the end of the login process. Then it crashes and automatically reboots. This goes on in an endless loop, and is caused by the phone running software not compatible with OCS 2007 R2. I post this hoping that it will be of help to others on a similar quest.
The problem
For about a year now, since we started our OCS pilot, I’ve had a Polycom CX700 on my desk. Actually, it has spent the bulk of that time on a shelf at my desk. It was purchased as a test device along with a collection of headsets and other OCS enabled devices. Most of them worked fine, but this one I just couldn’t seem to get working. After a lot of fiddling about I was able to log in successfully with a test user that had never ever logged on to any other device or computer. I happily thought I had it fixed, but as soon as I switched to another user it went back to the same old pattern: logon, crash and reboot in an endless loop.
Giving up
Pretty early in the process I was informed that the firmware (1.0.522.73) was built for OCS 2007, and as we were running 2007 R2 it would not work. Later, I was told by Polycom support that it was impossible to update this version to a 2007R2 compatible firmware without having an OCS 2007 device update server. At that point I gave up, put it on the shelf and concluded that we had to swap it for another one with newer firmware.
But the phone was still there. It just sat there on my shelf, and from time to time this annoyed me to the point that I researched it further. I am not accustomed to giving up, especially not on a technical problem. Mulling at the back of my head was the feeling that there had to be a solution somewhere, some hack to fix it or a bug blocking my path. But no matter what I tried, I had no luck.
At some point in my search I came across these articles on TechNet blogs by Rui Silva: http://blogs.technet.com/b/ucspotting/archive/2009/04/16/how-to-upgrade-polycom-cx700-1-0-452-0-using-the-ocs-2007-r2-device-update-service.aspx
I followed them both in detail (several times) since they describe a problem similar to mine, but to no avail. It just wouldn’t budge, it refused to give in. I had all the device update websites set up correctly, and the RequestHandlerAuditLog listed the device as connecting and identifying itself, but it never requested nor downloaded any updates. As far as I could tell everything was set up correctly at the server side, so the phone had to be the source of my trouble.
A new hope
I was close to giving it up for good, but then suddenly I had an epiphany: The RequestHandlerAuditLog samples other people were posting online always listed a request stating it was currently running version 0.0.0.0;01-01-1601 00:00:00, before it requested the interim version(1.0.522.103) which is the bridge version from OCS 2007 to OCS 2007R2.
Logging DateTime,User Name,User Host Address,Device Type,Request DateTime,Mac Address,Serial Number,Vendor,Model,Revision,Locale,Requested[# Seperated for Multiple],Response[# Seperated for Multiple]
01.03.2011 14:17:01,ocstest@domain.com,[IP],UCPhone,03.01.2011 14:17:01,"0004F2BADAFF","C191A000499","Polycom","CX700","A","ENU",cpe.nbt;0.0.0.0;01.01.1601 00:00:00,
01.03.2011 14:34:59,,[IP],UCPhone,03.01.2011 05:34:59,"0004F2BADAFF","C191A000499","Polycom","CX700","A","ENU",cpe.nbt;0.0.0.0;01.01.1601 00:00:00,
01.03.2011 14:41:27,,[IP]UCPhone,03.01.2011 05:41:28,"0004F2BADAFF","C191A000499","Polycom","CX700","A","ENU",cpe.nbt;0.0.0.0;01.01.1601 00:00:00,
01.03.2011 14:53:15,,[IP],UCPhone,03.01.2011 05:53:15,"0004F2BADAFF","C191A000499","Polycom","CX700","A","ENU",cpe.nbt;0.0.0.0;01.01.1601 00:00:00,
But my CX700 never got past this, it never requested the interim version, and still it reported 0x0;200 as the last update status. That means the server thinks the phone does not require any updates, even though clearly it does. The first number is a hexadecimal WinInet error code, and the second is a standard http status code. 0x0 translates to no error, and http status 200 means the request succeeded. See this site for an explanation of the last update status codes. Rui Silva mentions changing the UCPHONE client version filter to allow any 1.522 version which I did a long time ago, but then I thought: What if the server believes the phone is actually running version 0.0.0? To test this, I removed the UCPHONE client version filter completely, thus allowing anything identifying itself as a compatible phone to download updates. I then reset the phone to trigger an update attempt and logged in with the special phone test OCS user mentioned above. Nothing happened. It just contacted device update as usual , which resulted in a 0x0;200. I went home and meant to try again the next morning, but other more pressing problems required my time so the phone was just left untouched on my desk. Suddenly, about 22 hours after the last reset I noticed the screen on the phone blinked. Then it rebooted, and lo and behold; it was now running version 3! I had read online that the upgrade process could be slow, but that it could take 22 hours to complete didn’t occur to me. The phone had been left on for several days earlier though, so removing the client version filter seams to be the magic bullet. The successful RequestHandlerAuditLog:
Logging DateTime,User Name,User Host Address,Device Type,Request DateTime,Mac Address,Serial Number,Vendor,Model,Revision,Locale,Requested[# Seperated for Multiple],Response[# Seperated for Multiple]
01.04.2011 14:53:27,ocstest@domain.com,[IP],UCPhone,04.01.2011 14:53:26,"0004F2BADAFF","C191A000499","Polycom","CX700","A","ENU",cpe.nbt;0.0.0.0;01.01.1601 00:00:00,http://[server]/DeviceUpdateFiles_Int/UCPhone/Polycom/CX700/A/ENU/3.5.6907.222/CPE/CPE.nbt;3.5.6907.222;13.11.2010 00:29:16
01.04.2011 14:57:54,,[IP],UCPhone,04.01.2011 05:57:54,"0004F2BADAFF","C191A000499","Polycom","CX700","A","ENU",cpe.nbt;3.5.6907.222;13.11.2010 00:29:16,
01.04.2011 16:07:06,jan.kare.lokna@domain.com,[IP],UCPhone,04.01.2011 07:07:05,"0004F2BADAFF","C191A000499","Polycom","CX700","A","ENU",cpe.nbt;3.5.6907.222;13.11.2010 00:29:16,
01.04.2011 16:09:39,jan.kare.lokna@domain.com,[IP],UCPhone,04.01.2011 16:09:39,"0004F2BADAFF","C191A000499","Polycom","CX700","A","ENU",cpe.nbt;3.5.6907.222;13.11.2010 00:29:16,
And the IIS log:
#Software: Microsoft Internet Information Services 7.0
#Version: 1.0
#Date: 2011-01-04 13:53:27
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status time-taken
2011-01-04 13:53:27 [OCS Server IP] POST /RequestHandler/ucdevice.upx - 443 - [Phone IP] Microsoft+UCPhone+Device 200 0 0 1778
2011-01-04 13:56:33 [OCS Server IP] GET /DeviceUpdateFiles_Int/UCPhone/Polycom/CX700/A/ENU/3.5.6907.222/CPE/CPE.nbt - 80 - [Phone IP] Microsoft+UCPhone+Device 200 0 0 185857
2011-01-04 13:56:33 [OCS Server IP] GET /DeviceUpdateFiles_Int/UCPhone/Polycom/CX700/A/ENU/3.5.6907.222/CPE/CPE.cat - 80 - [Phone IP] Microsoft+UCPhone+Device 200 0 0 15
2011-01-04 13:57:54 [OCS Server IP] POST /requestHandler/ucdevice.upx - 80 - [Phone IP] Microsoft+UCPhone+Device 200 0 0 155
As far as I can tell, the phone goes directly from 1.0.522.073 to 3.5.6907.222. I guess 1.0.522.073 is a beta or RC of the interim version.
The solution
To sum it up, this is the key steps of what got it working for me:
- Make sure to log in to the phone using a complete domain name, e.g. “domain.local\username”. If I used domain\username I received certificate download errors on the phone. The same error occurred if I tried username@domain.local, but according to Microsoft this should work. I suppose this is a bug.
- Remove the UCPHONES client version filter from the update server
- Set the default client filter to Allow
- Use a special upgrade user that has NEVER been logged in anywhere other than on the phone.
- Leave the phone on, signed in and untouched for at least 24 hours
this was very helpful, I just wanted to point others who may show up here: http://www.microsoft.com/en-us/download/details.aspx?id=23569
since many of you probably aren’t running ocs 2007 (or r2) anymore
the above trial from Microsoft worked perfectly, the only trick was changing the clock back on all the VMs to before December 2013 since all of the certs were expired, I was able to upgrade to 3.5 from this hyper-v environment after that did the 2 step CU5 to CU7 via lync 2013.