This is a story in the “Knights of Hyper-V” series, an attempt at humor with actual technical content hidden in the details.
A proclamation had been issued several moons ago by the gremlins of the blue window, declaring that a new version of the Virtual Machine Manager had been released. This had mostly been ignored by our merry knights, they were all busy building new systems, putting out fires and slaying dragons. You know, the usual stuff. Thus they had no time to spare for doing such things as maintenance on systems that were chugging along nicely without issues. But when a second proclamation appeared about an even newer version, it was decided to spend some time trying to do an upgrade in the lab, down in the spare dungeon.
Alas, this was not to be an easy task. The lab servers were in dire need of some maintenance as well, and one of the host flat out refused to respond to commands. Closer inspection revealed a “No bootable device” error on the local console. The results of a botched patching run a long time ago. For some reason the main partition was no longer marked active, a relatively easy fix in diskpart. But on to the main quest. Rumors would have it that there was no in place upgrade from SCVMM 2016 to SCVMM 1801. Those rumors were true indeed.
A knight was sent into the maze of documentation to look for answers. He came upon several dead ends and a lot of references to the hidden cat of 404, but he persisted and finally ended up at https://docs.microsoft.com/en-us/system-center/vmm/upgrade-vmm?view=sc-vmm-1801. Just as in the upgrade from SCVMM 2012 to SCVMM 2016, an uninstall/reinstall was required.
A cunning plan is devised
The SCVMM 1801 scroll of system requirements were reviewed to make sure that our systems were supported. The spare dungeon contains a single VM running both SCVMM and SQL Server and some old hosts. The VMM VM has the following setup:
- Windows Server 2016
- SQL Server 2012 SP4
- SCVMM 2016 4.0.2314.0 (UR5)
After some pondering around the table reading the scroll of upgrade instructions mentioned above, the following plan was agreed upon:
- Checkpoint/snapshot the VMM server.
- Create a Copy-Only backup of the VMM database.
- Reboot the VMM Server to make sure there are no pending reboots or other nasty stuff lurking in memory.
- Uninstall VMM 2016.
- Restart the server.
- Install VMM 1801.
- Upgrade VMM to 1807.
- Remove the checkpoint.
- Update the VMM Agent on the hosts.
- Turn off Diagnostic and Usage data.
Note: If you are running other System Center products, make sure that you review the upgrade sequence. Especially noteworthy is the fact that Operations Manager should be upgraded before VMM.
The king is dead…
Preparations
To prepare for removal, a SQL wizard was sent on a sub-quest to create a backup. He sent back this picture and a backup file:
The knight made a checkpoint of the VM to have a way back in case of complete and utter disaster, and restarted the VM to clear out any goblins lurking in the system memory. Be aware that creating checkpoints of running production SQL Servers is a risky procedure (remember, the lab server has SQL and VMM on the same VM). Creating the checkpoint will be successful, but an attempt to roll it back may lead to database corruption, and if things really go the way of the dodo you may have to reinstall the entire VM to recover. As such unnecessary side quests cuts in to our relaxation-time, it is better to create a copy of the VM in such cases. It is also of course recommended to host the VMM database on a separate SQL Server cluster in production.
Removal
After all these preparations came the time for the actual removal of the old version. Make sure to use the old installer to perform this quest. The new installer will pretend to do it, but it will ultimately fail and require yet another reboot to clean up the mess. If you are lucky, you will be able to try again, but that is why we created the checkpoint and secured the database backup.
We went looking for the old installer and located it at C:\System Center 2016 Virtual Machine Manager. Inside we found a setup.exe file that displays this menu:
Clicking the “Install” option summoned the VMM Setup Wizard. The Wizard gave us two options, “Remove features”, or “Cancel”.
Cautiously hopeful we asked the wizard to remove both the VMM Management server and the VMM Console. He countered with a question about the database, would we like to keep it. As it serves as storage for all our configuration scrolls and links to the VMs and hosts we urged the wizard not to remove it. The wizard grunted something, turned his back to us and started to mumble weird incantations.
Some time passed, and the wizard returned to us with a scroll of warnings about SPNs and SCPs:
The Service Principal Name (SPN) could not be registered in Active Directory Domain Services (AD DS) for the VMM management server.
1) Use setspn.exe to create SPN for vmmserver using following command “C:\WINDOWS\system32\setspn.exe -D redacted[]”.
2) Add SPN values to following registry key “Software\Microsoft\Microsoft System Center Virtual Machine Manager Server\Setup\VmmServicePrincipalNames”.
3) Run “C:\Program Files\Microsoft System Center 2016\Virtual Machine Manager\setup\ConfigureSCPTool.exe -uninstall” to configure SCP.
If SPN and SCP are not registered, VMM consoles on other computers will not be able to connect to this VMM management server and deploying a Hyper-V host to a bare-metal computer will not work.
We suspect this happened because the account running the installer is not a domain admin. Otherwise, the wizard claimed the removal to be successful. Only time will tell if he was correct…
..long live the king!
We turned our attention to the new installer for SCVMM 1801, in hope of getting the VMM up and running again. A new wizard was summoned and asked to install both the management server and the console on our VM.
You shall not pass without a valid key! boomed the wizard. A minion was sent on a sub-quest to rummage through the scrolls of system documentation. We went into the woods to hunt down some lunch, and when we returned, so had the minion and the secret key.
The wizard accepted the key and returned with a surprisingly short scroll of license. You usually have to go through a loong scroll basically agreeing to sell your soul and most of your armor to the gremlins of the blue window, but this was not even a page long. Closer inspection of the scroll revealed that this was a sly plot of the gremlins to make us just agree and move on. In fact, the scroll was entirely made up of references to mile-long scrolls that they claimed we had already accepted. Then came a notice telling us that the gremlins would use VMM to spy on us, or “Collect diagnostics and usage data” unless we remembered to turn it off. Thus we went back to our cunning plan and added “Turn off spying feature”.
The wizard turned around and mumbled incantations for a while, apparently checking for prerequisites. He soon came back, seemingly content with his findings so far, and asked about the database. We requested to use the existing one, and handed over the credentials for connecting to it.
The wizard warned us that this was an old database that had to be upgraded. We asked him to continue, and he came back requesting a service account for VMM. We specified the account used for the previous version:
Then came some questions about network ports and library shares, we told the wizard to keep using the previous settings. The fact that these settings were available from the database was a good sign. The wizard returned with a final warning, and after the fact told that not using the old service account would have been a bad idea.
We asked him to continue, and the mumbling resumed. There was nothing left to do, other than staring at the wizard (or the ceiling) and waiting for him to finish. Hopefully with a positive result…
Cleanup and patching
First, we removed the checkpoint. This is a simple and quick process if you do it in a timely fashion (less than 24h), and may be a total mess if you forget it and have to fix it later. This time, we remembered and removed it as soon as we confirmed that VMM 1801 was up an running. We also remembered to disable the spying feature, or Diagnostic and Usage data as the gremlins call it. The setting is located under Settings, General in the VMM console. There is also a PowerShell command if you want to script it, Set-SCVMMServer -DiagnosticsAndUsageData $false.
Patching
Then came the time for patching. We are using PSWindowsupdate, a nice PowerShell module for manual patching on Windows Server 2016. It detected the 1807 upgrade and installed it automagically:
Upgrading the VMM agent
We have had some issues with rouge VMs during previous VMM agent upgrades. Especially so when VMs with virtual fibre channel are concerned. Thus, we wait until everything else is finished and upgrade the agent on one host at the time while the host is in maintenance mode. This may be overkill, but it is a small price to pay compared with the wrath of an angry DBA whose VM died. To make it worse the SQL Instance may refuse to fail over to the backup VM because the SAN volume is locked by the not completely dead primary VM. The solution for that particular predicament was to restart all the hosts in the cluster. Yes, all of them. But enough about old war stories. Triggering the VMM agent update is quite a simple task:
- Enter the Fabric VMM console.
- Select all hosts.
- The hosts should all have the “Needs attention” status. This can sometimes take some time to appear, so if you do not see it, refresh your clusters.
- Repeat this procedure for each host:
- Put the host in maintenance mode.
- Right-click the host and select update agent.
- When the update is finished, restart the host
- Take the host out of maintenance.
At this point the knights went for a relaxing ride through the forest, satisfied with a job well done and looking forward to a well deserved dinner. But in the back of their minds loomed the bureaucracy needed to implement the same upgrade in production. Maybe that will result in another chapter in The Adventures of the knights of Hyper-V.
2 thoughts on “Upgrade to VMM 1801, a knights tale”