Upgrade to VMM 2019, another knight’s tale

This is a story in the “Knights of Hyper-V” series, an attempt at humor with actual technical content hidden in the details.

The gremlins of the blue window had been up to their usual antics. In 2018 they promised a semi-annual update channel for System Center Virtual Machine Manager. After a lot of badgering (by angry badgers) the knights had caved and installed SCVMM 1807. (That adventure has been chronicled here). As you are probably aware, the gremlins of the blue window are not to be trusted. Come 2019 they changed their minds and pretended to never have mentioned a semi-annual channel. Thus, the knights were left with a soon-to-be unsupported installation and had to come up with a plan of attack. They could only hope for time to implement it before the gremlins changed the landscape again. Maybe a virtual dragon to be slain next time? Or dark wizards? The head knight shuddered, shrugged it off, and went to study the not so secret scrolls of SCVMM updates. It was written in gremlineese and had to be translated to the common tongue before it could be implemented. The gremlins was of the belief that everyone else was living in a soft and cushy wonderland without any walls of fire, application hobbits or networking orcs and wrote accordingly. Thus, if you just followed their plans you would sooner or later be stuck in an underground dungeon filled with stinky water without a floatation-spell or propulsion device.

Scroll of prerequisites

• SQL Server 2016 or 2017
• Windows 2016
• SCVMM 2016 UR6 or SCVMM 1801/1807

The knights were surprised to find that an operating system upgrade did not appear to be necessary this time. The current version was still supported, and as an in-place OS upgrade is a pain in the back armor plates it was decided to test this in the lab without an upgrade. As you may remember, the lab is located deep down in the spare dungeon, together with old swords, discarded tankards, old scrolls of documentation and other such stuff that may one day be needed. Thus, some cleaning and patching was in order to prepare the lab for testing.

System Center Update Order

1. Service Management Automation
2. Orchestrator
3. Service Manager
4. Data Protection Manager
5. Operations Manager
6. Configuration Manager
7. Virtual Machine Manager
8. Service Provider Foundation

From <https://docs.microsoft.com/en-us/system-center/vmm/upgrade-vmm?view=sc-vmm-2019#upgrade-the-vmm-sql-server-database>

The lab was a stand-alone SCVMM without any other System Center applications and was thus free to be upgraded at any time, but the production system also contained SCOM and SCCM which would have to be fixed first. A messenger was sent out to the other teams, requesting that they got their asses, horses or oxen in gear and got on with patching.

Sly plan of attack

Back it all up. All of it.

The server running VMM. The database. The license keys. The shopping scroll for the celebratory feast. The alternate shopping scroll for the failure feast in case we did not succeed. All of it.

Disconnect from SCOM

In production we would have to disconnect SCVMM from SCOM before the upgrade.

Upgrade SQL Server to 2016 or 2017

Both the lab and production system was running SQL Server 2012. This is a fairly old, but still serviceable version. It could probably serve its purpose for a couple of years still, but the gremlins demanded it be updated to at least the 2016 version. An envoy was sent to the Wizards of SQL Server, requesting an upgrade of the VMM SQL Server instance. He reviewed the compatibility chart at https://docs.microsoft.com/en-us/sql/database-engine/install-windows/supported-version-and-edition-upgrades?view=sql-server-2017 and determined that an in-place upgrade was possible. He also advised against it, but as building a new machine would require an epic battle with the networking orcs controlling the walls of fire we decided to try anyway. An upgrade to 2016 was chosen as the wizard disliked version 2017 and version 2019 was not supported. The plan was written down in arrow-point format:

• Acquire the correct ISO from the gremlins running the great library of the blue window and feed it to the virtual ghost running VMM.
• Awaken the installation wizard.
• Beseech the installation wizard to Upgrade from a previous version of SQL Server.

• Install the latest cumulative updates.

The SQL Server Wizard and installation Wizard started talking in an unintelligible combination of dark incantations and gremlineese ultimately demanding that a chilled red or white monster was located and offered as payment while the button of next was hammered in repeatedly. A red monster was delivered, and the SQL Wizard immediately slammed the button of upgrade with a banhammer. He preceded to sit down into the closest comfy chair while simultaneously keeping an eye on the installation wizard and devouring the monster. The installation wizard was painting a green band across the screen while talking in an incomprehensible language that the SQL Wizard seemed to understand and enjoy. He smiled and hummed positively and appeared to be in a good mood for once. The knights went out for lunch, and when they came back the SQL Server was running version 2016 SP2 CU7.

The SQL Wizard asked about compatibility levels. The head knight went into the dungeon and studied the scrolls of VMM updates. The only levels mentioned was cluster functional levels. As SQL Server 2016 was supported, the SQL Server wizard set the compatibility level of the VMM database to 130 which was code for 2016. Don’t ask, the SQL Wizards always chant strange codes. The SQL Wizard went back to his own dungeon, and the knights were ready for the next step in their plan of attack…

Side quest

It was decided to wait for the SCOM upgrade in production to be planned before the quest main quest continued. The SCOM oxen were moving very slowly across the field, and no one had heard from the SCCM dungeon for a while, so the knights decided to go on a side quest while they waited. The knights were worried that the gremlins of the blue window may change everything again before the push to production, thus making the progress null and void and restarting the quest from the beginning. There were rumors whispered in the dark of a mighty potion that if consumed, could grant you the power to halt the gremlins in their tracks. For a while. Or maybe just for a bit.

The scroll of ingredients was long and included a fabled blueberry battery, a yellow monster chilled in arctic sea-ice, wild boar bacon smoked over an aspen fire and powder of Emulex. The last ingredient made the knights shudder. Emulex parts were made by evil wizards to wreak havoc in the server-dungeons of gullible knights stupid enough to install them, and making the potion would require handling such an evil implement. But as luck would have it, many years back one of the knights had been part in a heroic battle against the evil wizards. His spoils-of-war chest included a small jar of Emulex chunks. The knights visited the wizard of Badgerville who possessed a mighty mortar that could safely reduce the chunks to dust. After procuring the ingredients and consuming the potion, the knights were at last ready to continue the main quest.

Uninstall SCVMM

The scroll of upgrade specifies that we should remove the current version using control panel, programs and features. If this should fail for any reason, you need to get a hold of the iso of installation for the previous version and launch the installer from there. The exact same ISO that was used during installation.

We fired up the installer and chose to remove both the management server and the console:

And of course, we beseeched the setup wizard to retain the database, as otherwise we would have to start over from scratch without any of our prior data. After that, it was a waiting game, twiddling thumbs while singing merry songs to encourage the setup wizard. When it completed successfully the wizard mumbled something about SPN records that the knights chose to ignore for the moment. On to the next waypoint we went!

Update from production 2020-03-06: If your active directory is filled with group policy scrolls of hardening and grumpy privileged access mananagement orcs, un-installation will fail hard. The Setup Wizard will reference his rather unintelligible SetupWizard.log file and otherwise refuse to give an explanation. If you spend some time reading the log scroll and manage to stay awake long enough, you will find references to “Out of impersonation” and “constraint violations” when trying to remove the SPNs using setspn.exe. The solution is not pretty, but it works:

• Create a special SCVMM upgrade OU.
• Enable block inheritance on this OU.
• Move the SCVMM server and the main SCVMM service account into this OU.
• Create or obtain an account that is a member of Domain Admins. NB: Do not use the deafault ‘Administrator’ account for this purpose. Or any other purpose for that matter.
• Make sure that this account or the Domain Admins group is a member of the local administrators group on the SCVMM server. It should be by default, but you would not be in this predicament if things were as they should be. Pam orcs are devious and unpredictable.
• Log in to the SCVMM server using the domain admin account.
• Run the SCVMM Setup Wizard.
• The uninstall process should complete successfully.
• Run the upgrade using the same domain admin account (see the next chapter for details).
• If you created an account for this purpose only, disable or delete it.
• Move the SCVMM server and the service account back to their original OU.

Install SCVMM

We triggered a restart of the VMM server just to be on the safe side. Then came the time to search for the previously acquired SCVMM 2019 ISO. After some time it was discovered on a thumb drive hiding in the bottom of a saddlebag in the mudroom. Happily, without any actual mud on the drive.

To reinstall SCVMM we need the following loot:

• The SCVMM installer found on the aforementioned ISO.
• The name of the SQL Server instance containing the VMM database usually named VirtualManagerDB
• Credentials for a service account with access to the VMM database
• Credentials for the VMM service account.
• A license key for System Center. Could be the same key as for SCVMM 2016. Or maybe not. The gremlins of the blue window are fickle fiends. Check with your local customer relations gremlin if your code doesn’t work.
• Current backups, as we did a side quest lasting a couple of months since we ran the last special backup.

Loot gathered, we went to the setup wizard and forked it over. The setup wizard went on with his usual shenanigans and wanted us to confirm that we really wanted to upgrade the database. As we had a backup, we solemnly swore that we were up to no good and confirmed our intentions. The setup wizard got the remaining loot and started mumbling again. And we had to wait. And wait some more. So we found it best to sing merry songs of encouragement interspersed with encouraging cheers of “You’re our favorite wizard” and such to increase our chances of success. (The knights had grown increasingly superstitious after the Emulex involvement on their side quest.)

This time the knights took notes when the setup wizard warned them about missing SPNs at the end of the installation.

Verify SPNs

Using the notes from the setup wizard, the knights went on to verify the SPNs.

• Please do note that this is an upgrade scroll. The values should already be in place, but they should be checked.
• On a domain controller, run setspn.exe -S [Values from the setup wizard]. (If you get an error talking about duplicate values, the entry already exists. Confirm by running setspn -L Domain\[VMM Service Account].
• Check the registry value “HKLM\Software\Microsoft\Microsoft System Center Virtual Machine Manager Server\Setup\VmmServicePrincipalNames” on the VMM server. It should contain the text SCVMM/[VMM Server Netbios name],SCVMM/[VMM SERVER FQDN].
• Run “C:\Program Files\Microsoft System Center\Virtual Machine Manager\setup\ConfigureSCPTool.exe -install” to configure SCP.

VMM Console will not start with error 1604

After restarting the VMM server, the knights were suddenly barred from logging in by an evil spirit. The console just sputtered meaningless errors:

“You cannot access Virtual Machine Manager server localhost. Contact the Virtual Machine Manager administrator to verify that your account is a member of a valid user role and then try the operation again.“

After banging on the door for several hours using different hammers and axes, and even trying to talk nicely to the door in all the languages and dialects the knights could muster, they went about searching for an answer on the spy network better known as scroogle search. That only provided garbage and ample ads for armor-itch ointments. They went on to kidnap a gremlin from the blue window in hope that he could provide answers. He demanded a special key for access and that the knights signed a scroll of license. Furthermore they had to agree to random spies in the cookie-jar. At least the gremlins were forthcoming about this in contrast to the sneaky creatures of scroogle. Anyways, the gremlins blamed the SPN setup. But the SPN setup was verified, and everything worked just peachy until the restart of the VMM server! After some digging around in old technet forum posts, the knights came across this: https://social.technet.microsoft.com/Forums/systemcenter/en-us/c3e5fd36-5de2-413f-87b8-538411d8dc6d/virtual-machine-manager-error-1604. It did not provide a direct solution, but the honorable DavyPierson mentioned going in to the depths of AD and looking for a key. A keyword to be exact, that had to include the name of the service account running VMM. We did not change the service account, but we chose to check it out anyway. The service account was OK as expected, but to our horror we found a pile of garbage where the SPNs should have been defined as a keyword:

We changed that keyword to SPNs:SCVMM/[VMM Server Netbios name],SCVMM/[VMM SERVER FQDN, restarted the server and voila, it was working again.

Note: If you do not know how to perform the task above in ADSI Edit, seek help. It is very easy to completely destroy yur domain beyond repair messing around in there.

Upgrade VMM Agents

At last an easy task for our merry, but at this point in the quest rather tired knights. All that was needed was to enter the Fabric part of VMM, right click each host and select update agent. The dialog will ask you for a RunAs account. Said account has to be a local admin on the host. If you are superstitious it might be a good idea to put the host in maintenance mode while you upgrade the agent.

Sounds easy? Well, it should be. But the knights were not to be so lucky. The first production server was updated without issue. And because of the Emulex scare, the knights had used maintenance mode and even restarted the host afterwards. The load was failed back and everything was peachy. Then came the second node. It was not happy. It spewed out errors. Error 2912 to be exact…

Update agent failed with error 2912

If you receive error 2912 while trying to upgrade the agent, there may be several possible issues blocking your path:

• A certificate issue. (see KB971264)
• WinRM problems (worthy of it’s own story).
• A problem with the DHCP extension.

If everything was working fine before the upgrade, chances are good that the DHCP extension is the culprit. The solution is quite simple:

• Acquire a copy of the current DHCP Extension. When you ran the SCVMM installation it extracted all files to “C:\System Center Virtual Machine Manager” by default, or another folder of your choosing. Inside you should find the subfolder “amd64\Setup\msi\DHCPExtension”, and inside that the file called DHCPExtn.msi.
• Copy the DHCPExtn.msi file to each host and run it. (Maintenance mode is recommended).
• Try the agent upgrade again.

If that did not help, try installing the agent and DHCP extension manually.

• Remove both the agent and the extension using programs and features.
• Restart the node.
• Reinstall both. You will find the Agent msi in a folder next to the DHCPExtension folder.
• You may need to trigger the agent installation from an administrative command prompt.

Connect to SCOM

In production, the SCOM MP for VMM had to be reattached. This is a SCOM problem, so it was outsourced to the SCOM farmers.

Disable spying feature

The knights prefer not to be spied upon, so the usage data gathering was disabled. They grumpily noted that it had to be disabled again, even though it had already been disabled in the previous version.