Catching the ghosts of forgotten VMs

This is a story in the “Knights of Hyper-V” series, an attempt at humor with actual technical content hidden in the details.

It was nearing the end of summer, but most of the Knights of Hyper-V were still on vacation. There was of course always one knight on call, but the others were lazily roaming the countryside, or lounging along the bank of a river pretending to be on a fishing trip. Some even went on expeditions to far away realms looking for trouble, relaxation, fancy fishing gear, VMWare-proof armor, or new riding boots. The all-seeing monitors however, were not on vacation. To be honest we do not even know if they ever sleep, they just seem to take turns going into hibernation mode. Instead they had spent the summer installing new crystal orbs, automated all-seeing eyes and such. One of their new contraptions was some kind of network enabled spooky ghost detector. Its purpose was to send probes into The Wasteland of Nexus and attempt to locate signs of the ghosts of forgotten VMs and other security problems.

This came about as flaws had been discovered in the procedure for disposal of outdated VMs. The minions responsible for dealing with outdated VM disposal had gotten increasingly bureaucratic, spending most of their time hassling others with demands of forms filled in triplicate to update documentation. And such tasks are of course important, but the most important thing is to actually dispose of the old VM. The result was a number of undocumented (as the documentation had been updated) VMs roaming The Wasteland of Nexus without updated security software, making the entire realm vulnerable to outside attacks from beyond the wall. Firewall that is.

Such was the back-story, when one dark and gloomy midsummer morning, a trouble ticket landed in the inbox of the knight on call with a loud boom. It was another list of suspect activities detected in the wasteland. A couple of probes had returned during the night, complaining about servers without patches several years old. To add a little spice to the mix, this was ghost servers. If you nocked on the right door they would answer, but they were not listed anywhere. Not in the labyrinthine CMDB, and certainly not in any of the address books. For all intents and purposes they did not exist. Except of course for the undeniable fact that they most certainly did. This was something that could provide days, if not months of confused contemplation for social studies majors, human resources, project managers and others of similar ilk. But the knight was an engineer and simply scoffed at such irrelevancies. To him this was simply a problem looking for a solution. But which solution? The available information pointed to an ancient server from 2010. That is a very long time ago, and at least two documentations systems has been sent off to Valhalla by the way of funeral pyre in the meantime. The current buzzword-friendly variant was named after the Chinese philosopher Confucius. He was the inventor of the term “Do not do to others what you do not want done to yourself”, but if such terms was to be enforced in documentation systems, violent outbreaks would be the norm, as most documentation can be interpreted as a form of torture. Anyways, no trace of the ghosts were found in the current system, and the old ones were burned. There was always a faint hope that someone had kept a personal log mentioning the ghosts former names, but no such luck was to be had this time around.

The knight went back to the all-seeing monitors and requested more information to aid him in his search. Another probe was dispatched into the spirit world, this time with instructions to look for identifying marks instead of fuzzing about missing security updates, foul stenches and gates left open. While waiting for the probes to return, the knight identified an old long forgotten storage system. The storage minions swore it had been properly decommissioned and disposed of years ago, but it was found to be chugging along under a desk, consuming power and collecting dust.

Another sub-quest expedition to the physical realm of Hyper-V hosts revealed that someone had been re-inserting old decommissioned servers that were kept around for spare parts into the magic cabinet of the silver slanted ‘E’. Or it could of course bee that they had never been removed in the first place due to bureaucratic loops and lost scrolls of Todo. Anyways, the knight had bagged two ghosts.

We rejoin our knight the next morning. For once it was a good morning. The sun was shining, and the success of yesterday’s sub-quests were still lingering in the knights mind. Sadly, that would soon change. The probes were back, and they were happily reporting that the former names of the ghosts had been decoded. This identified the responsible service team, but the service team minions were all relatively new and had never heard of these old ghosts. Armed with new knowledge the knight went straight to the VMM daemons to demand an explanation. But to his great alarm, he found that the VMM daemons too had never heard of these ghosts. Feverishly the knight searched the scrolls of physical servers, in a vain hope that the servers nevertheless were physical beings, but no. No such server had ever existed. With that, only one possible solution remained; the ghost were located in the realm of VMWare…

There was no choice other than to beseech the man with the crowbar to borrow his Hazard Suit and plan an expedition to the toxic fields of vCenter. Once there, the ghosts were immediately detected. On a closer (but hasty) inspection of the remaining area, the knight also identified two other ghosts. He quickly filled out a scroll identifying the ghosts, and went back to more pleasing surroundings. He then updated the trouble ticket and forwarded it to the unholy riders of VMWare, hoping that he wouldn’t have to go back for a long, long time.

Author: DizzyBadger

SQL Server DBA, Cluster expert, Principal Analyst

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.