A new (and improved?) wasteland

This is a story in the “Knights of Hyper-V” series, an attempt at humor, with actual technical content hidden in the details. This particular one is just for fun though. Any resemblance of actual trademarks or people or events (real or those  that can only be found residing inside your mind), is purely coincidental and should be disregarded.

The knights of Hyper-V were doing some spring cleaning. Or, it was actually summer and thus to late in the year to call it spring cleaning anymore. Project setbacks, slow equipment deliveries and the plague-that-shall-not-be-named had severely hampered progress. But finally, the day had arrived to replace some of the hard working VMs with fresh new ones, running updated software versions glistening in the summer sun. Or covered in the more gloomy, but oh so common summer rain. And perhaps snow, locusts or other more or less funny local phenomenon.

The old servers was not really all that old, but a change in networking politics had ushered in an early swap over. We were leaving the The Wasteland of Nexus for a new and supposedly better (and cheaper) Wasteland. With software-defined wasteland processors or something to that effect. The knights did not really care, all they knew was that new network armor plate connections were required, and that was always a pain in the backside. The application minions would be grumpy as they would have to write scroll after scroll of requests beseeching for safe paths through the walls of fire.

But enough of the backstory. After a long, looong time the imposed quest for a new wasteland was nearing its end, and it was time for cleanup. Most of the knights were finally at summer vacation, preparing to queue along the congested paths to the beach, waiting in line to look over a cliff, visiting distant relations or hiding in a deep dungeon to escape the aforementioned plague. Only a skeleton crew (not composed of actual skeletons this time) remained to watch over the systems and do the odd cleanup job. A passing minstrel wrote an ode to one of the old servers in exchange for a late breakfast, or early lunch depending on  your point of view.:

Ode to server sixteen

New servers come in, and old ones get phased out.
It exists now only as a memory
Vanished into thin air
Like a fleeting ghost in the machine
Binary code rearranged to form new beginnings
It will always remain in our hearts

For the time being, all was well in the kingdom. All the VMs were kept in line by the automated all-seeing eye of OM, and it was time to relax, read, and practice dragon slaying if one was such inclined. Till next time, enjoy your life such as it is. Remember, things could always be worse. Before you know it, the roars of a three-headed Application bug dragon and the distant horrified screams of application team minions could wake you from your slumber…

Upgrade to VMM 2019, another knight’s tale

This is a story in the “Knights of Hyper-V” series, an attempt at humor with actual technical content hidden in the details.

The gremlins of the blue window had been up to their usual antics. In 2018 they promised a semi-annual update channel for System Center Virtual Machine Manager. After a lot of badgering (by angry badgers) the knights had caved and installed SCVMM 1807. (That adventure has been chronicled here). As you are probably aware, the gremlins of the blue window are not to be trusted. Come 2019 they changed their minds and pretended to never have mentioned a semi-annual channel. Thus, the knights were left with a soon-to-be unsupported installation and had to come up with a plan of attack. They could only hope for time to implement it before the gremlins changed the landscape again. Maybe a virtual dragon to be slain next time? Or dark wizards? The head knight shuddered, shrugged it off, and went to study the not so secret scrolls of SCVMM updates. It was written in gremlineese and had to be translated to the common tongue before it could be implemented. The gremlins was of the belief that everyone else was living in a soft and cushy wonderland without any walls of fire, application hobbits or networking orcs and wrote accordingly. Thus, if you just followed their plans you would sooner or later be stuck in an underground dungeon filled with stinky water without a floatation-spell or propulsion device.

Continue reading “Upgrade to VMM 2019, another knight’s tale”

Catching the ghosts of forgotten VMs

This is a story in the “Knights of Hyper-V” series, an attempt at humor with actual technical content hidden in the details.

It was nearing the end of summer, but most of the Knights of Hyper-V were still on vacation. There was of course always one knight on call, but the others were lazily roaming the countryside, or lounging along the bank of a river pretending to be on a fishing trip. Some even went on expeditions to far away realms looking for trouble, relaxation, fancy fishing gear, VMWare-proof armor, or new riding boots. The all-seeing monitors however, were not on vacation. To be honest we do not even know if they ever sleep, they just seem to take turns going into hibernation mode. Instead they had spent the summer installing new crystal orbs, automated all-seeing eyes and such. One of their new contraptions was some kind of network enabled spooky ghost detector. Its purpose was to send probes into The Wasteland of Nexus and attempt to locate signs of the ghosts of forgotten VMs and other security problems.

This came about as flaws had been discovered in the procedure for disposal of outdated VMs. The minions responsible for dealing with outdated VM disposal had gotten increasingly bureaucratic, spending most of their time hassling others with demands of forms filled in triplicate to update documentation. And such tasks are of course important, but the most important thing is to actually dispose of the old VM. The result was a number of undocumented (as the documentation had been updated) VMs roaming The Wasteland of Nexus without updated security software, making the entire realm vulnerable to outside attacks from beyond the wall. Firewall that is.

Such was the back-story, when one dark and gloomy midsummer morning, a trouble ticket landed in the inbox of the knight on call with a loud boom. It was another list of suspect activities detected in the wasteland. A couple of probes had returned during the night, complaining about servers without patches several years old. To add a little spice to the mix, this was ghost servers. If you nocked on the right door they would answer, but they were not listed anywhere. Not in the labyrinthine CMDB, and certainly not in any of the address books. For all intents and purposes they did not exist. Except of course for the undeniable fact that they most certainly did. This was something that could provide days, if not months of confused contemplation for social studies majors, human resources, project managers and others of similar ilk. But the knight was an engineer and simply scoffed at such irrelevancies. To him this was simply a problem looking for a solution. But which solution? The available information pointed to an ancient server from 2010. That is a very long time ago, and at least two documentations systems has been sent off to Valhalla by the way of funeral pyre in the meantime. The current buzzword-friendly variant was named after the Chinese philosopher Confucius. He was the inventor of the term “Do not do to others what you do not want done to yourself”, but if such terms was to be enforced in documentation systems, violent outbreaks would be the norm, as most documentation can be interpreted as a form of torture. Anyways, no trace of the ghosts were found in the current system, and the old ones were burned. There was always a faint hope that someone had kept a personal log mentioning the ghosts former names, but no such luck was to be had this time around.

The knight went back to the all-seeing monitors and requested more information to aid him in his search. Another probe was dispatched into the spirit world, this time with instructions to look for identifying marks instead of fuzzing about missing security updates, foul stenches and gates left open. While waiting for the probes to return, the knight identified an old long forgotten storage system. The storage minions swore it had been properly decommissioned and disposed of years ago, but it was found to be chugging along under a desk, consuming power and collecting dust.

Another sub-quest expedition to the physical realm of Hyper-V hosts revealed that someone had been re-inserting old decommissioned servers that were kept around for spare parts into the magic cabinet of the silver slanted ‘E’. Or it could of course bee that they had never been removed in the first place due to bureaucratic loops and lost scrolls of Todo. Anyways, the knight had bagged two ghosts.

We rejoin our knight the next morning. For once it was a good morning. The sun was shining, and the success of yesterday’s sub-quests were still lingering in the knights mind. Sadly, that would soon change. The probes were back, and they were happily reporting that the former names of the ghosts had been decoded. This identified the responsible service team, but the service team minions were all relatively new and had never heard of these old ghosts. Armed with new knowledge the knight went straight to the VMM daemons to demand an explanation. But to his great alarm, he found that the VMM daemons too had never heard of these ghosts. Feverishly the knight searched the scrolls of physical servers, in a vain hope that the servers nevertheless were physical beings, but no. No such server had ever existed. With that, only one possible solution remained; the ghost were located in the realm of VMWare…

There was no choice other than to beseech the man with the crowbar to borrow his Hazard Suit and plan an expedition to the toxic fields of vCenter. Once there, the ghosts were immediately detected. On a closer (but hasty) inspection of the remaining area, the knight also identified two other ghosts. He quickly filled out a scroll identifying the ghosts, and went back to more pleasing surroundings. He then updated the trouble ticket and forwarded it to the unholy riders of VMWare, hoping that he wouldn’t have to go back for a long, long time.

Dark magic at the backup site

This is a story in the “Knights of Hyper-V” series, an attempt at humor with actual technical content hidden in the details.

It was a nice Friday afternoon, and the Knights of Hyper-V were holding an end-of-week council meeting. The mood was light, as cunning and not so cunning plans were laid for the days ahead. There were misbehaving servers in need of an attitude adjustment, and misbehaving minions in need of a reality check. Just as the head knight was delivering a lengthy speech on the proper use of Virtual Message Queue-Incantations and the correct grammar for Receive Side Scaling spells of power, a stressed out envoy from the all-seeing monitors arrived and demanded immediate audience. They had lost all communication with one of the ghosts in the offsite backup dungeon. “Not another network armor issue!” exclaimed one of the Knights. She knew that some of the hosts at the offsite dungeon still used the inferior (and highly unstable) Broadcom plates as a connection to The Wasteland of Nexus.

The Knights leapt into action and went to interrogate the hosts. To their big surprise the hosts were in uproar. The cluster event scroll was gushing with blood red critical alerts, and host1 was completely unreachable. Undoubtedly, some dark magic was veiling this information from the gaze of the all-seeing monitors. The Knights bade the VMM daemons to put the missing host in maintenance mode. This proved to be a great mistake. As the VMM daemons tried to herd all the ghosts over to hosts 2 and 3, the cluster log cried out in red agony once more, but this time with storage problems on host3 as well. Thus the cluster was left with only one working node, and perhaps a cursed node spewing corrupt data into the storage. The Knights had no other choice but to take it all down and put all the ghosts to sleep. What started out as a quiet afternoon had suddenly turned into a frenzy of unruly runaway hosts and buzzards circling in the sky above the backup site.

Continue reading “Dark magic at the backup site”

The quest for the source of the foul stench, part 2

Part 1:https://lokna.no/?p=1837

On a rainy day, one adventurous minion traveled the dark and lonesome road to the physical realm of the troublesome hypervisor with the itchy connection to The Wastelands of Nexus. The minion scratched and scratched the hypervisors itchy back, until the rotten residue from the horrors of Nexus was all gone. He then presented the troublesome hypervisor with a new and shiny armor plate, and it was yet again prepared to rejoin The Wastelands of Nexus with a fresh and healthy connection.

Alas, the minions work had all been in vain. When the poor hypervisor on host 2 tried sending data through The Wasteland of Nexus and into the spirit world, it was unable to get a response. The ghost within was still restless, and upon investigation the foul stench came back and seemed to originate from the pipeline better known as HyperV1.

I borrowed a hazardous environment suit from a man with a red crowbar and went to investigate. A foul-smelling substance oozed out of the pipeline, and I wondered how the poor servers were able to survive in such an environment. I plugged the pipeline, and as soon as I had washed off the foul-smelling substance and placed the rags in an airproof yellow hazmat-container, the stench subsided and I was able to breathe freely. The virtual servers were able to get a stable connection to The Wasteland of Nexus through the spare pipeline known as HyperV2, and the ghosts rejoiced in happiness, as they were again able to communicate with the spirit world.

But the adventure was not yet over. We were no closer to identifying the source of the smelly substance. All we know is that it enters the hypervisor in the physical realm outside the host itself. May it be that the connections going into the physical server has gone bad? I would have to dispatch another minion into the physical realm of the hypervisors to look for answers…

Armed with this knowledge, an infrastructure minion scurried over to the physical realm of the hypervisor once again. This time she tried changing the place of Host 2 with Host 4. As feared, the failure followed the slot over to Host 4. This meant one of two things. May it be that the people of The Wasteland of Nexus had deceived us, and the connection was actually faulty at their end after all? Or could it be even worse; a problem in the magic cabinet of the silver slanted E? Into which all these hosts had to be inserted? The mere thought was enough to make the minions shudder, as everything branded with the mark of the silver slanted E was notoriously difficult to troubleshoot.

At long last, the Knights of Hyper-V decided that enough was enough. They ordered the minions to move host 4 into another functioning slot, and vitrify both the slot in the magic cabinet and the pipe going to The Wasteland of Nexus to make sure they were never used again. They further swore to expel anyone caught trying to bring more equipment bearing the mark of the silver slanted E into the physical domain. At last, the ghosts within the hypervisors were happy again, and the minions rejoiced for a couple of minutes before they went on to the next trouble ticket.

The quest for the source of the foul stench, part 1

Late one cloudy evening, I was alerted by one of the helmet-clad application team minions, who claimed that one of their spirits were not responding. It was time for an adventure!

Venturing into the land of the hypervisors, I noticed a foul stench of failure radiating from one of the hosts. Something was definitely rotten in the kingdom of Hyper-V host 2. I tried communicating with the ghosts within, but my calls went unanswered. The connection to the spirit world of the virtual servers residing in host 2 was down. I worried that the servers had run off to greener pastures, but The Wizard of VMM insisted that they were still in place. The Wizard also adamantly claimed that there was no problem whatsoever, and snorted offended that the foul smell had to originate elsewhere.

The horror struck me: I may have to go down to The Wasteland of Nexus to investigate. Such an adventure would require a companion fluent in IOS, the almost incomprehensible gibberish spoken by the dark-greenish inhabitants. May it be that the host was working fine as The Wizard claimed, but that the data just disappeared into the vast nothingness of DEV/NULL? I teleported in to the host console to interrogate the servers locally. After waiting in front of the server for a long time, the doors suddenly sprung open, and I was let inside. The poor thing was clearly in turmoil. The event scroll was running red with error messages complaining about everything from time sync to database connections. Some meditation in front of the scroll revealed that most of the problems were caused by a poor connection to the physical realm. This had led to an identity crisis; the server was not even sure of who it was anymore.

Just to make sure, I beseeched The Wizard of VMM to recite the incantations required to move the server out of host 2 and into a temporary host. After some time, the ghosts were once again responding to calls. I commanded the application minions to perform some tests. They sprang into action, and was shortly rejoicing, happy that their application was once again fully redundant. However, the quest was not over yet. Why were the ghosts so unhappy at host 2 and what was the source of the foul stench?

To be continued…

Part 2: https://lokna.no/?p=1842