The Wake-Up Call

Joe was woken by an ear wrecking sound. He accidentally tripping over a couple of empty cans of Redbull as he flew back in his chair, grabbing both armrests, like he was trying to avoid falling out of a plane, with no parachute.

Joe was still sitting at his computer, with several lines of square imprints on his right cheek, indicating, that he had been sleeping there for some time.

After a few seconds of disorientation, he finally came to the realization that it was the guard phone from work, that had woken him up. He instinctively turned off the alarm, leaving the room quiet once more. The alarm only rang in case of an emergency. It could be something as innocent as a critical server failure, at a customer site, to a catastrophic Lights Out Incident that would cause a global Internet outage, leaving everyone without an Internet connection. Joe quietly hoped for an emergency that could help advance his career and increase his paycheck, something that his regular working hours, had not helped him achieve.

Joe had worked at Vitrix Inc. for just over five years, and during that time, he had made little to no career progression. In the meantime, several of his co-workers had been promoted and gotten generous, without demonstrating any skills or work effort that would cause it. Working in tech, Joe had come to realize, was like stepping into a different world, with its own language and rules, that everyone began to live by after a few months. Promotions were based on hero-moments, making higher-ups’ look good, and not as a result of putting in a solid and stable work effort. Tech had shown to be a rigged game, that needed to be played, to advance.

Joe looked at his watch, it was December 31st, 9:32 PM. He looked back at the guard phone and quickly realized, that he was not on call Today. As a matter of fact, this was his day off, as part of his first consecutive days off, for the past 3 months. He knew, because his boss, Mr. Gretberg, had specifically told him not to log in between the 22nd and 31st of December.

“If I see you commit any code during that period, it will be your ass!” or something along those lines, was the subject of their last one-on-one chat. There had apparently been some complaints about Joe’s recent behavior. Spiteful comments, unstable working hours, rejected pull requests on a whim, password resets of co-workers’ accounts, Mr. Gretberg had gone on and on. Joe did not agree to any of this, of course, and tried to direct the attention to the co-workers’ incompetence and their jealousy of Joe’s superior skils, but the decision had already been made by the higher-ups, Mr. Gretberg had told him. “We need to set an example.”, and that was the end of it.

Joe looked at the message on the guard phone:

Guard Phone Alert Message

Received: December 31st, 9:31 PM



ALERT!

Come to the office!

This is urgent!

Respond to this message immediately with “YES” if you comply.

He had not been at the office, psychically that is, since last spring, where the Vitrix leadership team had decided, that everyone should work remotely, to prevent another incident.

Joe knew, that this was his chance to show his worth, a possible hero-moment, that could finally help him advance his career.

Joe replied immediately;

Guard Phone Alert Message Reply

Sent: December 31st, 9:34 PM

YES

The Drive

The Uber driver turned around and smiled, as Joe entered the black Honda Civic, that was waiting outside his apartment building. The driver had received an impressive five-star rating, but it was not for his communication skills or his ability to drive, Joe soon learned.

“Are you going to a New Year’s Eve party?” The driver asked.

“No, I am going to work.” Joe said as he looked out his side window, in the back of the car, trying to avoid further small talk.

“Ohh man, that sucks! On New Year’s Eve? Where do you work, on a nuclear power plant or something?”

“At Vitrix.” Joe replied.

“The server company?” The driver asked, in a way that implied a lack of understanding, why someone would have to go to work on New Year’s Eve, to tinker on a server.

“It is not a server company. Vitrix is the reason you are able to drive this Uber for a living.”

The driver gave Joe a questioning stare through the rearview mirror. After what seemed like an eternity of driving without the driver actually looking at the road, Joe quickly added; “Vitrix is a global Internet infrastructure provider. We keep the Internet running.”

“Ahh, then I better get you to work in a hurry, I can’t risk missing New Year’s Eve surge pricing!” the driver exclaimed, stepping harder on the gas, knocking Joe back in his seat.

“As long as I’m alive, when we get there!”, Joe whispered through his teeth.

The Situation Room

“I hope for a five-star rating.”, the Uber driver said, as he pulled up at their destination, with a smile so fake, it could scare a five-star rating out of anyone, and that was probably the cause of the rating in the first place, Joe thought.

Joe got out of the Uber and looked up at the 72-storage tall behemoth that was the Vitrix HQ. The only lights that were turned on in the building, was on the 42-floor, or, the Developer Hub, as the higher-ups liked to call it. Joe’s co-worker Ron liked to refer to it as the ‘Jump floor’, since the incident. Ron was the equivalent of the class clown, but this clown was racking in a yearly salary in the top 6-figures, making 30% more than him, with 99% less competence, according to Joe. Ron and Joe often talked about the irony of the development team being placed on the 42-floor, which often was referred to as the answer to all Vitrix’s problems, but was also, in many ways, the cause of Vitrix’s problems, when new servers or software was released - littered with sloppy code and breaking changes on external APIs, causing red alerts and guard phones ringing at off-hours.

When Joe exited the elevator on the 42-floor, he was met at the door by Amy, the desk clerk.

“Great! Finally! About time!” Amy uttered. She was dressed as if she had been on her way to a New Year’s Eve party, and she didn’t seem happy about the sudden change of plans.

“The entire team is waiting for you in the Situation room.”

“Who is?” Joe asked.

“The higher-ups and Ron, Mike, and Susan from your team.”

“What is going on?” Joe asked, surprised that the higher-ups were joining in, on a day like this. He straightened his shirt and put a hand through his hair. This was his time to make an impression they would remember.

When Joe entered the situation room, the well-known smell of engineer had already filled the 12-person meeting room to the brim.

“That was about goddamn time!” one of the higher-ups exclaimed when he saw Joe enter the room. “Ron, bring Joe up to speed, and when I get back here, you better have a patch ready! I have a fucking party to attend to.”

“Yes sir, Mr. Kuakass” Ron replied.

“It’s 10:45 now, that gives you one hour at most, so you have time to ship the patch.”, Mr. Kaukass stressed as he exited the situation room with Amy, and the rest of the higher-ups, leaving Joe, Ron, and Susan in the room.

After not receiving the welcome he had hoped for, Joe asked; “What did I miss? And why the hard deadline, don’t tell me it is because of some stupid party?”.

“This is the clusterfuck of all clusterfucks!” Ron said.

“We are about to cause a global Lights Out Incident”, Susan said, “and you need to help us find the cause - fast. If we don’t patch this before entering the new year, everything turns to shit. That includes our careers and this company!”.

“We know that a server update was released some time ago,” Ron added, “that somehow infected all Vitex servers, causing them to become bricked based on the date when entering the new year.”.

“Bricked how?” Joe asked.

“Like; dead, K.I.A, point of no return, nuked,” Ron answered spitefully, “you know what bricked means, or did the ‘vacation’ wipe your brain?”.

After a few seconds of silence, thinking through what the ramifications of an event like this would mean to the world, and to his career if he helped prevent it, Joe asked;

“How did you discover the bug?”.

“I ran a server update simulation on a new patch we were planning to release, after the code freeze in early January,” Ron began, “but I accidentally got the year wrong, causing a bricked server. After a lot of digging around, running multiple tests, and informing the higher-ups, we found the bug to be related to the date.”

“Specifically,” Susan continued “when entering a new year, at the exact time of midnight. We identified a previous server update, that seemed to be the cause of this, but we were not able to roll back or anything, the update seems to have corrupted previous versions for some reason. You, Me, and Ron are the only ones who have worked on the server update that caused this, so we got you here as fast as we could.”.

The Pull Request

Joe sat down and pulled his laptop out of his bag, looking up the several months’ old commits, that were associated with the faulty server update. He knew, that he had been working on it, but he hardly remembered anything about it. He had done most of the work at home during the night, with 12 cars of Redbull flowing through his system.

451 commits by Joe himself, 232 by Susan, and 142 by Ron, that lazy overpaid bastard. There was one other commit, out of all those, that caught Joe’s attention. It looked like a commit Ron had made, but it didn’t fit the pattern of date and time of Ron’s previous commit history. When looking closely, it had no Vitrix user associated with it. The account used Ron’s name but had a phony email address linked with it. Something like that would only be done if the committer deliberately wanted to hide their tracks. Joe knew, even before opening the commit, that this was what he was looking for, but he also knew not to tell his team, at least not right away.

Joe thought; “Was this Susan’s doing, and why frame Ron? Or maybe it was Ron, creating an alibi for himself by not using his own account?”.

Joe had never seen anything like this. The commit, by the unknown author, modified all lines of code in the codebase. The added code was not readable at all but scrambled on purpose. It was encrypted, enforced by several external libraries. If you did not look at the code in the commit, you would be none the wiser, since it appeared to create a mirror version of the previous code, hiding in plain sight. This explained why nobody had found it already. By looking at it, Joe knew that this was way above Ron and Susan’s skill level.

Joe thought, “This would have to have been done by an outsider, assisted by Ron or Susan. But to what end? Why bring the Internet to its knees and kick everyone back to the stone age? … This could only be done by Hualia, the Chinese Internet competitor, to demonstrate the instability of a centralized Internet, by making VItrix look bad, it would open up the market to them through enforced regulation!”.

After thinking through multiple scenarios, Joe realized that something like this could never pass a review in a Pull Request unless the reviewer was in on it. There are strict guardrails that prevent something like this from happening. That’s why a developer is not permitted to review or approve their own commits, people become lazy and sloppy, and the company, or the world, in this case, suffers as a result.

Joe looked at the line identifying the email of the Pull Request approver, and there is was, the ID of the accomplish; joe@vitrix.com.

The Patch

“This made no sense.”, Joe thought, “someone must have hacked my account and approved the request without my knowledge. Hualia would definitely have the motive to do such a thing, framing me in the process! There is not enough time to repair it before midnight, but something must be done.”.

Joe looked at the time; 11:23 PM. The higher-ups would be back at any minute, and Joe did not have an explanation or a fix, he was willing to share with anyone.

“Did you find anything?”, Ron asked, without looking away from his screen, sensing Joe’s sudden pause on the keyboard.

“No, not yet.”

“Just look through Susan’s commits,” Ron continued, directed at Joe “I am sure the bug is in there somewhere.”.

“Why don’t you shut up Ron?”, Susan snarled, “If Joe and I didn’t have to pull your weight as well as our own, and you would deliver an honest days work, just once in a while, we wouldn’t we in this mess, would we Joe?”.

Joe didn’t answer. He was busy trying to get out of this mess with his job intact, and maybe saving the Internet, while he was at it.

Ron and Susan continued to argue, while Joe hacked away on his keyboard.

Lights Out

“I fixed it!” Joe exclaimed, pushing himself away from his computer.

“What?” Ron said, realizing he had missed, what must have been, 15 minutes of hardcore coding, while he was busy arguing with Susan, about who was to blame for the entire incident, and who would have the best chance of getting hired at another company when everything had turned to shit.

Ron didn’t get a chance to ask any further questions, since the higher-ups slammed through the door to the situation room at that very second. It was clear on their faces, as they entered, that the smell of engineer had not gotten better, since Joe got there.

“Jeez”, Mr. Kaukass said, “open a window for god’s sake!”.

“All the windows on the 42-floor are glued shut, since the incident last December, they can’t be opened.”, Susan said, without realizing that Mr. Kaukass and the rest of the higher-ups knew all about that incident, and why the decision had been made, to glue the windows shut.

It was clear by the look Mr. Kaukass gave Susan, that he had not meant it in a literal sense, and didn’t need to be reminded about any other incidents, than the one they were currently trying to handle. Susan almost curled up on her chair and faced her computer to avoid any verbal outburst from Mr. Kaukass.

“We have a solution,”, Ron interjected, as the time passed 11:51 PM, hoping to save face, “tell them, Joe.”

Everyone turned and looked hopefully at Joe.

“Well,” Joe began, “I wouldn’t call it a solution, but I have identified the cause of the error, and who caused it.” Joe leaned back in his chair and pointed at his screen.

“It is not caused by a software bug, but a deliberate harmful commit, designed to harm Vitrix by a competitor, that is trying to gain market share, as you see here,” Joe scrolled down, pointed, and continued, “Ron committed this code a few months ago, and it was immediately reviewed and accepted without comment, in a Pull Request by…”

“Susan!”, Mr. Kaukass read from the screen.

As soon as he had uttered the words, the lights, power, and internet went out as far as they could see, from the glued windows of the 42-floor, where only bright fireworks lighted up the room as the clock struck 12:00 PM, in a very inappropriate New Year’s Eve celebration.

The Aftermath

It turned out, to the fortune of Uber-drivers everywhere, that the software was easier to fix, for Joe, than first anticipated. He managed to get all servers back online within a few hours in the new year, a real hero-moment.

Joe never saw Ron or Susan at work again after they were escorted out, not even on social media, which seemed kind of odd to Joe.

Joe got a promotion as a result of his efforts on New Year’s Eve, and his help getting the Internet up and running again. Joe even got an office, just for him, and a team under his direct supervision, something he had always dreamt of.

He smiled at his new employees as entered his office and closed the door, revealing his newly printed name tag on the door; “Mr. Joe Gretberg”.