The Y2k Bug (in People's Minds)
(continued)
For a couple months, I roamed the off-white tiles of the facility, finding machines on desks, shelves, and floors, figuring out what fixes they needed, and grinding through the process. For some, I had to install new software. For others, I had to edit configuration files. (Some applications had prepared for different date formats - there are different calendaring styles around the globe, after all.) For others, I needed to write shell scripts. The easiest part was running patches. The vendors supplied those. All I needed to do was follow instructions. Sometimes, though, I discovered problems like, a) the vendor had no patch, only a promise to write one, b) their patch didn't work, or c) the company responsible had gone out of business. When there wasn't a vendor fix, I'd research to find a rival vendor with a compatible fix or a hobbyist who'd written a script. A few times, I went to the source files on the disk and edited them.
It was fun work, in its way, but it was clear that our facility would get hit hard by the year change unless I took the needed steps. In all, I fixed around seventy systems, which I tracked in a list, and a dozen or so applications I tracked in a separate list.
"Are you worried?" my boss asked me as the end of December loomed.
"Nah." I folded my arms and surveyed the machine landscape. I was accustomed to the construction dust and the human smells from clinical trials, patient rooms with their antiseptics, the bright yellow-and-black warning tapes and signs, and the dull roars from air handlers down the halls. We also got occasional animal odors from the area of the future mouse imaging lab, where the vet techs were running tests, walking the landscape with blueprints in hand. They kept failing to install network wires for their instruments. I paused for a moment to feel irritated about having to catch their design flaws so often. In four months, I'd corrected four floor plans. "I've gotten everything."
"Really, everything?"
"Pretty sure." I harrumphed. The specialty medical apps had not been easy, some of them, and only the week before I'd caught someone using one I hadn't seen. We didn't have a software inventory list so I had to search each disk, sometimes each drawer, and watch the scientists in action.
At least our network was a stack of white-bodied, unmanaged hubs. They were dumb packet repeaters. They didn't log anything. They didn't manage packet traffic. I didn't have to upgrade them because, in fact, it wasn't possible.
My boss had me check again. I reviewed all the systems, moved the dates forward on one of every machine type, and declared it all fine and all verified. I went home feeling good about Y2K. On New Year's Eve, I spent the night with my friends, three-quarters of whom were also computer admins. We complained about the work, drank or smoked, watched the fireworks on television, listened to broadcasters panic about Y2K, complained to ourselves about the public Y2K complaints, and heard from one another that we were sure we had gotten it right.
When the next Monday came around, I drove in early. All the lights were on in the research facility. All the computers were humming. I ran one of our science applications as a test. It worked fine. I ran another, less popular analysis program. It performed precisely as it should.
I popped into SSH and connected to a remote machine. It worked fine. Our dumb network was all right - even though I couldn't upgrade it.
At around eleven in the morning, I added a couple of incoming staff to our user directory. I set their default passwords, configured the accounts to require the users to reset their password immediately after login, sorted them into their appropriate lab groups, and ran a backup of our Network Information Service (NIS). Since we didn't have anything better than tape backups, I kept a spare NIS server. That was what I used as my hot-swap failover. It doubled as the spare copy of all our user information.
The job got two lines into my backup script and crashed. When I pinged the backup server, I found it had gone offline.
"What the hell?" I pushed myself out of my chair by the armrests. I knew the old Sun SPARC 2 had been bought in 1994. Someone had placed it on a corner shelf of the next door office. In all my months, I hadn't bothered to move it. The little box was ancient but reliable.
When I got to it, I scrounged a grey, centronics cable and a monitor to login. I had to stand on one of the desks to work. The SPARC2 graphical interface crashed and dumped me to the command line. Above the system prompt, I saw an error about an "unknown date." I was pretty sure I knew what to do. First, I had to change the date to 1999 so it would boot normally. Then, well, we had another SPARC2 I'd patched. I grumbled my way back to my office for an external disk and the correct software.
"What happened?" my boss asked. He knew something was wrong by my scowl.
"The backup NIS server crashed."
"Was it the Y2K bug?"
"Yes." I sighed, hands on hips, and faced him to apologize. "Sorry, I thought I'd gotten everything."
He waved it off. "Can you fix it?"
"In about half an hour."
He nodded and marched away. Twenty minutes later, he strolled back to my desk, where I'd returned after getting the SPARC2 running with network ports activated. At this point, I could work on it remotely and fix the rest of the configuration.
"I've been testing everything," he announced. "The other equipment, I mean. All the instruments. They're good. It's all good. The whole place is fine except for your backup. Is the NIS server the only thing that broke from Y2K?"
"Yeah."
"Not too bad," he concluded. He set his hands on his hips and gazed out into the busy facility.
#
A couple days into the new year, the American media concluded that Y2K for computers was no big deal after all. I wasn't convinced because I'd been there. I'd done the work. I knew the talking heads were wrong.
Clearly, we had a bug in humankind.
An ever-present glitch in people's minds is that they don't give themselves credit for preventing a disaster. Like the steps taken to fix the ozone layer, the Y2K event was a pretty good victory for humanity. In the environmental case we didn't actually fix the problem. Instead, we managed to stop destroying part of the atmosphere for a little while. That was good enough. Our Y2K victory was a smaller one but hundreds of thousands of system administrators and programmers around the world united, at least for a while, to keep our infrastructure running. We managed to avoid collectively shooting ourselves in the foot.
The glitch is, we didn't give ourselves any credit. Even when we made the right decisions, collectively, we compared it against some imagined ideal of an unachievable outcome; and so we regretted doing our best. We downplayed our successes.
We pretty often still do. The Y2K bug was a small victory. But it was a victory.
No comments:
Post a Comment