Biomythography, Note 51
Command Line of Doom
Sunlight had started to turn amber through our back window. The playground behind our townhouse still looked bright enough to play in. My son stared at it from his highchair. He banged his spoon on his plate.
Next to our dining area, the phone hung on the wall. Its handset was beige and cheap. It was sturdy. But I never answered it. I hadn't answered a call in years. When it rang as we were sitting down, Diane diverted from the path to her seat to pick it up. After a moment, she handed it to me.
"It's Adam," she said.
I put a sippy cup in my child's hand and rose from the dinner table.
"I've got a problem," Adam said on the other end. "I thought I'd share."
"What is it?" I kind of liked having problems to solve. Adam knew that although, really, he liked solving things too, so he didn't always share.
"The Usenet server crashed."
"Yeah?" That was awfully fast. I did some counting on the calendar page next to the phone and saw we hadn't gone a full three weeks since we installed the node. Of course, I understood Usenet services had changed from 1991 when I'd first gotten to know them. The main thing was, the Eternal September had taken place.
During Eternal September, which began in the fall of 1993, Internet service providers started offering Usenet access to everyone who asked. That included Adam and me as of last month. But the explosion of nodes everywhere around the country, actually across the world, changed Usenet. The Usenet threads had been channels for semi-private chats involving professional academics. Now computer hobbyists and various other professionals were getting involved. In 1994, the AOL service decided to open a Usenet gateway, too. That added even more non-academics.
Everything was faster, bigger. Different.
"The stipend for the job was pretty good." Adam's voice took on a tone I recognized as leading into a set of logical statements.
"Yeah." It was true. I waited for his logic.
"I figure it should cover a session of troubleshooting."
Crap, that seemed reasonable. I didn't want to travel from Frederick to DC again but I was nodding even before he finished his sentence. I'd left the Usenet node running in fine shape. But maybe I'd created some sort of problem with my vanilla configuration. It was possible. Maybe I'd programmed the server to crash in some slow, non-obvious way.
"Okay," I sighed.
Since his office felt the situation was urgent, I agreed to report on Saturday morning. We made the same driving arrangements as before. After I got to Adam's place in Gaithersburg, we hopped into his car and he took us the rest of the way.
Once again, his offices looked dark but somewhat comfortable and well-used. We flipped on the lights. While Adam took care of other business, something about a color printer, I sat myself at the Usenet server. It didn't take much time to figure out what was wrong. The process table showed a bunch of necessary programs were missing including the Usenet service. Also, some of my commands to the server didn't work. Anything I ran that created a file seemed to fail. I checked the disks,
$ df -k
and read the usage report.
"The disk is full," I called to Adam. "Like, it's completely, one hundred percent full."
"Huh, well." The tone of his voice, even at a distance, told me Adam had suspected as much. "I know you said to keep an eye on it."
"I was busy." His voice got testy. He stalked from desk to desk. Something was wrong with the office printer and he wasn't having a straightforward time with it. "This is where I work. I have other jobs."
My fingers rested on the keyboard of the little Linux server.
"Yeah, but it's not just like you're supposed to keep watching and deleting whenever you notice it's getting full. You have to filter it. You have to edit the configuration files."
"Well, I don't know how to do that."
"You watched me do it. We wrote out notes."
He remained silent for almost half a minute. He shrugged.
"Okay," I said, doing math on a scrap of paper. "You've got seventeen days of the entire Usenet. That's the limit for this storage. The machine has got 180 megabytes of hard disk. Your real limit, one that gives you room for swap space, needs to be less than seventeen days. If you want decent swap space, I say you should keep ten days without any other filter setting except timestamps."
"Well, I had to buy all three machines the same."
"Yeah, but that means this is what you've got. One hundred eighty megs, unless you put in another disk. You can keep all the Usenet threads for a little while, but not for very long. Or you can filter out some of them and make the storage last for a month. But you can't keep all of the Usenet traffic forever."
"From what my boss has been saying, he wants at least three weeks."
His boss had business reasons, I figured. I started running directory usage summaries and doing the math.
$ cd comp
$ du -h
$ du -h ./graphics
$ cd ..
$ cd alt
$ du -h
After a few minutes, the pattern of Usenet history began to emerge. I understood which threads used up most of the storage. I could see there wasn't much in the science directory tree besides a scattering of text files. The scientists at various research sites around the country were holding discussions but they weren't sharing microscope images or anything with large binaries to take up space on a disk.
"Do you know what threads you want to keep?" I called to Adam. When we'd configured it, he hadn't.
"Yeah." He wandered closer with a toner canister in his hands. "I think I do."
"Okay, then, I can tell you where most of the payoff is going to be. We can delete the big directories you don't need to save. And I can set up any filters you tell me."
Finally, his expression eased into a tentative smile.
"My boss probably wants to leave the social discussions and the talk threads about nuclear disarmament."
"Those are easy. They don't take up much room." I ran a du -h on the talk directory to make sure. It reported about what I expected, 9M of data, all of it in text files on the various threads. The same was sort of true with the computer discussion threads except in some of them, like /usr/local/bin/usenet/comp/graphics, people sent image files sizable enough that it would help to delete them. But Adam wanted to keep the graphics.
"That's the best part," he said.
"You know, I have to set up the filters for what to save. But really I'm asking what I can delete. If I can't delete comp.graphics I need to know what other big folders I can blow away."
"Let me see." He pulled up a chair next to me. When he launched into what to save, I tried to bring up a file editor for the Usenet configuration file.
"Wait!" I raised my right hand. "Wait, wait, wait. I have to make room on the disk first. Deletions. I need some, at least. I can't edit the configuration until I blow away some Usenet files."
"Well, don't remove the stuff I want."
I checked what I was doing. I'd lost track for a moment.
Somehow I'd ended up in the wrong directory. That happened all the time in the vanilla installation of Slackware. There was no customization of the command line prompt. Soon I got back to where I needed to be, though, and talked with Adam about the sizes of files, folders, and the details of the usenet directory structure.
$ cd /usr/local/bin/usenet
When I'd deleted enough files, I edited the configuration while Adam watched. He had to make decisions about what discussions his office wanted to participate in. He let me put a three week timer on the files, too. It was an option in the configuration he hadn't wanted to exercise before. With the decision made, though, whenever he wanted to archive a thread for his boss or a set of graphics files for himself, he would have to make the decision within twenty-one days. After that, the usenet service timer would purge the files and they'd be lost.
I had to go back and forth between changing directories, running pwd to make sure I was in the right place, and running rm -rf * for a long time.
$ cd /usr/local/bin/usenet/comp/VMS
$ rm -rf *
$ cd ..
$ cd lang
$ rm -rf *
$ cd ..
$ cd sys
$ rm -rf *
Adam's business wanted to save particular parts of the file trees. That meant I had to individually run recursive deletions in multiple spots. Each time I wrote 'rm -rf' I told the computer to remove files with the -r flag for doing it recursively (everything I specified and everything in the further along the directory path, too) and the -f flag (for 'force,' meaning I didn't want to get asked every time for ten thousand times about whether I wanted to delete something). Basically, I was telling the system to shut up and delete what I said.
Finally, after some negotiation, Adam decided alt.music could go. There was a ton of space used by the directory for it, so when I told it to delete, nothing happened for a while.
"How long is this going to take?" Adam asked. He'd been watching me for half an hour or more. He had wandered off and wandered back, too, announcing he'd fixed the printer. For at least twenty minutes, he'd done nothing with his hands.
"A while." I'd already done the math in my head. At the rate the disk could clean itself up, the big deletions would take an hour.
"What does it take to fire up Doom?" Adam wondered.
"We'd have to play on the other two machines," I pointed out.
"So? There are two of us."
I scooted over to the web server, logged in, and ran,
$ cd /usr/local/bin
$ ./doom -net 192.168.0.4 192.168.0.6
Adam laughed as he got it running on the mail server, too. Soon, we were each at our console and dashing around in the game arena. We blasted the hell out of each other with bazookas. Every now and then, after I killed Adam or I died, I'd hop up and run to the Usenet machine to blow away another directory.
After twenty minutes between deletions (I'd found a good sniper spot and crushed Adam for a while, both of us laughing because he couldn't take two steps), I dashed back to the Usenet server. Oddly, the drive light was flashing. The command prompt hadn't returned from my previous command, either.
I hit a CTRL-C to stop the removal. Then I ran a pwd.
"What's wrong?" Adam said.
I didn't answer right away. I was surveying the damage. My recursive deletion had been running without interruption on the main Usenet folder. I had been deleting everything in Usenet. Everything. Including all the stuff his boss wanted to save. Maybe. Probably.
"Sorry, man." I showed him the remaining folders. I tried to explain what had happened. Fortunately, the recursive deletions had gotten stuck in the alt folder, where there was too much to remove in twenty minutes. The routine had removed most of alt.binaries, though, something Adam had wanted to keep.
"It's all really gone?" Adam asked.
"Yeah, really. Until I fire up the server again, those folders will be empty."
He sighed. A moment later, he shrugged.
"Your boss won't be mad?"
"Yeah, but no," He gave a sardonic smile. "He won't be mad at me. He'll be mad at the consultant."
Hey, that was me. "Why?"
"I'll just tell him you made the decision. It's not like you'll ever see him"
"Oh." Well, that was that. "True."
"I meant to do that," I added and Adam mouthed it at the same time. The Pee-Wee Herman show had been off the air for a couple of years but every now and then, he referenced it like a pop culture reflex. I'd developed similar reflexes from him. It was no surprise to find him anticipating me.
"Is there any directory left to blow away?" he asked.
"A couple. The religion tree and the fido tree might as well go."
"Start on those. Then let's play some more Doom."
$ rm -rf *
A few seconds later, on a different window of the same computer, I typed,
$ ./doom -net 192.168.0.5 192.168.0.6