Date: Fri, 6 Jan 1995 23:05:28 -0700 From: chuqui@netcom.com (chuq von rospach) To: apple-internet-users@medraut.apple.com Subject: Admin: what happened. Message-ID: I just wrote this to someone I know. It explains pretty well what's been going on with this list the last week (and I want to apologize for the problems, glitches, delays, overloads and other hassles of the last week. I'm doing what I can to make sure this is ALL of the propblems you'll eve see on this list...), so I figured I'd share it with everyone. Hopefully, it make clearer why things have been a bit flakey. chu ----- >I've subscribed to your Mac/Internet list -- looking forward to reading it. >Thanks for doing it. (Saw the plug in TidBits.) You and the rest of the known universe. Wanna hear a fun story? No, but I'llt ell you anyway. Adam shipped off Tidbits around Sunday. I'd originally expected, oh, 500ish on the -users list. Around Christmas, last time I looked, it was about 350. When I got in Tuesday morning, about 8, I checked, and the list was abou 850. I went and gloated at a co-worker, blythely claiming that I'd probabl crack a thousand by, oh, three or four days. I cracked 1,000 about 11:30. By 3PM I was at 1,350. About then, my server, which hadn't been tuned for these kinds of loads, started barfing... I peaked about noon yesterday at about 1,550. The move from about 400 users is 100% attributable to TidBits. Since then, because volume have gon absolutely looney, I've dropped back down to about 1,475 as people run screaming into the night. I've spent two days trying to tune the damn Unix box so it wouldn't thrash itself into oblivion. At one point, I had about 120 -users messages pending, both for distribution and administrative, that the listproc simply could never catch up on. I've had to retune the kernel five times, re-adjust buffer space three, had five crashes, the listproc hung twice and ate the subscriber list once (thank GOD for backups! But I've now rewritten all my scripts to archive the last 30 days of subscriber lists, just in case. Useful for reporting, too) because I ran out of process slots and fork() starte failing.... Right now, the front end, listproc, is caught up, but I've got about 600 email packages queued into sendmail with 77,000 (!) messages waiting to be sent. I've got the little porker running with five separate looping sendmail setups, plus I started five others just to push it a little faster overnight. They'll time out eventually. Last I looked, there were 88 messages aimed just at ME sitting in the queue waiting to be sent out. You don't want to know what my mailbox has been like, but I've topped 600 email messages since 11PM Thursday night. What's amazing about this is the SPEED of information dissemination. How fast the information moved around. How fast users reacted to it. How QUICKLY a system can be innocently blown to pieces by users jumping on i all at once. If I'd been smart, I would have pre-tuned the system in anticipation, but it'd been working fine for literally months as is and I expected a more gradual growth, not 400% or so in three days (with th "everyone asks their first question when they sign on" problem). Besides, there was no way I could have known what needed to be tuned until I saw what was going on. I could have made it better, but not avoided it all. Or even most. Fortunately, I've been doing Unix long enough that I could tell what to do just from watching the system twitch, but one wonders how mer mortals deal with this stuff (more likely, mere mortals don't get INTO this kind of stuff, or DON'T figure it out and it dies on them). Larry Niven did a set of stories many years ago about the problems with this kind of flash point, only he did it with people and instantaneous teleportation. On the Internet, you don't move the body around, but a person's email (ghost?) does the same kind of thing. Damn, I'll have to go find those stories again. I never thought I'd live one. And I'll have to drop him a letter and let him know about this. He'll likely bust a major gut over it. What's even MORE interesting is that as I watched what was going on, all sorts of information started popping out of the flow. I could tell when each time zone showed up for work; who reads mail/usenet first thing in th morning (most of us); who reads at lunch; who reads last thing before packing it in and going home; who reads from home in the evening. You could see waves of postings, subscription requests and other administrative dat flowing as different regions, countries, continenents all woke up, stompe on the server, and went back to bed. The statistical anthropologist would have a field day pulling all this data together and building demographs just from a sociological standpoint. I'm amused that the first thing many folks do is pop up their email and read things like TidBits, even after a long holiday or perhaps a Christmas break. I now better understand what goes on at places like Info-Mac. And, for tha matter, services like Netcom who are dealing with the front-end of all thi growth. It's scary, too. Wait until the net doubles again. Wait until the waves become data tsunamis. Unwary servers will simply implode. Mine came real close, because I hadn't properly installed a couple of load-levellers. I hadn't needed to. And if I'm REALLY lucky, I'll be completely caught up by the end of the weekend, and then life will be fine again. But I doubt it. I think it'll be Monday or Tuesday before this big wad of data completely gets out to its new owners. There's a story in there somewhere..... Chuq Von Rospach