Time to time I help my friends with system configuration and other “wtf? it worked yesterday!” kind of issues. One of them runs a thematic torrent tracker that has a certain popularity and medium-sized community around it. The tracker itself is based on a heavily modified TBDev code with some performance fixes and functional additions, and it was forked so long ago that I’m not even sure how closely it relates to the official TBDev branch now. And these guys have about 30,000 registered users stacking up to 100,000 simultaneous upload/download activities.
But that was all quite irrelevant, just to give a broad picture. A couple of monthes ago this tracker was upgraded to the second server – it was running on a single server before that has started to choke. And after the upgrade everything was running fine and fast with a plenty of extra resources to spare until yesterday, when this guy came to me with the words: “Damn, man, looks like we already need yet another server, everything is hell slow and just dying here”.
Of course, my first reaction was “WTF?”. They were running happily last week, and suddenly – overloaded. Hell no, this stuff just never happens.