Nix Bits

This interesting article talks about the kernel.org infrastructure used to maintain the Linux kernel. Overall it’s a fascinating little bit of history. It’s also intriguing because it gives an example of running an extremely bandwidth and processor intensive site. This quote is especially interesting regarding an earlier verision of kernel.org hosted on a dual PIII.

Serving data with http and ftp is is not very CPU intensive, but over time the amount of rsync traffic being fed by the kernel.org server continued to increase, and rsync is CPU intensive. “That’s what rsync does” Peter said, “it trades bandwidth for CPU horsepower. We were getting to the point where we had all the bandwidth, but the Dual PIII 1.1’s couldn’t really keep up.” He noted that the load average kept growing, well into triple digits. Referring to 32-bit systems, Peter noted, “we learned that the Linux load average rolls over at 1024. And we actually found this out empirically.”

That’s fairly amazing. Also noteworthy is the bare number of software optimizations they’ve thrown at the problem, which basically consisted of mounting their filesystems with the noatime attribute. Have to double check that one on some of my busier http boxes.

Slashdot has an article in their FAQ, detailing their hardware and software mix as well. It probably hasn’t been updated in a while but the basic config described probably remains as detailed in the FAQ entry.