My Latest Hardware Adventure - Iglulik
Iglulik is an Inuit word meaning "place of homes". It seemed an appropriate name for this new server I was putting together because it would be home to a number of virtual machines that would perform functions currently being serviced by one 32-bit dual Xeon machine and a bunch of UltraSparc Sun machines.
I opted not to go with a CPU and motherboard that required registered ECC memory purely from an expense perspective. I decided on Asus based on the fact that I have never had an Asus motherboard fail. Even in a box where the fans went south and the motherboard got so hot that electrolytic capacitors vented, the motherboard continued to function.
My choice of CPU for this server was an Intel i7-2600. I went with the i7-2600 and not the i7-2600K because I wanted full hardware virtualization that the i7-2600 offered. I was not going to overclock a server CPU anyway and that's really the main advantage of the i7-2600K. The i7-2600K has an unlocked multiplier but lacks some of the I/O virtualization instructions that the i7-2600 has.
The motherboard I chose was an Asus P8Z68 Pro/Gen3. My choice of motherboards was dictated by the need to support a large amount of RAM, an i7-2600 CPU, an ATX form factor, a number of SATA-3 ports, built in Ethernet, and built-in video because this was a server not a machine I'd be gaming on so high-end video was not required.
Before ordering this motherboard, I read quite a few reviews to see how well it was working under Linux for other people. Some people had adverse experiences, problems with it cycling several times while booting, problems with one of the SATA ports, etc. None of these things seemed as if they'd be show stoppers for my application so I went ahead and ordered it.
Then there was the problem with memory. I wanted to put 32GB on this board, there were four DIMM slots. That meant I'd have to use 8GB DIMMS. The approved vendors listed by Asus didn't sell 8GB DIMMS. So I took a chance and ordered a 32GB memory kit from Komputer Bay, not an approved vendor, which consisted of 4 8GB PC 1600 DDR3 DIMMS.
I ran memtest86 for several hours, not a glitch. Even ran it overclocked for a while running both the CPU and the memory at above it's rated clock speed, with no errors. I put the clock back to spec before installing the machine as a server but just wanted to test it past it's rated speed to stress any marginal components before it was placed into service.
The Komputer Bay memory has peformed flawlessly, not a glitch. Asthetically, it seemed a shame that this machine was going to sit hidden away in a server cabinet. The Asus motherboard had these rather unique blue heat sinks on various components. The Komputer Bay memory had similar metallic blue heat spreaders that went beautifully with the motherboard. Then to top it off we put a number of Thermaltake fans which are clear with bright blue LEDs in the case. At any rate, if anyone has a need for 32GB of memory, the Komputer Bay memory is a solid performer in this motherboard.
The one problem I had with this machine is physical rather than electrical. The way the CPU cooler is held down is with these four plastic locking screws. The problem with this arrangement is that they come lose with just a little vibration. They came lose in the car ride to the co-location facility and the machine powered itself down about 30 seconds after I powered it up. But in the absence of vibration, it works fine. The machine is in service now and runs very cool:
temp1: +82.0°F (crit = +210.2°F) temp2: +85.6°F (crit = +210.2°F)
I used a Western Digital Caviar Black 2TB 7200 RPM drive for the main storage. For the mail spool, which has heavy I/O requirements because our customers tend to keep very large mailboxes, I went with a 1TB 10,000 RPM Western Digital Velociraptor drive. I put the mail spool on the first half of the drive where it would have the highest I/O rate and used the second half for various miscellaneous functions that aren't frequently accessed, especially during the day.
I've been very happy with the overall performance of this box. A full Linux install takes about 12 minutes. SElinux takes about a minute for a relable. The i7-2600 is a quad-core CPU with hyperthreading. The Linux kernel sees each thread as a CPU, so it sees eight CPU's. Compiling is a whole different experience with this box. Adding a -j8 to GNU make's arguments so it uses the CPU fully and it just flies. Makes porting old C code where you've got to fix this, compile, fix that, compile, a much less burdonsome task than it used to be.
At present, I have six virtual machines running as guests using Qemu/KVM. Getting the network working was a bit tricky, mostly due to lack of adequate documentation, but otherwise the machine has been extremely stable even in the face of much poking and proding on my part.
Other Hardware Linux Is Running On
Other than this latest addition, I have CentOS 6.2 Linux running on a SuperMicro dual Xeon box, a bunch of Sun Ultra-2's running Aurora 2.99 and Redhat 6.1, a couple of Ultra-1's running Redhat 6.1, a dual-CPU (ross 125Mhz Hypersparc modules) SS-10 running Redhat 4.1, and a Sun LX running Redhat 4.1.
The dual Xeon has been stable as an NFS file server but for some reason tends to crash when I exit a vnc session. I've gone in and out of vnc all day on the new i7-2600 box, with the exact same version of vnc and CentOS except 64-bit rather than 32-bit versions, and it's stable, so why exiting vnc on the 32-bit machine crashes it sometimes (and not everytime) is one of life's great mysteries.
Aurora 2.99 on Sparc-2 is mostly stable, however, certain versions of eggdrop will panic the kernel. Eggdrop is an IRC bot and not particularly resource intensive so why this causes problems is a mystery. It also is only certain versions that are problematic.
The UltraSparc hardware has good I/O capability in general but the Ethernet is a piece of garbage and tends to drop packets. This causes problems with the old versions of NFS present in the distributions of Linux I've been using on these machines. As the web has migrated from static content to more interactive content, more CPU capability was required than what these old machines provide.
I hope this is helpful to someone who might be considering putting a Linux server together based on this hardware. I haven't had any of the issues with the multiple cycle reboots some have described so either there is an issue with older versions or there is some form of operator error.