It won't happen to me...

Really. Everyone believes it won't happen to them because it hasn't happened. Well, it happened. To me. *twice* in the same week. *sigh*.

Both of my linux servers lost a hard drive; the root volume. I do keep regular backups, but what one learns when something like this happens is how much info you've been forgetting. Settings and other one-offs that you forgot to backup.

The way you noticed the failures is never how you expect. I had always imagined that I'd be doing something on the system when it failed, or that the system wouldn't boot. In this case, it started with the media frontend not finding some files. Further inspection pointed to the SMB share wasn't responding. Then, a ping to the host failed... panic building... Check the console... and I/O errors rolling across the screen. *sigh*

I grabbed some older IDE drives, two 250G drives and reinstalled Ubuntu 9.04. Clearly no one installs a RAID1 Root mirror because the process is a royal pain in the ass.

You can't simply partition one drive and then point to a second and say "mirror this one here." Rather, you create a partition, indicate that its type is Linux RAID. You must do this for each partition, and then a second time exactly the same on the second drive. Only then can you enter the RAID menu where it will populate the various RAID options (0,1 and 5). Quizically, it asks if you have any spares when creating a RAID1. This threw me off since I didn't think about actually having a hot-spare for a mirror. After creating new devices, md0, md1, mdX, etc which is composed of a partition from each of the two drives in mirror, ensuring that you match of the correct partition then you can return to the more "normal" partitioner setup; namely selecting which device is mounted where with what filesystem etc. *whew*.

Note that once you format these devices Linux makes the raid *active*. What this means in a mirror secenario is that all writes to the first go to the second. What this means practically is that you'll be doing *double* the writes during the install. Assuming you have reasonably new system and the drives are SATA, you probably won't have much trouble; however I initially did my mirror operation with two IDE drives, on the same channel. This was horrendously slow. Normally a server install takes maybe 20 minutes. The install over RAID1 took over an hour. And the drive hadn't been completely mirrored after the install. After the final boot into Ubuntu 9.04 for the first time the system was unsuable for about a day as the software RAID continued to replicate the data to the second drive.

After having survived the lost of a root drive on one of my two servers I was ready for a break. Non such luck. I discovered the second root failure in a similar manner. Files not accessible, followed by failed ping, followed by console inspection resulting in a screen full of ext3 errors. This time I didn't give up hope and booted a live CD and fscked then ran badblocks -c hoping that I could mark the bad blocks, relocate the data and move on. I met with little success. After waiting over a day while badblocks ran I finially gave up and ordered another two 250G SATA drives for the server.

Now, this second server is the home of the major part of the total storage. I have a 4-port 3Ware RAID card with four 500G drives and four 1.5TB drives, LVM'ed together, plus one root drive for a total of 9 drives. The motherboard has 7 SATA ports. A 5-port Nvidia fakeraid and a 2-port JMicron. The JMicron ports are separate since the chipset provides a SATA to IDE conversion for legacy OSes that don't support booting from SATA. The current setup occupied 4 of the 5 Nvidia ports and one of the JMicron ports. After replacing the busted root drive with two 250G SATA drives, I now had the same 4 1.5TB drives in the Nvidia controller and the 250s in the Jmicron. I proceeded to the Ubuntu installer only to find that it would only "see" one of the two root drives. Hrm, that's odd. OK, pull one of the cables for the root drive, now it "sees" that drive file; maybe one of the ports on the JMicron is bad... no, same drive can be seen in either port. Either drive can be seen in either port... So strange. I tried to use the 5th port of the Nvidia; no dice. That port didn't even seem active. Well, maybe there isn't enough power. I disconnected first the Nvidia controller. Success!

I installed 9.04 onto a RAID1 mirror and on my laptop started shopping for a larger power supply. The next day I log into the system to find I/O errors all over the place. Sometime during the night the SATA controller encountered an error and dropped on of the root drives. Frantic google searching only turned up that the SATA controller might be bad. I was still concerned that I might have a power issue so I unplug *all* drives except *one* 250G drive. I plug that single drive into the Nvidia controller and install Ubuntu. I wait another day and I'm greeted with the same SATA failure. I can't be the power supply since I'm pulling next to nothing here; just the main board and a single SATA drive. Reluctantly I place an order for a new motherboard which can use the same CPU (AMD Sempron) and RAM (4Gs DDR2).

In my haste, I screwed up and got a new board with only 2 DIMM slots instead of four; however, after swapping the board, I load the system up with all 10 drives; install Ubuntu and the system is restored with greater protection than before. Space is cheap, do yourself a favor and use disk mirroring.

I love it when a plan comes together

Long time, no post.

Well, things have become significantly more busy and hectic than before. My family has grown by one and things at work were incredibly busy for 2009. However, Christmas 2009 had everything coming together, family, extended family and technology in the home.

This past fall, my parents moved closer to where I live. They ended up moving themselves so it was a long and grueling process with multiple trips over a period of two months. The last part of the move was completed on December 23rd as the last trailer load was delivered to their retirement home. What a relief for my father as he completed his last day on his job on the 24th and drove through the night to be home for Christmas morning.

I'll briefly cover what's changed tech-wise but will save most of the details for separate posts. Over the summer, it was time to upgrade the front-end HTPC machine. Replacing the system was nice, but ultimately missed one of the key reason for the upgrade; fitting within the media cabinet.

My servers at home both suffered hard drive loss; the root partition on *both* systems. After recovering; both systems both run RAID 1 (mirror) root drives.

I delved into the realm of Boxee for frontend media playback. Verdict? It has some quirks, but overall it's very nice; I'm still using Boxee today and eagerly awaiting the transition from alpha to beta!

I've become a big fan of the tvnamer package. While the default naming format isn't to my liking, it's usable and recognizable by Boxee; one of the key features is media discovery which is based on filename.

I've been turned on to Pandora for streaming music. In general I really like the service. The free account does have a few annoyances but none are enough to make me want to look for a different streaming service.

Christmas 2009 was really good for me hardware-wise; a massive upgrade so I'll have reviews of the equipment and setup. 55" Vizio VF550M, Sony HT CT-500 Soundbar/Reciever, PS3 Slim 120G, Sanus LL-11 Wallmount, Powerbridge remote power outlet (in-wall power, cabling). Logitech Bluetooth/IR adapter for the PS3.

Hopefully you had a great 2009 and I hope you have a Happy 2010.