Jason Buscema

Apple xServe RAID

One thing is consistent when working in the advertising world… the need for storage, then for even more storage. Once you’ve got the storage situation under control, double it… then maybe you’ll be good for a few more months.

Alright, so I’m exaggerating a little. The thing is no matter how much storage I put in place I always seem to need more. It’s a sick disease. The project files we work on seem to be getting exponentially larger over a short amount of time. I remember when an InDesign package for a larger presentation might top off around 20-30MBs. Then I remember when it soon eclipsed 100MBs. Now I see InDesign package files anywhere from 500MBs to over 5GBs. Then there’s the verbal requirement to “backup everything” which means 10TB of additional storage really means around 25TBs including backup space.

Apple xServe RAIDs and Promise VTrak in rack

I have five very pretty looking Apple xServe RAIDs in the server room. They first debuted in February of 2003 and were discontinued by Apple in February of 2008. Many of the drives in these xServe RAIDs were original 250GB or 500GB drives which have been spinning between 3-5 years. Can you say “end of life”? I had to decide what path I was going to take to meet our ever-growing storage needs. I could ditch the outdated xServe RAIDs and go with all new Promise Technology RAIDs or I could look at refurbishing the xServe RAIDs and breath new life into them.

The xServe RAID uses PATA drives and not the newer SATA drives. PATA drives are much harder to come by in the larger sizes and unfortunately much more expensive on a per GB level. Originally these raid units were around $12,000 for 7TBs (14x 500GB). Eventually the 750GB drives were available and for around the same price point this gave you 10.5TB (14x 750GB).

xServe RAID drive module

A new Promise VTrak E-Class 16x SATA raid is $14,999 MSRP and provides 16TBs of raw data. The Apple xServe RAID is 2Gbps fibre while the Promise is 4Gbps. The xServe RAID can support hardware RAID 0,1,5 and JBOD while the Promise can support RAID 0,1,5,6 and more. xServe RAID is limited to PATA while the Promise is SATA or SAS. Clearly, the Promise VTrak system offers quite a bit more than the xServe RAID which is probably why Apple decided to stop competing in this market.

So once again I turned to eBay where I’ve had much success in the past for work related purchases. I found a vendor selling brand new Seagate 750GB PATA drives for around $150/each. Over the course of this past year I purchase a total of 75 drives from this vendor at a total cost of $11, 250. This gave me enough drives to fully refurbish all five xServe RAIDs with five drives set aside as spares. That’s a total cost of $2100 per RAID giving me 10.5TB of raw storage per RAID.

Upgrading the drives is dead simple for anyone with average computer hardware experience. Here’s the basics:

1. Remove any currently configured arrays from the RAID controller via the GUI tool
2. Pull out the seven drives associated with the above now defunct array
3. Pop the end cap off of the each tray with your fingers
4. Unscrew the four screws under the tray holding the drive in place
5. Remove the IDE cable and the power cable
6. Insert the new 750GB drive, reconnect cables, tighten the screws and replace the end cap
7. Make sure the firmware on the xServe RAID is at 1.5.1 or the RAID will not recognize any drive over 500GB
8. Insert all the drives back into the RAID and make sure the blue light shows up on each
9. Setup a new array to your specifications
10. Make sure to enable background initialization if you plan to use the array immediately

Compaq DPS-450CB-1 power supply

I also bought a spare xServe RAID chassis which contains two power supplies, two controllers and a mid-plan logic board. In addition to the spare chassis I also have two more spare controllers and two more spare power supplies, all purchased from eBay over the past year. Some Compaq systems use the exact same power supply but have a different latching mechanism on them. If you are replacing a dead xServe RAID power supply or wish to have a spare, you can buy the Compaq DPS-450CB-1 and just swap out the latching mechanism. These power supplies run around $30 on eBay vs the Apple ones for over $100.

Each newly refurbished xServe RAID is configured into two RAID5 arrays each with 6 drives and a hot spare. This yields 3.41TB of active storage per RAID controller for a total of 6.82TB of available RAID5 storage per RAID.

I purchased a new Promise VTrak E-Class 16x SATA raid which has become our main production storage RAID. Four of the Apple xServe RAIDs are dedicated to our backup system which is running Atempo Time Navigator and the fifth houses low traffic shares. With the four xServe RAIDs connected to the backup server I now have 27.28TBs of available storage!

The bottom-line is simply that I needed lots more storage but didn’t have the budget to purchase all new RAID units. The 4Gbps fibre channel speed increase that the Promise offers over the xServe RAID would never be utilized in the backup environment where the storage was most needed. The biggest downside to going this direction is that the xServe RAIDs are no longer covered under warranty where-as a new Promise RAID would have full coverage. As long as eBay is around, I’m not concerned in the least bit not having my equipment under warranty.

Refurbishing older equipment gives you these benefits:

• Higher ROI
• Lower TCO
• Keeps the boss and finance folks happy
• Mad eBay skills
• It’s green and helps keep equipment out of landfills

Cheap Apple xServe RAIDs can still be found on eBay as many companies abandon old equipment. Do some digging and see how much it would cost to refurbish one of these RAIDs and give it a new life!

I’ve done more Cisco configuration in the past month then I have done in my entire 10 year stint at Lambesis. I’ve found the Cisco IOS can be very daunting at times (all the time) but once you’ve got a handle on it, it’s incredibly powerful and robust. One project that I’ve had on my plate for quite some time now has been a firewall upgrade for both the main office (HQ) and our datacenter. This was not a simple project as I couldn’t screw up any of the VPN tunnels currently in place between the locations. Our existing VPN setup used the Cisco Concentrator 3000 Series. This is a great VPN device but it’s just that and no firewall. I was looking at the Cisco ASA 5510 or better however the price tag is insanely high. It is a great device but not worth the money over what you can get with a used PIX. Just like all technology there is a premium for the latest and greatest.

I turned to eBay to purchase all of the gear. I went with a PIX 515E Unrestricted Failover Pair w/ VAC+, 512MB ram at the datacenter, a single PIX 515E w/ VAC+, 512MB RAM at our headquarters, and the PIX 506e for all remote locations. The PIX 515E VAC+ can handle 190Mbps cleartext throughput and 135Mbps IPSec VPN throughput. Our datacenter is on a 100Mbps pipe and our HQ is currently on a 6Mbps pipe with plans to push it over 40Mbps within the next 6 months. I’ll eventually add a Failover PIX at the headquarters office to be on the safe side. Just for comparison’s sake, a comparable new ASA 5510 runs around $3,500/each while the above PIX configuration ran around $600/each. I was able to purchase three PIX 515E units and three 506e units for around $1000 less than the price of a single new ASA 5510 device. Note: there is no warranty or Cisco support on these devices.

First off, I am not a Cisco certified engineer or anything close to it. I learn by trial and error and luckily I have the opportunity most of the time to do just that. Our existing system was not cutting it and I’m too embarrassed to even tell you what it was. I will tell you that our Cisco 3620 router was doing NAT for us and had some aggressive ACLs. I was asking the router to do way too much and anytime I would fire up an rsync session over the VPN, the router would just crap out.

With a little help from a good friend at Nextlevel Internet, I was able to get the PIX 515e setup at our headquarters and start testing the setup. Keep in mind I have remote locations and users connected over the VPN back to our headquarters. If I changed the gateway to the newly installed PIX, it would cripple those remote locations as they would not be able to see the DNS servers for starters.

Here is what I did:

1. Configure PIX 515e at headquarters and setup an unused private IP for testing
2. Setup a couple test machines on our local network and use the PIX 515e for the gateway
3. Once happy with results, I had to work on the datacenter setup because of the VPN tunnel between the two locations
4. Replaced the 506e firewall at the datacenter with the PIX 515e FO pair
5. Establish a PTP tunnel back to the PIX 515e at the HQ
6. Configure remote VPN capabilities on PIX 515e at HQ for software VPN clients
7. Add a Guest network at HQ with access only to the public interface
8. Remove the private LAN, NAT and the ACL from the Cisco 3620 router at HQ
9. Change the IP of the PIX 515e at HQ to be the gateway IP
10. Go home (ok, I was already there actually)

There are definitely some differences between the PIX IOS and the Cisco Concentrator 3000 configuration which gave me some trouble. On the Concentrator my VPN client IP pool was the same subnet as my private LAN which worked fine for many years. On the PIX, I had to create a new subnet for my VPN clients and ensure my ACLs are setup correctly for proper routing and access. In addition I discovered I need to have separate ACLs for each PTP or remote VPN configuration which was different then how the Concentrator 3000 was configured.

One thing that bite me the hardest was arp. After I moved a server from the 506E to the 515E at the datacenter it no longer worked. Turns out that I needed to have my ISP for the datacenter run a “clear arp” on the uplink router. Once I figured that out I was able to coordinate all my upgrades with them for zero downtime.

After a lot of trial and error, I am happy to report a very successful stable firewall/VPN upgrade. My Cisco 3620 router is happy once again just routing.

What I still have to do:

1. Enable hairpin routing to work so all VPN locations and traverse one another. This will allow an authorized user to VPN into HQ and get into the Datacenter, for example.
2. Setup AAA against our Mac OS X 10.5 Leopard server (Open Directory) for the remote VPN clients (software)
3. Setup proper QoS for voice traffic across all devices
4. Add a PIX 515 FO unit to HQ and configure accordingly.