SanGod.Com

The painful part is I have a lot to do, but it’s mindless drivel that doesn’t stimulate the geek in me nearly enough.

Interesting thing from this week though - I found that I tend to over-engineer clariion designs.  I guess such a large part of me wishes I was still working on the Symmetrix/SRDF configs of my past that when presented with what turned out to be a simple Clariion config, I spread the raid-groups a little too efficiently, which, according to the second set of eyes I had working with me, can actually cause more problems than it fixes.

Silly me - I assumed that spreading the IO across both spindles and DAE’s was a good thing.  (It used to be, but apparently not so much now)

Someone give me a DMX to play with.

Anyone heard of Fusion-IO?  I’ve been doing some reading and it seems like if the press is reliable they may have hit that breakthrough I’ve been looking for.

Essentially it’s up to (currently) 320GB of NAND-Flash storage memory that fits into a PCI-E x4 slot.  Booting from this is supposed to come sometime in Q3, but when it does, watch out.  The numbers they are quoting:

• Over 125,000 IOPS of random sustained read/write packets
• 700MB per second of bandwidth - Sustained

make them easily one of the best performers in the game.  You can transfer 4.5GB (a single-sided DVD) in about 5 seconds.

The fact that they have a 320GB card out, and a 640G card due later this year, means they’ve got conventional SSD beat.  The only serious drawback I can see is the lack of a 2.5″ form-factor SATA interface for laptops.  But who knows - that may also be coming down the pipeline soon.  I’d replace the 80GB SATA-II drive in my Latitude D620 with one of the 80G ioDrives in a heartbeat.

Check them out at www.fusionio.com and let me know what you think.

Ok - maybe that was too much.  My only challenges this week had to do with my own personal infrastructure.  The powers that be, in their infinite wisdom, brought me in to do Symm work and then after my onboarding handed me a dozen or so Clariion / Celerra designs to do.  When I asked them about the mix-up, they simply stated that they hired a bunch of DMX-only people for projects coming down the pipeline that never closed, so now they have a glut of Symm people who don’t know anything else.

Myself being a Symm person who also happens to know the basics of Clariion / Celerra I get stuck with working with the low-end projects.

I can’t complain too much, since April 1st I’ve been on a total of 1 assignment away from home (to install ECC) and the rest I’ve been working out of my home-office (read basement data-center) and spending much more time with my kids than I think I could have hoped for.

But at heart I’m a geek.  I need to get my hands on the hardware to keep my skills sharp.  And since I don’t think EMC’s going to put a DMX in my garage (hint, hint - even a 1000 will do nicely) I’m stuck trying to figure out a way to get involved with the Symm projects one way or another.

The only Symm experience I’ve realy gotten in the last month was an SRDF/A implementation that mysteriously(?) dropped and no-one could figure out why.  Turns out they needed about 6 times the bandwidth they had allocated to SRDF and the SRDF/A session was terminating because the WP limit was being reached during a database restore.  (Doctor - it hurts when I do this…)

That was handled with a succession of phone-calls and even when I asked if they wanted me on-site during the production cut-over (they went ahead with it anyway) they didn’t respond.  Not even a ‘thanks but no thanks’ just a great big ignore.

Oh well.

Jg

Well - for the last three weeks I’ve designed more NAS/iSCSI systems than I really want to admit to.

I understand what is so alluring about network storage, it’s cheap, plentiful, and it utilizes a technology that every data-center already has in place.

I have to admit that the customer I’m currently working on has done what none other has dared do until now. iSCSI on Symmetrix.

My first question of course is…..Why? Does it make sense to spend millions on a storage array only to refuse to spend $25k on a couple of switches? The only time I could see this as being even remotely useful would be a situation where you have something along the lines of a giant linux distributed computing cluster, though for simplicity’s sake I would probably opt for NFS over iSCSI.

Customer of course asked me what I thought - and I bit my tongue so hard I was afraid he’d caught that I wasn’t sold on the idea. I gave them the politic answer, of course being that “It’s a great, inexpensive way to connect a large number of hosts to a single pool of storage.” which of course neither answered his question nor committed me to an opinion.

Truthfully - I’ve seen it work a few times now, and with a few exceptions about powerpath and different subnets, it seems fairly robust. (Powerpath for windows works out of the box, PowerPath for linux requires that each interface be in a different subnet, which complicates things)

The only thing I can suggest if you’re planning an iSCSI SAN, is that at the very minimum you want to create a separate VLAN for the iSCSI traffic or risk having issues with broadcast traffic interfering with production traffic. Ideally you’ll create a separate physical storage network and n’ere the twain shall meet.

A few have noticed the site was down for an extended period this week.  I learned a few things this week.

I set up my FC system and was so excited to get it moving that I neglected to adequately test my equipment.  I bought used equipment, with used drives, and put real data on them after a whopping 2 days of light testing.  I never stress-tested the drives, didn’t do any kind of exercizing of them to validate that they were worthy of production data.

I also neglected to functionally test the array.  While it did offer the ability to configure a hot-spare, I didn’t check to see if the hot-spare was functional before I moved data over to it.  (Seeing that it was configured was enough for me)

So what happened was this.  I was running on the system and all was well until a drive failed.  The hot-spare didn’t invoke on it’s own, and while one drive was in a failed state, the second drive failed.  Needless to say I lost half my luns and three of them were corrupted beyond repair. 

Luckily I’m one of the old hold-outs.  I have a tape backup system consisting of a Veritas 6.0 environment with an ATL tape library.  I was able to restore to within 48 hours of the failure using tape.

My *NEW* storage back-end of consists of a Dell 2650 with 5x 146G drives.  I installed CentOS5 with a 512GB NFS-mount partition and mounted them to my VMWare servers.  The most interesting part is I realized that by bonding the network interfaces I’m getting the same bandwidth I got out of the 2x 1Gig fibrechannel ports.

Not being a network guy though, does anyone have any suggestions for optimising NFS for storage applications?

IBM ‘Blindsided’ by Federal Contract Suspension

If there ever was an *OUCH* moment this is it.  IBM - who running jokes have referred to as “Inferior but Marketable” may not be marketable anymore.  It seems that they’ve been caught up in an $80 million bid-rigging scandal that has them blacklisted from bidding on new projects with the federal government.

I would say this is a good thing, but the main alternative now is EDS…..

Ok, changed venues again.  Going to do some work for some smaller customers.  Down side is I won’t be involved in the huge datacenter migrations I’ve done in the past, up side is that I’ll get more hands-on stick time and be able to keep the skills sharp.  So far it’s a *LOT* of NAS work, which is fine by me.

One thing I noticed is that the big data centers, you know, the big Telco sites that seem to run amok in the mid-Atlantic and north-east have an inexplicable “hands-off” approach to systems management.

I was in one site recently when I found that not even the local admins had access to half the systems in the data center. They had outsourced almost ALL of their SA work to India. Even the storage manager, who had WAY above average skills in the Systems Administration arena, was not allowed to log into a box even to check the status of PowerPath.

Now there is a right-way and a wrong-way to do things. Forcing your storage guy to sit on his hands while someone half-a-world away troubleshoots a storage problem is insane. Trust your employees to do the job you pay them to do, and that includes consultants.

Nothing is worse than being in a ‘hands-off’ data center. You can quadruple the amount of time it’s going to take to get something done. The best consulting jobs are the ones where they turn the keyboard over to you and walk away, expecting it to be completed when they get back from their coffee break. Because nine times out of ten, without someone triple-checking everything you do, you can get it done in no-time flat and have time to catch a movie afterwards.

What’s worse is forcing someone to wait. It’s bad enough when an employee has to wait days for something to get done, but when a consultant you’re paying upwards of $200/hr to have on-site, don’t make him wait two days for outputs he needs to do his job - Let’s see, using the above as an example, $200/hr x 8 hrs x 2 days = $3,200….. As a stockholder in some of these companies, I object. I don’t like wasting other people’s money any more than I like wasting my own.

(By the way - $200 / hr falls into the category of guesses - I go out of my way not to know exactly what I’m costing the customer in the end.)

/SG

This was written a while ago but never got posted for some reason - I want to say at least two months or so have passed and the solution below has been allmost bulletproof.

About once a month I have to reboot it.  So much for Linux being the anti-Microsoft.

————————————-

Just a two minute post while I sit in the Atlanta airport waiting for a connecting flight that might never come.

To the folks at Fry’s Electronics: You are evil. That’s the only explanation for putting a Fry’s on the road going to DFW (Dallas, TX) where geeks like me will get sucked into the vortex.

I bought a portable NAS box, (CIFS/NFS, nothing too glamorous) there for $209. Add two inexpensive 500G SATA drives in a Raid-1 config I now have a half-terabyte of static storage.

It’s a linux based system, very solidly put together, (IE Steel case, no rattles) and it performs very well, utilizing most of the Gig-E interface on the back of it.

In case anyone is interested, they have a few larger boxes as well, including support for iSCSI.

 www.sansdigital.com

A couple of years ago, I picked up an old Clariiion FC5300 wholesale (free) from a junk-pile at one of my customers.  I played with it, it was nice, but I couldn’t figure out why I should use it when I had 73+G drives available to me.

I started the FC@Home project then.  Because I thought it would be cool to have fibrechannel running in my home system.

Well I got rid of the FC5300 because the 30 x 18G Full-Height drives were just too much to power and cool.

A few weeks ago I decided that I needed to do it again.  (I posted something of it earlier)  Got an old EMC/Brocade DS16B2 switch, a PowerVault 224F JBOD, and started playing.

Well the first thing I found is that I could never use JBOD for the purposes I wanted to.  I wanted to put together some redundant shared storage for my VMWare servers so I could play with VMotion and Clustering.  While I could share individual disks, RAID wasn’t an option and I refuse to use unprotected storage.

So I scoured Ebay and found a PowerVault 660F to add to the 224F.  Now the 224F came with 14x 18G drives, the 660F came with 14×36G drives.  I paid under $200 (not including shipping) for each of the two racks.  They are 3U units, don’t pull a tremendous amount of power, and are as difficult to cool as any drive array (it’s the drives that cause the heat, not the array)

Another $100 or so in cables (The HSSDC->HSSDC jumpers that were required for between the units) and I was good to go.  I already had some DB-9 FC –> SC-Duplex converts, as well as some SC–>LC cables, so that part was easy.  I found someone who off-loaded a bunch of old Qlogic QLA2200 HBA’s (9 for $50) and the whole things was done.

I initially had issues getting it recognized, but on a whim I called Dell support.  The tech informed me that this was so far out of support he really couldn’t help me, then proceeded to spend about an hour helping me out.  Turns out that the Array Manager software that you use to manage the thing doesn’t work with the latest / greatest QLogic drivers.  I had to back-rev them to v8.x and suddenly it worked perfectly.  (He also told me it was never ever going to work on Win2k3 - a fact I’ve happily disproven.)

I just got it carved, and all but one of my VM’s are moved over to it.  I have about 500G of Raid-5 Storage available with 2 Hot-Spares (Since I don’t know the history of the drives, I figured better safe than sorry).

So far so good.  Performance is great, though I’m only going through one switch I have redundant RAID controllers, so that’s at least something.  As soon as I find someone dumping a second DS16B2 I’ll probably incorporate that into the mix as well.

So I set up the 2-node VMWare cluster, and set it for DRS just to see if it works the way they say it will.  (I’m also curious because I have less memory in the second node than the first, if it will be aware of that.)  I have a third 2650 I got here because some newbie on Ebay didn’t realize that the particular error message he got on boot meant simply that there was no operating system on the disks.  As soon as I get the rail-kit I’m going to mount this puppy up and make it a 3-node cluster.

I’m such a geek.

Brocade bought SBS.

I don’t know how many of you happen to have looked at the resume I had posted - but I spent a couple of years at Strategic Business Systems (www.sbsplanet.com).

I’m not sure what Brocade is hoping to get out of this.  SBS doesn’t do sales, and doesn’t even really have any influence in the buying process.

SBS has been a pretty successful company - grown by leaps and bounds.  I would never go back to them because they wield their non-compete agreement like a battle-axe and use every opportunity as a chance to hook someone in.

The real problem is that Brocade as a switch manufacturer is on it’s way out.  From a 90% install base they really have nowhere to go but down, and Cisco is gaining very quickly.

I’m not a big fan of Brocade.  I have a brocade switch in my home SAN not because of any preference, but because they are cheap on Ebay.   Their ASIC’s are slow and their licensing is oppressive.

Does this make sense to anyone?