My file server has been a pretty frequent topic on this blog and it probably won’t be a surprise when I’m bringing it up again. This time there’s been an issue with firmware, although I didn’t realize it until a just a few days ago. I’ve always had some minor issues with my file server, most prominently in performance. It used to stall now and then, but I read and read and ZFS seemed to have some problems with “bursty” writes etc. so I didn’t pay that much attention to it. Not until about a week ago. I had to replace one of the disks in the RAID, I had a WD Green of the same model as the others laying as a cold spare so I used it and started resilvering. I got about 2-5 MB/s in resilvering speed, something was definitely wrong. But what as it?
My first though was “this is probably an alignment problem, I knew the disks were 4k (Advanced Format) drives with 4k sectors reporting 512k sectors to the OS. This can cause a bit of a problem, so I decided to backup all my important data from the RAIDZ store and align the disks, like this, using the gnop utility in FreeBSD. However this didn’t make a difference. The RAID was still AWFULLY slow.
I started reading blog after blog, but couldn’t figure out what the problem was. Then it hit me, I’d heard about the problem with WD Green disks a couple of years ago and some of my disks were from around that time. I decided to try them out one by one and check if there was a difference in performance between the separate disks. My two oldest 1,5 TB disks seemed to be the problem. Now and then they stalled for about 5-20 seconds, no data could be read or written. This seemed to indicate I had disks with the Intellipark bug, extremely aggressive power savings that made the disks pretty much useless. I tried running a configuration tool from WD that would sort out the problem but with no luck. It seemed the RAID was kind of OK with one of the disks with buggy firmware but when I used two of them it just went bananas and stopped working at all, pretty much.
What to do? I replaced the drives, plain and simple. Two fresh WD Green 2 TB disks. OK, the other disks are 1,5 TB so I’m ending up loosing 1 TB but the RAID works! What about the performance? I used to get stalls and about really bad performance with bonnie++ (25 MB/s write and 100 MB/s read from the RAID in average).
If you have a look at the screen shot above I now get 179 MB/s write and 237 MB/s read with bonnie++ on my RAIDZ2. By no means extremely fast, but it’s not bad either.
If you have disks from the WD Green series performing strangely, make sure it isn’t bad firmware that’s the problem. In my case I had problems with WD15EARS-00Z5B1 using firmware 80.00A80. It seems that’s the firmware causing the most problems.