Tag Archives: freebsd

The firmware bug that made me crazy

Leave a reply

My file server has been a pretty frequent topic on this blog and it probably won’t be a surprise when I’m bringing it up again. This time there’s been an issue with firmware, although I didn’t realize it until a just a few days ago. I’ve always had some minor issues with my file server, most prominently in performance. It used to stall now and then, but I read and read and ZFS seemed to have some problems with “bursty” writes etc. so  I didn’t pay that much attention to it. Not until about a week ago. I had to replace one of the disks in the RAID, I had a WD Green of the same model as the others laying as a cold spare so I used it and started resilvering. I got about 2-5 MB/s in resilvering speed, something was definitely wrong. But what as it?

My first though was “this is probably an alignment problem, I knew the disks were 4k (Advanced Format) drives with 4k sectors reporting 512k sectors to the OS. This can cause a bit of a problem, so I decided to backup all my important data from the RAIDZ store and align the disks, like this, using the gnop utility in FreeBSD. However this didn’t make a difference. The RAID was still AWFULLY slow.

I started reading blog after blog, but couldn’t figure out what the problem was. Then it hit me, I’d heard about the problem with WD Green disks a couple of years ago and some of my disks were from around that time. I decided to try them out one by one and check if there was a difference in performance between the separate disks. My two oldest 1,5 TB disks seemed to be the problem. Now and then they stalled for about 5-20 seconds, no data could be read or written. This seemed to indicate I had disks with the Intellipark bug, extremely aggressive power savings that made the disks pretty much useless. I tried running a configuration tool from WD that would sort out the problem but with no luck. It seemed the RAID was kind of OK with one of the disks with buggy firmware but when I used two of them it just went bananas and stopped working at all, pretty much.

What to do? I replaced the drives, plain and simple. Two fresh WD Green 2 TB disks. OK, the other disks are 1,5 TB so I’m ending up loosing 1 TB but the RAID works! What about the performance? I used to get stalls and about really bad performance with bonnie++ (25 MB/s write and 100 MB/s read from the RAID in average).

Screen shot from a round of bonnie++

Screen shot from a round of bonnie++

If you have a look at the screen shot above I now get 179 MB/s write and 237 MB/s read with bonnie++ on my RAIDZ2. By no means extremely fast, but it’s not bad either.

Conclusion

If you have disks from the WD Green series performing strangely, make sure it isn’t bad firmware that’s the problem. In my case I had problems with WD15EARS-00Z5B1 using firmware 80.00A80. It seems that’s the firmware causing the most problems.

Adding DTrace support to FreeBSD

Leave a reply

I’ve been interested in learning about DTrace for quite a while now, to start with I thought I’d get support for it on my file server running FreeBSD. The generic FreeBSD kernel that comes with the standard installation doesn’t have DTrace support enabled. Time to get that sorted.

1. To start with, make sure your system has the kernel source installed under /usr/src/sys.

2. Then, because I run a 64-bit system, I’m changing to /usr/src/sys/amd64/conf and copy GENERIC to a new conf file called DTRACE. At the bottom of the DTRACE file I add the following.


# Dtrace support
options KDTRACE_HOOKS # all architectures - enable general DTrace hooks
options DDB_CTF # all architectures - kernel ELF linker loads CTF data
options KDTRACE_FRAME # amd64 - ensure frames are compiled in
makeoptions DEBUG="-g" # amd64? - build kernel with gdb(1) debug symbols
makeoptions WITH_CTF=1

This adds the DTrace support to the new kernel I’m compiling. I also removed some crap (like WiFi support) since I’ll never need that in a file server. Now, time to recompile the kernel.

  1. Run config /usr/src/sys/amd64/conf/DTRACE
  2. Go to /usr/src/sys/amd64/compile/DTRACE and run make cleandepend; make depend; make; make install; reboot

There we go, last run kldload dtraceall and dtrace -l hopefully gives you a list of all the DTrace probes avaliable.

Loads o' probes

Loads o' probes

NFS or iSCSI for use as backend storage on ESXi?

Leave a reply

VMware, back in the days of early 2001, when there was only VMware ESX Server was built with ONE storage system in mind. A NetApp using NFS. They added support for FibreChannel, iSCSI, etc. but NFS was the main thing for VMware. In many cases it still is! For an average home enthusiast like myself FibreChannel isn’t an option because of price, that means it’s either NFS or iSCSI that’ll be used for storage. Unless you use internal disks in the server itself, but why would you want to do that? Your file server is there for a reason! What are the pros and cons of NFS vs. iSCSI? Here are my thoughts on the matter.

NFS

+

VMware was built for this.

It’s fast when it comes to reading compared to iSCSI, at least in my experience.

The virtual machines are stored as files on the file server and not in an image using VMFS.

-

NFS does sync writes, this tends to be slow. Can be solved by using ZIL, ZFS Intent Log if using ZFS.

Security isn’t the best unless using NFSv4 with authentication, use on a dedicated storage network instead.

 

iSCSI

+

Easier to share between more than one host if using multiple ESXi server.

Can be configured with good security.

-

Slow. At least that’s what my experience tells me.

Not that flexible since the VMs are stored in an image using VMFS, the VMware File System.

 

What to choose?

For me it was an easy choice, I went for NFS. I have used iSCSI before with ESXi and I wasn’t content. It was slow and more or less a pain to backup the machines. With NFS I can just power off the machine and copy the whole thing to another directory on the file server (or a take a snapshot with ZFS). I might add I did use iSCSI under Solaris with the COMSTAR iSCSI target, haven’t tried it under FreeBSD which I’m currently using.

Which ever one you go for I recommend using a dedicated network for storage. This is mostly for security reasons, you don’t want anyone to mess with the storage traffic. Storage also tends to use quite a bit of bandwidth, you probably don’t want this on your LAN. Since I only have two machines, the file server and the ESXi server, I use a single cable between them. No need for a network switch.

Normal load on my ZFS storage system

Normal load on my ZFS storage system

As you can see above I tend to get a lot of reads, I’d say 99% of my storage I/O are read requests.

Upgrading FreeBSD from 9.0-RC3 to 9.0-RELEASE

Leave a reply

Since I’ve been running FreeBSD 9.0-RC3 for a couple of weeks it was just a matter of time before 9.0-RELEASE was coming out and 12th of January it did. It was time to upgrade from release candidate to the release version. This is a pretty painless procedure, but I’ll still give you a quick walk through.

There’s a guide to follow at http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/updating-upgrading-freebsdupdate.html, but I’ll give a quick description.

I’ll be using a command called freebsd-update and first I had to tell FreeBSD I wanted to upgrade it to 9.0-RELEASE.

freebsd-update -r 9.0-RELEASE upgrade

You’ll get a question if the components to be upgraded looks reasonable, answer y. (If you’re using a custom kernel, FreeBSD will switch to a GENERIC one under the upgrade process.)

Once it’s done, type the following command to begin installation of the upgrades.

freebsd-update install

If the system was running with a custom built kernel, use the following command.
nextboot -k GENERIC

Otherwise, just reboot the computer to use the new kernel.

Run freebsd-update again to continue updating the rest of the system.

freebsd-update install

When this is complete, use port-upgrade to upgrade all packages installed via ports. port-upgrade is found under /usr/ports/ports-mgmt/portupgrade.
portupgrade -f ruby
rm /var/db/pkg/pkgdb.db
portupgrade -f ruby18-bdb
rm /var/db/pkg/pkgdb.db /usr/ports/INDEX-*.db
portupgrade -af

This will take a while to compelte, depending on how much software you installed via ports.

When done, run freebsd-update once more.

freebsd-update install

You’re done! If you need to build a new version of your custom kernel, you can do it now.

My fileserver Titan is upgraded to 9.0-RELEASE

My fileserver Titan is upgraded to 9.0-RELEASE

FreeBSD 9.0 released

Leave a reply

FreeBSD 9.0 is out! It’s been a long wait to get the latest open ZFS version, version 28, in FreeBSD and noew it’s finally here. The SATA drivers now support AHCI, more good news for us storage nerds plus NFS support for NFSv4. You can read the offical announcement here.

Time to upgrade from 9.0-RC3 to 9.0-RELEASE, I know what I’ll be up to tonight for sure. :)

Benchmarking the new file server

Leave a reply

Just for the fun of it I had to benchmark the RAID on my new file server. Or well… newly reinstalled file server.

Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
titan 16G 88 99 54883 17 112916 38 243 99 306316 40 366.9 12
Latency 166ms 92611ms 1555ms 45494us 412ms 770ms
Version 1.96 ------Sequential Create------ --------Random Create--------
titan -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 16698 99 +++++ +++ 16332 99 16003 98 +++++ +++ 16690 99

This is the output from bonnie++. Sequential write and read is at 112 MB/s and 306 MB/s. No complaints here, I’m more than pleased with the performance. I haven’t done any ZFS tuning or something like that, it’s all just standard settings.

Re-adding L2ARC and ZIL to the ZFS storage pool

Leave a reply

As I wrote in an earlier post I’ve had a file server crash. After some investigating it turned out that one of my SATA controller cards was the problem. I’ve now bought a new card and installed it.

Prepared with tea!

Prepared with tea!

The SSD disks are once again installed in my fileserver

The SSD disks are once again installed in my fileserver

The new SATA controller

The new SATA controller

I decided to go for a card using the Marvel 88SX7042 chipset since it’s well supported in FreeBSD. I had to add a line to /boot/loader.conf though:

hw.hptrr.attach_generic=0

Without this FreeBSD uses the driver for a RocketRAID card from HighPoint (which is not what I have). This line tells the FreeBSD kernel not to use that driver as a generic driver, i.e. for my card.

Adding L2ARC and ZIL

The next step was to attach the disks to the storage pool for usage as L2ARC and ZIL. Only two commands are needed.

zpool add storage log mirror ada1 ada2
zpool add storage cache ada0

The first line adds ada1 and ada2 as a mirrored ZIL (ZFS Intent Log) and the second line ada0 as L2ARC (Layer 2 Adaptive Replacement Cache).

I’m up and running again! Feels kind of good. :)

Fileserver crash

2 Replies

Hmm… Yesterday my fileserver crashed and I lost all my data. It was all because of a faulty SATA-controller. All my data was corrupt. I didn’t realise this until today, that the card was the problem, so I installed FreeBSD instead of Solaris last night.

I think I’ll keep FreeBSD and run ZFS on that instead. Seems to work really great and to be honest, FreeBSD is a lot easier to maintain than Solaris. The ports system makes my life a lot easier than the very few precompiled packages for Solaris.

Screen shot 2011-02-05 at 20.01.13

Speeding up your internet connection with WAN load balancing

9 Replies

To start with this either requires you to have multiple Internet connections OR, like me, an ISP that gives you 100 Mbit/s download and 10 Mbit/s upload. The thing is that my ISP gives me a 100 Mbit full duplex connection but limits my upload to 10 Mbit/s per IP. Notice, per IP address! I can get five IP addresses, this means 5 x 10 Mbit/s upload. I could have five different computers all uploading at 10 Mbit/s or 5 network cards (NICs) in one computer with load balancing. Of course I use the load balancing solution with pfSense as OS for my router.

To make things even better you can do this by using a virtual, I use VMware ESXi with a dedicated NIC connected to the Internet.

Internet connection in ESXi

Now, create five virtual NICs on that virtual machine like this.

Five virtual NICs

As you can see there are five NICs connected to Internet and one to my LAN. Continue by installing pfSense. Connect LAN and the first WAN NIC when installing pfSense. Your Internet connection should be working (but without load balancing) to go to the next step.

Go to Interfaces > Assign and add your remaining NICs as OPT1-4. When activating them, don’t forget to check “Enable optional interface” at the top of the page and check “Disable userland FTP-proxy application” too.

NICs assigned to opt1-4

All your NICs should get an IP address (hopefully). Check this under Status > Interfaces. Should look something like the picture below.

All NICs have IP addresses

The next step is to create a pool for Load balancing containing all the WAN NICs. This is done under Services > Load balancer.

Use the following settings:
Type: Gateway
Behavior: Load balancing
Monitor: DNS Server 1 (use this if your NIC are all connected to the same ISP, otherwise the gateway of each NIC)

Now, add the interfaces and set an appropriate name for the pool.

Load balancing pool

The last step is to make sure the pool is used, this is done by a firewall rule. Go to Firewall > Rules and the LAN tab. Edit the existing rule and change the gateway to your pool’s name.

Gateway changed

You’re done! This is what I get when using speedtest.net (which doesn’t show my real upload speed, I’ve been achieving 30 MBit/s at least with multiple upload connections).

Speed test

Speedtest nr 2

Now, there’s one thing left. HTTPS can be a bit cranky when it’s load balanced, so you can add a firewall rules to make sure HTTPS is always running from/to the first WAN NIC. Add a new rule under Firewall > Rules under the LAN tab. Choose destination port range and select HTTPS, gateway should be set to Default (not the LB pool). Save the rule and move it to the top of the rules under the LAN tab. I’ve added rules for both HTTPS and HTTP.

HTTP and HTTPS rules