Tag Archives: nfs

Virtual machine storage performance, NFS vs. iSCSI

I’m running VMware ESXi to virtualize my computers and FreeBSD 9.0 with ZFS as backend storage. I’ve done some tests a couple of years ago telling me to always choose NFS over iSCSI for virtual machine storage. Now, is this the best solution for performance? I had to test this, totally unscientific but still worth reading I hope.

I did two different tests, one with bonnie++ and one with dd. Each test was performed inside a virtual machine (Ubuntu 12.04 LTS 64 bit) running off iSCSI storage and one stored on NFS. They were both stored on the same storage pool. At last the test was performed directly on the FreeBSD backend server to see the potential maximum. This isn’t true though as I was limited by my 1 Gb/s Ethernet connecting the ESXi machine and the FreeBSD server.

I just ran bonnie++ with standard settings, no tweaking at all. Here are the results.

Bonnie++ results

Bonnie++ results

The second test was done with dd. I used dd if=/dev/zero of=./test bs=`expr 1024 \* 1024` count=4096 to create a 4 GB file just measuring the write performance.

dd results

dd results

Conclusion

As you can see there was a slight advantage for NFS, although not a huge one. In my opinion, the fact that the virtual machines are stored as files on the FreeBSD servers filesystem in NFS, is a giant reason for NFS as the first choice. You can take backup of your machines with ease and won’t have to use VMFS at all.

NFS or iSCSI for use as backend storage on ESXi?

VMware, back in the days of early 2001, when there was only VMware ESX Server was built with ONE storage system in mind. A NetApp using NFS. They added support for FibreChannel, iSCSI, etc. but NFS was the main thing for VMware. In many cases it still is! For an average home enthusiast like myself FibreChannel isn’t an option because of price, that means it’s either NFS or iSCSI that’ll be used for storage. Unless you use internal disks in the server itself, but why would you want to do that? Your file server is there for a reason! What are the pros and cons of NFS vs. iSCSI? Here are my thoughts on the matter.

NFS

+

VMware was built for this.

It’s fast when it comes to reading compared to iSCSI, at least in my experience.

The virtual machines are stored as files on the file server and not in an image using VMFS.

-

NFS does sync writes, this tends to be slow. Can be solved by using ZIL, ZFS Intent Log if using ZFS.

Security isn’t the best unless using NFSv4 with authentication, use on a dedicated storage network instead.

 

iSCSI

+

Easier to share between more than one host if using multiple ESXi server.

Can be configured with good security.

-

Slow. At least that’s what my experience tells me.

Not that flexible since the VMs are stored in an image using VMFS, the VMware File System.

 

What to choose?

For me it was an easy choice, I went for NFS. I have used iSCSI before with ESXi and I wasn’t content. It was slow and more or less a pain to backup the machines. With NFS I can just power off the machine and copy the whole thing to another directory on the file server (or a take a snapshot with ZFS). I might add I did use iSCSI under Solaris with the COMSTAR iSCSI target, haven’t tried it under FreeBSD which I’m currently using.

Which ever one you go for I recommend using a dedicated network for storage. This is mostly for security reasons, you don’t want anyone to mess with the storage traffic. Storage also tends to use quite a bit of bandwidth, you probably don’t want this on your LAN. Since I only have two machines, the file server and the ESXi server, I use a single cable between them. No need for a network switch.

Normal load on my ZFS storage system

Normal load on my ZFS storage system

As you can see above I tend to get a lot of reads, I’d say 99% of my storage I/O are read requests.

Swap in VMware ESXi, do or don’t?

I’ve never really thought about the swap when running virtual machines, I’ve pretty much just gone with the standard settings during the installation. I’ve never, but I should have, asked myself “do I need swap on this machine?”. My web server (the one you’re on right now actually, reading this) had 512 MB of RAM and a couple of GB swap. The virtual machine just got slower and slower and the RAID on my file server was crazy busy, but I couldn’t figure out why. Then it struck me, swapping! Since I run ESXi over NFS this is also where the swap is read and written. By no means fast, especially when it comes to latency. TCP/IP and NFS plus a ZFS RAID system, there was a huge bottle neck here!

How to solve it

Easy, more RAM. After that? Even more RAM and disable the swap, or at least don’t store it on an NFS volume on slow, spinning disks. OK, if there’s only one machine swapping and nothing else reading or writing to the RAID it’ll be fine, but when you have loads of virtual machine swapping and a bunch of other stuff getting read and written it gets painfully slooow. You can even add separate swap disks (preferably SSDs).

My web server now has 2 GB of RAM and the swap disabled. What a difference! I’ve disabled the swap on my other virtual machines (DNS, IPv6 gateway, etc.) too, no need for them to have swap really.

htop output on the web server

htop output on the web server

If you have a look at the image above you can see it has 0 MB swap space. I have more than enough free RAM and the machine runs a LOT smoother.

FreeBSD 9.0 released

FreeBSD 9.0 is out! It’s been a long wait to get the latest open ZFS version, version 28, in FreeBSD and noew it’s finally here. The SATA drivers now support AHCI, more good news for us storage nerds plus NFS support for NFSv4. You can read the offical announcement here.

Time to upgrade from 9.0-RC3 to 9.0-RELEASE, I know what I’ll be up to tonight for sure. :)

Network filing protocol test

Most people using a file server or a NAS are using CIFS. Why? Because it’s the only protocol supported by Windows by default and it’s been de facto standard for a long time now. But there are alternatives, especially if you run a Mac or use Linux (or any other *NIX system for that matter). Since I use a Mac that’s what’s been used during my tests as the main client hardware and software platform. Maybe it’s time to try something new instead of CIFS, like AFP from Apple or the old work horse NFS that’s been around since 1984. Let the testing begin.

Test implementation

I’ve done two different tests, one using rsync from the console and one using the Finder in Mac OS X. Each of these test’s were done in three sets:

  • Read from the file server
  • Write to the file server
  • Read and write simultaneously to the file server

Each file system (CIFS, AFP and NFS) were mounted to different folders under /Volumes and I used a 10 GB file with random data generated with dd if=/dev/urandom of=./testfile.bin bs=1024k count=10240 as the file to copy back and fourth.

rsync

The rsync test was done using rsync –progress command to be able to monitor the transfer speed during the transfer itself. I then used the average transfer speed reported by rsync as result. When doing the read/write test there was one rsync program reading and one writing.

Finder

The Finder test was done using drag and drop plus a stop watch. Not 100% precise, but good enough.

Test equipment and configuration

This section describes my test equipment and configuration used.

Client

  • Macbook Pro 13″
  • Core 2 Duo 2.53 GHZ
  • 8 GB RAM
  • 128 GB Apple SSD
  • Mac OS X 10.6.6

Server

  • Standard PC
  • Cure 2 Quad
  • 8 GB RAM
  • 6x 1.5 TB WD Green in RAID-Z with 2x Corsair F60 mirrored as ZIL and 1x Corsair F60 as L2ARC
  • Oracle Solaris 11 Express
  • Using built in CIFS and NFS servers and Netatalk 2.2 alpha for AFP connectivity

No tuning was done during the tests and no special software was used, just rsync and Finder. The computers are connected via 1 Gbit/s Ethernet.

Result

These are the results from the tests.

Read performance

Read performance

Read performance

AFP wins in both rsync and Finder with a lot better performance in the Finder test. NFS is the only one of the protocols performing worse in Finder than with rsync. The 99 MB/s that AFP delivers is REALLY good, in fact I dare to say it’s pretty rare from a home built system.

Write performance

Write performance

Write performance

The test show pretty much the same as in read performance. (I know it says read MB/s, it should say write of course. But the figures are still correct.) AFP is the fastest with CIFS second and NFS in third. The 91 MB/s I got with AFP is pretty much what you can expect as a maximum from 1 Gbit/s Ethernet.

Simultaneous read and write performance

Read/write performance

Read/write performance

Something strange is going on here, NFS get 61 MB/s read in the Finder test while only getting 30 MB/s read in the pure read test. No other surprises, about 30-40 MB/s.

Conclusion

If you have a Mac and your NAS or file server can use AFP, use it! It’s much faster than the others, especially when it comes to pure reads.

As you can see there’s a lot of difference between rsync and Finder tests, why this is I don’t know. I don’t use rsync that much, all my backup is done with Time Machine. The strangest thing is the read performance if the read/write test with NFS. Suddenly I got 60 MB/s for some reason. The fact that NFS is so slow in writes is because it always does synchronous writes which is a great deal slower (but more safe when it comes to losing data due to power loss etc.). Another limiting factor in the read/write test is the SSD in the client, I should have used a RAM disk since the SSD may limit the performance in the test to about 40 MB/s read/write as the result shows.

I’d love to do this test, especially the CIFS tests, on a Windows computer but since I don’t have one I can’t do it.