Tag Archives: iscsi

Virtual machine storage performance, NFS vs. iSCSI

4 Replies

I’m running VMware ESXi to virtualize my computers and FreeBSD 9.0 with ZFS as backend storage. I’ve done some tests a couple of years ago telling me to always choose NFS over iSCSI for virtual machine storage. Now, is this the best solution for performance? I had to test this, totally unscientific but still worth reading I hope.

I did two different tests, one with bonnie++ and one with dd. Each test was performed inside a virtual machine (Ubuntu 12.04 LTS 64 bit) running off iSCSI storage and one stored on NFS. They were both stored on the same storage pool. At last the test was performed directly on the FreeBSD backend server to see the potential maximum. This isn’t true though as I was limited by my 1 Gb/s Ethernet connecting the ESXi machine and the FreeBSD server.

I just ran bonnie++ with standard settings, no tweaking at all. Here are the results.

Bonnie++ results

Bonnie++ results

The second test was done with dd. I used dd if=/dev/zero of=./test bs=`expr 1024 \* 1024` count=4096 to create a 4 GB file just measuring the write performance.

dd results

dd results

Conclusion

As you can see there was a slight advantage for NFS, although not a huge one. In my opinion, the fact that the virtual machines are stored as files on the FreeBSD servers filesystem in NFS, is a giant reason for NFS as the first choice. You can take backup of your machines with ease and won’t have to use VMFS at all.

NFS or iSCSI for use as backend storage on ESXi?

Leave a reply

VMware, back in the days of early 2001, when there was only VMware ESX Server was built with ONE storage system in mind. A NetApp using NFS. They added support for FibreChannel, iSCSI, etc. but NFS was the main thing for VMware. In many cases it still is! For an average home enthusiast like myself FibreChannel isn’t an option because of price, that means it’s either NFS or iSCSI that’ll be used for storage. Unless you use internal disks in the server itself, but why would you want to do that? Your file server is there for a reason! What are the pros and cons of NFS vs. iSCSI? Here are my thoughts on the matter.

NFS

+

VMware was built for this.

It’s fast when it comes to reading compared to iSCSI, at least in my experience.

The virtual machines are stored as files on the file server and not in an image using VMFS.

-

NFS does sync writes, this tends to be slow. Can be solved by using ZIL, ZFS Intent Log if using ZFS.

Security isn’t the best unless using NFSv4 with authentication, use on a dedicated storage network instead.

 

iSCSI

+

Easier to share between more than one host if using multiple ESXi server.

Can be configured with good security.

-

Slow. At least that’s what my experience tells me.

Not that flexible since the VMs are stored in an image using VMFS, the VMware File System.

 

What to choose?

For me it was an easy choice, I went for NFS. I have used iSCSI before with ESXi and I wasn’t content. It was slow and more or less a pain to backup the machines. With NFS I can just power off the machine and copy the whole thing to another directory on the file server (or a take a snapshot with ZFS). I might add I did use iSCSI under Solaris with the COMSTAR iSCSI target, haven’t tried it under FreeBSD which I’m currently using.

Which ever one you go for I recommend using a dedicated network for storage. This is mostly for security reasons, you don’t want anyone to mess with the storage traffic. Storage also tends to use quite a bit of bandwidth, you probably don’t want this on your LAN. Since I only have two machines, the file server and the ESXi server, I use a single cable between them. No need for a network switch.

Normal load on my ZFS storage system

Normal load on my ZFS storage system

As you can see above I tend to get a lot of reads, I’d say 99% of my storage I/O are read requests.

Using iSCSI for extra storage

Leave a reply

I’ll start with the background, I’ve built myself a gaming PC… again. Mostly from old bits and pieces I already had. Now, the hard drive in my new PC isn’t very big, it gives my about 120 GB usable storage after Windows 7 and all that crap is installed. Is this a good place for my games? Probably not, since games tend to use a lot of space. I could either get another disk or use some sort of network storage. Disks make noise and get warm, don’t want that. Then, network storage it is!

You might wonder… iSCSI, what? Why? Basically I could go for CIFS storage (normal Windows compatible shares from my NAS running Solaris). However there’s a problem, some programs and games don’t want to be installed to a network share. Idiotic, I know. The solution for this is to use iSCSI instead. iSCSI sends raw SCSI commands (the language of most storage systems) and sends them encapsulated over the network. Essentially, you lengthen your hard drive connection via the IP network.

Solaris

I use the COMSTAR iSCSI target to share a part of my zpool via iSCSI. If you happen to use Solaris too, have a look here. Right now I’m sharing a 500 GB iSCSI disk with thin provisioning (even though Windows sees it as 500 GB, it only uses the amount actually stored on the iSCSI disk).

Windows

For once Microsoft has done something right, Windows 7 has built in iSCSI support. You can find it under Administrative Tools in the Control Panel. Just add the NAS as a portal under “Discovery” and the target should be found. Now you can use it just as a local disk (make sure to format it).