Tag Archives: vmware

Blog moved to SmartOS, but what’s the point of SmartOS really?

Leave a reply

So, I’ve moved my blog to SmartOS. This means that if you’re reading this, it worked! I’m still using WordPress, Apache, PHP, MySQL etc. but Linux is gone. Why? Why would anyone replace VMware ESXi and a VM running Linux with SmartOS? I’ll tell you why!

Why.

It’s quite simple, performance. Running SmartOS I’m using operating system-level virtualization. The difference between running a full blown hypervisor based VM and OS-level virtualization (from now on called a zone) in SmartOS? There are several differences, but the main ones are these:

  • SmartOS and all it’s zones are using the same kernel, but are isolated from each other
  • There’s no overhead, zones are all interacting with the OS directly
  • The biggest problem is usually flexibility, you can’t run a different OS in a zone

If you need to run lots of different OSes, zones isn’t for you. The nice thing about SmartOS is that you can run KVM and get that functionality as well.

Deploying SmartOS in a home environment

18 Replies

What’s the most common hypervisor for home use? I’d say probably the “free” version of VMware vSphere. I’ve been using VMware for a couple of years now and it has served me well, but. There’s always a but, but what is it? For me it’s the overhead, I want to be able to run operating system level virtualization. To do this I had to ditch VMware, even though it works good for full virtualization, it just doesn’t cut it. The next question was what to replace it with? The answer was simple, SmartOS. I’ve always been a big fan of Solaris and ZFS. SmartOS is basically the best parts of Solaris, ZFS, KVM and Dtrace bundled and customised by Joyent. All these wonderful software in one package. I was sold, it was time to get this deployment going!

Do I need new hardware?

It depends. If you’re using a newer Intel Core i3/i5/i7 you’re probably fine. If you’re, like me, using an AMD CPU it’s time to invest. SmartOS won’t run on AMD CPU’s or older Intel CPU’s. I got a motherboard from ASUS (no server grade, just cheap and simple) and a Core i5. To this I added 8 GB RAM and two SATA disks which will be used to store the virtual machines. Why not use iSCSI you might say? If you do you’re missing out on all the wonderful self healing capabilities of ZFS, hence it’s best to use internal disks. OK, hardware’s sorted!

Installing SmartOS

The easiest way to do this is to download an USB image and write it to a pen drive. SmartOS is made to be run from a USB pen drive or SD card. Have a look at the SmartOS wiki for instructions.

Converting virtual machines from VMware to SmartOS

If you’re running VMware ESXi 4.x this isn’t a big deal (in theory). There’s a way to convert and import the vmdk-files. Once again, the SmartOS wiki has the answer.

Now, was I using ESXi 4.x? Of course not, I was using 5.1. Converting a vmdk from 5.1? Nooooo. Won’t work. So, how do you get the machine(s) out of VMware and into SmartOS? I used a combination of dd, netcat and bzip2.

Start by booting up a Linux live cd on the VMware VM you want to convert. When it’s started you’ll need to start a receiving server using netcat on another machine, preferably one with enough harddrive space to fit the VM. Use the following command:

nc -l 19000|bzip2 -d|dd bs=16M of=vm_image.img

The server is ready to receive data, let’s send some!

dd bs=16M if=/dev/sda|bzip2 -c|nc [server ip] 19000

This will send the contents from sda on the Linux VM to an image file on the server. I’m sending it through bzip2 to avoid sending unnecessary data, it’s better to just compress all them zero blocks! OK, the data is extracted from VMware and now resides in an image file. The next step is to transfer the data into SmartOS. Have a look here at Ed Plese’s blog, he explains how to set up a new VM using KVM. It boots an iso file, which is perfect. Boot the same live cd as on VMware earlier and we’ll transfer the image to the virtual disk in SmartOS. Just make sure the virtual disk in SmartOS is big enough to fit the data on the image. When booted, issue the following on the VM:

nc -l 19000|bzip2 -d|dd bs=16M of=/dev/vda

Now, make sure you type /dev/vda instead of /dev/sda since the disk is called vda in SmartOS instead of sda. Now, send the data from the server like this:

dd bs=16M if=vm_image.img|bzip2 -c|nc [vm ip] 19000

When the transfer is done, just reboot the VM and the machine should boot up. This will only work if you’re using UUID’s in /etc/fstab on the virtual machine, if not you’ll have to change from sda to vda before doing the initial transfer out of VMware.

So far?

I’ve had the system up for about three hours, works like it should. No complaints yet, but if you’re a point-and-click kind of guy, this probably isn’t for you. SmartOS is text console all the way. Just the way I like it. Plus the fact that the CPU/memory restrictions in the free license of VMware won’t be an issue. No restrictions here! At last, thank you Joyent! This is one sweet piece of software!

VM's running in SmartOS

VM’s running in SmartOS

Virtual machine storage performance, NFS vs. iSCSI

4 Replies

I’m running VMware ESXi to virtualize my computers and FreeBSD 9.0 with ZFS as backend storage. I’ve done some tests a couple of years ago telling me to always choose NFS over iSCSI for virtual machine storage. Now, is this the best solution for performance? I had to test this, totally unscientific but still worth reading I hope.

I did two different tests, one with bonnie++ and one with dd. Each test was performed inside a virtual machine (Ubuntu 12.04 LTS 64 bit) running off iSCSI storage and one stored on NFS. They were both stored on the same storage pool. At last the test was performed directly on the FreeBSD backend server to see the potential maximum. This isn’t true though as I was limited by my 1 Gb/s Ethernet connecting the ESXi machine and the FreeBSD server.

I just ran bonnie++ with standard settings, no tweaking at all. Here are the results.

Bonnie++ results

Bonnie++ results

The second test was done with dd. I used dd if=/dev/zero of=./test bs=`expr 1024 \* 1024` count=4096 to create a 4 GB file just measuring the write performance.

dd results

dd results

Conclusion

As you can see there was a slight advantage for NFS, although not a huge one. In my opinion, the fact that the virtual machines are stored as files on the FreeBSD servers filesystem in NFS, is a giant reason for NFS as the first choice. You can take backup of your machines with ease and won’t have to use VMFS at all.

Backup of virtual machines with ZFS snapshots and rsync

4 Replies

I use VMware ESXi to run virtual machines, they’re all stored via NFS on my file server which uses ZFS. Even though I use RAID-Z it would be a good idea to back them up to a remote site. I do this by snapshoting the ZFS volume and rsyncing the snapshotted machines to my Xserve at work.

#!/usr/local/bin/bash

#Create snapshot
backupdate=$(date "+%Y-%m-%d")
zfs snapshot storage/vmware@$backupdate

#rsync virtual machines
rsync -avz /storage/vmware/.zfs/snapshot/$backupdate/Callisto marcus@[remote host]:/Volumes/Storage/backup/VMs/
rsync -avz /storage/vmware/.zfs/snapshot/$backupdate/Ganymede marcus@[remote host]:/Volumes/Storage/backup/VMs/
rsync -avz /storage/vmware/.zfs/snapshot/$backupdate/Mercury_1 marcus@[remote host]:/Volumes/Storage/backup/VMs/

This is by no means a perfect solution. I’m backing up live machines, this has a potential of going wrong. But since I’ve snapshotted them I hope the problem is negligible. As you can see in the script I’ve changed the remote host to [remote host], replace with host of choice plus modify the rest of the paths. I think you get it. :)

Right now I’ve added a cron job to do a backup once a week.

Swap in VMware ESXi, do or don’t?

2 Replies

I’ve never really thought about the swap when running virtual machines, I’ve pretty much just gone with the standard settings during the installation. I’ve never, but I should have, asked myself “do I need swap on this machine?”. My web server (the one you’re on right now actually, reading this) had 512 MB of RAM and a couple of GB swap. The virtual machine just got slower and slower and the RAID on my file server was crazy busy, but I couldn’t figure out why. Then it struck me, swapping! Since I run ESXi over NFS this is also where the swap is read and written. By no means fast, especially when it comes to latency. TCP/IP and NFS plus a ZFS RAID system, there was a huge bottle neck here!

How to solve it

Easy, more RAM. After that? Even more RAM and disable the swap, or at least don’t store it on an NFS volume on slow, spinning disks. OK, if there’s only one machine swapping and nothing else reading or writing to the RAID it’ll be fine, but when you have loads of virtual machine swapping and a bunch of other stuff getting read and written it gets painfully slooow. You can even add separate swap disks (preferably SSDs).

My web server now has 2 GB of RAM and the swap disabled. What a difference! I’ve disabled the swap on my other virtual machines (DNS, IPv6 gateway, etc.) too, no need for them to have swap really.

htop output on the web server

htop output on the web server

If you have a look at the image above you can see it has 0 MB swap space. I have more than enough free RAM and the machine runs a LOT smoother.

Install VMware ESXi without a CDROM

2 Replies

For some reason VMware only offers ESXi as an iso. But… My computer running ESXi has no CDROM! How to solve this? Install VMware ESXi in VMware Workstation or Fusion! I run ESXi from a USB stick so it’s very easy to attach it to a virtual machine.

1. Download the iso.

2. Create a new Virtual machine i Workstation/Fusion and attach the iso image. It should set it to ESXi as guest operating system automatically.

3. Remove the virtual harddrive. Insert a USB stick that you want to install ESXi on.

4. Attach the USB stick to the virtual machine.

5. Boot the iso in VMware Workstation/Fusion and install onto the USB stick.

ESXi being installed to a USB stick in VMware Fusion

ESXi being installed to a USB stick in VMware Fusion

Getting a second Internet connection

1 Reply

I’ve been a customer of swedish Bredbandsbolaget for many years now and I thought I’d try something new. Fortunately I can get internet access from two different ISP through Ethernet in my building, Bredbandsbolaget and Netatonce. Since I didn’t want to give up my current connection I went for both of them. Now, I only have one Cat5E from the central in the building to my apartment and VLAN isn’t an option. Both ISPs offers 100 Mbit and the solution was to use an Ethernet splitter. 100 Mbit only uses two of the four pairs in a standard Cat5 cable, i.e. I can run two 100 Mbit connections in one cable with a splitter. Perfect! Since the technician, who installed (patched really) the connection, is an old friend of mine I just asked him to hook up the splitter.

Everything works flawlessly! I use a switch to VLAN the ISPs and connect them to my ESXi machine running pfSense. BBB (Bredbandsbolaget) is untagged VLAN 10 at port 1, Netatonce untagged VLAN 11 at port two and they both merge through one cable with both VLANs tagged. Then I create two different Virtual LANs in ESXi and connect virtual NICs to each of them.

In ESXi it looks like this.

So, how about speed? Is the second ISP delivering? Oh yes, this is from a test done just a couple of minutes ago.

In english that’s 93 Mbit download speed and 76 Mbit upload. Quite nice.

Screen shot 2011-02-05 at 20.01.13

Speeding up your internet connection with WAN load balancing

9 Replies

To start with this either requires you to have multiple Internet connections OR, like me, an ISP that gives you 100 Mbit/s download and 10 Mbit/s upload. The thing is that my ISP gives me a 100 Mbit full duplex connection but limits my upload to 10 Mbit/s per IP. Notice, per IP address! I can get five IP addresses, this means 5 x 10 Mbit/s upload. I could have five different computers all uploading at 10 Mbit/s or 5 network cards (NICs) in one computer with load balancing. Of course I use the load balancing solution with pfSense as OS for my router.

To make things even better you can do this by using a virtual, I use VMware ESXi with a dedicated NIC connected to the Internet.

Internet connection in ESXi

Now, create five virtual NICs on that virtual machine like this.

Five virtual NICs

As you can see there are five NICs connected to Internet and one to my LAN. Continue by installing pfSense. Connect LAN and the first WAN NIC when installing pfSense. Your Internet connection should be working (but without load balancing) to go to the next step.

Go to Interfaces > Assign and add your remaining NICs as OPT1-4. When activating them, don’t forget to check “Enable optional interface” at the top of the page and check “Disable userland FTP-proxy application” too.

NICs assigned to opt1-4

All your NICs should get an IP address (hopefully). Check this under Status > Interfaces. Should look something like the picture below.

All NICs have IP addresses

The next step is to create a pool for Load balancing containing all the WAN NICs. This is done under Services > Load balancer.

Use the following settings:
Type: Gateway
Behavior: Load balancing
Monitor: DNS Server 1 (use this if your NIC are all connected to the same ISP, otherwise the gateway of each NIC)

Now, add the interfaces and set an appropriate name for the pool.

Load balancing pool

The last step is to make sure the pool is used, this is done by a firewall rule. Go to Firewall > Rules and the LAN tab. Edit the existing rule and change the gateway to your pool’s name.

Gateway changed

You’re done! This is what I get when using speedtest.net (which doesn’t show my real upload speed, I’ve been achieving 30 MBit/s at least with multiple upload connections).

Speed test

Speedtest nr 2

Now, there’s one thing left. HTTPS can be a bit cranky when it’s load balanced, so you can add a firewall rules to make sure HTTPS is always running from/to the first WAN NIC. Add a new rule under Firewall > Rules under the LAN tab. Choose destination port range and select HTTPS, gateway should be set to Default (not the LB pool). Save the rule and move it to the top of the rules under the LAN tab. I’ve added rules for both HTTPS and HTTP.

HTTP and HTTPS rules

Upgrade no 2, the ESXi server

2 Replies

The time had come when it was time to change the components of my ESXi server. Why? Well, for starters it used to crash about once a week for nu apparent reason. Hopefully this is now a thing of the past. The second reason to upgrade is just for the fun of it, more memory and a faster CPU. You might be wondering what new hardware I’ve put into the machine. I’ll tell you!

Motherboard

I first tried a board from Gigabyte, but it refused to boot from my USB stick containing ESXi. What to do? Couldn’t be bothered with trying to get it to boot, so instead i went for the ASUS M4A89GTD. It uses the AMD 890GX chipset and has support for Phenom II X6, which is a must as you’ll see further down. It also has a good amount of expandability with many PCI express slots for upgrades, NICs, etc. Last but not least, it boots my USB/ESXi stick with perfection. My old ESXi server also had an ASUS motherboard, so I think i made the right choice. Since it’s a desktop motherboard I disabled the sound card, SATA controller, onboard ethernet, etc. in BIOS.

CPU

AMD Phenom II X6 1055T, a fast CPU with six cores at a reasonable price. When running a lot of VMs I’d rather have six cores than two or four and because the CPU is cheap it wasn’t a hard decision.

RAM

2x 4 GB Corsair XMS3 1066 MHz DDR3, in total 8 GB RAM. Since my local computer store didn’t have ECC memory I went for these, might replace them with ECC RAM though.

Conclusion

So far everything is running smoothly, although I’ve only had the machine up and running for 16 hours. At the moment it has three VMs, two Linux (Ubuntu) and one Windows XP. With most of the memory and CPU unused I have a lot of power to spare and hopefully I don’t have to upgrade for quite some time. Last but not least, sorry for the lack of pictures. With the Gigabyte motherboard refusing to boot I got so frustrated I forgot to take pictures.

The complete server setup

Leave a reply

At home I have two servers, a file server and a VMware ESXi server. If you’ve read my previous posts you’ve noticed that my file server runs Solaris. But what about the hardware?

File server

At the moment the file server consists of the following:

Core 2 Quad 2.5 GHz
8 GB RAM
6x Western Digital 1.5 TB hard disks
3x Corsair F60 60 GB SSD disks
1x Western Digital 160 GB IDE hard disk
1x Intel dual port NIC

The CPU is quite nice, have’t had any problems with it. Earlier the machine used a Core 2 Duo, but the quads were becoming so cheap that I couldn’t resist it. The amount of RAM is OK for my needs, i get about 6 GB of ARC usage. Just wish I had ECC RAM, but I guess that’s something I’ll have to consider for my next mayor upgrade.

When it comes to storage there’s pretty much just one big RAIDZ-pool made from the six 1.5TB-disks. The pool also utilizes a mirrored pair of SSDs for ZIL and a single SSD for L2ARC. The system is on a single IDE disk at the moment, which isn’t an optimal solution.

Finally the network connectivity is provided by a dualport PT1000 NIC, one port for LAN and one for my IP-SAN. The IP-SAN is only connected to the ESXi-server right now.

ESXi server

Athlon X2 6000+
4 GB RAM
No harddisks
4 GB USB memory to hold ESXi and configuration
1x Intel PT1000 card and 1x Broadcom NetXtreme BCM5721

This server doesn’t have as much goodies in it, it just runs the VMs. In my opinion I does that rather well, since I only have three VMs running on it. The RAM isn’t a problem yet and everything else works as expected. The connectivity to the file server is done through a separate network, so it can’t be affected by whatever is going on on the LAN. I use NFS for my datastore against the file server, I feel it gives me the best speed and the best flexibility. I have used iSCSI, but the problem with iSCSI is that it isn’t as easy to extract VMDK-files if needed.