LarsBlog

After playing with the same Windows 2000 Virtual Machine in both VmWare Player and VirtualBox, I thought I'd take the opportunity to do some basic performance testing between the two, to see whether I should make the jump to VirtualBox and take advantage of it's seamless windowing feature. While I was at it, I decided to enable SMP support in the Windows 2000 VM and try benchmarking this on my intel dual-core host PC with 2 Virtual CPUs configured in VmWare.

The performance I was specifically interested in was running the WebSphere Commerce development environment inside the VM. As a *very* approximate measure of performance, I decided to measure the start-up time of the WebSphere Commerce test server in each of the VMs - hardly a proper benchmarking tool but I figure this would be a reasonable estimate as it involves a lot of CPU usage and disk IO, and the allocation of a few hundred MBs of RAM.

The results were a little surprising. VmWare outperformed VirtualBox by quite a margin. Also, in VmWare, the SMP-enabled VM using both CPU cores performed more slowly than the singl-CPU version of the VM. The VM was able to utilise both CPU cores of my host machine, but somehow this didn't translate to increased performance. In my older version of VmWare player, the VmWare process on the host machine seemed to sit at 20-30% CPU utilisation even when the guest VMwas idle, so I re-did the tests with the latest version of VmWare player (which vmware explicitly state has SMP support) and this fixed the idle-CPU issue but not the performance. I have read that multi-core performance in a virtualised environment is very tricky, and it seems that one should be very careful before enabling it. Perhaps there are scenarios where it would result in a performance gain.

The raw figures are listed below. Multiple timings were taken for each VM as the WebSphere Commerce test environment has a tendency to take longer for the initial start-up than for subsequent ones, but the times start to stabilise after 3 or 4 attempts. The times taken are between the first time-stamp in the SystemOut log after clicking 'start' and the time-stamp of the line that states 'server1 is open for e-business'. Also, each server was started in debug mode.

VmWare Player 1.02 Build: 59824 - Dual-CPU VM: (DEBUG MODE)
1: [11/03/08 03:16:53:999 GMT] - [11/03/08 03:19:50:874 GMT] = 2:57
2: [11/03/08 03:22:35:780 GMT] - [11/03/08 03:24:43:858 GMT] = 2:08
3: [11/03/08 03:26:21:015 GMT] - [11/03/08 03:28:39:874 GMT] = 2:18
[Guest VM rebooted]
4: [11/03/08 03:53:03:343 GMT] - [11/03/08 03:56:16:640 GMT] = 3:13
5: [11/03/08 04:01:03:234 GMT] - [11/03/08 04:03:33:406 GMT] = 2:30
6: [11/03/08 04:06:26:640 GMT] - [11/03/08 04:08:55:218 GMT] = 2:29
7: [11/03/08 04:48:43:030 GMT] - [11/03/08 04:51:02:561 GMT] = 2:14

VmWare Player 1.02 Build: 59824 - Single CPU VM: (DEBUG MODE)
1: [11/03/08 03:35:32:452 GMT] - [11/03/08 03:37:15:999 GMT] = 1:43
2: [11/03/08 03:39:02:906 GMT] - [11/03/08 03:40:28:702 GMT] = 1:30
3: [11/03/08 03:42:01:406 GMT] - [11/03/08 03:43:24:296 GMT] = 1:23
[Guest VM rebooted]
4: [11/03/08 04:31:58:843 GMT] - [11/03/08 04:38:18:109 GMT] = 6:20
5: [11/03/08 04:39:45:375 GMT] - [11/03/08 04:41:21:656 GMT] = 1:36
6: [11/03/08 04:43:09:984 GMT] - [11/03/08 04:44:51:718 GMT] = 1:42

[Re-did above 2 tests with VmWare 2.03, which is supposed to have better SMP support]

VmWare Player 2.03 Build: 80004 - Single CPU VM: (DEBUG MODE)
1: [03/04/08 14:42:41:602 BST] - [03/04/08 14:45:08:649 BST] = 2:27
2: [03/04/08 14:50:55:618 BST] - [03/04/08 14:52:18:540 BST] = 1:23
3: [03/04/08 14:58:35:821 BST] - [03/04/08 15:00:00:243 BST] = 1:25
4: [03/04/08 15:03:21:337 BST] - [03/04/08 15:04:49:680 BST] = 1:28

VmWare Player 2.03 Build: 80004 - Dual-CPU VM: (DEBUG MODE)
1: [03/04/08 15:45:41:437 BST] - [03/04/08 15:48:56:890 BST] = 3:15
2: [03/04/08 15:59:25:337 BST] - [03/04/08 16:02:05:587 BST] = 2:40
3: [03/04/08 16:42:31:806 BST] - [03/04/08 16:49:08:087 BST] = 6:23
4: [03/04/08 20:28:33:315 BST] - [03/04/08 20:31:41:003 BST] = 3:08
5: [03/04/08 20:39:42:159 BST] - [03/04/08 20:42:08:175 BST] = 2:26
6: [03/04/08 20:54:24:312 BST] - [03/04/08 20:57:09:419 BST] = 2:45

VirtualBox 1.56 - Single CPU VM: (DEBUG MODE)
1: [11/03/08 05:34:55:070 GMT] - [11/03/08 05:38:23:892 GMT] = 3:28
2: [17/03/08 15:42:28:547 GMT] - [17/03/08 15:45:44:078 GMT] = 3:16
3: [17/03/08 15:47:36:638 GMT] - [17/03/08 15:49:51:648 GMT] = 2:04
4: [17/03/08 16:32:17:747 GMT] - [17/03/08 16:34:39:097 GMT] = 2:22
5: [17/03/08 16:36:19:082 GMT] - [17/03/08 16:38:43:591 GMT] = 2:24

Note: Lower times are better :) 

Note2: VirtualBox does not support Dual-CPU VMs at this time, so I could not test this.

Well, these figures need to be taken with a grain of salt. These results aren't performed under 'laboratory conditions', and some of the performance difference between single and dual-core might be in part due to the different Hardware Abstraction Layer used in Windows 2000. Nevertheless, it seems as though for performance, old-school single-core VmWare is still the king, with VirtualBox and it's seamless mode a good option for scenarios where performance isn't so critical. And for those of us that want to run heavy-weight development tools with the convenience of a VM whilst taking advantage of modern multi-core processors - we'll have to wait a bit longer. :(