Author: Chaitanya M Bhatt

After working in a virtualized performance test bed for a certain project, I observed that few professionals have conspicuously used incorrect testing process in this context and I would like to jot my words of wisdom out here on this topic.

When I discuss about virtualization of performance test bed with performance engineers, the common reaction is that they think “It sounds cool.” I wouldn’t disagree with that. Never. Running a virtual guest operating system on a host operating system obviously sounds cool. You would be capitalizing on those ‘idle’ resources by pooling resources and making them shared. But, what engineers forget is that virtualization works best only in certain type of workloads. To be specific — it works best with diverse workloads. However, in a load testing bed, the load injector machine or load generator machines during execution always tend to work with a fixed workload with resource utilization almost reaching the threshold level while emulating higher load levels.

So, when you plan to create a test environment with load generators running on a virtualized platform, the first thing which you ought to know is that VMware or any virtualization software for instance has too many overheads like excessive context switching, too many interrupts, shared network resources, high IO activity etc. All these factors sum up to become an “Irrepressible effect” on your virtualized LoadGenerator machines. The whole virtualization paradigm causes processes to generally run slower than when on a physical machine. Hence, it causes a Vusers to run slower than usual. In fact because of the aforementioned points the integrity of your test results will be questionable.
Having said that, I feel, one should have load generators on virtual machines only if it is inevitable for them (Like, due to a stubborn client maybe!).

If you are moving ahead accepting the digressions from best practices, then, it is recommended to be wary of the limitation, document them and then act upon it carefully in order minimize the shortcomings of the approach.

Best practices:

Create multiple virtual LoadGenerator machine instances.
Having multiple virtualized LoadGenerators on a host machine is better than having a single virtual LoadGenerator machine because the hypervisor of a virtual platform scheduler works better when there is a diverse workload than when there is a homogeneous workload. Remember that virtualization software like VMware – be it whether it is a ‘Type-1’ or ‘Type-2’ hypervisor — it is designed to support as many virtual instances as possible on a single physical machine. A simple illustration: Load generator working on single virtual machine emulating 100 Vusers is likely to use more CPU resource than when the same 100 Vusers are split across into 2 virtual machines – 2 load generators.

Have an eye on the CPU utilization.
Ensure that all your virtualized load injector boxes are running at an optimal CPU utilization watermark (say, within 80%), this can avoid annoying issues in recorded response time like those negative response time for transactions which is an infamous problem in this context! This issue triggers because the VMware guest OS clocks are synchronized with host operating system time (which intern depends on your physical clock) and if the CPU is 100% busy while VMware guest operating system is attempting to sync the clock, then the process gets placed in the run queue of the processor causing a clock drift in guest operating system which intern messes up your transaction response time recorded by LoadRunner.
Tap those Load Generator machine resources metrics while executing tests and make sure none of them are starving for resources.

Never forget to add think time and pacing.
Also make sure that you strictly follow your business requirement than simply firing requests which otherwise, would be more like doing a stress test. Incorporate realistic think time in your scripts with appropriate pacing values. Since, CPU is a shared resource in a virtualized environment the above suggestion ensures that no virtual OS instance is deprived of CPU resource at any point in time unnecessarily.