Archive for November, 2009

Author: Chaitanya M Bhatt
Email: chaitanya.bhatt@gmail.com

Let us try looking at this issue from another perspective. If it were that training was a solitary solution to fixing this issue, then the industry would have never faced a problem called “deficit of skill/incompetency” which unarguably persists across not just a specific country but every geographical area in the world. I have seen
situations where in spite of giving massive training programs to professionals before being deployed to projects, resources still have shown severe lack aptitude and skills when put to work.

According to me, any technical field is very capricious per se and new technologies crop up every day; in such a juncture, I feel, it is every professionals responsibility to align his/her skills to the evolving technologies on their own by the method of self-learning. Today, loadrunner statistically has a market share of just over 63% with other emerging small players like Compuware, Boarland, Radview etc. sharing the rest of the market and it wouldn’t be a surprise to me if in another 5 years we see loadrunner market share to have dwindled to less than 50%, so does this mean that all the professionals need to be trained officially by their employers on every competing tool which enters the industry? I work in a company where different clients prefer different tools, ranging from completely open sources tools like Grinder, Jmeter to enterprise level tool sets from HP, IBM etc., now if every resource were to be trained on all these tools before being deployed to projects then the employer would never ever get his ROI; not to forget that these days ‘Training Programs’ are priced exorbitantly high.

The solution is in the hands of every professional; if a professional has the right kind of approach in problem solving and a good interest level in the subject then he/she would learn any tool/technology without any hassle.

I recently registered to Google and Yahoo LoadRunner groups and saw a disturbing habit of several members in the group of posting basic questions of LoadRunner like “What is LoadRunner?”. This is ridiculous! If people are capable to register in Google/Yahoo groups with an intention to learn loadrunner from the members of it then I think they are definitely also capable to register in http://itrc.hp.com portal which is a gateway to acquire tons of Loadrunner resources, and these resources are undoubtedly the best anyone can ever get.

If recruiters pay a little more attention to a candidate’s interest level towards the job profile, his/her true problem solving capability, smartness and general attitude rather than just looking at the “University” he/she graduated from and his/her GPA — then I would say they have hired a candidate who is already 90% qualified to do the job and the rest 10% would automatically come up as he becomes more experienced. “Training Programs” I feel, contributes more in refining the capabilities of a resource than in shaping one.

Author: Chaitanya M Bhatt
Email: chaitanya.bhatt@performancecompetence.com

In my observation, persistently there has been a lack of awareness of the difference between “Simultaneously connected active users” and “Simultaneously connected passive users”. Commonly this problem is observed both from executive stakeholder side as well as the performance engineering/testing team side.

Impact of not understanding these concepts makes a lot of difference when it comes to meeting the actual performance test requirement and can make a huge difference on the legitimacy of your load test report.

For your information: Simultaneously connected active users can be also termed as “active concurrent application users” and simultaneously connected passive users can be termed as “connected users in passive mode”.

Connected users in passive mode:
I will quickly explain the differences and the nuances with an example: Imagine if “Facebook” was your AUT (Application Under Test); Many end users of Facebook.com often set www.facebook.com as their default browser page and hence when the browser is opened — a connection (may not be persistent) would be maintained between the end user and the Facebook.com server. The user may leave the Facebook.com session idle without firing any query like searching for a friend or adding a friend. This case is a typical example of connected user who is in a passive state.
Facebook.com might have 100’s of thousands of such users who are in this state and it is important to simulate this condition during load tests because even passive sessions or connections can hold certain resources of the server as well as can have impact on the overall thread pool size and can also affect various tunable configuration parameters of web/app/db servers.

Active concurrent application users:
Continuing with the same Facebook.com example, it is known that there are also users who would not just be connected to Facebook.com but would also be actively using facebook.com features like adding a friend, poking a friend, updating status, uploading photos etc. It is during this kind of user connection state that the actual payload gets affected heavily. Higher the rate of user activity higher would be the transactional throughput.
Care should be taken while setting a test execution window, because clients might say facebook .com has 200,000 concurrent active connections in a day with an average user base executing 9,600,000 transactions daily (this means 10 million transactions such as adding friend, poking friend, photo upload etc.), but when you break the transactions to fit your test window – you often forget to think about the persistence duration of users per login-logout sessions against your test window.
What I meant to say, is that, if you are executing a 60 minutes load test and for scenario configuration purpose if you’re breaking the daily stats to fit your 60 minutes test window by dividing total users per day by 24 (approx. 8300 virtual users). And, also you would have calculated total transactions fired per hour, which is equal to 2 (derived the result after dividing 9.6 million by 200,000 = 48 transactions/user/day; 48/24 = 2 transactions/hour), but ignored the persistence of transactions; you may face a situation where all virtual users executing those 2 transactions (like: adding a friend and poking a friend) would finish execution within seconds leaving the rest of the test window void without any transactions, which will surely cause over-stressing of the server leading to inaccurate result. In the other extreme case without thoughtfully setting virtual user transaction persistence can cause a situation, specifically with ‘login – Perform a transaction – logout’ scenario type, where one user finishes a cycle quickly before the other enters; this would actually lead to lesser active user concurrency and resulting in inaccurate result.
The solution to maintain realistic transaction persistence is by making virtual users to stay back in the connected session with proper “pacing” between transactions and “realistic think time” , so that transactions are optimally(try following actual/live pattern) spaced within the test execution duration/window.
Emulating realistic concurrency of user at application usage level is the key to understand the true application behavior during load test.

Author: Chaitanya M Bhatt

Many people say that process oriented work is the key for success. Even I strongly support that statement but I always wonder why people do not give prominence to those 7 basic quintessential points of performance test activity which can truly make or break a project.

When a resource is allocated to a performance test project the first thing the project manager would ask him/her to do is to start working on preparation of a test plan. But during test plan preparation phase, it is very often seen that test managers and his/her subordinates would heedlessly prepare the document with most of the contents of the document copied from previous projects or from arbitrary template available in the internet and proceed to the next phase without giving any prominence to those requirements/specifications mentioned in the test plan. It is undeniably a known fact that preparation of a test plan is always done as a formality obligated to the company’s process standards (like ISO tick it) and as a consequence this document is never really used in conjunction with project execution.

Time to be invested for a test plan preparation I would say should be a significant part of the total project execution time. But all these facts are unfortunately talked more in theoretical sense and less in professional sense and hence practitioners often fail to strictly bind their activities as per test plan intimidated by the large overhead of process associated with it and also believe that it would slow them down.

This undoubtedly is a bad practice, but even if you negate the project planning phase – I feel we can still expect light in the end of the tunnel if the test engineers incorporates those 7 basic quintessential facets of performance test activity religiously during project execution phase.

The 7 tenets are:
1. Know what the SLA states.
2. Understand the real user usage patterns.
3. Know how to load the server.
4. Know how much to the load server.
5. Know what type of load needs to be induced on the AUT.
6. Know your test tool well to maximize on its capabilities.
7. Know your test environment.

The best part of these tenets is that you just have to make sure they are a part of your consciences in your professional frontier. You need not document these things neither would you have to present it to the client; you just have to practice it. Because in the end, what the client expects is not how good or bad was your load test plan nor would the client be bothered of those different standards that you followed, instead the client would always be more bothered about the degree of accuracy of the result that you have presented to them and how much of help would these statistics be to improve the performance of the application. And trust me knowing each of aspects mentioned in the tenets and acting upon it carefully can definitely help achieve positive outcome of the load test project.
Conclusion: On failure to comply by the standard processes of performance testing, at least incorporate the 7 tenets of performance testing which could help you to effectively compensate for those digressions from best practices.

Author: Chaitanya M Bhatt
Email: chaitanya.bhatt@performancecompetence.com

One of my internet friend asked me a question“How do I calibrate the total number of Loadgenerators required to generate a load of 300 Virtual user if I have each loadgenerators equiped with 3Gb of RAM and 3.3GHz of cpu clock speed?”

My answer:

This is a highly debatable topic I must say. It is very subjective. Memory foot print which HP as a vendor provides is not very accurate in a general sense. The impact of Memory and CPU footprint of any protocol on a Loadgenerator also depends greatly on the length of the business flow and on the workload profile which the client expects the system to achieve. Example: Hypothetically, if in a SAP GUI application if you have 100 virtual users just entering employee name and number in a field and submitting it to the server and are iterating it only 2 times in a half hour period (which if is your test duration) then I would say you may not need more than 1 load generator which is sized with 3Gb of RAM and 3.3 Ghz dual core Intel processor.

Adding to the above, HP does not provide any kind of CPU footprint report. Hence I generally go about telling people to calculate the heaviness/payload of loadrunner process on their own. This can be done by identifying metrics of resource consumption by 1 user and theoretically determining total resource needed by multiplying the observed consumption for single virtual user with the expected number of concurrent users.

Using Windows Task manager(Process tab) while executing a single Vu will help you determine the memory consumed by a single ‘MDRV’ process which if multiplied with expected concurrent virtual user count will help you in calibrating total memory required to run a load test. Similarly to estimate the CPU resource for a certain protocol, I would recommend you to monitor and record few CPU counters of your LoadGenerator through ‘perfmon’ while deriving baseline result and multiply the metrics with the target Vu count which is mentioned in the workload profile.

It is ideally recommended to have as many boxes as possible to match the theoretical requirement, but as we all know, one of the many challenges a performance tester faces is lack of sufficient loadgenerator machines. Hence for the benefit of time and to avoid further pestering from your boss to somehow resolve the issue – try utilizing your loadgenerator boxes as efficiently as possible by:

1. Log only when error occurs.
2. Precisely calculate think-time between transactions. This can help
in exerting less load on the load generators. Strictly follow real user screen pause patterns.(note this point as
very important!).
3. Uninstall any daemon applications running on Loadgenerators
which is not a part of AUT.
4. Avoid excessively using the “Show Vuser” option during test
execution.
5. Declaration of variables with large size should be avoided, which
directly helps in conserving overall memory. Consult the development
team if your unable to estimate the right size(This is a major concern
in Java Vuser and VB Vuser scripts).
6. Never over-iterate a script.

Why use WAN Emulation using WANem?

WANem is free software. Developed by TCS, published by Free Software Foundation. It can modify or redistributed on terms of GNU General Public License version 2.

Rigorous performance testing and optimization is a critical factor in the successful delivery of any business application. Yet frequently the performance of deployed applications doesn’t live up to business requirements or end-user expectations. One reason behind these unpleasant “surprises” is the fact that most performance staging labs only test the application with local users (in a local area network (LAN) environment), while the fully deployed application is used by a variety of end-users, some local and others accessing the application remotely over different network links. The different network conditions that exist between end-users and application servers have a tremendous effect on the overall performance that remote end-users experience. This deviation from performance test results obtained in the lab is further exacerbated for N-Tier applications where each tier may reside in a different geographical location with its own unique set of network conditions.

Network Emulation tools can be used to accurately replicate existing or projected conditions in the distributed production environment – including infrastructure, application traffic and the distribution of end-users.

To sum up on a high level — the benefits of using WAN Emulation tools are:
- WANem is a freeware.
- mitigate applications deployment risk
- find errors before deployment
- test new WAN topologies and technologies
- emulate remote users experience
- stress models of the network to find vulnerabilities

2.1 System Requirements:
Minimum an i386 based PC with 1 CPU, 512 MB RAM and 1 Network interface card – 100 Mbps (preferably 1 Gbps).

2.1 Applications supported by the WANem(but not limited to)
1. Web applications,
2. Video Streaming
3. Interactive applications – telnet like application.

2.2 Setting up WANem
WANem is distributed in the form of a bootable CD with Linux Knoppix O/S. This CD comes with WANem preinstalled. There are no installation steps. When an i386 architecture based PC is booted with the PC WANem is ready for use.

*Refer WANem user guide for more information on launching WANem and usage.

Drawbacks of WANem:

1. Multiple NIC cards when there is need for performing distributed load generation.
2. Dedicated PC required for WANem setup.
3. Network address translation needs to be done when a client application is running on a different network.
4. Cannot be integrated with LoadRunner.
5. Installation is relatively difficult.

Conclusion

WANem is free software and hence cost effective. Advanced network options present in WANem keeps trust and usability of the tool on the same level as of any other contemporary network emulation tool.

Author:Chaitanya Bhatt
Email:chaitanya.bhatt@performancecompetence.com

Recently a newbie to Loadrunner asked me the following question: “I have seen you somewhere in the loadrunner groups. I like to know if you
are aware of some sites where we can have a look at with different protocols
that uses.

But Like to know more about other protocols.”

My Answer to the above question in a general sense:
Let me tell one thing frankly. According to me LoadRunner is a small tool but the technologies it supports is very vast. I haven’t come across any website which is comprehensive enough to help people deal with rarely used protocols like Oracle 2 tier or Tuxedo for instance. When people asked me whether it is possible to directly connect to the database and work on it – using Toad or SQL query – they put a smile on my face. People, you can connect to the database for sure using ODBC protocol, Oracle 2 tier protocol but you surely need not have any SQL query analyzer or Toad to do that. Vugen alone is sufficient.

If you are looking forward to really learn the different protocols of Loadrunner then start learning the technologies up front on which these protocol works. Example if it is SAP GUI, start googling information on SAP GUI and keep you basic understanding of SAP GUI good enough so that if you challenged to work on it tomorrow – you would at least have fair amount of idea of where to start with.

Most of the LR professionals I would say are self thought. You need to have the aptitude and drive to perform R&D on something you feel is challenging you. In this particular case I would suggest you install Oracle DB on your machine and try connecting to the Db using ODBC protocol of LR and once you establish connection – write queries to fetch and insert data. You would love the way you learnt new things by this method. Believe me, it is a tried and tested method of learning Loadrunner. It just works.

Author: Chaitanya M Bhatt

Basically when programmers goes for .NET/VB/Java applet driven
development – most oftenly the application would not use open
standards like HTTP and hence loadrunner protocols would fail to
capture any of the messages sent through the application layer.

Developers would generally use custom messaging formats when they
develop an application using .NET/VB/Java applets, so this means when
the client and server are speaking with each other none of the
loadrunner protocols can identify the message because it hasn’t been
predefined.

Solution: Since contemporary record-replay option is ruled you can use
decompilers to make those DLL’s of your application more readable and
you should also consider getting help from the developers to build
workflow scripts using the source code of the application itself
in .NET Vu or VB Vu. This task could be challenging if the developers
have shrouded/obfuscated the code intentionally for security purpose,
hence i strongly suggest you get developer’s support during script
preperation
.
This tool can come in handly if your application :http://
www.remotesoft.com/salamander/

Worst case if nothing works, for the benifit of time you may choose
the following approach only if your workload scenario is light weight
with 10-50 virtual users, you may use RDP(analog recording) protocol
which records inputs of Keyboards and Mouse Clicks. You may need
higher end loadgenerator hardware if your targetting a higher range of
virtual user.

You can also try Winsock protocol. You may be lucky if your
application buffer passing through the sockets is small in size and
probably in which case your winsock script might be more readable and
may relatively be more feasible than any of the above mentioned methods
(again…it is subjective).

Author:Chaitanya M Bhatt

If your integrating QTP with LoadRunner, the only advantage compared
to RDP protocol is that the rate of success of replays of your
workflow script would be higher due to object recognition capability
of QTP(strictly avoid using analogue mode of recording, which would
defeat the whole purpose of this ordeal!)

Remember QTP approach never gives accurate transaction response time
results what so ever! All the client side activities like page
rendering time, client side internal processing time etc. would be
added to your response time.

Using QTP with LoadRunner is challenging. But I would like to give you few tips to make your job a little more simpler.

1. Make sure of the reason why you have chosen QTP approach. (This is very
very important.)
2. Now that you’re sure you’re left with only one option, make sure
you know the limitation of this approach.(also make sure the client is
informed about this!)
3. Make sure you have Windows NT installed on all the LoadGenerators
(Server OS can allow multiple user accounts to be created, which can
help you reduce total number of LoadGenerators. Example: 1
LoadGenerator with 32 Gb of memory and 2 quadcore processors with
Windows NT or Windows 2003 can probably have approximately 15 user
accounts with QTP installed seperately on all the user accounts.)
4. In the controller loadgenerator options, make sure you have atleast
2 or more terminals per session enabled in your terminal session
manager. This can help you spawn more than one qtp session on one user account.
( This option is very tricky! Do a very detailed proof of
concept before you go ahead with this.)
5. Make sure you have good amount of think time between page loads
which can help you avoid page sync failures.

Author: Mark McWhinney
Company: Portata, Inc. (http://www.portata.com)
Text give below is an extract from the white paper created by Mark McWhinney.

The intent of load testing is the same as functional testing or any other type of testing: to find problems before the end users do. However, load testing is not the same as functional testing. It has its own unique issues and its own critical success factors. This white paper describes some of the critical success factors that are unique or are particularly meaningful to load testing.

Planning
As with any project, the planning stage is the most critical to the project’s success. For load testing projects, the planning stage is even more critical. Load testing involves more parts of an organization than any other activity in application development. The people involved often include business owners, developers, architects, QA, database administrators, system administrators, network administrators, data center support staff, various managers, third party organizations and sometimes customers. Planning, coordinating and communicating must be done well to ensure the project’s success.

To further complicate the planning process, load test activities and load test results tend to be nebulous by their nature. Without good planning, it is all too easy to reach the end of a load test, find that the answer is “42″, and realize that you really do not understand what it means.

Goal Setting. The first step in a successful load test planning process is to define the specific goals of the project in objective terms. We find it useful to express the goals in the form of questions whose answer can only be a number or a yes/no. For example, “how many users can run order entry while maintaining 8 second response time or better for 90% of their transactions?”

SEI process. The goal setting step is the first step in the overall planning process that produces the load test plan. Portata uses the SEI Load Test Planning process. See our white paper on this process.

Staff Skills
Load testers need a variety of skills including many skills that functional testers usually do not have.

Experience. Load testers need in-depth knowledge about load testing and how to use the load test tools. The load test team should have at least one member who has run several successful load test projects.

Test Tool Certifications. Some tool vendors have load test tool certifications. Though the certifications are not a substitute for experience, they indicate that the load testers at least understand how the tools work.

Application. Load testing does not require as much knowledge of the application as functional testing does. In functional testing the testers cover every function, but in load testing the testers will execute just a few common paths through the application. However, the load testers should have some operational knowledge of the application to be tested or at least have access to a knowledgeable application user.

Application Usage. The intent of load testing is to emulate what real users will be doing with the application when it is deployed. Therefore, the load testers must also have access to someone who knows how the application is actually used in production or can at least make an informed estimate. That person will supply information about how users typically accomplish a particular task, how often tasks are performed, and how many users perform the tasks.

Systems. Load testers must have a good working knowledge of the software and hardware components of the application. These components may include web servers, application servers, database servers, operating systems, networks and network elements such as load balancers. The load testers need not have “guru” level knowledge of each of the components but should have operational knowledge and an understanding of the performance issues associated with the components. For example, a load tester should know what multi-way joins, indexes and spin counts are and what affect they have on a database server.

Protocols. Load tests are usually driven at the protocol level instead of the GUI level. This allows load generation machines to emulate many users. Load testers should understand the protocol(s) used at the point where the load test tool interfaces with the application to be tested. For web applications this is typically between the web browser and the web server. For load testing web applications a load tester should know both HTTP and HTML. Other commonly used protocols are SQL*Net, ODBC, and DCOM.

Communication. Load testers will work with many parts of an organization to coordinate activities, schedules and resources. Load testing is not a heads down coding exercise. Daily interaction with a variety of people requires good oral and written communication skills as well as good people skills.

If the load testers do not have sufficient communication and people skills, the project will require more management time to handle the planning, coordination and communication.

Support Staff
The load testers must have access to DBAs, application server administrators, network administrators, and system administrators. They are used on an as-needed basis to set up the test environment, provide access to the components, solve problems, and monitor the load test runs.

Load Test Lab
Test Servers. Management usually will not allow load tests to be performed on productions systems since the high load levels would interfere with production activities. Also, production activities may skew the load test results.

Unlike a lab for functional testing, a load test lab must be configured the same as the production environment in terms of component capacity and speed, otherwise the load test may not be accurate or even meaningful. For example, if a production machine has four CPUs then the test lab should be configured with a four CPU test machine.

Often it is cost prohibitive to duplicate the full production system in the test lab. Some possible workarounds are to use the production equipment in the test lab prior to deployment or to use the production infrastructure during off-business hours. Equipment can be rented or leased but there is typically a long leadtime for acquisition and set up so plan accordingly.

Databases. The database(s) in the test lab must be preloaded with either a copy of production data or dummy data that is similar in size and content to the production data. Databases that are too small will tend to give erroneously fast results and can obscure table scan and index problems.

Load Test Tools. Load test tools range from the low-end URL pingers to commercial-grade multi-protocol load generation and monitoring systems. Most applications that are worth load testing include business critical functions and dynamic data, which require the higher end tools.

The important features to have in a load test tool are:

Ability to parameterize data.
Ability to capture dynamic data and use on subsequent requests.
Application infrastructure monitoring.
Support for the application’s protocols.
The load test tools are not inexpensive, so allocate sufficient budget during the budget planning process.

Load Test Monitoring
Response time measurements indicate whether you have a problem and how bad it is, but they do not tell you why you have a problem, what the root cause is, or how to get rid of it. For that information you need to monitor the individual components in the application architecture. The components may include a web server, application server, database server, network and network elements such as firewalls, operating systems and server hardware.

The better load test tools have built-in monitors which measure the various components in real-time and report results verses the load levels applied at the time. If the load test tool that you will be using has the capability to monitor, the tool may require that monitoring agents be installed on the components and/or authorizations be granted to access the components in real time. If the tool does not have this capability, you must make provisions for staff to manually monitor the key components and integrate the results.

Author: James Pulley (www.powertest.com)
Text pulled out from LoadRunner google groups.

Geography is not destiny. Were that so I would likely be sitting on a farm
tractor bringing in a crop of cotton or tobacco. There are incredibly
competent LoadRunner/Performance testing professionals located all over the
world, India included. What draws the ire of many in the profession,
where I am just one individual willing to call it out, is that there are
many people working in the profession who lack the primary skills to be
successful. Yes, we can answer a question of how to lay a brick, so to
speak, but we really should be working with architects who have already
mastered these primary skills for construction. Such dramatically poor
skill bases damages the reputation of the firms providing the deficient
resources. The lack of success in performance testing efforts damages the
reputation of the industry. The lack of success with a given vendors’ tool
damages the reputation of the tool vendor. The low skill base contributes
dramatically increases the risk of deploying a new application into what
should be a very risk-averse discipline.

I should note that those individuals in India who are very skilled in the
performance discipline command rates which are equal to or higher than what
that same person would receive for superlative services anywhere in the
world. The market for solid performance testing skills is fairly uniform
no matter where you acquire those skills. So….for the price leaders
that do not train, do not mentor, toss your employees to the proverbial
performance test wolves and attack the very value foundation of the
profession to seek to draw revenue from, if you would just hire a little
more carefully, invest in proper education and mentoring, adhere to all of
the tenets of the software licenses for the software that you use (even if
you don’t like some of the restrictions) then you too could bill at a much
higher rate than what you currently command and you would have industry
professionals clamoring to be a part of your organization where they could
learn and expand their skills. Recruitment costs down, Bill Rates Up,
Margins Up, Customers better served, Total amounts paid by customer
engagement hold steady or decline due to improved efficiencies, Employees
higher up on Maslow’s hierarchy of needs, Vendors tool not tarnished by poor
implementation skills. Seems like a Win (employee), Win (organization),
Win (customer), Win (vendor), Win (industry) to me.