Software Load Test Scripting: Best Practices

1.0 Plan your Business Process scripting effort

Successful performance testing relies on in-depth understanding of the application, the business requirements of the customer and careful planning of the script development phase.

1.1  Gather relevant data
Make sure you get the document that describes the business objectives of the application, maps the customer environment and provides the most appropriate solution at each stage of deployment. This document provides the foundation for the scripting effort and must be created before the script planning and development by the business analysts/Solution Architects.
Understand the organization’s goals
Identify the key IT-enabled business processes to achieve these goals
Identify the target Performance Center users’ Organization Structure (from strategic business champions to technical experts).
Meet the stakeholders
Identify pain points and current state of environment
Map and analyze the tested applications
Define the project deliverables and objectives
Plan the test environment
For further details refer to the Performance Testing Product Best Practices document.

4.1.1    Map the application
Get familiar with the application you’re about to test. Understanding the real user workflows that map together and form a business process is the most important step to creating the script design.   With a thorough script design, the emulation of real user interaction is legitimate, traceable, and logical.  Technical errors are more easily discovered through good test design, and logical errors do not become an issue. Additionally such step might be helpful to detect certain actions that may need special attention (e.g. a web page that often returns an error in a web based application). Interview the system administrator and the application developers (if available) and learn how the application works and its internal architecture. Focus on gathering the following data for each application:
How the applications is being used by its regular users
Application architecture (2/3 tier client/server, ASP/Java, etc)
The internal process of the application and how they work
Protocols used (HTTP, JavaScript/JSP, Tuxedo/JOLT, etc)
Development Source: Vendor or in-house, or combination?
Communication paths to the application (WAN/LAN, link speed, known congestion issues)
Security issues (firewalls, username/passwords etc.)
Data required to interact with the application.

4.2    Create the Performance Testing requirements document
The Performance Testing Requirements document is the cornerstone of any Performance Testing project. It must include precise definitions of what would be achieved during the project implementation and set the Performance Testing strategy. Within the Performance Testing Requirements document should reside Use Case scenarios that define the business processes necessary to emulate the load.  Thus, this document becomes the basis for the script design, and should contain the following aspects:
Business processes that need to be tested.
The flow of each business process.
Key Performance Indicators (KPI)s per business process.  KPIs map what the expected results of the script execution should indicate.  For example, a KPI for a script that emulates a login-logout business process would be the number of seconds that is appropriate for the transaction to be considered acceptable.
Make sure you get the stakeholders approval for the test plan before moving forward. For further details refer to the Performance Testing Product Best Practices document.
4.3    Plan the Performance Testing
Performance testing in a pre-production environment should be the final step before deploying applications into production. One of the significant contributors to the high cost of application testing is duplication of script development efforts.  In order to reduce such inefficiencies, the planning phase should begin weeks before the actual script development and can be performed during other functional testing efforts. In this stage create the detailed Performance testing plan. The plan should cover the following aspects of performance testing:
Decide which functions should be tested.  These functions represent the business processes of real users interacting with the system under test
Identify the required Performance test types. Refer to the Performance Testing Product Best Practices document for a list of commonly used performance test types and choose the ones that make sense in your business environment.
Align the application lifecycle with the available tests. Make sure that the performance tests you have chosen are suitable for the application lifecycle (development, pre-production or production). Refer to the Performance Testing Product Best Practices document for recommended test types per the application lifecycle.
Define in detail the performance test scenarios.
For further details refer to the Performance Testing Product Best Practices document.
4.4    Validate the test environment
4.4.1    Align with the production environment
It is important to run Performance tests against an environment that is as similar as possible to the production environment; otherwise this could lead to inaccurate test results.
The performance test environment requires at least two sets of testing environments:
Production infrastructure
Test environment

Note: Normally most of the performance testing would be done in the test environment; however there are situations where the business requirement would be to run certain performance tests on the production environment to validate its readiness for the next business day.
For further details refer to the Performance Testing Product Best Practices document.

4.4.2    Decide what to script
Analyze the Use Case scenarios and understand the impact they have on the system under test.  Oftentimes, several Use Case scenarios have the same effect when executed, making the scripting of them redundant.  Therefore, it is efficient to identify these redundancies and decide to script only a small subset of the larger set of business processes.  For example, the “login” use case and the “logout” use case both impact the same middleware code and the same database tables, so it may not be necessary to script both processes and execute them.  There are three criteria for deciding what to script:
Volume:  the most common business processes that generate the highest total number of interaction.
Dynamic Content:  The business processes that cause the most workload for the system, especially workload that involves dynamic information to be produced.
Mission Critical:  Not necessarily those business processes which fulfill the above two requirements, but are use cases that are considered the most important aspects of the application usage.
Normally you will find that the Pareto 80-20 rule applies in such situations: 20 percent of the transactions cause 80 percent of the system load. Identifying those 20 percent in the script design phase (and deciding they are the most relevant to your script) ensures that every performance test accurately mimics the load generated by users.

4.5    Scripting design document
Prepare a script design document which explains the Use Case Scenarios and fulfills the framework of the overall Performance Test Plan.  The script design document should contain, at minimum, the following details:
The Business Process name (and the technical script name that represents its emulation).
The Performance Test to which the script is being executed for.
The specific transactions within the script that are being measured for performance (the script may itself be measured, as well as individual steps within it).
The KPIs that will contrast the results of the transactions against expectations.
Technical issues (such as protocol considerations, network considerations, etc).
Environmental configuration (mapped to the Run-Time Settings of the Performance test scenario).
Data considerations

4.5.1    The Business Process and the technical script name
As in any serious development effort keep the program/script names as meaningful as possible. Such approach allows fast identification of relevant scripts and better code re-use. Use the following sample name conventions:

Business process    Script main purpose    Script technical name
Search the portal content    Web transaction: Login, search, log-out    portalSearch_01_1
Customer records update    SAP GUI transaction – update customer records    SAP_custom_rec_02_03

The first part of the script name contains the script protocol and the main transaction that would be performed while the next part has the script revision number (for simple version control).

4.5.2    The Performance Test
Each script should be mapped to the relevant performance test as described in the Performance Testing requirements document. Identify per each performance test the applicable scripts and write it down in the Script design document.
4.5.3    The specific transactions that would be measured
Document the business transactions and their workflow for each script. For better granularity and future code re-use plan to have purpose focused scripts that do a limited number of transactions. Another benefit of such approach is that in case of a problem, it would be easier to identify where the script failed and what area of the application needs fine tuning.
4.5.4    KPIs
Write down the KPIs that would be tested and their expected values per each script such as the expected response time of the GUI during the various transactions.
4.5.5    Technical issues (if applicable)
Document any known technical issues such as protocol-related issues that need to be addressed or avoided during the script development phase.
4.5.6    Environmental configuration
Specify environmental configuration mapped to the Run-Time Settings of the Performance test scenario that would be utilized during the execution of scripts. Mention any special or non-default run-time settings.
4.5.7    Data considerations
Document the data considerations required for running the script: will the test use unique data or the same data within each iteration?  What data will cause the system under test to respond in ways that match the three requirements of deciding which script to emulate?).
4.6    Customer approval
Present samples from the Scripting design document to the stakeholders and the silo experts of the application. Make sure that the silo experts are satisfied with the level of the planned scripts and that the development direction is acceptable.

5    Script development
Before starting the actual scripting development phase make sure that the following documents have been created and approved by the customer:
The Performance testing design document
The Scripting design document
Don’t proceed with any massive scripting development before these documents are completed as it may lead to duplication of development efforts. For instance if no extensive mapping of the business processes was made and no critical transactions have been identified and approved by the stakeholders, most likely that the scripts would need to be completely re-written. In such cases there would be many iterations to align with inputs from the stakeholders and to deal with application aspects that were not covered by the first version of the scripts.
5.1    Record a script
In order to reduce the initial development script cycle, use the Performance Center VuGen technology to record the business process. The scripts produced from the recording must match the test design and use case scenarios.  Adherence to this match ensures that scripts can be reused in the future. The recorded scripts would serve as a framework for further script development.
5.1.1    Script recording best practices
Use the following options to enhance the script recording:
Use the Multi-protocol option to record a test that utilizes several protocols (i.e. Web and Oracle).
Additional manipulation options for Actions such as creating new actions, sorting, renaming and deleting actions are available for selected protocols (such as Web, Oracle NCA, SAP Web, Web Services and others). Older protocols may not have such options supported, therefore check in advance for availability of the action manipulation features for protocols you plan to use.
Record several actions in a sequence (rather then recording one action, then stopping and starting with another action). This ensures continuity of correlation of session IDs and other data being passed in parameters within the script.
Use previously created and tested scripts to create new functionality. Open the proven script, save it under a new name and modify it as needed.
Note that every time a change is made in a script, that script needs to be recompiled.
For further discussion on script recording considerations refer to KBA 28412.

5.2    Script structure considerations
5.2.1    The default script structure
By default, VuGen divides each script into three main blocks:
Vuser_init
This part of the script should hold all the initialization steps required to prepare the system for the transactions specified in the action block. Within the Vuser_init section, all recorded activities are executed once at the beginning of the script execution.  Activities in the Vuser_init are not repeated once executed, even if the script is programmed to iterate. Typical activities allocated to this section are one-time events based on the logic of the script.
Action
Here are located most of the business transactions (Vuser actions) intended for iteration . All the main functions are included here and are transparent to the VuGen user.
Vuser_end
This part holds all actions required to gracefully complete the emulation. Within the Vuser_end section, all recorded activities are executed once and only once at the end of the script execution.  Activities in the Vuser_end are not repeated once executed, even if the script is programmed to iterate. Typical activities allocated to this section are one-time events based on the logic of the script.
For instance, in order to emulate a user’s entire working day which requires logging-into the application in the morning and logging-out at the end of the day, place all the steps emulating logging in and logging out into “Vuser_init” and “Vuser_end” sections respectively. All other actions are performed throughout the working day and are placed in the “Action” section.
The “Action” itself can be broken into different blocks or you can add more actions to the default sections created by VuGen.

5.2.2    Script structure recommendations
Follow the default script structure when you run performance tests using the same user. If several users are emulated you would have to record or script the log-in phase of the next user after the first user has logged-out.
For detailed discussion about how to handle actions, transactions and blocks in VuGen, refer to the Runtime Settings chapter in the VuGen User’s manual.
5.2.2.1    Script structure options
The script must reflect the steps of a business process case but also be capable of interacting with the application in a manner real users would. A common pattern is a Vuser that needs to emulate a login, perform a number of test case iterations, logout and repeat the process again. Sometimes the Vusers are required to login and maintain an open user-session throughout the test run. To accommodate for actual system usage the engineer needs to consider the use (or not) of the Script’s vuser_init and vuser_end sections or even the use of additional functions. The following is a set of possible cases seen so far: