Automated Testing

Table of Contents

1. Introduction
2. Installation
3. Configuring and Running Test Scenarios
4. Performing Automated Continuous and Nightly Builds and Tests
5. Automatically Regenerating The Scenarios
6. Appendix: The Templates
   1. gnu.xml.template
   2. gnu-ipp.xml.template
   3. iphone.xml.template
   4. msvc.xml.template
   5. symbian.xml.template
   6. symbian-aps.xml.template
   7. symbian-vas.xml.template

PJSIP has been equipped with an automated testing script (called test scenarios) since version 1.6. This article explains what it is and how to configure it.

Introduction

The automated testing framework automatically and continuously updates, configures, builds, and performs various unit tests on PJSIP distribution, and submit the result to ​PJSIP's CDash test dashboard. The scenario uses ​CCDash (client for CDash; note the extra "c") framework to automate the tasks.

Supported targets:

• GNU/Makefile desktop based (Linux, Mac OS X, etc.)
• Windows or Visual Studio including Windows Mobile
• Symbian S60 (with/without APS/VAS)
• iPhone/iPad

Installation

1. Checkout PJSIP source code from SVN
2. Download and install ​CCDash (click the link for the info)
3. Download and install any features to be used/tested, e.g. Intel IPP, Symbian APS/VAS plugin, etc.

Configuring and Running Test Scenarios

The general steps to configure automated test script on a platform are as follows:

1. Go to directory tests/automated inside pjsip distribution.
2. Choose the appropriate scenario template for your target/test objective. A template is an XML file containing ccdash commands to perform automated tests for a particular target. The PJSIP distribution have been equipped with the following templates:

Target platform	Template file name	Description	Requirements
Makefile based (Linux, Mac OS X)	gnu.xml.template	Default settings	-
Linux, Mac OS X with IPP	gnu-ipp.xml.template	Intel IPP	Intel IPP and samples
MSVC/Win32 inc. WM	msvc.xml.template	Default settings	-
Symbian/S60 3rd Ed	symbian.xml.template	Default settings	-
Symbian/S60 3rd Ed	symbian-aps.xml.template	Nokia APS and APS-Direct	Nokia APS plugin
Symbian/S60 3rd Ed	symbian-vas.xml.template	Nokia VAS and VAS-Direct	Nokia VAS plugin
iPhone	iphone.xml.template	iPhoneOS	one iPhone SDK must be installed
all	prepare.xml.template	See Automatically Regenerating The Scenarios below	See inside the file

2. From the template, you will then create the actual scenario file by running configure.py. This is needed to instantiate the template using configurations that are specific to your host. Sample session:

    $ cd tests/automated
    $ ./configure.py -o gnu.xml gnu.xml.template
   

   The above command will produce gnu.xml scenario file.
3. For testing, run the scenario by running:

    $ ./run_scenario.py gnu.xml
   

   This will perform the automated tests and submit the result under "Experimental" group on CDash. Depending on which test is being run, it may take between two minutes to two hours to complete. You may press Ctrl-C to stop the execution.

Performing Automated Continuous and Nightly Builds and Tests

Once you're satisfied that the scenario(s) will work correctly, the next step is to run a script to automatically do the Continuous and Nightly builds. The run_continuous.py script continuously monitors the repository for changes, and perform the builds and tests when it does. It also performs Nightly builds at every 00:00:00 GMT.

The run_continuous.py script takes the scenario file(s) as argument:

  $ ./run_continuous.py gnu.xml [iphone.xml ...]

However there is a problem with the above invocation, namely if the scenario template changes, it won't be picked up by run_continuous.py, since we need to re-run configure.py for that. This will be discussed below.

Automatically Regenerating The Scenarios

Use prepare.xml.template template scenario to generate prepare.xml scenario. The prepare.xml scenario contains commands to run configure.py for all other scenarios, effectively regenerating them. To use this:

1. Instantiate prepare.xml from prepare.xml.template:

   $ ./configure.py -t gnu -o prepare.xml prepare.xml.template
   

2. Edit the prepare.xml file, and remove scenarios (each scenario is represented as a <Test> item) which you don't want to run.
3. For each scenario that you do want to run, make sure the requirements are met. For example, you'd need to set some environment variables before running the prepare.xml scenario.
4. Run the prepare.xml scenario to regenerate other scenarios:

   $ ./run_scenario.py prepare.xml
   

5. Now you can run prepare.xml scenario before other scenarios in run_continuous.py:

     $ ./run_continuous.py prepare.xml gnu.xml [iphone.xml ...]
   

Appendix: The Templates

The detailed steps to configure each template will be explained below.

gnu.xml.template

This configures, build, and tests with default settings. To perform this test:

 $ ./configure.py -t gnu -o gnu.xml gnu.xml.template
 $ ./run_scenario.py gnu.xml

gnu-ipp.xml.template

Template for testing IPP.

Requirements:

• Intel IPP and samples
• Environment variables:
  • IPPROOT
  • IPPSAMPLES
  • IPPARCH (optional)
• IPP path added to LD_LIBRARY_PATH

Sample session:

 $ export IPPROOT=/opt/intel/ipp/6.1.2.051/em64t/
 $ export IPPSAMPLES=/home/bennylp/Desktop/opt/ipp-samples
 $ (optional) export IPPARCH=em64t
 $ export LD_LIBRARY_PATH='$LD_LIBRARY_PATH:/opt/intel/ipp/6.1.2.051/em64t/sharedlib'
 $ ./configure.py -t gnu -o gnu-ipp.xml gnu-ipp.xml.template
 $ ./run_scenario.py gnu-ipp.xml

iphone.xml.template

Template for testing iPhone.

Requirement(s):

• on Mac OS X
• at least one iPhone SDK must be installed

Sample session:

 $ ./configure.py -o iphone.xml iphone.xml.template
 $ ./run_scenario.py iphone.xml

Note:

• currently there's no way to select which iPhone SDK to use, as the configure.py automatically picks up the latest version of SDK installed.

msvc.xml.template

Visual Studio targets.

Requirements:

1. Visual Studio tools (cl.exe, vcbuild.exe) are set in PATH

Sample session:

 C:> python configure.py-o msvc.xml msvc.xml.template
      .. and answer the configure question ..
 C:> python run_scenario.py msvc.xml

symbian.xml.template

Symbian S60 targets. Set EPOCROOT environment variable to point to the SDK location before running configure.py.

Sample session:

 C:> set EPOCROOT=\S60\devices\S60_5th_Edition_SDK_v1.0
 C:> python configure.py -t s60 -o symbian.xml symbian.xml.template
      .. and answer the configure question ..
 C:> python run_scenario.py symbian.xml

symbian-aps.xml.template

Symbian S60 with APS target.

TBD.

symbian-vas.xml.template

Symbian S60 with VAS target.

TBD.