wiki:AutomatedTesting

Version 5 (modified by bennylp, 14 years ago) (diff)

--

Automated Testing

Table of Contents

  1. Introduction
  2. Installation
  3. Configuring and Running Test Scenarios
    1. gnu.xml.template
    2. gnu-ipp.xml.template
    3. iphone.xml.template
    4. msvc.xml.template
    5. symbian.xml.template
    6. symbian-aps.xml.template
    7. symbian-vas.xml.template

PJSIP has been equipped with an automated testing script (called test scenarios) since version 1.6. This article explains what it is and how to configure it.


Introduction

The automated testing framework automatically and continuously updates, configures, builds, and performs various unit tests, and submit the result to PJSIP's CDash test dashboard. The scenario uses CCDash (client for CDash; note the extra "c") framework to automate the tasks.

Supported targets:

Installation

  1. Checkout PJSIP source code from SVN
  2. Download and install CCDash (click the link for the info)
  3. Download and install any features to be used/tested, e.g. Intel IPP, Symbian APS/VAS plugin, etc.

Configuring and Running Test Scenarios

The general steps to configure automated test script on a platform:

  1. Go to directory tests/automated inside pjsip distribution.
  2. Choose the appropriate scenario template for your target/test objective. A template is an XML file containing command to perform automated tests for a particular target. The templates are listed below:
Target platform Template file name Description Requirements
Makefile based (Linux, Mac OS X) gnu.xml.template Default settings -
Linux, Mac OS X gnu-ipp.xml.template Intel IPP Intel IPP and samples
MSVC/Win32 msvc.xml.template Default settings -
Symbian/S60 3rd Ed symbian.xml.template Default settings -
Symbian/S60 3rd Ed symbian-aps.xml.template Nokia APS and APS-Direct Nokia APS plugin
Symbian/S60 3rd Ed symbian-vas.xml.template Nokia VAS and VAS-Direct Nokia VAS plugin
iPhone iphone.xml.template iPhoneOS one iPhone SDK must be installed
  1. From the template, you will then create the actual scenario file by running configure.py. This is needed to instantiate the template using configurations that are specific to your host. Sample session:
     $ cd tests/automated
     $ ./configure.py scenario.xml.template -o scenario.xml
    
  2. For testing, run the scenario by running:
     $ ./run_scenario.py scenario.xml
    
    This will perform the automated tests and submit the result under "Experimental" group. Depending on which test is being run, it may take between two minutes to two hours to complete. You may press Ctrl-C to stop the execution.
  3. Once you're satisfied that the scenario(s) will work correctly, run this to perform both "Continuous" and "Nightly" builds on the specified scenario(s):
     $ ./run_continuous.py scenario1.xml [scenario2.xml ...]
    
    The script will run the automated tests everytime changes occur in the repository, or at every 00:00:00 GMT for the "Nightly" build.
  4. Go to PJSIP's CDash test dashboard to check the test results. Note that test results are only submitted once all tests are completed.
  5. Now the automated test will run everytime there is a new checkin in the repository, or at every 00:00:00 GMT for the "Nightly" build.

The detailed steps for each template will be explained below.

gnu.xml.template

This configures, build, and tests with default settings. To perform this test:

 $ ./configure.py -t gnu -o gnu.xml gnu.xml.template
 $ ./run_scenario.py gnu.xml

gnu-ipp.xml.template

Template for testing IPP.

Requirements:

  • Intel IPP and samples
  • Environment variables:
    • IPPROOT
    • IPPSAMPLES
    • IPPARCH (optional)
  • IPP path added to LD_LIBRARY_PATH

Sample session:

 $ export IPPROOT=/opt/intel/ipp/6.1.2.051/em64t/
 $ export IPPSAMPLES=/home/bennylp/Desktop/opt/ipp-samples
 $ (optional) export IPPARCH=em64t
 $ export LD_LIBRARY_PATH='$LD_LIBRARY_PATH:/opt/intel/ipp/6.1.2.051/em64t/sharedlib'
 $ ./configure.py -t gnu -o gnu-ipp.xml gnu-ipp.xml.template
 $ ./run_scenario.py gnu-ipp.xml

iphone.xml.template

Template for testing iPhone.

Requirement(s):

  • on Mac OS X
  • at least one iPhone SDK must be installed

Sample session:

 $ ./configure.py -o iphone.xml iphone.xml.template
 $ ./run_scenario.py iphone.xml

Note:

  • currently there's no way to select which iPhone SDK to use, as the configure.py automatically picks up the latest version of SDK installed.

msvc.xml.template

Visual Studio targets.

TBD.

symbian.xml.template

Symbian S60 targets.

TBD.

symbian-aps.xml.template

Symbian S60 with APS target.

TBD.

symbian-vas.xml.template

Symbian S60 with VAS target.

TBD.

Attachments (3)