{{{
#!html
<!-- MAIN TABLE START -->
<table border=0 width="90%" align="center"><tr><td>
}}}

= ~~Automated Testing~~ (deprecated) =

**Moved to GitHub CI: https://github.com/pjsip/pjproject**

[[Image(pjsip on cdash.png, align=center, title=PJSIP Dashboard)]]

'''Table of Contents'''
[[PageOutline(2-3,,inline)]]

PJSIP has been equipped with an automated testing script (called test scenarios) since version 1.6. This article explains what it is and how to configure it.

[[BR]]

== Introduction ==

The automated testing framework automatically and continuously updates, configures, builds, and performs various unit tests on PJSIP distribution, and submit the result to [http://my.cdash.org/index.php?project=PJSIP PJSIP's CDash] test dashboard. The scenario uses [https://wush.net/trac/pjsip-labs/ CCDash] (client for CDash; note the extra "c") framework to automate the tasks.

Supported targets:
 - [wiki:Getting-Started/Autoconf GNU/Makefile] desktop based (Linux, Mac OS X, etc.)
 - [wiki:Getting-Started/Windows Windows] or Visual Studio including [wiki:Getting-Started/Windows-Mobile Windows Mobile]
 - [wiki:Getting-Started/Symbian Symbian S60] (with/without [wiki:APS]/[wiki:VAS])
 - [wiki:Getting-Started/iPhone iPhone/iPad]

[[BR]]

== Installation ==
 1. Checkout PJSIP source code from SVN
 1. Download and install [https://wush.net/trac/pjsip-labs/ CCDash] (click the link for the info)
 1. Download and install any features to be used/tested, e.g. Intel IPP, Symbian APS/VAS plugin, etc.

[[BR]]

== Configuring and Running Test Scenarios ==

The general steps to configure automated test script on a platform are as follows:

 0. Go to directory '''{{{tests/automated}}}''' inside pjsip distribution.
 1. Choose the appropriate scenario ''template'' for your target/test objective. A template is an XML file containing ccdash commands to perform automated tests for a particular target. The PJSIP distribution have been equipped with the following templates:

    || '''Target platform ''' ||  '''Template file name ''' ||  '''Description ''' ||  '''Requirements ''' ||
    || Makefile based (Linux, Mac OS X) || {{{gnu.xml.template}}} || Default settings || - ||
    || Linux, Mac OS X with IPP ||  {{{gnu-ipp.xml.template}}} || Intel IPP || [wiki:Intel_IPP_Codecs Intel IPP and samples] ||
    || MSVC/Win32 inc. WM ||  {{{msvc.xml.template}}} || Default settings || - ||
    || Symbian/S60 3rd Ed||  {{{symbian.xml.template}}} || Default settings || - ||
    || Symbian/S60 3rd Ed ||  {{{symbian-aps.xml.template}}} || Nokia APS and APS-Direct || [wiki:APS Nokia APS plugin] ||
    || Symbian/S60 3rd Ed ||  {{{symbian-vas.xml.template}}} || Nokia VAS and VAS-Direct || [wiki:VAS Nokia VAS plugin] ||
    || iPhone ||  {{{iphone.xml.template}}} || iPhoneOS || one iPhone SDK must be installed ||
    || all ||  {{{prepare.xml.template}}} || See [#prep Automatically Regenerating The Scenarios] below || See inside the file ||



 2. From the template, you will then create the actual scenario file by running '''{{{configure.py}}}'''. This is needed to ''instantiate'' the template using configurations that are specific to your host. Sample session:
    {{{
     $ cd tests/automated
     $ ./configure.py -o gnu.xml gnu.xml.template
    }}}
    The above command will produce '''{{{gnu.xml}}}''' scenario file.
 3. For testing, run the scenario by running:
    {{{
     $ ./run_scenario.py gnu.xml
    }}}
    This will perform the automated tests and submit the result under "Experimental" group on CDash. Depending on which test is being run, it may take between two minutes to two hours to complete. You may press Ctrl-C to stop the execution.

== Performing Automated Continuous and Nightly Builds and Tests ==

Once you're satisfied that the scenario(s) will work correctly, the next step is to run a script to automatically do the Continuous and Nightly builds. The '''{{{run_continuous.py}}}''' script continuously monitors the repository for changes, and perform the builds and tests when it does. It also performs Nightly builds at every 00:00:00 GMT.

The '''{{{run_continuous.py}}}''' script takes the scenario file(s) as argument:

{{{
  $ ./run_continuous.py gnu.xml [iphone.xml ...]
}}}

However there is a problem with the above invocation, namely if the scenario template changes, it won't be picked up by {{{run_continuous.py}}}, since we need to re-run {{{configure.py}}} for that. This will be discussed below.


== Automatically Regenerating The Scenarios == #prep

Use {{{prepare.xml.template}}} template scenario to generate {{{prepare.xml}}} scenario. The {{{prepare.xml}}} scenario contains commands to run {{{configure.py}}} for all other scenarios, effectively regenerating them. To use this:
 1. Instantiate {{{prepare.xml}}} from {{{prepare.xml.template}}}:
 {{{
$ ./configure.py -t gnu -o prepare.xml prepare.xml.template
 }}}
 1. Edit the {{{prepare.xml}}} file, and remove scenarios (each scenario is represented as a <Test> item) which you don't want to run.
 1. For each scenario that you '''do''' want to run, make sure the requirements are met. For example, you'd need to set some environment variables before running the {{{prepare.xml}}} scenario.
 1. Run the {{{prepare.xml}}} scenario to regenerate other scenarios:
 {{{
$ ./run_scenario.py prepare.xml
 }}}
 1. Now you can run {{{prepare.xml}}} scenario before other scenarios in {{{run_continuous.py}}}:
{{{
  $ ./run_continuous.py prepare.xml gnu.xml [iphone.xml ...]
}}}
 

[[BR]]

== Appendix: The Templates ==

The detailed steps to configure each template will be explained below.

[[BR]]

=== gnu.xml.template ===

This configures, build, and tests with default settings. To perform this test:
{{{
 $ ./configure.py -t gnu -o gnu.xml gnu.xml.template
 $ ./run_scenario.py gnu.xml
}}}

[[BR]]

=== gnu-ipp.xml.template ===

Template for testing IPP.

Requirements:
 - [wiki:Intel_IPP_Codecs Intel IPP and samples]
 - Environment variables:
    - IPPROOT
    - IPPSAMPLES
    - IPPARCH (optional)
 - IPP path added to LD_LIBRARY_PATH

Sample session:
{{{
 $ export IPPROOT=/opt/intel/ipp/6.1.2.051/em64t/
 $ export IPPSAMPLES=/home/bennylp/Desktop/opt/ipp-samples
 $ (optional) export IPPARCH=em64t
 $ export LD_LIBRARY_PATH='$LD_LIBRARY_PATH:/opt/intel/ipp/6.1.2.051/em64t/sharedlib'
 $ ./configure.py -t gnu -o gnu-ipp.xml gnu-ipp.xml.template
 $ ./run_scenario.py gnu-ipp.xml
}}}


[[BR]]

=== iphone.xml.template ===

Template for testing [wiki:Getting-Started/iPhone iPhone].

Requirement(s):
 - on Mac OS X
 - at least one iPhone SDK must be installed

Sample session:
{{{
 $ ./configure.py -o iphone.xml iphone.xml.template
 $ ./run_scenario.py iphone.xml
}}}

Note:
 - currently there's no way to select which iPhone SDK to use, as the {{{configure.py}}} automatically picks up the latest version of SDK installed.


[[BR]]

=== msvc.xml.template ===

Visual Studio targets.

Requirements:
 1. Visual Studio 2005 (not Express)
 1. Visual Studio has been setup (i.e. you can build PJSIP successfully from Visual Studio)
 1. Visual Studio tools are set in PATH (e.g. cl.exe, vcbuild.exe)

Sample session:
{{{
 C:> python configure.py-o msvc.xml msvc.xml.template
      .. and answer the configure question ..
 C:> python run_scenario.py msvc.xml
}}}


[[BR]]

=== symbian.xml.template ===

Symbian S60 targets. Set EPOCROOT environment variable to point to the SDK location before running configure.py.

Sample session:
{{{
 C:> set EPOCROOT=\S60\devices\S60_5th_Edition_SDK_v1.0
 C:> python configure.py -t s60 -o symbian.xml symbian.xml.template
      .. and answer the configure question ..
 C:> python run_scenario.py symbian.xml
}}}

[[BR]]

=== symbian-aps.xml.template ===

Symbian S60 with APS target.

TBD.


[[BR]]

=== symbian-vas.xml.template ===

Symbian S60 with VAS target.

TBD.



{{{
#!html
<!-- MAIN TABLE END -->
</td></tr></table>
}}}