Changes between Version 4 and Version 5 of AutomatedTesting


Ignore:
Timestamp:
Aug 18, 2010 5:49:58 AM (9 years ago)
Author:
bennylp
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • AutomatedTesting

    v4 v5  
    1616== Introduction == 
    1717 
    18 The automated testing script/scenario updates, configures, builds, and performs various unit tests, and submit the result to [http://www.cdash.org CDash] test dashboard. The scenario uses [http://trac2.pjsip.org/ccdash CCDash] framework to automate the tasks, and supports !Windows/Visual Studio build system, GNU Makefile based build system (for Linux, Mac OS X, and possibly iPhone), as well as Symbian targets. 
    19  
    20 Supported operations: 
    21  - SVN update 
    22  - configure 
    23  - build 
    24  - running unit tests 
    25  - upload the result to CDash 
     18The automated testing framework automatically and continuously updates, configures, builds, and performs various unit tests, and submit the result to [http://my.cdash.org/index.php?project=PJSIP PJSIP's CDash] test dashboard. The scenario uses [http://trac2.pjsip.org/ccdash CCDash] (client for CDash; note the extra "c") framework to automate the tasks. 
    2619 
    2720Supported targets: 
    28  - GNU/Makefile based (Linux, Mac OS X, iPhone, etc.) 
    29  - Windows or Visual Studio including Windows Mobile 
    30  - Symbian S60 
     21 - [wiki:Getting-Started/Autoconf GNU/Makefile] desktop based (Linux, Mac OS X, etc.) 
     22 - [wiki:Getting-Started/Windows Windows] or Visual Studio including [wiki:Getting-Started/Windows-Mobile Windows Mobile] 
     23 - [wiki:Getting-Started/Symbian Symbian S60] (with/without [wiki:APS]/[wiki:VAS]) 
     24 - [wiki:Getting-Started/iPhone iPhone/iPad] 
    3125 
    3226== Installation == 
    33  
    34  1. Create a CDash project, either by registering your project [http://www.cdash.org/CDash/ here] or by installing CDash locally. 
    3527 1. Checkout PJSIP source code from SVN 
    3628 1. Download and install [http://trac2.pjsip.org/ccdash CCDash] (click the link for the info) 
     
    3931== Configuring and Running Test Scenarios == 
    4032 
    41 The general steps to perform a test: 
     33The general steps to configure automated test script on a platform: 
    4234 
    43  1. Choose the appropriate scenario ''template'' for your target/test objective. The templates are listed below: 
     35 0. Go to directory '''{{{tests/automated}}}''' inside pjsip distribution. 
     36 1. Choose the appropriate scenario ''template'' for your target/test objective. A template is an XML file containing command to perform automated tests for a particular target. The templates are listed below: 
    4437 
    4538    || '''Target platform ''' ||  '''Template file name ''' ||  '''Description ''' ||  '''Requirements ''' || 
    46     || Makefile based (Linux, Mac OS X, iPhone) || {{{gnu.xml.template}}} || Default settings || - || 
     39    || Makefile based (Linux, Mac OS X) || {{{gnu.xml.template}}} || Default settings || - || 
    4740    || Linux, Mac OS X ||  {{{gnu-ipp.xml.template}}} || Intel IPP || [wiki:Intel_IPP_Codecs Intel IPP and samples] || 
    4841    || MSVC/Win32 ||  {{{msvc.xml.template}}} || Default settings || - || 
     
    5043    || Symbian/S60 3rd Ed ||  {{{symbian-aps.xml.template}}} || Nokia APS and APS-Direct || [wiki:APS Nokia APS plugin] || 
    5144    || Symbian/S60 3rd Ed ||  {{{symbian-vas.xml.template}}} || Nokia VAS and VAS-Direct || [wiki:VAS Nokia VAS plugin] || 
    52  2. Create a scenario from a template by running: 
     45    || iPhone ||  {{{iphone.xml.template}}} || iPhoneOS || one iPhone SDK must be installed || 
     46 2. From the template, you will then create the actual scenario file by running '''{{{configure.py}}}'''. This is needed to ''instantiate'' the template using configurations that are specific to your host. Sample session: 
    5347    {{{ 
    5448     $ cd tests/automated 
    55      $ ./configure.py file.xml.template -o file.xml 
     49     $ ./configure.py scenario.xml.template -o scenario.xml 
    5650    }}} 
    57  3. Run the scenario by running: 
     51 3. For testing, run the scenario by running: 
    5852    {{{ 
    59      $ ./run_scenario.py file.xml 
     53     $ ./run_scenario.py scenario.xml 
    6054    }}} 
     55    This will perform the automated tests and submit the result under "Experimental" group. Depending on which test is being run, it may take between two minutes to two hours to complete. You may press Ctrl-C to stop the execution. 
     56 4. Once you're satisfied that the scenario(s) will work correctly, run this to perform both "Continuous" and "Nightly" builds on the specified scenario(s): 
     57    {{{ 
     58     $ ./run_continuous.py scenario1.xml [scenario2.xml ...] 
     59    }}} 
     60    The script will run the automated tests everytime changes occur in the repository, or at every 00:00:00 GMT for the "Nightly" build. 
     61 5. Go to [http://my.cdash.org/index.php?project=PJSIP PJSIP's CDash] test dashboard to check the test results. Note that test results are only submitted once all tests are completed. 
     62 6. Now the automated test will run everytime there is a new checkin in the repository, or at every 00:00:00 GMT for the "Nightly" build. 
     63 
    6164 
    6265The detailed steps for each template will be explained below. 
     
    9598=== iphone.xml.template === 
    9699 
    97 Template for testing [Getting-Started/iPhone iPhone]. 
     100Template for testing [wiki:Getting-Started/iPhone iPhone]. 
    98101 
    99102Requirement(s):