Changes between Version 5 and Version 6 of AutomatedTesting


Ignore:
Timestamp:
Aug 18, 2010 5:58:55 AM (9 years ago)
Author:
bennylp
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • AutomatedTesting

    v5 v6  
    1616== Introduction == 
    1717 
    18 The automated testing framework automatically and continuously updates, configures, builds, and performs various unit tests, and submit the result to [http://my.cdash.org/index.php?project=PJSIP PJSIP's CDash] test dashboard. The scenario uses [http://trac2.pjsip.org/ccdash CCDash] (client for CDash; note the extra "c") framework to automate the tasks. 
     18The automated testing framework automatically and continuously updates, configures, builds, and performs various unit tests on PJSIP distribution, and submit the result to [http://my.cdash.org/index.php?project=PJSIP PJSIP's CDash] test dashboard. The scenario uses [http://trac2.pjsip.org/ccdash CCDash] (client for CDash; note the extra "c") framework to automate the tasks. 
    1919 
    2020Supported targets: 
     
    2424 - [wiki:Getting-Started/iPhone iPhone/iPad] 
    2525 
     26[[BR]] 
     27 
    2628== Installation == 
    2729 1. Checkout PJSIP source code from SVN 
     
    2931 1. Download and install any features to be used/tested, e.g. Intel IPP, Symbian APS/VAS plugin, etc. 
    3032 
     33[[BR]] 
     34 
    3135== Configuring and Running Test Scenarios == 
    3236 
    33 The general steps to configure automated test script on a platform: 
     37The general steps to configure automated test script on a platform are as follows: 
    3438 
    3539 0. Go to directory '''{{{tests/automated}}}''' inside pjsip distribution. 
    36  1. Choose the appropriate scenario ''template'' for your target/test objective. A template is an XML file containing command to perform automated tests for a particular target. The templates are listed below: 
     40 1. Choose the appropriate scenario ''template'' for your target/test objective. A template is an XML file containing ccdash commands to perform automated tests for a particular target. The PJSIP distribution have been equipped with the following templates: 
    3741 
    3842    || '''Target platform ''' ||  '''Template file name ''' ||  '''Description ''' ||  '''Requirements ''' || 
    3943    || Makefile based (Linux, Mac OS X) || {{{gnu.xml.template}}} || Default settings || - || 
    40     || Linux, Mac OS X ||  {{{gnu-ipp.xml.template}}} || Intel IPP || [wiki:Intel_IPP_Codecs Intel IPP and samples] || 
    41     || MSVC/Win32 ||  {{{msvc.xml.template}}} || Default settings || - || 
     44    || Linux, Mac OS X with IPP ||  {{{gnu-ipp.xml.template}}} || Intel IPP || [wiki:Intel_IPP_Codecs Intel IPP and samples] || 
     45    || MSVC/Win32 inc. WM ||  {{{msvc.xml.template}}} || Default settings || - || 
    4246    || Symbian/S60 3rd Ed||  {{{symbian.xml.template}}} || Default settings || - || 
    4347    || Symbian/S60 3rd Ed ||  {{{symbian-aps.xml.template}}} || Nokia APS and APS-Direct || [wiki:APS Nokia APS plugin] || 
     
    4953     $ ./configure.py scenario.xml.template -o scenario.xml 
    5054    }}} 
     55    The above command will produce '''{{{scenario.xml}}}''' scenario file. 
    5156 3. For testing, run the scenario by running: 
    5257    {{{ 
    5358     $ ./run_scenario.py scenario.xml 
    5459    }}} 
    55     This will perform the automated tests and submit the result under "Experimental" group. Depending on which test is being run, it may take between two minutes to two hours to complete. You may press Ctrl-C to stop the execution. 
    56  4. Once you're satisfied that the scenario(s) will work correctly, run this to perform both "Continuous" and "Nightly" builds on the specified scenario(s): 
     60    This will perform the automated tests and submit the result under "Experimental" group on CDash. Depending on which test is being run, it may take between two minutes to two hours to complete. You may press Ctrl-C to stop the execution. 
     61 4. Once you're satisfied that the scenario(s) will work correctly, run '''{{{run_continuous.py}}}''' to perform both "Continuous" and "Nightly" builds on the specified scenario(s): 
    5762    {{{ 
    5863     $ ./run_continuous.py scenario1.xml [scenario2.xml ...] 
    5964    }}} 
    60     The script will run the automated tests everytime changes occur in the repository, or at every 00:00:00 GMT for the "Nightly" build. 
    61  5. Go to [http://my.cdash.org/index.php?project=PJSIP PJSIP's CDash] test dashboard to check the test results. Note that test results are only submitted once all tests are completed. 
     65    The script will run the automated tests in the scenario(s) everytime changes occur in the repository, or at every 00:00:00 GMT for the "Nightly" build. 
     66 5. Go to [http://my.cdash.org/index.php?project=PJSIP PJSIP's CDash] test dashboard to check the test results. Note that test results are only submitted once all tests in a scenario are completed. 
    6267 6. Now the automated test will run everytime there is a new checkin in the repository, or at every 00:00:00 GMT for the "Nightly" build. 
    6368 
    6469 
    65 The detailed steps for each template will be explained below. 
     70[[BR]] 
     71 
     72== The Templates == 
     73 
     74The detailed steps to configure each template will be explained below. 
     75 
     76[[BR]] 
    6677 
    6778=== gnu.xml.template === 
     
    7283 $ ./run_scenario.py gnu.xml 
    7384}}} 
     85 
     86[[BR]] 
    7487 
    7588=== gnu-ipp.xml.template === 
     
    96109 
    97110 
     111[[BR]] 
     112 
    98113=== iphone.xml.template === 
    99114 
     
    114129 
    115130 
     131[[BR]] 
     132 
    116133=== msvc.xml.template === 
    117134 
     
    121138 
    122139 
     140[[BR]] 
     141 
    123142=== symbian.xml.template === 
    124143 
     
    126145 
    127146TBD. 
     147 
     148[[BR]] 
    128149 
    129150=== symbian-aps.xml.template === 
     
    133154TBD. 
    134155 
     156 
     157[[BR]] 
    135158 
    136159=== symbian-vas.xml.template ===