Changes between Version 4 and Version 5 of AutomatedTesting

Timestamp:

Aug 18, 2010 5:49:58 AM (14 years ago)

Author:

bennylp

Comment:

--

AutomatedTesting

@@ -16 +16 @@
 == Introduction ==
 
-The automated testing script/scenario updates, configures, builds, and performs various unit tests, and submit the result to [http://www.cdash.org CDash] test dashboard. The scenario uses [http://trac2.pjsip.org/ccdash CCDash] framework to automate the tasks, and supports !Windows/Visual Studio build system, GNU Makefile based build system (for Linux, Mac OS X, and possibly iPhone), as well as Symbian targets.
-
-Supported operations:
- - SVN update
- - configure
- - build
- - running unit tests
- - upload the result to CDash
+The automated testing framework automatically and continuously updates, configures, builds, and performs various unit tests, and submit the result to [http://my.cdash.org/index.php?project=PJSIP PJSIP's CDash] test dashboard. The scenario uses [http://trac2.pjsip.org/ccdash CCDash] (client for CDash; note the extra "c") framework to automate the tasks.
 
 Supported targets:
- - GNU/Makefile based (Linux, Mac OS X, iPhone, etc.)
- - Windows or Visual Studio including Windows Mobile
- - Symbian S60
+ - [wiki:Getting-Started/Autoconf GNU/Makefile] desktop based (Linux, Mac OS X, etc.)
+ - [wiki:Getting-Started/Windows Windows] or Visual Studio including [wiki:Getting-Started/Windows-Mobile Windows Mobile]
+ - [wiki:Getting-Started/Symbian Symbian S60] (with/without [wiki:APS]/[wiki:VAS])
+ - [wiki:Getting-Started/iPhone iPhone/iPad]
 
 == Installation ==
-
- 1. Create a CDash project, either by registering your project [http://www.cdash.org/CDash/ here] or by installing CDash locally.
  1. Checkout PJSIP source code from SVN
  1. Download and install [http://trac2.pjsip.org/ccdash CCDash] (click the link for the info)
@@ -39 +31 @@
 == Configuring and Running Test Scenarios ==
 
-The general steps to perform a test:
+The general steps to configure automated test script on a platform:
 
- 1. Choose the appropriate scenario ''template'' for your target/test objective. The templates are listed below:
+ 0. Go to directory '''{{{tests/automated}}}''' inside pjsip distribution.
+ 1. Choose the appropriate scenario ''template'' for your target/test objective. A template is an XML file containing command to perform automated tests for a particular target. The templates are listed below:
 
     || '''Target platform ''' ||  '''Template file name ''' ||  '''Description ''' ||  '''Requirements ''' ||
-    || Makefile based (Linux, Mac OS X, iPhone) || {{{gnu.xml.template}}} || Default settings || - ||
+    || Makefile based (Linux, Mac OS X) || {{{gnu.xml.template}}} || Default settings || - ||
     || Linux, Mac OS X ||  {{{gnu-ipp.xml.template}}} || Intel IPP || [wiki:Intel_IPP_Codecs Intel IPP and samples] ||
     || MSVC/Win32 ||  {{{msvc.xml.template}}} || Default settings || - ||
@@ -50 +43 @@
     || Symbian/S60 3rd Ed ||  {{{symbian-aps.xml.template}}} || Nokia APS and APS-Direct || [wiki:APS Nokia APS plugin] ||
     || Symbian/S60 3rd Ed ||  {{{symbian-vas.xml.template}}} || Nokia VAS and VAS-Direct || [wiki:VAS Nokia VAS plugin] ||
- 2. Create a scenario from a template by running:
+    || iPhone ||  {{{iphone.xml.template}}} || iPhoneOS || one iPhone SDK must be installed ||
+ 2. From the template, you will then create the actual scenario file by running '''{{{configure.py}}}'''. This is needed to ''instantiate'' the template using configurations that are specific to your host. Sample session:
     {{{
      $ cd tests/automated
-     $ ./configure.py file.xml.template -o file.xml
+     $ ./configure.py scenario.xml.template -o scenario.xml
     }}}
- 3. Run the scenario by running:
+ 3. For testing, run the scenario by running:
     {{{
-     $ ./run_scenario.py file.xml
+     $ ./run_scenario.py scenario.xml
     }}}
+    This will perform the automated tests and submit the result under "Experimental" group. Depending on which test is being run, it may take between two minutes to two hours to complete. You may press Ctrl-C to stop the execution.
+ 4. Once you're satisfied that the scenario(s) will work correctly, run this to perform both "Continuous" and "Nightly" builds on the specified scenario(s):
+    {{{
+     $ ./run_continuous.py scenario1.xml [scenario2.xml ...]
+    }}}
+    The script will run the automated tests everytime changes occur in the repository, or at every 00:00:00 GMT for the "Nightly" build.
+ 5. Go to [http://my.cdash.org/index.php?project=PJSIP PJSIP's CDash] test dashboard to check the test results. Note that test results are only submitted once all tests are completed.
+ 6. Now the automated test will run everytime there is a new checkin in the repository, or at every 00:00:00 GMT for the "Nightly" build.
+
 
 The detailed steps for each template will be explained below.
@@ -95 +98 @@
 === iphone.xml.template ===
 
-Template for testing [Getting-Started/iPhone iPhone].
+Template for testing [wiki:Getting-Started/iPhone iPhone].
 
 Requirement(s):