Index: docs/manual/Makefile ================================================================== --- docs/manual/Makefile +++ docs/manual/Makefile @@ -30,15 +30,15 @@ # asciidoc -b html5 -a icons -a iconsdir=$(DISPATH)/images/icons -a toc2 design_spec.txt # all : server.ps megatest_manual.html client.ps complex-itemmap.png -megatest_manual.html : megatest_manual.txt $(INPAGES) writing_tests.txt installation.txt *png +megatest_manual.html : megatest_manual.txt *.txt installation.txt *png asciidoc -b html5 -a icons -a iconsdir=$(DISPATH)/images/icons -a toc2 megatest_manual.txt # dos2unix megatest_manual.html -megatest.pdf : megatest_manual.txt $(INPAGES) writing_tests.txt *png +megatest.pdf : megatest_manual.txt *.txt *png a2x -a toc -f pdf megatest_manual.txt server.ps : server.dot dot -Tps server.dot > server.ps DELETED docs/manual/getting_started.in Index: docs/manual/getting_started.in ================================================================== --- docs/manual/getting_started.in +++ /dev/null @@ -1,115 +0,0 @@ -// This file is part of Megatest. -// -// Megatest is free software: you can redistribute it and/or modify -// it under the terms of the GNU General Public License as published by -// the Free Software Foundation, either version 3 of the License, or -// (at your option) any later version. -// -// Megatest is distributed in the hope that it will be useful, -// but WITHOUT ANY WARRANTY; without even the implied warranty of -// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -// GNU General Public License for more details. -// -// You should have received a copy of the GNU General Public License -// along with Megatest. If not, see . -// -// Copyright 2006-2012, Matthew Welland. - -Getting Started ---------------- - -[partintro] -.Getting started with Megatest --- -Creating a testsuite or flow and your first test or task. --- - -After installing Megatest you can create a flow or testsuite and add some -tests using the helpers. Here is a quickstart sequence to get you up and -running your first automated testsuite. - -Creating a Megatest Area -~~~~~~~~~~~~~~~~~~~~~~~~ - -Choose Target Keys -^^^^^^^^^^^^^^^^^^ - -First choose your "target" keys. These are used to organise your runs in a -way that is meaningful to your project. If you are unsure about what to use -for keys just use a single generic key such as "RUNTYPE". These keys will be -used to hand values to your tests via environment variables so ensure they -are unique. Prefixing them with something such as PROJKEYS_ is a good -strategy. - -Examples of keys: - -.Example keys -[width="60%",options="header"] -|============================================== -| Option | Description -| RELEASE/ITERATION | This example is used by Megatest for its internal QA. -| ARCH/OS/RELEASE | For a software project targeting multiple platforms -| UCTRLR/NODETYPE | Microcontroller project with different controllers -running same software -|============================================== - -Create Area Config Files -^^^^^^^^^^^^^^^^^^^^^^^^ - -You will need to choose locations for your runs (the data generated every -time you run the testsuite) and link tree. For getting started answer the -prompts with "runs" and "links". We use the Unix editor "vi" in the examples -below but you can use any plain text editor. - -.Using the helper to create a Megatest area ------------------- -megatest -create-megatest-area - -# optional: verify that the settings are ok -vi megatest.config -vi runconfigs.config ------------------- - -Creating a Test -~~~~~~~~~~~~~~~ - -Choose the test name for your first test and run the helper. You can edit -the files after the initial creation. You will need to enter names and -scripts for the steps to be run and then edit the -tests//testconfig file and modify the logpro rules to properly -process the log output from your steps. For your first test just hit enter -for the "waiton", "priority" and iteration variable prompts. - -Hint: for geting started make your logpro rules very liberal. expect:error -patterns should match nothing and comment out expect:required rules. - -.Using the helper to create a Megatest test ---------------- -megatest -create-test myfirsttest - -# then edit the generated config -vi tests/myfirsttest/testconfig ---------------- - -Running your test -~~~~~~~~~~~~~~~~~ - -First choose a target and runname. If you have a two-place target such as -RELEASE/ITERATION a target would look like v1.0/aff3 where v1.0 is the -RELEASE and aff3 is the ITERATION. For a run name just use something like -run1. - -.Running all tests (testpatt of "%" matches all tests) ---------------- -megatest -run -target v1.0/aff3 -runname run1 -testpatt % -log run1.log ---------------- - -Viewing the results -~~~~~~~~~~~~~~~~~~~ - -Start the dashboard and browse your run in the "Runs" tab. - -.Starting dashboard ----------------- -dashboard -rows 24 ----------------- ADDED docs/manual/getting_started.txt Index: docs/manual/getting_started.txt ================================================================== --- /dev/null +++ docs/manual/getting_started.txt @@ -0,0 +1,115 @@ +// This file is part of Megatest. +// +// Megatest is free software: you can redistribute it and/or modify +// it under the terms of the GNU General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// Megatest is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU General Public License for more details. +// +// You should have received a copy of the GNU General Public License +// along with Megatest. If not, see . +// +// Copyright 2006-2012, Matthew Welland. + +Getting Started +--------------- + +[partintro] +.Getting started with Megatest +-- +Creating a testsuite or flow and your first test or task. +-- + +After installing Megatest you can create a flow or testsuite and add some +tests using the helpers. Here is a quickstart sequence to get you up and +running your first automated testsuite. + +Creating a Megatest Area +~~~~~~~~~~~~~~~~~~~~~~~~ + +Choose Target Keys +^^^^^^^^^^^^^^^^^^ + +First choose your "target" keys. These are used to organise your runs in a +way that is meaningful to your project. If you are unsure about what to use +for keys just use a single generic key such as "RUNTYPE". These keys will be +used to hand values to your tests via environment variables so ensure they +are unique. Prefixing them with something such as PROJKEYS_ is a good +strategy. + +Examples of keys: + +.Example keys +[width="60%",options="header"] +|============================================== +| Option | Description +| RELEASE/ITERATION | This example is used by Megatest for its internal QA. +| ARCH/OS/RELEASE | For a software project targeting multiple platforms +| UCTRLR/NODETYPE | Microcontroller project with different controllers +running same software +|============================================== + +Create Area Config Files +^^^^^^^^^^^^^^^^^^^^^^^^ + +You will need to choose locations for your runs (the data generated every +time you run the testsuite) and link tree. For getting started answer the +prompts with "runs" and "links". We use the Unix editor "vi" in the examples +below but you can use any plain text editor. + +.Using the helper to create a Megatest area +------------------ +megatest -create-megatest-area + +# optional: verify that the settings are ok +vi megatest.config +vi runconfigs.config +------------------ + +Creating a Test +~~~~~~~~~~~~~~~ + +Choose the test name for your first test and run the helper. You can edit +the files after the initial creation. You will need to enter names and +scripts for the steps to be run and then edit the +tests//testconfig file and modify the logpro rules to properly +process the log output from your steps. For your first test just hit enter +for the "waiton", "priority" and iteration variable prompts. + +Hint: for geting started make your logpro rules very liberal. expect:error +patterns should match nothing and comment out expect:required rules. + +.Using the helper to create a Megatest test +--------------- +megatest -create-test myfirsttest + +# then edit the generated config +vi tests/myfirsttest/testconfig +--------------- + +Running your test +~~~~~~~~~~~~~~~~~ + +First choose a target and runname. If you have a two-place target such as +RELEASE/ITERATION a target would look like v1.0/aff3 where v1.0 is the +RELEASE and aff3 is the ITERATION. For a run name just use something like +run1. + +.Running all tests (testpatt of "%" matches all tests) +--------------- +megatest -run -target v1.0/aff3 -runname run1 -testpatt % -log run1.log +--------------- + +Viewing the results +~~~~~~~~~~~~~~~~~~~ + +Start the dashboard and browse your run in the "Runs" tab. + +.Starting dashboard +---------------- +dashboard -rows 24 +---------------- DELETED docs/manual/howto.in Index: docs/manual/howto.in ================================================================== --- docs/manual/howto.in +++ /dev/null @@ -1,208 +0,0 @@ -// This file is part of Megatest. -// -// Megatest is free software: you can redistribute it and/or modify -// it under the terms of the GNU General Public License as published by -// the Free Software Foundation, either version 3 of the License, or -// (at your option) any later version. -// -// Megatest is distributed in the hope that it will be useful, -// but WITHOUT ANY WARRANTY; without even the implied warranty of -// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -// GNU General Public License for more details. -// -// You should have received a copy of the GNU General Public License -// along with Megatest. If not, see . -// -// Copyright 2006-2012, Matthew Welland. - -How To Do Things ----------------- - -Process Runs -~~~~~~~~~~~~ - -Remove Runs -^^^^^^^^^^^ - -From the dashboard click on the button (PASS/FAIL...) for one of the tests. From the test control panel that -comes up push the clean test button. The command field will be prefilled with a template command for removing -that test. You can edit the command, for example change the argument to -testpatt to "%" to remove all tests. - -.Remove the test diskperf and all it's items ----------------- -megatest -remove-runs -target ubuntu/nfs/none -runname ww28.1a -testpatt diskperf/% -v ----------------- - -.Remove all tests for all runs and all targets ----------------- -megatest -remove-runs -target %/%/% -runname % -testpatt % -v ----------------- - -Archive Runs -^^^^^^^^^^^^ - -Megatest supports using the bup backup tool (https://bup.github.io/) to archive your tests for efficient storage -and retrieval. Archived data can be rapidly retrieved if needed. The metadata for the run (PASS/FAIL status, run -durations, time stamps etc.) are all preserved in the megatest database. - -For setup information see the Archiving topic in the reference section of this manual. - -To Archive -++++++++++ - -Hint: use the test control panel to create a template command by pushing the "Archive Tests" button. - -.Archive a full run ----------------- -megatest -target ubuntu/nfs/none -runname ww28.1a -archive save-remove -testpatt % ----------------- - -To Restore -++++++++++ - -.Retrieve a single test ----------------- -megatest -target ubuntu/nfs/none -runname ww28.1a -archive restore -testpatt diskperf/% ----------------- - -Hint: You can browse the archive using bup commands directly. - ----------------- -bup -d /path/to/bup/archive ftp ----------------- - -Submit jobs to Host Types based on Test Name -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.In megatest.config ------------------------- -[host-types] -general ssh #{getbgesthost general} -nbgeneral nbjob run JOBCOMMAND -log $MT_LINKTREE/$MT_TARGET/$MT_RUNNAME.$MT_TESTNAME-$MT_ITEM_PATH.lgo - -[hosts] -general cubian xena - -[launchers] -envsetup general -xor/%/n 4C16G -% nbgeneral - -[jobtools] -launcher bsub -# if defined and not "no" flexi-launcher will bypass launcher unless there is no -# match. -flexi-launcher yes ------------------------- - -Tricks ------- - -This section is a compendium of a various useful tricks for debugging, -configuring and generally getting the most out of Megatest. - -Limiting your running jobs -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The following example will limit a test in the jobgroup "group1" to no more than 10 tests simultaneously. - -In your testconfig: - ----------------- -[test_meta] -jobgroup group1 ----------------- - -In your megatest.config: - ---------------- -[jobgroups] -group1 10 -custdes 4 ---------------- - -Debugging Tricks ----------------- - -Examining The Environment -~~~~~~~~~~~~~~~~~~~~~~~~~ - -Test Control Panel - xterm -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -From the dashboard click on a test PASS/FAIL button. This brings up a test control panel. Aproximately near the center left of the -window there is a button "Start Xterm". Push this to get an xterm with the full context and environment loaded for that test. You can run -scripts or ezsteps by copying from the testconfig (hint, load up the testconfig in a separate gvim or emacs window). This is the easiest way -to debug your tests. - -During Config File Processing -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -It is often helpful to know the content of variables in various -contexts as Megatest does the actions needed to run your tests. A handy technique is to force the startup of an xterm in the context being examined. - -For example, if an item list is not being generated as expected you -can inject the startup of an xterm as if it were an item: - -.Original items table ------------------ -[items] -CELLNAME [system getcellname.sh] ------------------ - -.Items table modified for debug ------------------ -[items] -DEBUG [system xterm] -CELLNAME [system getcellnames.sh] ------------------ - -When this test is run an xterm will pop up. In that xterm the -environment is exactly that in which the script "getcellnames.sh" -would run. You can now debug the script to find out why it isn't -working as expected. - -Organising Your Tests and Tasks -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The default location "tests" for storing tests can be extended by -adding to your tests-paths section. - ----------------------------- -[misc] -parent #{shell dirname $(readlink -f .)} - -[tests-paths] -1 #{get misc parent}/simplerun/tests ----------------------------- - -The above example shows how you can use addition sections in your -config file to do complex processing. By putting results of relatively -slow operations into variables the processing of your configs can be -kept fast. - -Alternative Method for Running your Job Script -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.Directly running job in testconfig -------------------- -[setup] -runscript main.csh -------------------- - -The runscript method is essentially a brute force way to run scripts where the -user is responsible for setting STATE and STATUS and managing the details of running a test. - -Debugging Server Problems -~~~~~~~~~~~~~~~~~~~~~~~~~ - -Some handy Unix commands to track down issues with servers not -communicating with your test manager processes. Please put in tickets -at https://www.kiatoa.com/fossils/megatest if you have problems with -servers getting stuck. - ----------------- -sudo lsof -i -sudo netstat -lptu -sudo netstat -tulpn ----------------- ADDED docs/manual/howto.txt Index: docs/manual/howto.txt ================================================================== --- /dev/null +++ docs/manual/howto.txt @@ -0,0 +1,208 @@ +// This file is part of Megatest. +// +// Megatest is free software: you can redistribute it and/or modify +// it under the terms of the GNU General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// Megatest is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU General Public License for more details. +// +// You should have received a copy of the GNU General Public License +// along with Megatest. If not, see . +// +// Copyright 2006-2012, Matthew Welland. + +How To Do Things +---------------- + +Process Runs +~~~~~~~~~~~~ + +Remove Runs +^^^^^^^^^^^ + +From the dashboard click on the button (PASS/FAIL...) for one of the tests. From the test control panel that +comes up push the clean test button. The command field will be prefilled with a template command for removing +that test. You can edit the command, for example change the argument to -testpatt to "%" to remove all tests. + +.Remove the test diskperf and all it's items +---------------- +megatest -remove-runs -target ubuntu/nfs/none -runname ww28.1a -testpatt diskperf/% -v +---------------- + +.Remove all tests for all runs and all targets +---------------- +megatest -remove-runs -target %/%/% -runname % -testpatt % -v +---------------- + +Archive Runs +^^^^^^^^^^^^ + +Megatest supports using the bup backup tool (https://bup.github.io/) to archive your tests for efficient storage +and retrieval. Archived data can be rapidly retrieved if needed. The metadata for the run (PASS/FAIL status, run +durations, time stamps etc.) are all preserved in the megatest database. + +For setup information see the Archiving topic in the reference section of this manual. + +To Archive +++++++++++ + +Hint: use the test control panel to create a template command by pushing the "Archive Tests" button. + +.Archive a full run +---------------- +megatest -target ubuntu/nfs/none -runname ww28.1a -archive save-remove -testpatt % +---------------- + +To Restore +++++++++++ + +.Retrieve a single test +---------------- +megatest -target ubuntu/nfs/none -runname ww28.1a -archive restore -testpatt diskperf/% +---------------- + +Hint: You can browse the archive using bup commands directly. + +---------------- +bup -d /path/to/bup/archive ftp +---------------- + +Submit jobs to Host Types based on Test Name +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.In megatest.config +------------------------ +[host-types] +general ssh #{getbgesthost general} +nbgeneral nbjob run JOBCOMMAND -log $MT_LINKTREE/$MT_TARGET/$MT_RUNNAME.$MT_TESTNAME-$MT_ITEM_PATH.lgo + +[hosts] +general cubian xena + +[launchers] +envsetup general +xor/%/n 4C16G +% nbgeneral + +[jobtools] +launcher bsub +# if defined and not "no" flexi-launcher will bypass launcher unless there is no +# match. +flexi-launcher yes +------------------------ + +Tricks +------ + +This section is a compendium of a various useful tricks for debugging, +configuring and generally getting the most out of Megatest. + +Limiting your running jobs +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The following example will limit a test in the jobgroup "group1" to no more than 10 tests simultaneously. + +In your testconfig: + +---------------- +[test_meta] +jobgroup group1 +---------------- + +In your megatest.config: + +--------------- +[jobgroups] +group1 10 +custdes 4 +--------------- + +Debugging Tricks +---------------- + +Examining The Environment +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Test Control Panel - xterm +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +From the dashboard click on a test PASS/FAIL button. This brings up a test control panel. Aproximately near the center left of the +window there is a button "Start Xterm". Push this to get an xterm with the full context and environment loaded for that test. You can run +scripts or ezsteps by copying from the testconfig (hint, load up the testconfig in a separate gvim or emacs window). This is the easiest way +to debug your tests. + +During Config File Processing +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +It is often helpful to know the content of variables in various +contexts as Megatest does the actions needed to run your tests. A handy technique is to force the startup of an xterm in the context being examined. + +For example, if an item list is not being generated as expected you +can inject the startup of an xterm as if it were an item: + +.Original items table +----------------- +[items] +CELLNAME [system getcellname.sh] +----------------- + +.Items table modified for debug +----------------- +[items] +DEBUG [system xterm] +CELLNAME [system getcellnames.sh] +----------------- + +When this test is run an xterm will pop up. In that xterm the +environment is exactly that in which the script "getcellnames.sh" +would run. You can now debug the script to find out why it isn't +working as expected. + +Organising Your Tests and Tasks +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The default location "tests" for storing tests can be extended by +adding to your tests-paths section. + +---------------------------- +[misc] +parent #{shell dirname $(readlink -f .)} + +[tests-paths] +1 #{get misc parent}/simplerun/tests +---------------------------- + +The above example shows how you can use addition sections in your +config file to do complex processing. By putting results of relatively +slow operations into variables the processing of your configs can be +kept fast. + +Alternative Method for Running your Job Script +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.Directly running job in testconfig +------------------- +[setup] +runscript main.csh +------------------- + +The runscript method is essentially a brute force way to run scripts where the +user is responsible for setting STATE and STATUS and managing the details of running a test. + +Debugging Server Problems +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Some handy Unix commands to track down issues with servers not +communicating with your test manager processes. Please put in tickets +at https://www.kiatoa.com/fossils/megatest if you have problems with +servers getting stuck. + +---------------- +sudo lsof -i +sudo netstat -lptu +sudo netstat -tulpn +---------------- Index: docs/manual/megatest_manual.html ================================================================== --- docs/manual/megatest_manual.html +++ docs/manual/megatest_manual.html @@ -1359,10 +1359,190 @@
dashboard -rows 24
+ +
+

Study Plan

+
+

Megatest is an extensive program with a lot to learn. Following are some paths through the material to smooth the learning path.

+
+

Basic Concepts (suggest you pick these up on the way)

+
    +
  • +

    +Components of automation; run, test, iteration +

    +
  • +
  • +

    +Selectors; target, runname, and testpatt +

    +
  • +
+
+
+

Running Testsuites or Automation

+
    +
  • +

    +Using the dashboard gui (recommended) +

    +
      +
    • +

      +Using the "Runs" panel. +

      +
    • +
    • +

      +Using the "Run Control" panel. +

      +
    • +
    • +

      +Using a test control panel +

      +
    • +
    • +

      +The Right Mouse Button menu +

      +
    • +
    • +

      +Debug features +

      +
        +
      • +

        +xterm +

        +
          +
        • +

          +pstree +

          +
        • +
        • +

          +log files; mt_copy.log, mt_launch.log +

          +
        • +
        • +

          +variables; megatest.csh, megatest.sh +

          +
        • +
        • +

          +testconfig dump, *testconfig +

          +
        • +
        +
      • +
      • +

        +State/status buttons +

        +
      • +
      • +

        +Run, Clean, KillReq +

        +
      • +
      • +

        +ReRunClean +

        +
      • +
      +
    • +
    +
  • +
  • +

    +Using the command line +

    +
      +
    • +

      +Getting help; megatest -h, megatest -manual +

      +
    • +
    • +

      +Starting runs; megatest -run +

      +
        +
      • +

        +Selection controls; -target, -runname and -testpatt +

        +
      • +
      +
    • +
    +
  • +
+
+
+

Writing Tests and Flows

+
+
+
+

Advanced Topics

+
+
+

Writing Tests

@@ -2009,10 +2189,30 @@
runscript main.csh
+
+
+

Iteration

+
+
Sections for iteration
+
+
# full combinations
+[items]
+A x y
+B 1 2
+
+# Yields: x/1 x/2 y/1 y/2
+
+# tabled
+[itemstable]
+A x y
+B 1 2
+
+# Yields x/1 y/2
+

Requirements section

Header
@@ -2048,11 +2248,10 @@ was historically called "itemwait" mode. The terms "itemwait" and "itemmatch" are synonyms.

[requirements]
-waiton precedingtest
 mode itemmatch
@@ -2071,18 +2270,17 @@

Itemmap Handling

For cases were the dependent test has a similar but not identical -itempath to the downstream test an itemmap can allow for itemwait/itemmatch -mode by specifying rules on how to map the waiton’s item name to the current test’s item name.

+itempath to the downstream test an itemmap can allow for itemmatch +mode

example for removing part of itemmap for waiton test (eg: item foo-x/bar depends on waiton’s item y/bar)
[requirements]
 mode itemwait
-waiton precedingtest
 # itemmap <item pattern for this test>  <item replacement pattern for waiton test>
 itemmap .*x/ y/
example for removing part of itemmap for waiton test (eg: item foo/bar/baz in this test depends on waiton’s item baz)
@@ -2091,11 +2289,10 @@ # # ## Example # ## Remove everything up to the last / [requirements] mode itemwait -waiton precedingtest # itemmap <item pattern for this test> <nothing here indicates removal> itemmap .*/
example replacing part of itemmap for (eg: item foo/1234 will imply waiton’s item bar/1234)
@@ -2103,11 +2300,10 @@
#
 # ## Example
 # ## Replace foo/ with bar/
 [requirements]
 mode itemwait
-waiton precedingtest
 # itemmap <item pattern for this test>  <item replacement pattern for waiton test>
 itemmap foo/ bar/
example for backreference (eg: item foo23/thud will imply waiton’s item num-23/bar/thud
@@ -2115,11 +2311,10 @@
#
 # ## Example
 # ## can use \{number} in replacement pattern to backreference a (capture) from matching pattern similar to sed or perl
 [requirements]
 mode itemwait
-waiton precedingtest
 # itemmap <item pattern for this test>  <item replacement pattern for waiton test>
 itemmap foo(\d+)/ num-\1/bar/
example multiple itemmaps
@@ -2127,12 +2322,10 @@
# multi-line; matches are applied in the listed order
 # The following would map:
 #   a123b321 to b321fooa123 then to 321fooa123p
 #
 [requirements]
-mode itemwait
-waiton precedingtest
 itemmap (a\d+)(b\d+) \2foo\1
   b(.*) \1p

Complex mapping

@@ -2157,27 +2350,24 @@
Testconfig for Test E has
[requirements]
 waiton C
-mode itemwait
 itemmap (\d+)/res \1/bb
Testconfig for Test D has
[requirements]
 waiton C
-mode itemwait
 itemmap (\d+)/res \1/aa
Testconfig for Test C has
[requirements]
 waiton A B
-mode itemwait
 
 [itemmap]
 A (\d+)/aa aa/\1
 B (\d+)/bb bb/\1
@@ -2662,10 +2852,10 @@

Index: docs/manual/megatest_manual.txt ================================================================== --- docs/manual/megatest_manual.txt +++ docs/manual/megatest_manual.txt @@ -82,24 +82,26 @@ the distributed compute platform in use. A template script is provided which can launch jobs on local and remote Linux hosts. Currently megatest uses the network filesystem to call home to your master sqlite3 database. -include::plan.in[] +include::plan.txt[] // to allow the getting_started.txt to be a stand-alone document use level shifting, note that the preceding blank line is needed. // :leveloffset: 2 include::installation.txt[] -include::getting_started.in[] +include::getting_started.txt[] + +include::study_plan.txt[] :leveloffset: 0 include::writing_tests.txt[] -include::howto.in[] -include::reference.in[] +include::howto.txt[] +include::reference.txt[] Megatest Internals ------------------ ["graphviz", "server.png"] DELETED docs/manual/plan.in Index: docs/manual/plan.in ================================================================== --- docs/manual/plan.in +++ /dev/null @@ -1,232 +0,0 @@ -Road Map --------- - -// This file is part of Megatest. -// -// Megatest is free software: you can redistribute it and/or modify -// it under the terms of the GNU General Public License as published by -// the Free Software Foundation, either version 3 of the License, or -// (at your option) any later version. -// -// Megatest is distributed in the hope that it will be useful, -// but WITHOUT ANY WARRANTY; without even the implied warranty of -// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -// GNU General Public License for more details. -// -// You should have received a copy of the GNU General Public License -// along with Megatest. If not, see . - -// Copyright 2006-2012, Matthew Welland. - -Note 1: This road-map continues to evolve and subject to change without notice. - -Here is a smattering of ideas for Megatest 2.0 -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -. Add variable $MT_RUNPATH = $MT_LINKTREE/$MT_TARGET/$MT_RUNNAME ----------------- -[db] -api legacy|new ----------------- -. One big lesson from the 1.63-1.65 generation was that the main.db, 1.db … model was really good at scaling. I’d like to combine that model with the current also-very-good model. Obviously this is a disruptive change. I think making the old model the default and the new model an option for at least one generation would be fair. -. Rigorous megatest.config and runconfig.config caching. -.. Cache the configs in $MT_RUNPATH -.. Following invocations of –run, -rerun* will calculate the new config but only overwrite the cached file IF changed -. If the cached file changes ALL existing tests go from COMPLETED -> STALE, I’m not sure what to do about RUNNING tests -. !VARS in runconfigs are not exported to the environment. They are accessed via rget as if the ! was not there. -. Per test copy commands (crude example below is not correct). ----------------- -[testcopy] -%/iind% unison SRC DEST -% cp –r SRC DEST ----------------- -. Test management via pkts (optional?) -.. Control pkt types: run, kill, rerunclean, clean, archive, status? -.. Status pkt types: ack, step, status_change -. Add nanomsg as a transport option -. - -RFC M01: Add ability to move runs to other Areas -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -*Purpose*: allow shrinking megatest.db data by moving runs to an alternate - Megatest area with same keys. - -*Method*: extend db sync to take a different megatest area as a destination. - -*Design*: - -. add param -destination [area|path]. when specified runs are copied to new - area and removed from local db. -. the data move would involve these steps -.. copy the run data to destination area megatest.db -.. mark the run records as deleted, do not remove the run data on disk -. accessing the data would be by running dashboard in the satellite area -. future versions of Megatest dashboard should support displaying areas in a - merged way. -. some new controls would be supported in the config -.. [setup] => allow-runs [no|yes] <== used to disallow runs -.. [setup] => auto-migrate=[areaname|path] <== used to automatically - migrate data to a satellite area. - -*Branch*: This work is taking place on branch v1.65-reduce-records - -RFC M02: Move data into completed-runs.db -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -*Purpose*: shrink megatest.db data to enable lower load and higher performance. - -*Method*: add a completed-runs.db and automatically move runs data from megatest.db to that db - -*Design*: - -. completed-runs.db is a full megatest database with complete schema -. the data move would involve these steps -.. copy the run data to completed-runs.db -.. remove the run data, first from /tmp/…/megatest.db and /tmp/…/megatest_ref.db, followed by megatest.db -. accessing the data would be unchanged for most operations. -. a mode -full-db will be added which when specified would attach the completed-runs.db to megatest.db before doing the query -. mechanisms for moving runs to/from the megatest.db would be added -.. -reduce-records => move runs to completed-runs.db -.. -restore-records => move runs from completed-runs.db to megatest.db - -*Branch*: This work is taking place on branch v1.65-reduce-records - -RFC M03: Automatic homehost migrations -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -*Purpose*: Automatically migrate homehost. - -*Method*: Check that there are no tests running, launched or remotehoststart in past ½ hour then if not on homehost migrate the db to current host - -*Design*: - -. Check that the system is quiescent, i.e. that there are no runs in flight or recently run -. Create a lock -. Migrate the /tmp cache db to the current host -. Update the .homehost file -. Remove the lock - -*Branch*: This work not yet started - -Architecture Refactor -~~~~~~~~~~~~~~~~~~~~~ - -Goals -^^^^^ - -. Reduce load on the file system. Sqlite3 files on network filesystem can be - a burden. [green]#[DONE]# -. Reduce number of servers and frequency of start/stop. This is mostly an - issue of clutter but also a reduction in "moving parts". [green]#[DONE]# -. Coalesce activities to a single home host where possible. Give the user - feedback that they have started the dashboard on a host other than the - home host. [green]#[DONE]# -. Reduce number of processes involved in managing running tests. - -Changes Needed -^^^^^^^^^^^^^^ - -. ACID compliant db will be on /tmp and synced to megatest.db with a five - second max delay. [green]#[DONE]# -. Read/writes to db for processes on homehost will go direct to /tmp - megatest.db file. [green]#[DONE]# -. Read/wites fron non-homehost processes will go through one server. Bulk - reads (e.g. for dashboard or list-runs) will be cached on the current host - in /tmp and synced from the home megatest.db in the testsuite area. [green]#[DONE]# -. Db syncs rely on the target db file timestame minus some margin. [green]#[DONE]# -. Since bulk reads do not use the server we can switch to simple RPC for the - network transport. [green]#[DONE]# -. Test running manager process extended to manage multiple running tests. - -Current Items -~~~~~~~~~~~~~ - -ww05 - migrate to inmem-db -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -. Switch to inmem db with fast sync to on disk db's [green]#[DONE]# -. Server polls tasks table for next action -.. Task table used for tracking runner process [red]#[Replaced by mtutil]# -.. Task table used for jobs to run [red]#[Replaced by mtutil]# -.. Task table used for queueing runner actions (remove runs, - cleanRunExecute, etc) [red]#[Replaced by mtutil#] - - -// ww32 -// ~~~~ -// -// . Rerun step and or subsequent steps from gui -// . Refresh test area files from gui -// . Clean and re-run button -// . Clean up STATE and STATUS handling. -// .. Dashboard and Test control panel are reverse order - choose and fix -// .. Move seldom used states and status to drop down selector -// . Access test control panel when clicking on Run Summary tests -// . Feature: -generate-index-tree -// . Change specifing of state and status to use STATE1/STATUS1,STATE2/STATUS2 -// -// ww33 -// ~~~~ -// -// . http api available for use with Perl, Ruby etc. scripts -// . megatest.config setup entries for: -// .. run launching (e.g. /bin/sh %CMD% > /dev/null) -// .. browser "konqueror %FNAME% -// -// ww34 -// ~~~~ -// -// . Mark dependent tests for clean/rerun -rerun-downstream -// . On run start check for defunct tests in RUNNING, LAUNCHED or REMOTEHOSTSTART and correct or notify -// . Fix: refresh of gui sometimes fails on last item (race condition?) -// -// ww35 -// ~~~~ -// -// . refdb: Add export of csv, json and sexp -// . Convert to using call-with-environment-variables where possible. Should allow handling of parallel runs in same process. -// . Re-work text interface wizards. Several bugs on record. Possibly convert to gui based. -// . Add to testconfig requirements section; launchlimiter scriptname, calls scriptname to check if ok to launch test -// . Refactor Run Summary view, currently very clumsy -// . Add option to show steps in Run Summary view -// -// ww36 -// ~~~~ -// -// . Refactor guis for resizeablity -// . Add filters to Run Summary view and Run Control view -// . Add to megatest.config or testconfig; rerunok STATE/STATUS,STATE/STATUS... -// . Launch gates for diskspace; /path/one>1G,/path/two>200M,/tmp>5G,#{scheme *toppath*}>1G -// -// Bin List -// ~~~~~~~~ -// -// . Quality improvements -// .. Server stutters occasionally -// .. Large number of items or tests still has some issues. -// .. Code refactoring -// .. Replace remote process with true API using json (supports Web app also) -// . Streamline the gui -// .. Everything resizable -// .. Less clutter -// .. Tool tips -// .. Filters on Run Summary, Summary and Run Control panel -// .. Built in log viewer (partially implemented) -// .. Refactor the test control panel -// . Help and documentation -// .. Complete the user manual (I’ve been working on this lately). -// .. Online help in the gui -// . Streamlined install -// .. Deployed version (download a location independent ready to run binary bundle) -// .. Install Makefile (in progress, needed for Mike to install on VMs) -// .. Added option to compile IUP (needed for VMs) -// . Server side run launching -// . Support for re-running, cleaning etc. of individual steps (ezsteps makes this very easy to implement). -// . Launch process needs built in daemonizing (easy to do, just need to test it thoroughly). -// . Wizards for creating tests, regression areas (current ones are text only and limited). -// . Fully functional built in web service (currently you can browse runs but it is very simplistic). -// . Wildcards in runconfigs: e.g. [p1271/9/%/%] -// . Gui panels for editing megatest.config and runconfigs.config -// . Fully isolated tests (no use of NFS to see regression area files) -// . Windows version ADDED docs/manual/plan.txt Index: docs/manual/plan.txt ================================================================== --- /dev/null +++ docs/manual/plan.txt @@ -0,0 +1,232 @@ +Road Map +-------- + +// This file is part of Megatest. +// +// Megatest is free software: you can redistribute it and/or modify +// it under the terms of the GNU General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// Megatest is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU General Public License for more details. +// +// You should have received a copy of the GNU General Public License +// along with Megatest. If not, see . + +// Copyright 2006-2012, Matthew Welland. + +Note 1: This road-map continues to evolve and subject to change without notice. + +Here is a smattering of ideas for Megatest 2.0 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +. Add variable $MT_RUNPATH = $MT_LINKTREE/$MT_TARGET/$MT_RUNNAME +---------------- +[db] +api legacy|new +---------------- +. One big lesson from the 1.63-1.65 generation was that the main.db, 1.db … model was really good at scaling. I’d like to combine that model with the current also-very-good model. Obviously this is a disruptive change. I think making the old model the default and the new model an option for at least one generation would be fair. +. Rigorous megatest.config and runconfig.config caching. +.. Cache the configs in $MT_RUNPATH +.. Following invocations of –run, -rerun* will calculate the new config but only overwrite the cached file IF changed +. If the cached file changes ALL existing tests go from COMPLETED -> STALE, I’m not sure what to do about RUNNING tests +. !VARS in runconfigs are not exported to the environment. They are accessed via rget as if the ! was not there. +. Per test copy commands (crude example below is not correct). +---------------- +[testcopy] +%/iind% unison SRC DEST +% cp –r SRC DEST +---------------- +. Test management via pkts (optional?) +.. Control pkt types: run, kill, rerunclean, clean, archive, status? +.. Status pkt types: ack, step, status_change +. Add nanomsg as a transport option +. + +RFC M01: Add ability to move runs to other Areas +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +*Purpose*: allow shrinking megatest.db data by moving runs to an alternate + Megatest area with same keys. + +*Method*: extend db sync to take a different megatest area as a destination. + +*Design*: + +. add param -destination [area|path]. when specified runs are copied to new + area and removed from local db. +. the data move would involve these steps +.. copy the run data to destination area megatest.db +.. mark the run records as deleted, do not remove the run data on disk +. accessing the data would be by running dashboard in the satellite area +. future versions of Megatest dashboard should support displaying areas in a + merged way. +. some new controls would be supported in the config +.. [setup] => allow-runs [no|yes] <== used to disallow runs +.. [setup] => auto-migrate=[areaname|path] <== used to automatically + migrate data to a satellite area. + +*Branch*: This work is taking place on branch v1.65-reduce-records + +RFC M02: Move data into completed-runs.db +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +*Purpose*: shrink megatest.db data to enable lower load and higher performance. + +*Method*: add a completed-runs.db and automatically move runs data from megatest.db to that db + +*Design*: + +. completed-runs.db is a full megatest database with complete schema +. the data move would involve these steps +.. copy the run data to completed-runs.db +.. remove the run data, first from /tmp/…/megatest.db and /tmp/…/megatest_ref.db, followed by megatest.db +. accessing the data would be unchanged for most operations. +. a mode -full-db will be added which when specified would attach the completed-runs.db to megatest.db before doing the query +. mechanisms for moving runs to/from the megatest.db would be added +.. -reduce-records => move runs to completed-runs.db +.. -restore-records => move runs from completed-runs.db to megatest.db + +*Branch*: This work is taking place on branch v1.65-reduce-records + +RFC M03: Automatic homehost migrations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +*Purpose*: Automatically migrate homehost. + +*Method*: Check that there are no tests running, launched or remotehoststart in past ½ hour then if not on homehost migrate the db to current host + +*Design*: + +. Check that the system is quiescent, i.e. that there are no runs in flight or recently run +. Create a lock +. Migrate the /tmp cache db to the current host +. Update the .homehost file +. Remove the lock + +*Branch*: This work not yet started + +Architecture Refactor +~~~~~~~~~~~~~~~~~~~~~ + +Goals +^^^^^ + +. Reduce load on the file system. Sqlite3 files on network filesystem can be + a burden. [green]#[DONE]# +. Reduce number of servers and frequency of start/stop. This is mostly an + issue of clutter but also a reduction in "moving parts". [green]#[DONE]# +. Coalesce activities to a single home host where possible. Give the user + feedback that they have started the dashboard on a host other than the + home host. [green]#[DONE]# +. Reduce number of processes involved in managing running tests. + +Changes Needed +^^^^^^^^^^^^^^ + +. ACID compliant db will be on /tmp and synced to megatest.db with a five + second max delay. [green]#[DONE]# +. Read/writes to db for processes on homehost will go direct to /tmp + megatest.db file. [green]#[DONE]# +. Read/wites fron non-homehost processes will go through one server. Bulk + reads (e.g. for dashboard or list-runs) will be cached on the current host + in /tmp and synced from the home megatest.db in the testsuite area. [green]#[DONE]# +. Db syncs rely on the target db file timestame minus some margin. [green]#[DONE]# +. Since bulk reads do not use the server we can switch to simple RPC for the + network transport. [green]#[DONE]# +. Test running manager process extended to manage multiple running tests. + +Current Items +~~~~~~~~~~~~~ + +ww05 - migrate to inmem-db +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +. Switch to inmem db with fast sync to on disk db's [green]#[DONE]# +. Server polls tasks table for next action +.. Task table used for tracking runner process [red]#[Replaced by mtutil]# +.. Task table used for jobs to run [red]#[Replaced by mtutil]# +.. Task table used for queueing runner actions (remove runs, + cleanRunExecute, etc) [red]#[Replaced by mtutil#] + + +// ww32 +// ~~~~ +// +// . Rerun step and or subsequent steps from gui +// . Refresh test area files from gui +// . Clean and re-run button +// . Clean up STATE and STATUS handling. +// .. Dashboard and Test control panel are reverse order - choose and fix +// .. Move seldom used states and status to drop down selector +// . Access test control panel when clicking on Run Summary tests +// . Feature: -generate-index-tree +// . Change specifing of state and status to use STATE1/STATUS1,STATE2/STATUS2 +// +// ww33 +// ~~~~ +// +// . http api available for use with Perl, Ruby etc. scripts +// . megatest.config setup entries for: +// .. run launching (e.g. /bin/sh %CMD% > /dev/null) +// .. browser "konqueror %FNAME% +// +// ww34 +// ~~~~ +// +// . Mark dependent tests for clean/rerun -rerun-downstream +// . On run start check for defunct tests in RUNNING, LAUNCHED or REMOTEHOSTSTART and correct or notify +// . Fix: refresh of gui sometimes fails on last item (race condition?) +// +// ww35 +// ~~~~ +// +// . refdb: Add export of csv, json and sexp +// . Convert to using call-with-environment-variables where possible. Should allow handling of parallel runs in same process. +// . Re-work text interface wizards. Several bugs on record. Possibly convert to gui based. +// . Add to testconfig requirements section; launchlimiter scriptname, calls scriptname to check if ok to launch test +// . Refactor Run Summary view, currently very clumsy +// . Add option to show steps in Run Summary view +// +// ww36 +// ~~~~ +// +// . Refactor guis for resizeablity +// . Add filters to Run Summary view and Run Control view +// . Add to megatest.config or testconfig; rerunok STATE/STATUS,STATE/STATUS... +// . Launch gates for diskspace; /path/one>1G,/path/two>200M,/tmp>5G,#{scheme *toppath*}>1G +// +// Bin List +// ~~~~~~~~ +// +// . Quality improvements +// .. Server stutters occasionally +// .. Large number of items or tests still has some issues. +// .. Code refactoring +// .. Replace remote process with true API using json (supports Web app also) +// . Streamline the gui +// .. Everything resizable +// .. Less clutter +// .. Tool tips +// .. Filters on Run Summary, Summary and Run Control panel +// .. Built in log viewer (partially implemented) +// .. Refactor the test control panel +// . Help and documentation +// .. Complete the user manual (I’ve been working on this lately). +// .. Online help in the gui +// . Streamlined install +// .. Deployed version (download a location independent ready to run binary bundle) +// .. Install Makefile (in progress, needed for Mike to install on VMs) +// .. Added option to compile IUP (needed for VMs) +// . Server side run launching +// . Support for re-running, cleaning etc. of individual steps (ezsteps makes this very easy to implement). +// . Launch process needs built in daemonizing (easy to do, just need to test it thoroughly). +// . Wizards for creating tests, regression areas (current ones are text only and limited). +// . Fully functional built in web service (currently you can browse runs but it is very simplistic). +// . Wildcards in runconfigs: e.g. [p1271/9/%/%] +// . Gui panels for editing megatest.config and runconfigs.config +// . Fully isolated tests (no use of NFS to see regression area files) +// . Windows version DELETED docs/manual/reference.in Index: docs/manual/reference.in ================================================================== --- docs/manual/reference.in +++ /dev/null @@ -1,839 +0,0 @@ -// This file is part of Megatest. -// -// Megatest is free software: you can redistribute it and/or modify -// it under the terms of the GNU General Public License as published by -// the Free Software Foundation, either version 3 of the License, or -// (at your option) any later version. -// -// Megatest is distributed in the hope that it will be useful, -// but WITHOUT ANY WARRANTY; without even the implied warranty of -// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -// GNU General Public License for more details. -// -// You should have received a copy of the GNU General Public License -// along with Megatest. If not, see . - -Reference ---------- - -Megatest Use Modes -~~~~~~~~~~~~~~~~~~ - -.Base commands -[width="80%",cols="^,2m,2m",frame="topbot",options="header"] -|====================== -|Use case | Megatest command | mtutil -|Start from scratch | -rerun-all | restart -|Rerun non-good completed | -rerun-clean | rerunclean -|Rerun all non-good and not completed yet | -set-state-status KILLREQ; -rerun-|clean | killrerun -|Continue run | -run | resume -|Remove run | -remove-runs | clean -|Lock run | -lock | lock -|Unlock run | -unlock | unlock -|killrun | -set-state-status KILLREQ; -kill-run | killrun -|====================== - -Config File Helpers -~~~~~~~~~~~~~~~~~~~ - -Various helpers for more advanced config files. - -.Helpers -[width="80%",cols="^,2m,2m,2m",frame="topbot",options="header"] -|====================== -|Helper | Purpose | Valid values | Comments -| #{scheme (scheme code...)} | Execute arbitrary scheme code | Any valid scheme | Value returned from the call is converted to a string and processed as part of the config file -| #{system command} | Execute program, inserts exit code | Any valid Unix command | Discards the output from the program -| #{shell command} or #{sh ...} | Execute program, inserts result from stdout | Any valid Unix command | Value returned from the call is converted to a string and processed as part of the config file -| #{realpath path} or #{rp ...} | Replace with normalized path | Must be a valid path | -| #{getenv VAR} or #{gv VAR} | Replace with content of env variable | Must be a valid var | -| #{get s v} or #{g s v} | Replace with variable v from section s | Variable must be defined before use | -| #{rget v} | Replace with variable v from target or default of runconfigs file | | -| #{mtrah} | Replace with the path to the megatest testsuite area | | -|====================== - -Config File Settings -~~~~~~~~~~~~~~~~~~~~ - -Settings in megatest.config - -Config File Additional Features -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Including output from a script as if it was inline to the config file: - -------------------------- -[scriptinc myscript.sh] -------------------------- - -If the script outputs: - -------------------------- -[items] -A a b c -B d e f -------------------------- - -Then the config file would effectively appear to contain an items section -exactly like the output from the script. This is useful when dynamically -creating items, itemstables and other config structures. You can see the -expansion of the call by looking in the cached files (look in your linktree -for megatest.config and runconfigs.config cache files and in your test run -areas for the expanded and cached testconfig). - -Wildcards and regexes in Targets - -------------------------- -[a/2/b] -VAR1 VAL1 - -[a/%/b] -VAR1 VAL2 -------------------------- - -Will result in: - -------------------------- -[a/2/b] -VAR1 VAL2 -------------------------- - -Can use either wildcard of "%" or a regular expression: - -------------------------- -[/abc.*def/] -------------------------- - -Disk Space Checks -^^^^^^^^^^^^^^^^^ - -Some parameters you can put in the [setup] section of megatest.config: - -------------------- -# minimum space required in a run disk -minspace 10000000 - -# minimum space required in dbdir: -dbdir-space-required 100000 - -# script that takes path as parameter and returns number of bytes available: -free-space-script check-space.sh -------------------- - -Trim trailing spaces -^^^^^^^^^^^^^^^^^^^^ - ------------------- -[configf:settings trim-trailing-spaces yes] ------------------- - -Job Submission Control -^^^^^^^^^^^^^^^^^^^^^^ - -Submit jobs to Host Types based on Test Name -++++++++++++++++++++++++++++++++++++++++++++ - -.In megatest.config ------------------------- -[host-types] -general nbfake -remote bsub - -[launchers] -runfirst/sum% remote -% general - -[jobtools] -launcher bsub -# if defined and not "no" flexi-launcher will bypass launcher unless -# there is no host-type match. -flexi-launcher yes ------------------------- - -host-types -++++++++++ - -List of host types and the commandline to run a job on that host type. - -.host-type => launch command ------------- -general nbfake ------------- - -launchers -+++++++++ - -.test/itempath => host-type ------------- -runfirst/sum% remote ------------- - -Miscellaneous Setup Items -+++++++++++++++++++++++++ - -Attempt to rerun tests in "STUCK/DEAD", "n/a", "ZERO_ITEMS" states. - -.In megatest.config ------------------- -[setup] -reruns 5 ------------------- - -Replace the default blacklisted environment variables with user supplied -list. - -Default list: USER HOME DISPLAY LS_COLORS XKEYSYMDB EDITOR MAKEFLAGS MAKEF MAKEOVERRIDES - -.Add a "bad" variable "PROMPT" to the variables that will be commented out -in the megatest.sh and megatest.csh files: ------------------ -[setup] -blacklistvars USER HOME DISPLAY LS_COLORS XKEYSYMDB EDITOR MAKEFLAGS PROMPT ------------------ - -Run time limit -++++++++++++++ - ------------------ -[setup] -# this will automatically kill the test if it runs for more than 1h 2m and 3s -runtimelim 1h 2m 3s ------------------ - -Tests browser view -~~~~~~~~~~~~~~~~~~ - -The tests browser (see the Run Control tab on the dashboard) has two views for displaying the tests. - -. Dot (graphviz) based tree -. No dot, plain listing - -The default is the graphviz based tree but if your tests don't view -well in that mode then use "nodot" to turn it off. - ------------------ -[setup] -nodot ------------------ - -Capturing Test Data -~~~~~~~~~~~~~~~~~~~ - -In a test you can capture arbitrary variables and roll them up in the -megatest database for viewing on the dashboard or web app. - -.In a test as a script ------------------------- -$MT_MEGATEST -load-test-data << EOF -foo,bar, 1.2, 1.9, > -foo,rab, 1.0e9, 10e9, 1e9 -foo,bla, 1.2, 1.9, < -foo,bal, 1.2, 1.2, < , ,Check for overload -foo,alb, 1.2, 1.2, <= , Amps,This is the high power circuit test -foo,abl, 1.2, 1.3, 0.1 -foo,bra, 1.2, pass, silly stuff -faz,bar, 10, 8mA, , ,"this is a comment" -EOF ------------------------- - -Alternatively you can use logpro triggers to capture values and inject them -into megatest using the -set-values mechanism: - -.Megatest help related to -set-values ------------------------- -Test data capture - -set-values : update or set values in the testdata table - :category : set the category field (optional) - :variable : set the variable name (optional) - :value : value measured (required) - :expected : value expected (required) - :tol : |value-expect| <= tol (required, can be <, >, >=, <= or number) - :units : name of the units for value, expected_value etc. (optional) ------------------------- - -Dashboard settings -~~~~~~~~~~~~~~~~~~ - -.Runs tab buttons, font and size ------------------- -[dashboard] -btn-height x14 -btn-fontsz 10 -cell-width 60 ------------------- - -Database settings -~~~~~~~~~~~~~~~~~ - -.Database config settings in [setup] section of megatest.config -[width="70%",cols="^,2m,2m,2m",frame="topbot",options="header"] -|====================== -|Var | Purpose | Valid values | Comments -|delay-on-busy | Prevent concurrent access issues | yes\|no or not defined | Default=no, may help on some network file systems, may slow things down also. -|faststart | All direct file access to sqlite db files | yes\|no or not defined | Default=yes, suggest no for central automated systems and yes for interactive use -|homehost | Start servers on this host | | Defaults to local host -|hostname | Hostname to bind to | \|- | On multi-homed hosts allows binding to specific hostname -|lowport | Start searching for a port at this portnum| 32768 | -|required | Server required | yes\|no or not defined | Default=no, force start of server always -|server-query-threshold | Start server when queries take longer than this | number in milliseconds | Default=300 -|timeout | http api timeout | number in hours | Default is 1 minute, do not change -|====================== - -The testconfig File -------------------- - -Setup section -~~~~~~~~~~~~~ - -Header -^^^^^^ - -------------------- -[setup] -------------------- - -The runscript method is a brute force way to run scripts where the -user is responsible for setting STATE and STATUS - -------------------- -runscript main.csh -------------------- - -Requirements section -~~~~~~~~~~~~~~~~~~~~ - -.Header -------------------- -[requirements] -------------------- - -Wait on Other Tests -^^^^^^^^^^^^^^^^^^^ - -------------------- -# A normal waiton waits for the prior tests to be COMPLETED -# and PASS, CHECK or WAIVED -waiton test1 test2 -------------------- - -Mode -^^^^ - -The default (i.e. if mode is not specified) is normal. All pre-dependent tests -must be COMPLETED and PASS, CHECK or WAIVED before the test will start - -------------------- -[requirements] -mode normal -------------------- - -The toplevel mode requires only that the prior tests are COMPLETED. - -------------------- -[requirements] -mode toplevel -------------------- - -A item based waiton will start items in a test when the same-named -item is COMPLETED and PASS, CHECK or WAIVED in the prior test. This -was historically called "itemwait" mode. The terms "itemwait" and -"itemmatch" are synonyms. - -------------------- -[requirements] -mode itemmatch -------------------- - -Overriding Enviroment Variables -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Override variables before starting the test. Can include files (perhaps generated by megatest -envdelta or similar). - --------------------- -[pre-launch-env-vars] -VAR1 value1 - -# Get some generated settings -[include ../generated-vars.config] - -# Use this trick to unset variables -#{scheme (unsetenv "FOOBAR")} --------------------- - -Itemmap Handling -~~~~~~~~~~~~~~~~ - -For cases were the dependent test has a similar but not identical -itempath to the downstream test an itemmap can allow for itemmatch -mode - -.example for removing part of itemmap for waiton test (eg: item +foo-x/bar+ depends on waiton's item +y/bar+) -------------------- -[requirements] -mode itemwait -# itemmap -itemmap .*x/ y/ - -------------------- - -.example for removing part of itemmap for waiton test (eg: item +foo/bar/baz+ in this test depends on waiton's item +baz+) -------------------- - -# ## pattern replacement notes -# -# ## Example -# ## Remove everything up to the last / -[requirements] -mode itemwait -# itemmap -itemmap .*/ -------------------- - -.example replacing part of itemmap for (eg: item +foo/1234+ will imply waiton's item +bar/1234+) -------------------- - -# -# ## Example -# ## Replace foo/ with bar/ -[requirements] -mode itemwait -# itemmap -itemmap foo/ bar/ - -------------------- - -.example for backreference (eg: item +foo23/thud+ will imply waiton's item +num-23/bar/thud+ -------------------- -# -# ## Example -# ## can use \{number} in replacement pattern to backreference a (capture) from matching pattern similar to sed or perl -[requirements] -mode itemwait -# itemmap -itemmap foo(\d+)/ num-\1/bar/ - -------------------- - -.example multiple itemmaps -------------------- - -# multi-line; matches are applied in the listed order -# The following would map: -# a123b321 to b321fooa123 then to 321fooa123p -# -[requirements] -itemmap (a\d+)(b\d+) \2foo\1 - b(.*) \1p -------------------- - - -Complex mapping -^^^^^^^^^^^^^^^ -Complex mappings can be handled with a separate [itemmap] section (instead if an itemmap line in the [requirements] section) - -Each line in an itemmap section starts with a waiton test name followed by an itemmap expression - -.eg: The following causes waiton test A item +bar/1234+ to run when our test's +foo/1234+ item is requested as well as causing waiton test B's +blah+ item to run when our test's +stuff/blah+ item is requested --------------- -[itemmap] -A foo/ bar/ -B stuff/ --------------- - - -Complex mapping example -^^^^^^^^^^^^^^^^^^^^^^^ - - - -// image::itemmap.png[] -image::complex-itemmap.png[] - - -We accomplish this by configuring the testconfigs of our tests C D and E as follows: - -.Testconfig for Test E has ----------------------- -[requirements] -waiton C -itemmap (\d+)/res \1/bb ----------------------- - -.Testconfig for Test D has ----------------------- -[requirements] -waiton C -itemmap (\d+)/res \1/aa ----------------------- - -.Testconfig for Test C has ----------------------- -[requirements] -waiton A B - -[itemmap] -A (\d+)/aa aa/\1 -B (\d+)/bb bb/\1 ----------------------- - -.Testconfigs for Test B and Test A have no waiton or itemmap configured -------------------- -------------------- - -.Walk through one item -- we want the following to happen for testpatt +D/1/res+ (see blue boxes in complex itemmaping figure above): - -. eg from command line +megatest -run -testpatt D/1/res -target mytarget -runname myrunname+ -. Full list to be run is now: +D/1/res+ -. Test D has a waiton - test C. Test D's itemmap rule +itemmap (\d+)/res \1/aa+ -> causes +C/1/aa+ to run before +D/1/res+ -. Full list to be run is now: +D/1/res+, +C/1/aa+ -. Test C was a waiton - test A. Test C's rule +A (\d+)/aa aa/\1+ -> causes +A/aa/1+ to run before +C/1/aa+ -. Full list to be run is now: +D/1/res+, +C/1/aa+, +A/aa/1+ -. Test A has no waitons. All waitons of all tests in full list have been processed. Full list is finalized. - - - -itemstable -^^^^^^^^^^ -An alternative to defining items is the itemstable section. This lets you define the itempath in a table format rather than specifying components and relying on getting all permutations of those components. - - - - - -Dynamic Flow Dependency Tree -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.Autogeneration waiton list for dynamic flow dependency trees -------------------- -[requirements] -# With a toplevel test you may wish to generate your list -# of tests to run dynamically -# -waiton #{shell get-valid-tests-to-run.sh} -------------------- - -Run time limit -^^^^^^^^^^^^^^ - ------------------ -[requirements] -runtimelim 1h 2m 3s # this will automatically kill the test if it runs for more than 1h 2m and 3s ------------------ - -Skip -^^^^ - -A test with a skip section will conditional skip running. - -.Skip section example ------------------ -[skip] -prevrunning x -# rundelay 30m 15s ------------------ - -Skip on Still-running Tests -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - ------------------ -# NB// If the prevrunning line exists with *any* value the test will -# automatically SKIP if the same-named test is currently RUNNING. The -# "x" can be any string. Comment out the prevrunning line to turn off -# skip. - -[skip] -prevrunning x ------------------ - -Skip if a File Exists -^^^^^^^^^^^^^^^^^^^^^ - ------------------ -[skip] -fileexists /path/to/a/file # skip if /path/to/a/file exists ------------------ - -Skip if test ran more recently than specified time -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.Skip if this test has been run in the past fifteen minutes and 15 seconds. ------------------ -[skip] -rundelay 15m 15s ------------------ - -Disks -^^^^^ - -A disks section in testconfig will override the disks section in -megatest.config. This can be used to allocate disks on a per-test or per item -basis. - -Controlled waiver propagation -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If test is FAIL and previous test in run with same MT_TARGET is WAIVED then apply the following rules from the testconfig: -If a waiver check is specified in the testconfig apply the check and if it passes then set this FAIL to WAIVED - -Waiver check has two parts, 1) a list of waiver, rulename, filepatterns and 2) the rulename script spec (note that "diff" and "logpro" are predefined) - ------------------ -###### EXAMPLE FROM testconfig ######### -# matching file(s) will be diff'd with previous run and logpro applied -# if PASS or WARN result from logpro then WAIVER state is set -# -[waivers] -# logpro_file rulename input_glob -waiver_1 logpro lookittmp.log - -[waiver_rules] - -# This builtin rule is the default if there is no .logpro file -# diff diff %file1% %file2% - -# This builtin rule is applied if a .logpro file exists -# logpro diff %file1% %file2% | logpro %waivername%.logpro %waivername%.html ------------------ - -Ezsteps -~~~~~~~ - -.Example ezsteps with logpro rules ------------------ -[ezsteps] -lookittmp ls /tmp - -[logpro] -lookittmp ;; Note: config file format supports multi-line entries where leading whitespace is removed from each line - ;; a blank line indicates the end of the block of text - (expect:required in "LogFileBody" > 0 "A file name that should never exist!" #/This is a awfully stupid file name that should never be found in the temp dir/) - ------------------ - -To transfer the environment to the next step you can do the following: - ----------------------------- -$MT_MEGATEST -env2file .ezsteps/${stepname} ----------------------------- - -Triggers -~~~~~~~~ - -In your testconfig or megatest.config triggers can be specified - ------------------ -[triggers] - -# Call script running.sh when test goes to state=RUNNING, status=PASS -RUNNING/PASS running.sh - -# Call script running.sh any time state goes to RUNNING -RUNNING/ running.sh - -# Call script onpass.sh any time status goes to PASS -PASS/ onpass.sh ------------------ - -Scripts called will have; test-id test-rundir trigger test-name item-path state status event-time, added to the commandline. - -HINT - -To start an xterm (useful for debugging), use a command line like the following: - ------------------ -[triggers] -COMPLETED/ xterm -e bash -s -- ------------------ - -NOTE: There is a trailing space after the -- - -There are a number of environment variables available to the trigger script -but since triggers can be called in various contexts not all variables are -available at all times. The trigger script should check for the variable and -fail gracefully if it doesn't exist. - -.Environment variables visible to the trigger script -[width="90%",cols="^,2m",frame="topbot",options="header"] -|====================== -|Variable | Purpose -| MT_TEST_RUN_DIR | The directory where Megatest ran this test -| MT_CMDINFO | Encoded command data for the test -| MT_DEBUG_MODE | Used to pass the debug mode to nested calls to Megatest -| MT_RUN_AREA_HOME | Megatest home area -| MT_TESTSUITENAME | The name of this testsuite or area -| MT_TEST_NAME | The name of this test -| MT_ITEM_INFO | The variable and values for the test item -| MT_MEGATEST | Which Megatest binary is being used by this area -| MT_TARGET | The target variable values, separated by '/' -| MT_LINKTREE | The base of the link tree where all run tests can be found -| MT_ITEMPATH | The values of the item path variables, separated by '/' -| MT_RUNNAME | The name of the run -|====================== - - -Override the Toplevel HTML File -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Megatest generates a simple html file summary for top level tests of -iterated tests. The generation can be overridden. NOTE: the output of -the script is captured from stdout to create the html. - - -.For test "runfirst" override the toplevel generation with a script "mysummary.sh" ------------------ -# Override the rollup for specific tests -[testrollup] -runfirst mysummary.sh ------------------ - -Archiving Setup ---------------- - -In megatest.config add the following sections: - -.megatest.config --------------- -[archive] -# where to get bup executable -# bup /path/to/bup - -[archive-disks] - -# Archives will be organised under these paths like this: -# / -# Within the archive the data is structured like this: -# /// -archive0 /mfs/myarchive-data/adisk1 --------------- - -Handling Environment Variables ------------------------------- - -It is often necessary to capture and or manipulate environment -variables. Megatest has some facilities built in to help. - -Capture variables -~~~~~~~~~~~~~~~~~ - -.Commands ------------------------------- -# capture the current enviroment into a db called envdat.db under -# the context "before" -megatest -envcap before - -# capture the current environment into a db called startup.db with -# context "after" -megatest -envcap after startup.db - -# write the diff from before to after -megatest -envdelta before-after -dumpmode bash ------------------------------- - -Dump modes include bash, csh and config. You can include config data -into megatest.config or runconfigs.config. - -.Example of generating and using config data ------------------------------- -megatest -envcap original -# do some stuff here -megatest -envcap munged -megatest -envdelta original-munged -dumpmode ini -o modified.config ------------------------------- - -Then in runconfigs.config - -.Example of using modified.config in a testconfig ------------------------------- -cat testconfig -[pre-launch-env-vars] -[include modified.config] ------------------------------- - -Managing Old Runs ------------------ - -It is often desired to keep some older runs around but this must be balanced with the costs of disk space. - -. Use -remove-keep -. Use -archive (can also be done from the -remove-keep interface) -. use -remove-runs with -keep-records - -.For each target, remove all runs but the most recent 3 if they are over 1 week old ---------------------- -# use -precmd 'sleep 5;nbfake' to limit overloading the host computer but to allow the removes to run in parallel. -megatest -actions print,remove-runs -remove-keep 3 -target %/%/%/% -runname % -age 1w -precmd 'sleep 5;nbfake'" ---------------------- - -Nested Runs ------------ - -A Megatest test can run a full Megatest run in either the same -Megatest area or in another area. This is a powerful way of chaining -complex suites of tests and or actions. - -If you are not using the current area you can use ezsteps to retrieve -and setup the sub-Megatest run area. - -In the testconfig: ---------------- -[subrun] - -# Required: wait for the run or just launch it -# if no then the run will be an automatic PASS irrespective of the actual result -run-wait yes|no - -# Optional: where to execute the run. Default is the current runarea -run-area /some/path/to/megatest/area - -# Optional: method to use to determine pass/fail status of the run -# auto (default) - roll up the net state/status of the sub-run -# logpro - use the provided logpro rules, happens automatically if there is a logpro section -# passfail auto|logpro -# Example of logpro: -passfail logpro - -# Optional: -logpro ;; if this section exists then logpro is used to determine pass/fail - (expect:required in "LogFileBody" >= 1 "At least one pass" #/PASS/) - (expect:error in "LogFileBody" = 0 "No FAILs allowed" #/FAIL/) - -# Optional: target translator, default is to use the parent target -target #{shell somescript.sh} - -# Optional: runname translator/generator, default is to use the parent runname -run-name #{somescript.sh} - -# Optional: testpatt spec, default is to first look for TESTPATT spec from runconfigs unless there is a contour spec -test-patt %/item1,test2 - -# Optional: contour spec, use the named contour from the megatest.config contour spec -contour contourname ### NOTE: Not implemented yet! Let us know if you need this feature. - -# Optional: mode-patt, use this spec for testpatt from runconfigs -mode-patt TESTPATT - -# Optional: tag-expr, use this tag-expr to select tests -tag-expr quick - -# Optional: (not yet implemented, remove-runs is always propagated at this time), propagate these actions from the parent -# test -# Note// default is % for all -propagate remove-runs archive ... - ---------------- - -Programming API ---------------- - -These routines can be called from the megatest repl. - -.API Keys Related Calls -[width="70%",cols="^,2m,2m,2m",frame="topbot",options="header,footer"] -|====================== -|API Call | Purpose comments | Returns | Comments -|(rmt:get-keys run-id) | | ( key1 key2 ... ) | -| (rmt:get-key-val-pairs run-id) | | #t=success/#f=fail | Works only if the server is still reachable -|====================== - - -:numbered!: ADDED docs/manual/reference.txt Index: docs/manual/reference.txt ================================================================== --- /dev/null +++ docs/manual/reference.txt @@ -0,0 +1,860 @@ +// This file is part of Megatest. +// +// Megatest is free software: you can redistribute it and/or modify +// it under the terms of the GNU General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// Megatest is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU General Public License for more details. +// +// You should have received a copy of the GNU General Public License +// along with Megatest. If not, see . + +Reference +--------- + +Megatest Use Modes +~~~~~~~~~~~~~~~~~~ + +.Base commands +[width="80%",cols="^,2m,2m",frame="topbot",options="header"] +|====================== +|Use case | Megatest command | mtutil +|Start from scratch | -rerun-all | restart +|Rerun non-good completed | -rerun-clean | rerunclean +|Rerun all non-good and not completed yet | -set-state-status KILLREQ; -rerun-|clean | killrerun +|Continue run | -run | resume +|Remove run | -remove-runs | clean +|Lock run | -lock | lock +|Unlock run | -unlock | unlock +|killrun | -set-state-status KILLREQ; -kill-run | killrun +|====================== + +Config File Helpers +~~~~~~~~~~~~~~~~~~~ + +Various helpers for more advanced config files. + +.Helpers +[width="80%",cols="^,2m,2m,2m",frame="topbot",options="header"] +|====================== +|Helper | Purpose | Valid values | Comments +| #{scheme (scheme code...)} | Execute arbitrary scheme code | Any valid scheme | Value returned from the call is converted to a string and processed as part of the config file +| #{system command} | Execute program, inserts exit code | Any valid Unix command | Discards the output from the program +| #{shell command} or #{sh ...} | Execute program, inserts result from stdout | Any valid Unix command | Value returned from the call is converted to a string and processed as part of the config file +| #{realpath path} or #{rp ...} | Replace with normalized path | Must be a valid path | +| #{getenv VAR} or #{gv VAR} | Replace with content of env variable | Must be a valid var | +| #{get s v} or #{g s v} | Replace with variable v from section s | Variable must be defined before use | +| #{rget v} | Replace with variable v from target or default of runconfigs file | | +| #{mtrah} | Replace with the path to the megatest testsuite area | | +|====================== + +Config File Settings +~~~~~~~~~~~~~~~~~~~~ + +Settings in megatest.config + +Config File Additional Features +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Including output from a script as if it was inline to the config file: + +------------------------- +[scriptinc myscript.sh] +------------------------- + +If the script outputs: + +------------------------- +[items] +A a b c +B d e f +------------------------- + +Then the config file would effectively appear to contain an items section +exactly like the output from the script. This is useful when dynamically +creating items, itemstables and other config structures. You can see the +expansion of the call by looking in the cached files (look in your linktree +for megatest.config and runconfigs.config cache files and in your test run +areas for the expanded and cached testconfig). + +Wildcards and regexes in Targets + +------------------------- +[a/2/b] +VAR1 VAL1 + +[a/%/b] +VAR1 VAL2 +------------------------- + +Will result in: + +------------------------- +[a/2/b] +VAR1 VAL2 +------------------------- + +Can use either wildcard of "%" or a regular expression: + +------------------------- +[/abc.*def/] +------------------------- + +Disk Space Checks +^^^^^^^^^^^^^^^^^ + +Some parameters you can put in the [setup] section of megatest.config: + +------------------- +# minimum space required in a run disk +minspace 10000000 + +# minimum space required in dbdir: +dbdir-space-required 100000 + +# script that takes path as parameter and returns number of bytes available: +free-space-script check-space.sh +------------------- + +Trim trailing spaces +^^^^^^^^^^^^^^^^^^^^ + +------------------ +[configf:settings trim-trailing-spaces yes] +------------------ + +Job Submission Control +^^^^^^^^^^^^^^^^^^^^^^ + +Submit jobs to Host Types based on Test Name +++++++++++++++++++++++++++++++++++++++++++++ + +.In megatest.config +------------------------ +[host-types] +general nbfake +remote bsub + +[launchers] +runfirst/sum% remote +% general + +[jobtools] +launcher bsub +# if defined and not "no" flexi-launcher will bypass launcher unless +# there is no host-type match. +flexi-launcher yes +------------------------ + +host-types +++++++++++ + +List of host types and the commandline to run a job on that host type. + +.host-type => launch command +------------ +general nbfake +------------ + +launchers ++++++++++ + +.test/itempath => host-type +------------ +runfirst/sum% remote +------------ + +Miscellaneous Setup Items ++++++++++++++++++++++++++ + +Attempt to rerun tests in "STUCK/DEAD", "n/a", "ZERO_ITEMS" states. + +.In megatest.config +------------------ +[setup] +reruns 5 +------------------ + +Replace the default blacklisted environment variables with user supplied +list. + +Default list: USER HOME DISPLAY LS_COLORS XKEYSYMDB EDITOR MAKEFLAGS MAKEF MAKEOVERRIDES + +.Add a "bad" variable "PROMPT" to the variables that will be commented out +in the megatest.sh and megatest.csh files: +----------------- +[setup] +blacklistvars USER HOME DISPLAY LS_COLORS XKEYSYMDB EDITOR MAKEFLAGS PROMPT +----------------- + +Run time limit +++++++++++++++ + +----------------- +[setup] +# this will automatically kill the test if it runs for more than 1h 2m and 3s +runtimelim 1h 2m 3s +----------------- + +Tests browser view +~~~~~~~~~~~~~~~~~~ + +The tests browser (see the Run Control tab on the dashboard) has two views for displaying the tests. + +. Dot (graphviz) based tree +. No dot, plain listing + +The default is the graphviz based tree but if your tests don't view +well in that mode then use "nodot" to turn it off. + +----------------- +[setup] +nodot +----------------- + +Capturing Test Data +~~~~~~~~~~~~~~~~~~~ + +In a test you can capture arbitrary variables and roll them up in the +megatest database for viewing on the dashboard or web app. + +.In a test as a script +------------------------ +$MT_MEGATEST -load-test-data << EOF +foo,bar, 1.2, 1.9, > +foo,rab, 1.0e9, 10e9, 1e9 +foo,bla, 1.2, 1.9, < +foo,bal, 1.2, 1.2, < , ,Check for overload +foo,alb, 1.2, 1.2, <= , Amps,This is the high power circuit test +foo,abl, 1.2, 1.3, 0.1 +foo,bra, 1.2, pass, silly stuff +faz,bar, 10, 8mA, , ,"this is a comment" +EOF +------------------------ + +Alternatively you can use logpro triggers to capture values and inject them +into megatest using the -set-values mechanism: + +.Megatest help related to -set-values +------------------------ +Test data capture + -set-values : update or set values in the testdata table + :category : set the category field (optional) + :variable : set the variable name (optional) + :value : value measured (required) + :expected : value expected (required) + :tol : |value-expect| <= tol (required, can be <, >, >=, <= or number) + :units : name of the units for value, expected_value etc. (optional) +------------------------ + +Dashboard settings +~~~~~~~~~~~~~~~~~~ + +.Runs tab buttons, font and size +------------------ +[dashboard] +btn-height x14 +btn-fontsz 10 +cell-width 60 +------------------ + +Database settings +~~~~~~~~~~~~~~~~~ + +.Database config settings in [setup] section of megatest.config +[width="70%",cols="^,2m,2m,2m",frame="topbot",options="header"] +|====================== +|Var | Purpose | Valid values | Comments +|delay-on-busy | Prevent concurrent access issues | yes\|no or not defined | Default=no, may help on some network file systems, may slow things down also. +|faststart | All direct file access to sqlite db files | yes\|no or not defined | Default=yes, suggest no for central automated systems and yes for interactive use +|homehost | Start servers on this host | | Defaults to local host +|hostname | Hostname to bind to | \|- | On multi-homed hosts allows binding to specific hostname +|lowport | Start searching for a port at this portnum| 32768 | +|required | Server required | yes\|no or not defined | Default=no, force start of server always +|server-query-threshold | Start server when queries take longer than this | number in milliseconds | Default=300 +|timeout | http api timeout | number in hours | Default is 1 minute, do not change +|====================== + +The testconfig File +------------------- + +Setup section +~~~~~~~~~~~~~ + +Header +^^^^^^ + +------------------- +[setup] +------------------- + +The runscript method is a brute force way to run scripts where the +user is responsible for setting STATE and STATUS + +------------------- +runscript main.csh +------------------- + +Iteration +~~~~~~~~~ + +.Sections for iteration +------------------ +# full combinations +[items] +A x y +B 1 2 + +# Yields: x/1 x/2 y/1 y/2 + +# tabled +[itemstable] +A x y +B 1 2 + +# Yields x/1 y/2 +------------------ + + +Requirements section +~~~~~~~~~~~~~~~~~~~~ + +.Header +------------------- +[requirements] +------------------- + +Wait on Other Tests +^^^^^^^^^^^^^^^^^^^ + +------------------- +# A normal waiton waits for the prior tests to be COMPLETED +# and PASS, CHECK or WAIVED +waiton test1 test2 +------------------- + +Mode +^^^^ + +The default (i.e. if mode is not specified) is normal. All pre-dependent tests +must be COMPLETED and PASS, CHECK or WAIVED before the test will start + +------------------- +[requirements] +mode normal +------------------- + +The toplevel mode requires only that the prior tests are COMPLETED. + +------------------- +[requirements] +mode toplevel +------------------- + +A item based waiton will start items in a test when the same-named +item is COMPLETED and PASS, CHECK or WAIVED in the prior test. This +was historically called "itemwait" mode. The terms "itemwait" and +"itemmatch" are synonyms. + +------------------- +[requirements] +mode itemmatch +------------------- + +Overriding Enviroment Variables +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Override variables before starting the test. Can include files (perhaps generated by megatest -envdelta or similar). + +-------------------- +[pre-launch-env-vars] +VAR1 value1 + +# Get some generated settings +[include ../generated-vars.config] + +# Use this trick to unset variables +#{scheme (unsetenv "FOOBAR")} +-------------------- + +Itemmap Handling +~~~~~~~~~~~~~~~~ + +For cases were the dependent test has a similar but not identical +itempath to the downstream test an itemmap can allow for itemmatch +mode + +.example for removing part of itemmap for waiton test (eg: item +foo-x/bar+ depends on waiton's item +y/bar+) +------------------- +[requirements] +mode itemwait +# itemmap +itemmap .*x/ y/ + +------------------- + +.example for removing part of itemmap for waiton test (eg: item +foo/bar/baz+ in this test depends on waiton's item +baz+) +------------------- + +# ## pattern replacement notes +# +# ## Example +# ## Remove everything up to the last / +[requirements] +mode itemwait +# itemmap +itemmap .*/ +------------------- + +.example replacing part of itemmap for (eg: item +foo/1234+ will imply waiton's item +bar/1234+) +------------------- + +# +# ## Example +# ## Replace foo/ with bar/ +[requirements] +mode itemwait +# itemmap +itemmap foo/ bar/ + +------------------- + +.example for backreference (eg: item +foo23/thud+ will imply waiton's item +num-23/bar/thud+ +------------------- +# +# ## Example +# ## can use \{number} in replacement pattern to backreference a (capture) from matching pattern similar to sed or perl +[requirements] +mode itemwait +# itemmap +itemmap foo(\d+)/ num-\1/bar/ + +------------------- + +.example multiple itemmaps +------------------- + +# multi-line; matches are applied in the listed order +# The following would map: +# a123b321 to b321fooa123 then to 321fooa123p +# +[requirements] +itemmap (a\d+)(b\d+) \2foo\1 + b(.*) \1p +------------------- + + +Complex mapping +^^^^^^^^^^^^^^^ +Complex mappings can be handled with a separate [itemmap] section (instead if an itemmap line in the [requirements] section) + +Each line in an itemmap section starts with a waiton test name followed by an itemmap expression + +.eg: The following causes waiton test A item +bar/1234+ to run when our test's +foo/1234+ item is requested as well as causing waiton test B's +blah+ item to run when our test's +stuff/blah+ item is requested +-------------- +[itemmap] +A foo/ bar/ +B stuff/ +-------------- + + +Complex mapping example +^^^^^^^^^^^^^^^^^^^^^^^ + + + +// image::itemmap.png[] +image::complex-itemmap.png[] + + +We accomplish this by configuring the testconfigs of our tests C D and E as follows: + +.Testconfig for Test E has +---------------------- +[requirements] +waiton C +itemmap (\d+)/res \1/bb +---------------------- + +.Testconfig for Test D has +---------------------- +[requirements] +waiton C +itemmap (\d+)/res \1/aa +---------------------- + +.Testconfig for Test C has +---------------------- +[requirements] +waiton A B + +[itemmap] +A (\d+)/aa aa/\1 +B (\d+)/bb bb/\1 +---------------------- + +.Testconfigs for Test B and Test A have no waiton or itemmap configured +------------------- +------------------- + +.Walk through one item -- we want the following to happen for testpatt +D/1/res+ (see blue boxes in complex itemmaping figure above): + +. eg from command line +megatest -run -testpatt D/1/res -target mytarget -runname myrunname+ +. Full list to be run is now: +D/1/res+ +. Test D has a waiton - test C. Test D's itemmap rule +itemmap (\d+)/res \1/aa+ -> causes +C/1/aa+ to run before +D/1/res+ +. Full list to be run is now: +D/1/res+, +C/1/aa+ +. Test C was a waiton - test A. Test C's rule +A (\d+)/aa aa/\1+ -> causes +A/aa/1+ to run before +C/1/aa+ +. Full list to be run is now: +D/1/res+, +C/1/aa+, +A/aa/1+ +. Test A has no waitons. All waitons of all tests in full list have been processed. Full list is finalized. + + + +itemstable +^^^^^^^^^^ +An alternative to defining items is the itemstable section. This lets you define the itempath in a table format rather than specifying components and relying on getting all permutations of those components. + + + + + +Dynamic Flow Dependency Tree +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.Autogeneration waiton list for dynamic flow dependency trees +------------------- +[requirements] +# With a toplevel test you may wish to generate your list +# of tests to run dynamically +# +waiton #{shell get-valid-tests-to-run.sh} +------------------- + +Run time limit +^^^^^^^^^^^^^^ + +----------------- +[requirements] +runtimelim 1h 2m 3s # this will automatically kill the test if it runs for more than 1h 2m and 3s +----------------- + +Skip +^^^^ + +A test with a skip section will conditional skip running. + +.Skip section example +----------------- +[skip] +prevrunning x +# rundelay 30m 15s +----------------- + +Skip on Still-running Tests +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +----------------- +# NB// If the prevrunning line exists with *any* value the test will +# automatically SKIP if the same-named test is currently RUNNING. The +# "x" can be any string. Comment out the prevrunning line to turn off +# skip. + +[skip] +prevrunning x +----------------- + +Skip if a File Exists +^^^^^^^^^^^^^^^^^^^^^ + +----------------- +[skip] +fileexists /path/to/a/file # skip if /path/to/a/file exists +----------------- + +Skip if test ran more recently than specified time +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.Skip if this test has been run in the past fifteen minutes and 15 seconds. +----------------- +[skip] +rundelay 15m 15s +----------------- + +Disks +^^^^^ + +A disks section in testconfig will override the disks section in +megatest.config. This can be used to allocate disks on a per-test or per item +basis. + +Controlled waiver propagation +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If test is FAIL and previous test in run with same MT_TARGET is WAIVED then apply the following rules from the testconfig: +If a waiver check is specified in the testconfig apply the check and if it passes then set this FAIL to WAIVED + +Waiver check has two parts, 1) a list of waiver, rulename, filepatterns and 2) the rulename script spec (note that "diff" and "logpro" are predefined) + +----------------- +###### EXAMPLE FROM testconfig ######### +# matching file(s) will be diff'd with previous run and logpro applied +# if PASS or WARN result from logpro then WAIVER state is set +# +[waivers] +# logpro_file rulename input_glob +waiver_1 logpro lookittmp.log + +[waiver_rules] + +# This builtin rule is the default if there is no .logpro file +# diff diff %file1% %file2% + +# This builtin rule is applied if a .logpro file exists +# logpro diff %file1% %file2% | logpro %waivername%.logpro %waivername%.html +----------------- + +Ezsteps +~~~~~~~ + +.Example ezsteps with logpro rules +----------------- +[ezsteps] +lookittmp ls /tmp + +[logpro] +lookittmp ;; Note: config file format supports multi-line entries where leading whitespace is removed from each line + ;; a blank line indicates the end of the block of text + (expect:required in "LogFileBody" > 0 "A file name that should never exist!" #/This is a awfully stupid file name that should never be found in the temp dir/) + +----------------- + +To transfer the environment to the next step you can do the following: + +---------------------------- +$MT_MEGATEST -env2file .ezsteps/${stepname} +---------------------------- + +Triggers +~~~~~~~~ + +In your testconfig or megatest.config triggers can be specified + +----------------- +[triggers] + +# Call script running.sh when test goes to state=RUNNING, status=PASS +RUNNING/PASS running.sh + +# Call script running.sh any time state goes to RUNNING +RUNNING/ running.sh + +# Call script onpass.sh any time status goes to PASS +PASS/ onpass.sh +----------------- + +Scripts called will have; test-id test-rundir trigger test-name item-path state status event-time, added to the commandline. + +HINT + +To start an xterm (useful for debugging), use a command line like the following: + +----------------- +[triggers] +COMPLETED/ xterm -e bash -s -- +----------------- + +NOTE: There is a trailing space after the -- + +There are a number of environment variables available to the trigger script +but since triggers can be called in various contexts not all variables are +available at all times. The trigger script should check for the variable and +fail gracefully if it doesn't exist. + +.Environment variables visible to the trigger script +[width="90%",cols="^,2m",frame="topbot",options="header"] +|====================== +|Variable | Purpose +| MT_TEST_RUN_DIR | The directory where Megatest ran this test +| MT_CMDINFO | Encoded command data for the test +| MT_DEBUG_MODE | Used to pass the debug mode to nested calls to Megatest +| MT_RUN_AREA_HOME | Megatest home area +| MT_TESTSUITENAME | The name of this testsuite or area +| MT_TEST_NAME | The name of this test +| MT_ITEM_INFO | The variable and values for the test item +| MT_MEGATEST | Which Megatest binary is being used by this area +| MT_TARGET | The target variable values, separated by '/' +| MT_LINKTREE | The base of the link tree where all run tests can be found +| MT_ITEMPATH | The values of the item path variables, separated by '/' +| MT_RUNNAME | The name of the run +|====================== + + +Override the Toplevel HTML File +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Megatest generates a simple html file summary for top level tests of +iterated tests. The generation can be overridden. NOTE: the output of +the script is captured from stdout to create the html. + + +.For test "runfirst" override the toplevel generation with a script "mysummary.sh" +----------------- +# Override the rollup for specific tests +[testrollup] +runfirst mysummary.sh +----------------- + +Archiving Setup +--------------- + +In megatest.config add the following sections: + +.megatest.config +-------------- +[archive] +# where to get bup executable +# bup /path/to/bup + +[archive-disks] + +# Archives will be organised under these paths like this: +# / +# Within the archive the data is structured like this: +# /// +archive0 /mfs/myarchive-data/adisk1 +-------------- + +Handling Environment Variables +------------------------------ + +It is often necessary to capture and or manipulate environment +variables. Megatest has some facilities built in to help. + +Capture variables +~~~~~~~~~~~~~~~~~ + +.Commands +------------------------------ +# capture the current enviroment into a db called envdat.db under +# the context "before" +megatest -envcap before + +# capture the current environment into a db called startup.db with +# context "after" +megatest -envcap after startup.db + +# write the diff from before to after +megatest -envdelta before-after -dumpmode bash +------------------------------ + +Dump modes include bash, csh and config. You can include config data +into megatest.config or runconfigs.config. + +.Example of generating and using config data +------------------------------ +megatest -envcap original +# do some stuff here +megatest -envcap munged +megatest -envdelta original-munged -dumpmode ini -o modified.config +------------------------------ + +Then in runconfigs.config + +.Example of using modified.config in a testconfig +------------------------------ +cat testconfig +[pre-launch-env-vars] +[include modified.config] +------------------------------ + +Managing Old Runs +----------------- + +It is often desired to keep some older runs around but this must be balanced with the costs of disk space. + +. Use -remove-keep +. Use -archive (can also be done from the -remove-keep interface) +. use -remove-runs with -keep-records + +.For each target, remove all runs but the most recent 3 if they are over 1 week old +--------------------- +# use -precmd 'sleep 5;nbfake' to limit overloading the host computer but to allow the removes to run in parallel. +megatest -actions print,remove-runs -remove-keep 3 -target %/%/%/% -runname % -age 1w -precmd 'sleep 5;nbfake'" +--------------------- + +Nested Runs +----------- + +A Megatest test can run a full Megatest run in either the same +Megatest area or in another area. This is a powerful way of chaining +complex suites of tests and or actions. + +If you are not using the current area you can use ezsteps to retrieve +and setup the sub-Megatest run area. + +In the testconfig: +--------------- +[subrun] + +# Required: wait for the run or just launch it +# if no then the run will be an automatic PASS irrespective of the actual result +run-wait yes|no + +# Optional: where to execute the run. Default is the current runarea +run-area /some/path/to/megatest/area + +# Optional: method to use to determine pass/fail status of the run +# auto (default) - roll up the net state/status of the sub-run +# logpro - use the provided logpro rules, happens automatically if there is a logpro section +# passfail auto|logpro +# Example of logpro: +passfail logpro + +# Optional: +logpro ;; if this section exists then logpro is used to determine pass/fail + (expect:required in "LogFileBody" >= 1 "At least one pass" #/PASS/) + (expect:error in "LogFileBody" = 0 "No FAILs allowed" #/FAIL/) + +# Optional: target translator, default is to use the parent target +target #{shell somescript.sh} + +# Optional: runname translator/generator, default is to use the parent runname +run-name #{somescript.sh} + +# Optional: testpatt spec, default is to first look for TESTPATT spec from runconfigs unless there is a contour spec +test-patt %/item1,test2 + +# Optional: contour spec, use the named contour from the megatest.config contour spec +contour contourname ### NOTE: Not implemented yet! Let us know if you need this feature. + +# Optional: mode-patt, use this spec for testpatt from runconfigs +mode-patt TESTPATT + +# Optional: tag-expr, use this tag-expr to select tests +tag-expr quick + +# Optional: (not yet implemented, remove-runs is always propagated at this time), propagate these actions from the parent +# test +# Note// default is % for all +propagate remove-runs archive ... + +--------------- + +Programming API +--------------- + +These routines can be called from the megatest repl. + +.API Keys Related Calls +[width="70%",cols="^,2m,2m,2m",frame="topbot",options="header,footer"] +|====================== +|API Call | Purpose comments | Returns | Comments +|(rmt:get-keys run-id) | | ( key1 key2 ... ) | +| (rmt:get-key-val-pairs run-id) | | #t=success/#f=fail | Works only if the server is still reachable +|====================== + + +:numbered!: ADDED docs/manual/study_plan.txt Index: docs/manual/study_plan.txt ================================================================== --- /dev/null +++ docs/manual/study_plan.txt @@ -0,0 +1,68 @@ +// This file is part of Megatest. +// +// Megatest is free software: you can redistribute it and/or modify +// it under the terms of the GNU General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// Megatest is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU General Public License for more details. +// +// You should have received a copy of the GNU General Public License +// along with Megatest. If not, see . +// +// Copyright 2006-2012, Matthew Welland. + +Study Plan +---------- + +Megatest is an extensive program with a lot to learn. Following are some paths through the material to smooth the learning path. + +Basic Concepts (suggest you pick these up on the way) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +* Components of automation; run, test, iteration +* Selectors; target, runname, and testpatt + +Running Testsuites or Automation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +* Using the dashboard gui (recommended) +** Using the "Runs" panel. +** Using the "Run Control" panel. +** Using a test control panel +** The Right Mouse Button menu +** Debug features +*** xterm +**** pstree +**** log files; mt_copy.log, mt_launch.log +**** variables; megatest.csh, megatest.sh +**** testconfig dump, *testconfig +*** State/status buttons +*** Run, Clean, KillReq +*** ReRunClean +* Using the command line +** Getting help; megatest -h, megatest -manual +** Starting runs; megatest -run +*** Selection controls; -target, -runname and -testpatt + +Writing Tests and Flows +~~~~~~~~~~~~~~~~~~~~~~~ + +* tests/_testname_/testconfig link:megatest_manual.html#_the_testconfig_file[testconfig details] +** iteration (one test applied to many inputs) link:megatest_manual.html#_iteration[test iteration] +** dependencies link:megatest_manual.html#_requirements_section[test requirements] +* megatest areas +** megatest.config + +Advanced Topics +~~~~~~~~~~~~~~~ + +* Removing and keeping runs selectively link:megatest_manual.html#_managing_old_runs[managing runs] +* Subruns link:megatest_manual.html#_nested_runs[nested runs] +* Config file features link:megatest_manual.html#_config_file_helpers[config file features] + + +