ADDED docs/manual/getting_started.in Index: docs/manual/getting_started.in ================================================================== --- /dev/null +++ docs/manual/getting_started.in @@ -0,0 +1,99 @@ + +Getting Started +--------------- + +[partintro] +.Getting started with Megatest +-- +Creating a testsuite or flow and your first test or task. +-- + +After installing Megatest you can create a flow or testsuite and add some +tests using the helpers. Here is a quickstart sequence to get you up and +running your first automated testsuite. + +Creating a Megatest Area +~~~~~~~~~~~~~~~~~~~~~~~~ + +Choose Target Keys +^^^^^^^^^^^^^^^^^^ + +First choose your "target" keys. These are used to organise your runs in a +way that is meaningful to your project. If you are unsure about what to use +for keys just use a single generic key such as "RUNTYPE". These keys will be +used to hand values to your tests via environment variables so ensure they +are unique. Prefixing them with something such as PROJKEYS_ is a good +strategy. + +Examples of keys: + +.Example keys +[width="60%",options="header"] +|============================================== +| Option | Description +| RELEASE/ITERATION | This example is used by Megatest for its internal QA. +| ARCH/OS/RELEASE | For a software project targeting multiple platforms +| UCTRLR/NODETYPE | Microcontroller project with different controllers +running same software +|============================================== + +Create Area Config Files +^^^^^^^^^^^^^^^^^^^^^^^^ + +You will need to choose locations for your runs (the data generated every +time you run the testsuite) and link tree. For getting started answer the +prompts with "runs" and "links". We use the Unix editor "vi" in the examples +below but you can use any plain text editor. + +.Using the helper to create a Megatest area +------------------ +megatest -create-megatest-area + +# optional: verify that the settings are ok +vi megatest.config +vi runconfigs.config +------------------ + +Creating a Test +~~~~~~~~~~~~~~~ + +Choose the test name for your first test and run the helper. You can edit +the files after the initial creation. You will need to enter names and +scripts for the steps to be run and then edit the +tests//testconfig file and modify the logpro rules to properly +process the log output from your steps. For your first test just hit enter +for the "waiton", "priority" and iteration variable prompts. + +Hint: for geting started make your logpro rules very liberal. expect:error +patterns should match nothing and comment out expect:required rules. + +.Using the helper to create a Megatest test +--------------- +megatest -create-test myfirsttest + +# then edit the generated config +vi tests/myfirsttest/testconfig +--------------- + +Running your test +~~~~~~~~~~~~~~~~~ + +First choose a target and runname. If you have a two-place target such as +RELEASE/ITERATION a target would look like v1.0/aff3 where v1.0 is the +RELEASE and aff3 is the ITERATION. For a run name just use something like +run1. + +.Running all tests (testpatt of "%" matches all tests) +--------------- +megatest -run -target v1.0/aff3 -runname run1 -testpatt % -log run1.log +--------------- + +Viewing the results +~~~~~~~~~~~~~~~~~~~ + +Start the dashboard and browse your run in the "Runs" tab. + +.Starting dashboard +---------------- +dashboard -rows 24 +---------------- ADDED docs/manual/howto.in Index: docs/manual/howto.in ================================================================== --- /dev/null +++ docs/manual/howto.in @@ -0,0 +1,191 @@ +How To Do Things +---------------- + +Process Runs +~~~~~~~~~~~~ + +Remove Runs +^^^^^^^^^^^ + +From the dashboard click on the button (PASS/FAIL...) for one of the tests. From the test control panel that +comes up push the clean test button. The command field will be prefilled with a template command for removing +that test. You can edit the command, for example change the argument to -testpatt to "%" to remove all tests. + +.Remove the test diskperf and all it's items +---------------- +megatest -remove-runs -target ubuntu/nfs/none -runname ww28.1a -testpatt diskperf/% -v +---------------- + +.Remove all tests for all runs and all targets +---------------- +megatest -remove-runs -target %/%/% -runname % -testpatt % -v +---------------- + +Archive Runs +^^^^^^^^^^^^ + +Megatest supports using the bup backup tool (https://bup.github.io/) to archive your tests for efficient storage +and retrieval. Archived data can be rapidly retrieved if needed. The metadata for the run (PASS/FAIL status, run +durations, time stamps etc.) are all preserved in the megatest database. + +For setup information see the Archiving topic in the reference section of this manual. + +To Archive +++++++++++ + +Hint: use the test control panel to create a template command by pushing the "Archive Tests" button. + +.Archive a full run +---------------- +megatest -target ubuntu/nfs/none -runname ww28.1a -archive save-remove -testpatt % +---------------- + +To Restore +++++++++++ + +.Retrieve a single test +---------------- +megatest -target ubuntu/nfs/none -runname ww28.1a -archive restore -testpatt diskperf/% +---------------- + +Hint: You can browse the archive using bup commands directly. + +---------------- +bup -d /path/to/bup/archive ftp +---------------- + +Submit jobs to Host Types based on Test Name +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.In megatest.config +------------------------ +[host-types] +general ssh #{getbgesthost general} +nbgeneral nbjob run JOBCOMMAND -log $MT_LINKTREE/$MT_TARGET/$MT_RUNNAME.$MT_TESTNAME-$MT_ITEM_PATH.lgo + +[hosts] +general cubian xena + +[launchers] +envsetup general +xor/%/n 4C16G +% nbgeneral + +[jobtools] +launcher bsub +# if defined and not "no" flexi-launcher will bypass launcher unless there is no +# match. +flexi-launcher yes +------------------------ + +Tricks +------ + +This section is a compendium of a various useful tricks for debugging, +configuring and generally getting the most out of Megatest. + +Limiting your running jobs +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The following example will limit a test in the jobgroup "group1" to no more than 10 tests simultaneously. + +In your testconfig: + +---------------- +[test_meta] +jobgroup group1 +---------------- + +In your megatest.config: + +--------------- +[jobgroups] +group1 10 +custdes 4 +--------------- + +Debugging Tricks +---------------- + +Examining The Environment +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Test Control Panel - xterm +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +From the dashboard click on a test PASS/FAIL button. This brings up a test control panel. Aproximately near the center left of the +window there is a button "Start Xterm". Push this to get an xterm with the full context and environment loaded for that test. You can run +scripts or ezsteps by copying from the testconfig (hint, load up the testconfig in a separate gvim or emacs window). This is the easiest way +to debug your tests. + +During Config File Processing +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +It is often helpful to know the content of variables in various +contexts as Megatest does the actions needed to run your tests. A handy technique is to force the startup of an xterm in the context being examined. + +For example, if an item list is not being generated as expected you +can inject the startup of an xterm as if it were an item: + +.Original items table +----------------- +[items] +CELLNAME [system getcellname.sh] +----------------- + +.Items table modified for debug +----------------- +[items] +DEBUG [system xterm] +CELLNAME [system getcellnames.sh] +----------------- + +When this test is run an xterm will pop up. In that xterm the +environment is exactly that in which the script "getcellnames.sh" +would run. You can now debug the script to find out why it isn't +working as expected. + +Organising Your Tests and Tasks +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The default location "tests" for storing tests can be extended by +adding to your tests-paths section. + +---------------------------- +[misc] +parent #{shell dirname $(readlink -f .)} + +[tests-paths] +1 #{get misc parent}/simplerun/tests +---------------------------- + +The above example shows how you can use addition sections in your +config file to do complex processing. By putting results of relatively +slow operations into variables the processing of your configs can be +kept fast. + +Alternative Method for Running your Job Script +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.Directly running job in testconfig +------------------- +[setup] +runscript main.csh +------------------- + +The runscript method is essentially a brute force way to run scripts where the +user is responsible for setting STATE and STATUS and managing the details of running a test. + +Debugging Server Problems +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Some handy Unix commands to track down issues with servers not +communicating with your test manager processes. Please put in tickets +at https://www.kiatoa.com/fossils/megatest if you have problems with +servers getting stuck. + +---------------- +sudo lsof -i +sudo netstat -lptu +sudo netstat -tulpn +---------------- ADDED docs/manual/plan.in Index: docs/manual/plan.in ================================================================== --- /dev/null +++ docs/manual/plan.in @@ -0,0 +1,189 @@ +Road Map +-------- + +Note 1: This road-map is still evolving and subject to change without notice. + +RFC M01: Add ability to move runs to other Areas +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +*Purpose*: allow shrinking megatest.db data by moving runs to an alternate + Megatest area with same keys. + +*Method*: extend db sync to take a different megatest area as a destination. + +*Design*: + +. add param -destination [area|path]. when specified runs are copied to new + area and removed from local db. +. the data move would involve these steps +.. copy the run data to destination area megatest.db +.. mark the run records as deleted, do not remove the run data on disk +. accessing the data would be by running dashboard in the satellite area +. future versions of Megatest dashboard should support displaying areas in a + merged way. +. some new controls would be supported in the config +.. [setup] => allow-runs [no|yes] <== used to disallow runs +.. [setup] => auto-migrate=[areaname|path] <== used to automatically + migrate data to a satellite area. + +*Branch*: This work is taking place on branch v1.65-reduce-records + +RFC M02: Move data into completed-runs.db +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +*Purpose*: shrink megatest.db data to enable lower load and higher performance. + +*Method*: add a completed-runs.db and automatically move runs data from megatest.db to that db + +*Design*: + +. completed-runs.db is a full megatest database with complete schema +. the data move would involve these steps +.. copy the run data to completed-runs.db +.. remove the run data, first from /tmp/…/megatest.db and /tmp/…/megatest_ref.db, followed by megatest.db +. accessing the data would be unchanged for most operations. +. a mode -full-db will be added which when specified would attach the completed-runs.db to megatest.db before doing the query +. mechanisms for moving runs to/from the megatest.db would be added +.. -reduce-records => move runs to completed-runs.db +.. -restore-records => move runs from completed-runs.db to megatest.db + +*Branch*: This work is taking place on branch v1.65-reduce-records + +RFC M03: Automatic homehost migrations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +*Purpose*: Automatically migrate homehost. + +*Method*: Check that there are no tests running, launched or remotehoststart in past ½ hour then if not on homehost migrate the db to current host + +*Design*: + +. Check that the system is quiescent, i.e. that there are no runs in flight or recently run +. Create a lock +. Migrate the /tmp cache db to the current host +. Update the .homehost file +. Remove the lock + +*Branch*: This work not yet started + +Architecture Refactor +~~~~~~~~~~~~~~~~~~~~~ + +Goals +^^^^^ + +. Reduce load on the file system. Sqlite3 files on network filesystem can be + a burden. [green]#[DONE]# +. Reduce number of servers and frequency of start/stop. This is mostly an + issue of clutter but also a reduction in "moving parts". [green]#[DONE]# +. Coalesce activities to a single home host where possible. Give the user + feedback that they have started the dashboard on a host other than the + home host. [green]#[DONE]# +. Reduce number of processes involved in managing running tests. + +Changes Needed +^^^^^^^^^^^^^^ + +. ACID compliant db will be on /tmp and synced to megatest.db with a five + second max delay. [green]#[DONE]# +. Read/writes to db for processes on homehost will go direct to /tmp + megatest.db file. [green]#[DONE]# +. Read/wites fron non-homehost processes will go through one server. Bulk + reads (e.g. for dashboard or list-runs) will be cached on the current host + in /tmp and synced from the home megatest.db in the testsuite area. [green]#[DONE]# +. Db syncs rely on the target db file timestame minus some margin. [green]#[DONE]# +. Since bulk reads do not use the server we can switch to simple RPC for the + network transport. [green]#[DONE]# +. Test running manager process extended to manage multiple running tests. + +Current Items +~~~~~~~~~~~~~ + +ww05 - migrate to inmem-db +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +. Switch to inmem db with fast sync to on disk db's [green]#[DONE]# +. Server polls tasks table for next action +.. Task table used for tracking runner process [red]#[Replaced by mtutil]# +.. Task table used for jobs to run [red]#[Replaced by mtutil]# +.. Task table used for queueing runner actions (remove runs, + cleanRunExecute, etc) [red]#[Replaced by mtutil#] + + +// ww32 +// ~~~~ +// +// . Rerun step and or subsequent steps from gui +// . Refresh test area files from gui +// . Clean and re-run button +// . Clean up STATE and STATUS handling. +// .. Dashboard and Test control panel are reverse order - choose and fix +// .. Move seldom used states and status to drop down selector +// . Access test control panel when clicking on Run Summary tests +// . Feature: -generate-index-tree +// . Change specifing of state and status to use STATE1/STATUS1,STATE2/STATUS2 +// +// ww33 +// ~~~~ +// +// . http api available for use with Perl, Ruby etc. scripts +// . megatest.config setup entries for: +// .. run launching (e.g. /bin/sh %CMD% > /dev/null) +// .. browser "konqueror %FNAME% +// +// ww34 +// ~~~~ +// +// . Mark dependent tests for clean/rerun -rerun-downstream +// . On run start check for defunct tests in RUNNING, LAUNCHED or REMOTEHOSTSTART and correct or notify +// . Fix: refresh of gui sometimes fails on last item (race condition?) +// +// ww35 +// ~~~~ +// +// . refdb: Add export of csv, json and sexp +// . Convert to using call-with-environment-variables where possible. Should allow handling of parallel runs in same process. +// . Re-work text interface wizards. Several bugs on record. Possibly convert to gui based. +// . Add to testconfig requirements section; launchlimiter scriptname, calls scriptname to check if ok to launch test +// . Refactor Run Summary view, currently very clumsy +// . Add option to show steps in Run Summary view +// +// ww36 +// ~~~~ +// +// . Refactor guis for resizeablity +// . Add filters to Run Summary view and Run Control view +// . Add to megatest.config or testconfig; rerunok STATE/STATUS,STATE/STATUS... +// . Launch gates for diskspace; /path/one>1G,/path/two>200M,/tmp>5G,#{scheme *toppath*}>1G +// +// Bin List +// ~~~~~~~~ +// +// . Quality improvements +// .. Server stutters occasionally +// .. Large number of items or tests still has some issues. +// .. Code refactoring +// .. Replace remote process with true API using json (supports Web app also) +// . Streamline the gui +// .. Everything resizable +// .. Less clutter +// .. Tool tips +// .. Filters on Run Summary, Summary and Run Control panel +// .. Built in log viewer (partially implemented) +// .. Refactor the test control panel +// . Help and documentation +// .. Complete the user manual (I’ve been working on this lately). +// .. Online help in the gui +// . Streamlined install +// .. Deployed version (download a location independent ready to run binary bundle) +// .. Install Makefile (in progress, needed for Mike to install on VMs) +// .. Added option to compile IUP (needed for VMs) +// . Server side run launching +// . Support for re-running, cleaning etc. of individual steps (ezsteps makes this very easy to implement). +// . Launch process needs built in daemonizing (easy to do, just need to test it thoroughly). +// . Wizards for creating tests, regression areas (current ones are text only and limited). +// . Fully functional built in web service (currently you can browse runs but it is very simplistic). +// . Wildcards in runconfigs: e.g. [p1271/9/%/%] +// . Gui panels for editing megatest.config and runconfigs.config +// . Fully isolated tests (no use of NFS to see regression area files) +// . Windows version ADDED docs/manual/reference.in Index: docs/manual/reference.in ================================================================== --- /dev/null +++ docs/manual/reference.in @@ -0,0 +1,798 @@ +Reference +--------- + +Config File Helpers +~~~~~~~~~~~~~~~~~~~ + +Various helpers for more advanced config files. + +.Helpers +[width="80%",cols="^,2m,2m,2m",frame="topbot",options="header"] +|====================== +|Helper | Purpose | Valid values | Comments +| #{scheme (scheme code...)} | Execute arbitrary scheme code | Any valid scheme | Value returned from the call is converted to a string and processed as part of the config file +| #{system command} | Execute program, inserts exit code | Any valid Unix command | Discards the output from the program +| #{shell command} or #{sh ...} | Execute program, inserts result from stdout | Any valid Unix command | Value returned from the call is converted to a string and processed as part of the config file +| #{realpath path} or #{rp ...} | Replace with normalized path | Must be a valid path | +| #{getenv VAR} or #{gv VAR} | Replace with content of env variable | Must be a valid var | +| #{get s v} or #{g s v} | Replace with variable v from section s | Variable must be defined before use | +| #{rget v} | Replace with variable v from target or default of runconfigs file | | +| #{mtrah} | Replace with the path to the megatest testsuite area | | +|====================== + +Config File Settings +~~~~~~~~~~~~~~~~~~~~ + +Settings in megatest.config + +Config File Additional Features +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Including output from a script as if it was inline to the config file: + +------------------------- +[scriptinc myscript.sh] +------------------------- + +If the script outputs: + +------------------------- +[items] +A a b c +B d e f +------------------------- + +Then the config file would effectively appear to contain an items section +exactly like the output from the script. This is useful when dynamically +creating items, itemstables and other config structures. You can see the +expansion of the call by looking in the cached files (look in your linktree +for megatest.config and runconfigs.config cache files and in your test run +areas for the expanded and cached testconfig). + +Wildcards and regexes in Targets + +------------------------- +[a/2/b] +VAR1 VAL1 + +[a/%/b] +VAR1 VAL2 +------------------------- + +Will result in: + +------------------------- +[a/2/b] +VAR1 VAL2 +------------------------- + +Can use either wildcard of "%" or a regular expression: + +------------------------- +[/abc.*def/] +------------------------- + +Disk Space Checks +^^^^^^^^^^^^^^^^^ + +Some parameters you can put in the [setup] section of megatest.config: + +------------------- +# minimum space required in a run disk +minspace 10000000 + +# minimum space required in dbdir: +dbdir-space-required 100000 + +# script that takes path as parameter and returns number of bytes available: +free-space-script check-space.sh +------------------- + +Trim trailing spaces +^^^^^^^^^^^^^^^^^^^^ + +------------------ +[configf:settings trim-trailing-spaces yes] +------------------ + +Job Submission Control +^^^^^^^^^^^^^^^^^^^^^^ + +Submit jobs to Host Types based on Test Name +++++++++++++++++++++++++++++++++++++++++++++ + +.In megatest.config +------------------------ +[host-types] +general nbfake +remote bsub + +[launchers] +runfirst/sum% remote +% general + +[jobtools] +launcher bsub +# if defined and not "no" flexi-launcher will bypass launcher unless +# there is no host-type match. +flexi-launcher yes +------------------------ + +host-types +++++++++++ + +List of host types and the commandline to run a job on that host type. + +.host-type => launch command +------------ +general nbfake +------------ + +launchers ++++++++++ + +.test/itempath => host-type +------------ +runfirst/sum% remote +------------ + +Miscellaneous Setup Items ++++++++++++++++++++++++++ + +Attempt to rerun tests in "STUCK/DEAD", "n/a", "ZERO_ITEMS" states. + +.In megatest.config +------------------ +[setup] +reruns 5 +------------------ + +Replace the default blacklisted environment variables with user supplied +list. + +Default list: USER HOME DISPLAY LS_COLORS XKEYSYMDB EDITOR MAKEFLAGS MAKEF MAKEOVERRIDES + +.Add a "bad" variable "PROMPT" to the variables that will be commented out +in the megatest.sh and megatest.csh files: +----------------- +[setup] +blacklistvars USER HOME DISPLAY LS_COLORS XKEYSYMDB EDITOR MAKEFLAGS PROMPT +----------------- + +Run time limit +++++++++++++++ + +----------------- +[setup] +# this will automatically kill the test if it runs for more than 1h 2m and 3s +runtimelim 1h 2m 3s +----------------- + +Tests browser view +~~~~~~~~~~~~~~~~~~ + +The tests browser (see the Run Control tab on the dashboard) has two views for displaying the tests. + +. Dot (graphviz) based tree +. No dot, plain listing + +The default is the graphviz based tree but if your tests don't view +well in that mode then use "nodot" to turn it off. + +----------------- +[setup] +nodot +----------------- + +Capturing Test Data +~~~~~~~~~~~~~~~~~~~ + +In a test you can capture arbitrary variables and roll them up in the +megatest database for viewing on the dashboard or web app. + +.In a test as a script +------------------------ +$MT_MEGATEST -load-test-data << EOF +foo,bar, 1.2, 1.9, > +foo,rab, 1.0e9, 10e9, 1e9 +foo,bla, 1.2, 1.9, < +foo,bal, 1.2, 1.2, < , ,Check for overload +foo,alb, 1.2, 1.2, <= , Amps,This is the high power circuit test +foo,abl, 1.2, 1.3, 0.1 +foo,bra, 1.2, pass, silly stuff +faz,bar, 10, 8mA, , ,"this is a comment" +EOF +------------------------ + +Alternatively you can use logpro triggers to capture values and inject them +into megatest using the -set-values mechanism: + +.Megatest help related to -set-values +------------------------ +Test data capture + -set-values : update or set values in the testdata table + :category : set the category field (optional) + :variable : set the variable name (optional) + :value : value measured (required) + :expected : value expected (required) + :tol : |value-expect| <= tol (required, can be <, >, >=, <= or number) + :units : name of the units for value, expected_value etc. (optional) +------------------------ + +Dashboard settings +~~~~~~~~~~~~~~~~~~ + +.Runs tab buttons, font and size +------------------ +[dashboard] +btn-height x14 +btn-fontsz 10 +cell-width 60 +------------------ + +Database settings +~~~~~~~~~~~~~~~~~ + +.Database config settings in [setup] section of megatest.config +[width="70%",cols="^,2m,2m,2m",frame="topbot",options="header"] +|====================== +|Var | Purpose | Valid values | Comments +|delay-on-busy | Prevent concurrent access issues | yes\|no or not defined | Default=no, may help on some network file systems, may slow things down also. +|faststart | All direct file access to sqlite db files | yes\|no or not defined | Default=yes, suggest no for central automated systems and yes for interactive use +|homehost | Start servers on this host | | Defaults to local host +|hostname | Hostname to bind to | \|- | On multi-homed hosts allows binding to specific hostname +|lowport | Start searching for a port at this portnum| 32768 | +|required | Server required | yes\|no or not defined | Default=no, force start of server always +|server-query-threshold | Start server when queries take longer than this | number in milliseconds | Default=300 +|timeout | http api timeout | number in hours | Default is 1 minute, do not change +|====================== + +The testconfig File +------------------- + +Setup section +~~~~~~~~~~~~~ + +Header +^^^^^^ + +------------------- +[setup] +------------------- + +The runscript method is a brute force way to run scripts where the +user is responsible for setting STATE and STATUS + +------------------- +runscript main.csh +------------------- + +Requirements section +~~~~~~~~~~~~~~~~~~~~ + +.Header +------------------- +[requirements] +------------------- + +Wait on Other Tests +^^^^^^^^^^^^^^^^^^^ + +------------------- +# A normal waiton waits for the prior tests to be COMPLETED +# and PASS, CHECK or WAIVED +waiton test1 test2 +------------------- + +Mode +^^^^ + +The default (i.e. if mode is not specified) is normal. All pre-dependent tests +must be COMPLETED and PASS, CHECK or WAIVED before the test will start + +------------------- +[requirements] +mode normal +------------------- + +The toplevel mode requires only that the prior tests are COMPLETED. + +------------------- +[requirements] +mode toplevel +------------------- + +A item based waiton will start items in a test when the same-named +item is COMPLETED and PASS, CHECK or WAIVED in the prior test. This +was historically called "itemwait" mode. The terms "itemwait" and +"itemmatch" are synonyms. + +------------------- +[requirements] +mode itemmatch +------------------- + +Overriding Enviroment Variables +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Override variables before starting the test. Can include files (perhaps generated by megatest -envdelta or similar). + +-------------------- +[pre-launch-env-vars] +VAR1 value1 + +# Get some generated settings +[include ../generated-vars.config] + +# Use this trick to unset variables +#{scheme (unsetenv "FOOBAR")} +-------------------- + +Itemmap Handling +~~~~~~~~~~~~~~~~ + +For cases were the dependent test has a similar but not identical +itempath to the downstream test an itemmap can allow for itemmatch +mode + +.example for removing part of itemmap for waiton test (eg: item +foo-x/bar+ depends on waiton's item +y/bar+) +------------------- +[requirements] +mode itemwait +# itemmap +itemmap .*x/ y/ + +------------------- + +.example for removing part of itemmap for waiton test (eg: item +foo/bar/baz+ in this test depends on waiton's item +baz+) +------------------- + +# ## pattern replacement notes +# +# ## Example +# ## Remove everything up to the last / +[requirements] +mode itemwait +# itemmap +itemmap .*/ +------------------- + +.example replacing part of itemmap for (eg: item +foo/1234+ will imply waiton's item +bar/1234+) +------------------- + +# +# ## Example +# ## Replace foo/ with bar/ +[requirements] +mode itemwait +# itemmap +itemmap foo/ bar/ + +------------------- + +.example for backreference (eg: item +foo23/thud+ will imply waiton's item +num-23/bar/thud+ +------------------- +# +# ## Example +# ## can use \{number} in replacement pattern to backreference a (capture) from matching pattern similar to sed or perl +[requirements] +mode itemwait +# itemmap +itemmap foo(\d+)/ num-\1/bar/ + +------------------- + +.example multiple itemmaps +------------------- + +# multi-line; matches are applied in the listed order +# The following would map: +# a123b321 to b321fooa123 then to 321fooa123p +# +[requirements] +itemmap (a\d+)(b\d+) \2foo\1 + b(.*) \1p +------------------- + + +Complex mapping +^^^^^^^^^^^^^^^ +Complex mappings can be handled with a separate [itemmap] section (instead if an itemmap line in the [requirements] section) + +Each line in an itemmap section starts with a waiton test name followed by an itemmap expression + +.eg: The following causes waiton test A item +bar/1234+ to run when our test's +foo/1234+ item is requested as well as causing waiton test B's +blah+ item to run when our test's +stuff/blah+ item is requested +-------------- +[itemmap] +A foo/ bar/ +B stuff/ +-------------- + + +Complex mapping example +^^^^^^^^^^^^^^^^^^^^^^^ + + + +// image::itemmap.png[] +image::complex-itemmap.png[] + + +We accomplish this by configuring the testconfigs of our tests C D and E as follows: + +.Testconfig for Test E has +---------------------- +[requirements] +waiton C +itemmap (\d+)/res \1/bb +---------------------- + +.Testconfig for Test D has +---------------------- +[requirements] +waiton C +itemmap (\d+)/res \1/aa +---------------------- + +.Testconfig for Test C has +---------------------- +[requirements] +waiton A B + +[itemmap] +A (\d+)/aa aa/\1 +B (\d+)/bb bb/\1 +---------------------- + +.Testconfigs for Test B and Test A have no waiton or itemmap configured +------------------- +------------------- + +.Walk through one item -- we want the following to happen for testpatt +D/1/res+ (see blue boxes in complex itemmaping figure above): + +. eg from command line +megatest -run -testpatt D/1/res -target mytarget -runname myrunname+ +. Full list to be run is now: +D/1/res+ +. Test D has a waiton - test C. Test D's itemmap rule +itemmap (\d+)/res \1/aa+ -> causes +C/1/aa+ to run before +D/1/res+ +. Full list to be run is now: +D/1/res+, +C/1/aa+ +. Test C was a waiton - test A. Test C's rule +A (\d+)/aa aa/\1+ -> causes +A/aa/1+ to run before +C/1/aa+ +. Full list to be run is now: +D/1/res+, +C/1/aa+, +A/aa/1+ +. Test A has no waitons. All waitons of all tests in full list have been processed. Full list is finalized. + + +Dynamic Flow Dependency Tree +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.Autogeneration waiton list for dynamic flow dependency trees +------------------- +[requirements] +# With a toplevel test you may wish to generate your list +# of tests to run dynamically +# +waiton #{shell get-valid-tests-to-run.sh} +------------------- + +Run time limit +^^^^^^^^^^^^^^ + +----------------- +[requirements] +runtimelim 1h 2m 3s # this will automatically kill the test if it runs for more than 1h 2m and 3s +----------------- + +Skip +^^^^ + +A test with a skip section will conditional skip running. + +.Skip section example +----------------- +[skip] +prevrunning x +# rundelay 30m 15s +----------------- + +Skip on Still-running Tests +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +----------------- +# NB// If the prevrunning line exists with *any* value the test will +# automatically SKIP if the same-named test is currently RUNNING. The +# "x" can be any string. Comment out the prevrunning line to turn off +# skip. + +[skip] +prevrunning x +----------------- + +Skip if a File Exists +^^^^^^^^^^^^^^^^^^^^^ + +----------------- +[skip] +fileexists /path/to/a/file # skip if /path/to/a/file exists +----------------- + +Skip if test ran more recently than specified time +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.Skip if this test has been run in the past fifteen minutes and 15 seconds. +----------------- +[skip] +rundelay 15m 15s +----------------- + +Disks +^^^^^ + +A disks section in testconfig will override the disks section in +megatest.config. This can be used to allocate disks on a per-test or per item +basis. + +Controlled waiver propagation +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If test is FAIL and previous test in run with same MT_TARGET is WAIVED then apply the following rules from the testconfig: +If a waiver check is specified in the testconfig apply the check and if it passes then set this FAIL to WAIVED + +Waiver check has two parts, 1) a list of waiver, rulename, filepatterns and 2) the rulename script spec (note that "diff" and "logpro" are predefined) + +----------------- +###### EXAMPLE FROM testconfig ######### +# matching file(s) will be diff'd with previous run and logpro applied +# if PASS or WARN result from logpro then WAIVER state is set +# +[waivers] +# logpro_file rulename input_glob +waiver_1 logpro lookittmp.log + +[waiver_rules] + +# This builtin rule is the default if there is no .logpro file +# diff diff %file1% %file2% + +# This builtin rule is applied if a .logpro file exists +# logpro diff %file1% %file2% | logpro %waivername%.logpro %waivername%.html +----------------- + +Ezsteps +~~~~~~~ + +.Example ezsteps with logpro rules +----------------- +[ezsteps] +lookittmp ls /tmp + +[logpro] +lookittmp ;; Note: config file format supports multi-line entries where leading whitespace is removed from each line + ;; a blank line indicates the end of the block of text + (expect:required in "LogFileBody" > 0 "A file name that should never exist!" #/This is a awfully stupid file name that should never be found in the temp dir/) + +----------------- + +To transfer the environment to the next step you can do the following: + +---------------------------- +$MT_MEGATEST -env2file .ezsteps/${stepname} +---------------------------- + +Triggers +~~~~~~~~ + +In your testconfig or megatest.config triggers can be specified + +----------------- +[triggers] + +# Call script running.sh when test goes to state=RUNNING, status=PASS +RUNNING/PASS running.sh + +# Call script running.sh any time state goes to RUNNING +RUNNING/ running.sh + +# Call script onpass.sh any time status goes to PASS +PASS/ onpass.sh +----------------- + +Scripts called will have; test-id test-rundir trigger test-name item-path state status event-time, added to the commandline. + +HINT + +To start an xterm (useful for debugging), use a command line like the following: + +----------------- +[triggers] +COMPLETED/ xterm -e bash -s -- +----------------- + +NOTE: There is a trailing space after the -- + +There are a number of environment variables available to the trigger script +but since triggers can be called in various contexts not all variables are +available at all times. The trigger script should check for the variable and +fail gracefully if it doesn't exist. + +.Environment variables visible to the trigger script +[width="90%",cols="^,2m",frame="topbot",options="header"] +|====================== +|Variable | Purpose +| MT_TEST_RUN_DIR | The directory where Megatest ran this test +| MT_CMDINFO | Encoded command data for the test +| MT_DEBUG_MODE | Used to pass the debug mode to nested calls to Megatest +| MT_RUN_AREA_HOME | Megatest home area +| MT_TESTSUITENAME | The name of this testsuite or area +| MT_TEST_NAME | The name of this test +| MT_ITEM_INFO | The variable and values for the test item +| MT_MEGATEST | Which Megatest binary is being used by this area +| MT_TARGET | The target variable values, separated by '/' +| MT_LINKTREE | The base of the link tree where all run tests can be found +| MT_ITEMPATH | The values of the item path variables, separated by '/' +| MT_RUNNAME | The name of the run +|====================== + + +Override the Toplevel HTML File +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Megatest generates a simple html file summary for top level tests of +iterated tests. The generation can be overridden. NOTE: the output of +the script is captured from stdout to create the html. + + +.For test "runfirst" override the toplevel generation with a script "mysummary.sh" +----------------- +# Override the rollup for specific tests +[testrollup] +runfirst mysummary.sh +----------------- + +Archiving Setup +--------------- + +In megatest.config add the following sections: + +.megatest.config +-------------- +[archive] +# where to get bup executable +# bup /path/to/bup + +[archive-disks] + +# Archives will be organised under these paths like this: +# / +# Within the archive the data is structured like this: +# /// +archive0 /mfs/myarchive-data/adisk1 +-------------- + +Handling Environment Variables +------------------------------ + +It is often necessary to capture and or manipulate environment +variables. Megatest has some facilities built in to help. + +Capture variables +~~~~~~~~~~~~~~~~~ + +.Commands +------------------------------ +# capture the current enviroment into a db called envdat.db under +# the context "before" +megatest -envcap before + +# capture the current environment into a db called startup.db with +# context "after" +megatest -envcap after startup.db + +# write the diff from before to after +megatest -envdelta before-after -dumpmode bash +------------------------------ + +Dump modes include bash, csh and config. You can include config data +into megatest.config or runconfigs.config. + +.Example of generating and using config data +------------------------------ +megatest -envcap original +# do some stuff here +megatest -envcap munged +megatest -envdelta original-munged -dumpmode ini -o modified.config +------------------------------ + +Then in runconfigs.config + +.Example of using modified.config in a testconfig +------------------------------ +cat testconfig +[pre-launch-env-vars] +[include modified.config] +------------------------------ + +Managing Old Runs +----------------- + +It is often desired to keep some older runs around but this must be balanced with the costs of disk space. + +. Use -remove-keep +. Use -archive (can also be done from the -remove-keep interface) +. use -remove-runs with -keep-records + +.For each target, remove all runs but the most recent 3 if they are over 1 week old +--------------------- +# use -precmd 'sleep 5;nbfake' to limit overloading the host computer but to allow the removes to run in parallel. +megatest -actions print,remove-runs -remove-keep 3 -target %/%/%/% -runname % -age 1w -precmd 'sleep 5;nbfake'" +--------------------- + +Nested Runs +----------- + +A Megatest test can run a full Megatest run in either the same +Megatest area or in another area. This is a powerful way of chaining +complex suites of tests and or actions. + +If you are not using the current area you can use ezsteps to retrieve +and setup the sub-Megatest run area. + +In the testconfig: +--------------- +[subrun] + +# Required: wait for the run or just launch it +# if no then the run will be an automatic PASS irrespective of the actual result +run-wait yes|no + +# Optional: where to execute the run. Default is the current runarea +run-area /some/path/to/megatest/area + +# Optional: method to use to determine pass/fail status of the run +# auto (default) - roll up the net state/status of the sub-run +# logpro - use the provided logpro rules, happens automatically if there is a logpro section +# passfail auto|logpro +# Example of logpro: +passfail logpro + +# Optional: +logpro ;; if this section exists then logpro is used to determine pass/fail + (expect:required in "LogFileBody" >= 1 "At least one pass" #/PASS/) + (expect:fail in "LogFileBody" = 0 "No FAILs allowed" #/FAIL/) + +# Optional: target translator, default is to use the parent target +target #{shell somescript.sh} + +# Optional: runname translator/generator, default is to use the parent runname +run-name #{somescript.sh} + +# Optional: testpatt spec, default is to first look for TESTPATT spec from runconfigs unless there is a contour spec +test-patt %/item1,test2 + +# Optional: contour spec, use the named contour from the megatest.config contour spec +contour contourname ### NOTE: Not implemented yet! Let us know if you need this feature. + +# Optional: mode-patt, use this spec for testpatt from runconfigs +mode-patt TESTPATT + +# Optional: tag-expr, use this tag-expr to select tests +tag-expr quick + +# Optional: (not yet implemented, remove-runs is always propagated at this time), propagate these actions from the parent +# test +# Note// default is % for all +propagate remove-runs archive ... + +--------------- + +Programming API +--------------- + +These routines can be called from the megatest repl. + +.API Keys Related Calls +[width="70%",cols="^,2m,2m,2m",frame="topbot",options="header,footer"] +|====================== +|API Call | Purpose comments | Returns | Comments +|(rmt:get-keys run-id) | | ( key1 key2 ... ) | +| (rmt:get-key-val-pairs run-id) | | #t=success/#f=fail | Works only if the server is still reachable +|====================== + + +:numbered!: