Megatest

From Robert: 
      There is a real issue with the time it takes megatest to process runconfigs. 
 
      Every #{shell ...} command is processed in every target from every included 
file even if the target is not the one being run, or a simple -list-targets 
is given on the command line. 
 
To make matters worse, when running a regression, not only does the main 
megatest "mtest" call processes runconfigs, so does any server started by 
the regression run. 
 
And if it takes a LONG time to process the runconfigs, the regression run 
will spend a lot of time waiting for the server to start. 
      Every single call to megatest, whether its to start the regression, or start 
      a server, or whatever, is subject to the time it takes to process the 
      runconfig files. 
 
      The parsing of runconfigs should happen 1 time per session, and the 
      evaluation of value should only happen 1 time and when its needed/used 
      (there should not be a global evaluation of all parameters across all 
      targets unless there is some kind of dump operation occurring). 
 
While I appreciate that there was some original reason/intent for re-parsing 
the runconfigs the entire process is leading to a substantial performance 
issues in regression runs. Even if there is a need to re-evaluate a 
variable value, there should be a means of indicating so, like: 
 
      VAR #{eval ...} 
 
Something needs to be done to improve performance of reading the runconfigs.

When a variable is defined in a target earlier in the runconfig file, with no value, like 
 
MYENVVAR 
 
it appears with the name "MYENVVAR=" in the environment of a test run from a later target in the runconfig. It does not appear when doing megatest -show-runconfig -target <target> 
 
The workaround is to put "" for the value of the var.

For tests that have a step which is 2 commands joined with an && it's causing problems.  The workaround for now is to split the command into two steps.

Seeing instances of batch jobs running megatest far exceeding the runtime limit that is in the config.   
A couple of possible causes: 
megatest kills the job but the batch system doesn't detect that it's dead 
megatest does not honor the limit 
the limit could be getting overriden somewhere else that we're not seeing

If a test in the test list for a run has a typo like "items]" instead of "[items]" (missing bracket) it causes an env check to fail at the top level (precheck test) megatest.csh will contain "#setenv items]", and precheck will flag an illegal env var.

If the target has been removed from the runconfig, the run does not show up in dashboard, and cannot be removed using the dashboard.

When a user puts a !FOO variable into an include in the runconfig that matches everything, like [/.*/], it appears to match default as well, and when that happens, those variables leak into the environment.

../

Example problem: 
from runconfigs.config: 
[full] 
 ... 
[/.*/] 
THINGS_LIST [system #{getenv THING_ORACLE_EXE} listem] 
OKSTUFF    hello 
 
% megatest -showrunconfig 
THINGS_LIST expected_value 
OKSTUFF hello 
 
% megatest -showrunconfig -target full 
OKSTUFF hello 
  **THINGS_LIST does not appear**

All the target definitions have "runs" in the run tree browser.

I have observed that if you are running dashboard on a non-homehost it will work until the database size reaches some threshold, at which point it will freeze.  When restarting, it immediately freezes.  I suspect the database size is the critical dimension -- syncing likely takes longer than the action loop in dashboard. 
 
In the instance I see it, the megatest.db size is 5.4M with 4k tests and 11 runs. 
 
Running dashboard on homehost, there is no issue at all and dashboard responds very quickly. 
 
Here is the console log from dashboard when we see the freezing issue: 
 
 
Checking for user runconfigs config ... 
WARNING  lefkowit.runconfigs.config file ...doesn't exist 
do this if you care ### Add custom targets/overrides to global targets here to  ./configs/lefkowit.runconfigs.config 
 
Checking for required/common linktree path: /p/fdk/gwa/lefkowit/qa/libanaacatqa/links ... 
exists   /p/fdk/gwa/lefkowit/qa/libanaacatqa/links 
Defined disks: 
------------------------------------ 
exists   /p/fdk/gwa/lefkowit/qa/libanaacatqa/runs 
------------------------------------ 
Checking if megatest variables are already defined ... 
STARTING DASHBOARD ... 
WARNING: Current policy requires running dashboard on homehost: (10.38.45.46 . #f) 
INFO: Trying to start server (megatest -server 10.38.45.46 -m testsuite:libanaacatqa) ... 
INFO: (0) Starting server on 10.38.45.46, logfile is /nfs/pdx/disks/ch_icf_megatest_001/lefkowit/mtTesting/megatestqa/libanaacatqa/logs/server.log 
#====================================================================== 
# NBFAKE logging command to: /nfs/pdx/disks/ch_icf_megatest_001/lefkowit/mtTesting/megatestqa/libanaacatqa/logs/server.log 
#      megatest -server 10.38.45.46 -m testsuite:libanaacatqa 
#====================================================================== 
WARNING: problem with directory /p/fdk/gwa/lefkowit/fossil/ext/acatqa_ext/trunk/tests, dropping it from tests path 
WARNING: problem with directory /nfs/pdx/disks/ch_icf_megatest_001/lefkowit/mtTesting/megatestqa/relqa/tests, dropping it from tests path 
WARNING: problem with directory /nfs/pdx/disks/ch_icf_megatest_001/lefkowit/mtTesting/megatestqa/libanaacatqa/tests, dropping it from tests path 
NOTE: updates are taking a long time, 3.0s elapsed. 
NOTE: increasing poll interval from 1000 to 2000 
NOTE: updates are taking a long time, 3.0s elapsed. 
NOTE: increasing poll interval from 2000 to 4000 
NOTE: updates are taking a long time, 3.0s elapsed. 
NOTE: increasing poll interval from 4000 to 8000 
NOTE: updates are taking a long time, 3.0s elapsed. 
NOTE: increasing poll interval from 8000 to 16000 
NOTE: updates are taking a long time, 3.0s elapsed. 
NOTE: increasing poll interval from 16000 to 32000 
NOTE: updates are taking a long time, 3.0s elapsed. 
NOTE: increasing poll interval from 32000 to 64000 
NOTE: updates are taking a long time, 3.0s elapsed. 
NOTE: increasing poll interval from 64000 to 128000 
NOTE: updates are taking a long time, 3.0s elapsed. 
NOTE: increasing poll interval from 128000 to 256000 
NOTE: updates are taking a long time, 3.0s elapsed. 
NOTE: increasing poll interval from 256000 to 512000 
... 
NOTE: increasing poll interval from 1125899906842624000 to 2251799813685248000 
NOTE: updates are taking a long time, 3.0s elapsed. 
NOTE: increasing poll interval from 2251799813685248000 to 4503599627370496000 
NOTE: updates are taking a long time, 3.0s elapsed. 
 
        Call history: 
 
        db.scm:1972: call-with-current-continuation 
        db.scm:1972: with-exception-handler 
        db.scm:1972: ##sys#call-with-values 
        db.scm:1972: k3436 
        db.scm:1972: g3440 
        dashboard.scm:795: current-seconds 
        dashboard.scm:798: dboard:tabdat-allruns-by-id 
        dashboard.scm:798: hash-table-set! 
        dashboard.scm:803: debug:print 
        common_records.scm:131: debug:debug-mode 
        common_records.scm:132: with-output-to-port 
        dashboard.scm:804: iup-base#attribute 
        dashboard.scm:805: floor 
        dashboard.scm:3551: k7615 
        dashboard.scm:3551: g7619 
        dashboard.scm:3551: print-call-chain            <-- 
inexact number cannot be represented as an exact number 
Callback error in dashboard:runs-tab-updater 
Full condition info: 
((exn (location inexact->exact) (call-chain (#(db.scm:1978: loop #f #f) #(db.scm:1978: loop #f #f) #(db.scm:1978: loop #f #f) #(db.scm:1972: call-with-current-continuation #f #f) #(db.scm:1972: with-exception-handler #f #f) #(db.scm: 
1972: ##sys#call-with-values #f #f) #(db.scm:1972: k3436 #f #f) #(db.scm:1972: g3440 #f #f) #(dashboard.scm:795: current-seconds #f #f) #(dashboard.scm:798: dboard:tabdat-allruns-by-id #f #f) #(dashboard.scm:798: hash-table-set! #f # 
f) #(dashboard.scm:803: debug:print #f #f) #(common_records.scm:131: debug:debug-mode #f #f) #(common_records.scm:132: with-output-to-port #f #f) #(dashboard.scm:804: iup-base#attribute #f #f) #(dashboard.scm:805: floor #f #f))) (arg 
uments (9.00719925474099e+18)) (message inexact number cannot be represented as an exact number)) (type)) 
NOTE: updates are taking a long time, 3.0s elapsed. 
<stack dump repeats regularly, probably in the dashboard event loop>

If linktree is changed in megatest.config, lt link in MT_RUN_AREA_HOME should be updated to match.

In an active run, the run summary matrix view induces headaches in yours truly.  The column widths vary wildly from one second to the next.  Eg: on second, the column fits width of "REMOTEHOSTSTART" and the next fits "PASS".  multiply this delta by # of tests.

motivation - it is hard to debug when a server has gone wrong.  this is a feature indented for analysts and developers.

current behavior when client version does not match server version is for client to launch another server.  net result is a darwinian struggle of multiple version variants of megatest server.

<verbatim> 
SETUP: 
0) setup megatest build from tag v1.6407)  
1) under megatest folder, create ext-tests.  Poplulate subdir from megatest_qa fossil using commit d3e9c96dc307d7c941ea973d524adb4b267cbd56 (currently tip at time of writing) 
2) Using megatest_qa testsuite, perform two runs, say named "run1" and "run2".  run1 completes first.  All tests in both runs should resolve to COMPLETED/PASS. 
3) In run1, manually set tconfdisks/tom/n to RUNNING/PASS 
5) in run1, manually set tconfdisks/fred/d to RUNNING/FAIL 
6) in run1, manually set tconfdisks/bob/p to COMPLETED/FAIL 
7) Make sure "HIDE" mode is selected for filters. 
8) select state filter "RUNNING" 
9) select status filter "FAIL" 
 
OBSERVE: 
two columns 
a. run2 having tconfdisks topeleven and items tom/n and fred/d items as "RUNNING" 
b. run1 having tconfdisks toplevel and items tom/n and fred/d as "RUNNING" 
c. run1 having tconfdisks item bob/p "FAIL". 
 
ISSUE: 
item a above is unexpected, since they are in fact COMPLETED/PASS in the database. 
item b and c are expected and correct. 
 
 
</verbatim>

If a test has items, requesting update to the state/status of the top-level test to roll up.

Since default and misc and % expand to non-sensical values and are not valid targets for runs, we do not expect to see these as selectable targets in run control panel.

Following from xterm popped up from deleting runname 164am: (accessed by right clicking on a test in this run on dashboard->run->clean complete run 
 
INFO: (1) Recursively removing /nfs/pdx/disks/icf_env_disk001/bjbarcla/gwa/issues/mtdev/randy-slow/reproduce/qa/libanabeqa/links/p1275/2/PDK_r2.0/libana/164am/oalib_propbag_diff 
 
Error: (directory) cannot open directory - No such file or directory: "/nfs/pdx/disks/icf_env_disk001/bjbarcla/gwa/issues/mtdev/randy-slow/reproduce/qa/libanabeqa/links/p1275/2/PDK 
_r2.0/libana/164am/oalib_propbag_diff" 
 
        Call history: 
 
        common_records.scm:224: debug:debug-mode           
        common_records.scm:225: port?      
        common_records.scm:225: with-output-to-port        
        runs.scm:1903: file-exists?        
        runs.scm:1905: debug:print-info    
        common_records.scm:224: debug:debug-mode           
        common_records.scm:225: port?      
        common_records.scm:225: with-output-to-port        
        runs.scm:1906: file-exists?        
        runs.scm:1907: runs:safe-delete-test-dir           
        runs.scm:1662: directory-utils#directory-fold      
        directory-utils.scm:128: ##sys#get-keyword         
        directory-utils.scm:130: type-checks#check-procedure       
        directory-utils.scm:132: check-directory           
        directory-utils.scm:79: directory?         
        directory-utils.scm:132: directory              <--

<verbatim> 
 mtutil run -area ext-tests -target a/b/c/d 
 
Error: (null-list?) bad argument type - not a list: #<hash-table (2)> 
 
        Call history: 
 
        configf.scm:50: hash-table-ref/default 
        configf.scm:49: config:assoc-safe-add 
        configf.scm:41: ##sys#get-keyword 
        configf.scm:42: filter 
        configf.scm:43: append 
        configf.scm:48: hash-table-set! 
        mtut.scm:843: configf:lookup 
        configf.scm:416: hash-table? 
        configf.scm:417: hash-table-ref/default 
        mtut.scm:844: string-split 
        mtut.scm:845: hash-table-copy 
        mtut.scm:841: ##sys#call-with-values 
        mtut.scm:852: command-line->pkt 
        mtut.scm:392: current-seconds 
        mtut.scm:397: current-user-name 
        mtut.scm:410: filter            <-- 
</verbatim>

changing status from fail to pass works 
changing status from fail to waived brings up the waive dialog, but on "apply and close" click, the message "Error: in callback: unbound variable: rmt:test-set-state-status-by" appears on the console, and dialog does not close.  status does not change.

In the run tab, the width of matrix columns for runs is fixed.  If a field value for the target or the run name spans more than ~14 characters, it is cut off. 
 
This could be worked around in v1.60/29 by stretching the entire window horizontally by dragging the right corner out.  Since the matrix was bound to the right side, the column widths would scale out.  In current version (1.63/05) the matrix is not bound to the left edge, so stretching has no effect. 
 
Marking Severe; reported by end user.

In Run tab, when viewing test names on leftmost column, the test names look like they have blank spaces where one would expect underscores.  This was not the case in 1.60/29 
 
(marking severe - reported by two users)

For example, plxcm7003. 
 
Hang occurs when switching to dashboard "run" or "run summary" tab

The sort +testname and -testname in dashboard are not sorting testname alphabetically - this should take the itempath into account if it exists.

In megatest 1.6104 -list-targets attempts to create a symlink, so cannot check targets in another user's area. 
 
NEEDS A TEST?

Non-dashboard megatest commands are requiring DISPLAY to be set

creating a new refdb with "refdb set" does not create the required sxml files to be able to open the new refdb via "refdb edit"

refdb set command does not replace existing lines in the .dat file when updating existing keys: 
 
refdb set db row col data 
 
[] 
[col] 
row data 
 
... 
 
refdb set db row col data 
[] 
[col] 
row data 
row data

When having multiple runs, when I launch a new run, I click on the new run name that appears on the left pane of the Run Summary tab.  The pain pane still shows results for the run that was previously selected.  If I reselect the prior run, then select the new run again, it updates properly.

in my testconfig, I have a step which runs a script having the code "exit 6" 
 
In the test control panel, the Steps table correctly reflects Status=6 for the step. 
 
However, the overall status is "FAIL".  I expected "SKIP".

 
Warning (#<thread: Main loop>): in thread: call of non-procedure: #f 
 
        Call history: 
 
        dashboard.scm:1853: file-modification-time 
        dashboard.scm:1853: file-modification-time 
        dashboard.scm:1853: file-modification-time 
        dashboard.scm:1853: file-modification-time 
        dashboard.scm:1853: file-modification-time 
        dashboard.scm:1846: k3511 
        dashboard.scm:1846: g3515 
        dashboard.scm:1858: file-exists? 
        dashboard.scm:1859: file-modification-time 
        dashboard.scm:1861: current-seconds 
        dashboard.scm:1862: d:alldat-please-update 
        dashboard.scm:1862: d:alldat-last-db-update 
        dashboard.scm:1862: dashboard:recalc 
        dashboard.scm:1863: d:alldat-curr-tab-num 
        dashboard.scm:1871: d:alldat-curr-tab-num 
        dashboard.scm:1889: dashboard:update-new-view-tab               <-- 
 
Error: uncaught exception: #<condition: (exn type)> 
 
[6]    Exit 70                       dashboard -rows 25

When a testconfig is missing a specification of things to run, eg in [ezsetps] section, the test passes. 
 
II would expect the test to fail since its specification is incomplete. 
 
IIn my case, the error was a typo in spelling "ezsteps", but the test passed whereas it should have failed since the underlying step was actually failing but was masked by this issue.

Run data intermittently disappears from the megatest dashboard. It can be recovered by closing the dashboard and reopening it, but "Refresh" does nothing.

When I expand a test with many subtests (200), if I use the scrollbar to scroll to the bottom, some tests still do not show up. This is intermittent. Sometimes if I scroll to the top and then back down, it then shows all tests; other times I cannot see the additional tests at all.

Its the central run dir. Pls let me know if you need more info. 
 
Warning (#<thread: Main loop>): in thread: (finalize!) database is locked: #<sqlite3:statement sql="(5�"> 
 
        Call history: 
 
        db.scm:3181: thread-sleep! 
        db.scm:3182: db:delay-if-busy 
        db.scm:3165: ##sys#get-keyword 
        db.scm:3166: configf:lookup 
        configf.scm:288: hash-table? 
        configf.scm:289: hash-table-ref/default 
        db.scm:3169: db:dbdat-get-path 
        db.scm:3170: db:dbdat-get-db 
        db.scm:3171: conc 
        db.scm:3172: call-with-current-continuation 
        db.scm:3172: with-exception-handler 
        db.scm:3172: ##sys#call-with-values 
        db.scm:3178: file-exists? 
        db.scm:3172: k3877 
        db.scm:3172: g3881 
        db.scm:2039: sqlite3#for-each-row               <-- 
 
Error: uncaught exception: #<condition: (exn sqlite3)>

megatest -list-runs % -state COMPLETED -status FAIL,CHECK does not filter on state/status

With multiple waiton dependencies, the items are applied only to the last waiton dep.

When using itemmap and mode itemwait, if the mapped item contains a '%' the itemwait is behaving like toplevel - it is not starting the item when its dependencies have been met. 
 
Example, 
 
TESTA (waiton TESTB, mode itemwait) 
  ITEM1/fluff 
  ITEM2/fluff 
TESTB 
  repo/ITEM1 
  repo/ITEM2 
 
If I use itemmap to convert ITEM1/fluff to %/ITEM1, TESTA/ITEM1/fluff does not start until all items in TESTB are complete.

Tests with items are rolling up to UNKNOWN state rather than PASS/FAIL/WARN as applicable. 
 
This is happening with any number of items - I've tried 1 and 15.

I was helping a new user get set up today and I think we hit a first-use behavior in the dashboard. The behavior we noticed is that tests with test items start as expanded, which is fine. I had him select the collapse button and the test in progress collapsed. But as each successive test was populated with test items, that new test was presented as expanded. It appears that if there is nothing to collapse in a test and there is no previous regression using the test, the collapse is not applied.

That is a tough one. It seems it could be considered a bug in Megatest. I’m treating a toplevel test in the same way as I’m treating items and non-itemized tests. What I need to do is not apply the remove to the FAIL toplevel if there are any non-fail items. This will be a bit tough and I’d like to wait until 1.60 to fix it.

Runs with failed tests showing as running when a test launches a new megatest instance.

<verbatim>INFO: (1) Test exited in state=RUNNING, setting state/status based on exit code of #t and rollup-status of 127 
 
Error: (for-each) bad argument type - not a list: #<unspecified> 
 
        Call history: 
 
        http-transport.scm:274: thread-terminate!          
        http-transport.scm:275: debug:print-info           
        common_records.scm:68: debug:debug-mode    
        http-transport.scm:225: k682       
        http-transport.scm:225: g686       
        rmt.scm:59: db:string->obj         
        db.scm:1749: regex#regexp          
        db.scm:1748: regex#string-substitute       
        db.scm:1747: base64#base64-decode          
        db.scm:1746: with-input-from-string        
        db.scm:1750: s11n#deserialize      
        tests.scm:331: with-output-to-port         
        tests.scm:334: conc             <-- 
Obtained lock for megatest-rollup-sqlitespeed.html 
</verbatim>

<verbatim> 
Obtained lock for megatest-rollup-pycell_xor.html 
 
... stuff ... 
 
Error: uncaught exception: #<condition: (join-timeout-exception)> 
</verbatim>

<verbatim> 
[0] http-transport:client-connect -> ("192.168.0.35" 8080 #<request> #<request>)  
pid:1559 INFO: (1) Running -runstep, first change to directory /mfs/matt/data/megatest-inmem-with-per-run/tests/fullrun/tmp/mt_links/ubuntu/nfs/none/w08.7.21.45_aa/singletest 
 
Error: (string-split) bad argument type - not a string: #f 
 
        Call history: 
 
        margs.scm:17: hash-table-ref/default       
        megatest.scm:931: args:get-arg     
        margs.scm:17: hash-table-ref/default       
        megatest.scm:934: args:get-arg     
        margs.scm:17: hash-table-ref/default       
        megatest.scm:937: args:get-arg     
        margs.scm:17: hash-table-ref/default       
        megatest.scm:940: args:get-arg     
        margs.scm:17: hash-table-ref/default       
        megatest.scm:946: args:get-arg     
        margs.scm:17: hash-table-ref/default       
        megatest.scm:947: args:get-arg     
        margs.scm:17: hash-table-ref/default       
        megatest.scm:948: conc     
        megatest.scm:952: get-environment-variable         
        megatest.scm:952: string-split          <-- 
; tracing tasks:server-lock-slot 
</verbatim>

<verbatim> 
chlr12706> megatest -step showresults -setlog results.html :state end :status 0 
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory 
 
Error: (current-directory) cannot retrieve current directory 
 
        Call history: 
 
        megatest.scm:316: args:get-arg     
        margs.scm:17: hash-table-ref/default       
        megatest.scm:333: args:get-arg     
        margs.scm:17: hash-table-ref/default       
        megatest.scm:348: hash-table-keys          
        megatest.scm:346: lset-intersection        
        megatest.scm:352: setup-for-run    
        launch.scm:422: hash-table?        
        launch.scm:425: args:get-arg       
        margs.scm:17: hash-table-ref/default       
        launch.scm:427: get-environment-variable           
        launch.scm:424: find-and-read-config       
        configf.scm:253: ##sys#get-keyword         
        configf.scm:253: ##sys#get-keyword         
        configf.scm:253: ##sys#get-keyword         
        configf.scm:254: current-directory              <-- 
chlr12706> cd $PWD 
chlr12706> megatest -step showresults -setlog results.html :state end :status 0 
</verbatim>

	Warning (#&lt;thread: ezstep run single step gencurrgds&g...Warning (#<thread: ezstep run single step gencurrgds>): in thread: unbound variable: run-id 
 
        Call history: 
 
        fs-transport.scm:42: debug:print-info      
        common_records.scm:60: debug:debug-mode    
        fs-transport.scm:43: db:process-queue-item         
        db.scm:1849: g2314         
        db.scm:1850: g2318         
        db.scm:1851: g2322         
        db.scm:1852: g2326         
        db.scm:1853: alist-ref     
        db.scm:1855: debug:print-info      
        common_records.scm:60: debug:debug-mode    
        db.scm:1865: debug:print-info      
        common_records.scm:60: debug:debug-mode    
        db.scm:1866: server:reply          
        server.scm:108: debug:print-info           
        common_records.scm:60: debug:debug-mode    
        ezsteps.scm:167: cdb:roll-up-pass-fail-counts           <-- 
 
   
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory 
	4:58 PM

<verbatim> 
Launching /tmp/megatest/tests/fdktestqa/simplelinks/a/bigrun2/a/bigrun/890 
 
Error: (hash-table-ref) hash-table does not contain key 
tests-info 
#<hash-table (2)> 
 
	Call history: 
 
	dashboard.scm:1053: iup-base#attribute-set!	   
	dashboard.scm:1055: iup-base#attribute-set!	   
	dashboard.scm:1342: current-milliseconds	   
	dashboard.scm:1389: mutex-lock!	   
	dashboard.scm:1391: mutex-unlock!	   
	dashboard.scm:1381: mutex-lock!	   
	dashboard.scm:1385: mutex-unlock!	   
	dashboard.scm:1388: dashboard:run-update	   
	dashboard.scm:1308: file-modification-time	   
	dashboard.scm:1309: file-modification-time	   
	dashboard.scm:1310: current-seconds	   
	dashboard.scm:1311: dashboard:recalc	   
	dashboard.scm:1296: current-milliseconds	   
	dashboard.scm:1389: mutex-lock!	   
	dashboard.scm:1391: mutex-unlock!	   
	dashboard.scm:813: hash-table-ref	 
</verbatim>

<verbatim> 
[evn-override] 
# BEQASCRIPTS #{scheme (print *toppath*)}/scripts 
</verbatim>

<verbatim> 
================== 
Next we need to get the variables and values you wish to iterate this test over (blank for none) 
Enter the variable name: EGG_NAME 
Enter the space separated list of values for EGG_NAME: matchable readline apropos base64 regex-literals format "regex-case" "test" coops trace csv dot-locking posix-utils posix-extras directory-utils hostinfo tcp rpc csv-xml fmt json md5     
Enter the variable name:  
Hint; use .sh extension on the script names and we'll create placeholder scripts.#<unspecified>#<unspecified>#<unspecified>#<unspecified> 
</verbatim>

<hr /><i>matt added on 2013-07-21 22:32:16 UTC:</i><br />
More funky wording 
<verbatim> 
================== 
Now to enter the one or more steps that make up your test, note; you can add more later 
Enter the name for this step (blank to stop): 
</verbatim>

<hr /><i>matt added on 2013-07-23 04:06:14 UTC:</i><br />
Script name mapping is not handled right. foo -> foo.sh or ?? 
 
Executable bits not set on scripts

<verbatim> 
================== 
Now you need an initial place to store your runs. These are called "disks" and you 
can add more at any time. To get going provide a writeable directory name.  
Enter your test runs directory: #{getenv MT_TEST_RUN_AREA}/../runs 
WARNING: you have specified a path #{getenv MT_TEST_RUN_AREA}/../runs that does not exist. Attempting to create it... 
 
 
Error: (create-directory) cannot create directory - File exists: "#{getenv MT_TEST_RUN_AREA}/.." 
 
	Call history: 
 
	genexample.scm:49: read-line	   
	genexample.scm:51: directory?	   
	genexample.scm:57: conc	   
	genexample.scm:57: file-exists?	   
	genexample.scm:68: print	   
	genexample.scm:81: display	   
	genexample.scm:82: read-line	   
	genexample.scm:84: string-split	   
	genexample.scm:84: g25	   
	genexample.scm:84: g25	   
	genexample.scm:87: print	   
	genexample.scm:89: display	   
	genexample.scm:90: read-line	   
	genexample.scm:91: directory?	   
	genexample.scm:93: print	   
	genexample.scm:94: create-directory	  	<-- 
</verbatim>

<verbatim> 
"SELECT id,run_id,testname,state,status,event_time,host,cpuload,diskfree,uname,rundir,item_path,run_duration,final_logf,comment FROM tests WHERE run_id=? AND ((testname LIKE '%if %'' AND item_path LIKE '%')) ;" 
 
        Call history: 
 
        db.scm:564: string-intersperse 
        tests.scm:100: db:patt->like 
        db.scm:560: ##sys#get-keyword 
        db.scm:564: g779 
        substring-index 
        db.scm:566: conc 
        db.scm:564: string-intersperse 
        tests.scm:101: conc 
        tests.scm:104: reverse 
        tests.scm:104: append 
        tests.scm:104: string-intersperse 
        db.scm:796: conc 
        db.scm:792: conc 
        db.scm:802: debug:print-info 
        common_records.scm:58: debug:debug-mode 
        db.scm:803: sqlite3#for-each-row <-- 
</verbatim>

../envqa/tests/dumpndiffcdf/main.csh:megatest -test-status : state COMPLETED : status CHECK -setlog cdfdiff_prim/index.html

 
main test still says UNKNOWN 
If test is WARN, top-level should also be WARN 
ok. So SKIP is working as expected wrt item, but parent level test status is not correct (UNKNOWN) 
 
 
Ok. So the rollup is not correct 
Let me check that. 
 
 
For parent test, I think the conditions should be: 
All item PASS, parent PASS 
All item PASS or SKIP, parent PASS 
one item WARN, rest PASS/SKIP, parent WARN 
one item FAIL, parent FAIL 
one item WAIVED, no fail, parent WAIVED 
precedence is: FAIL, WARN, CHECK, PASS 
 
 
      The following rules need to apply to megatest for status for 
      parent tests with items 
 
      1 FAIL item                                      parent test FAIL 
      no FAIL, 1 CHECK                                 parent test CHECK 
      no FAIL, no CHECK, 1 WAIVED                      parent test WAIVED 
      no FAIL, no CHECK, no WAIVED, 1 WARN             parent test WARN 
      no FAIL, no CHECK, no WAIVED, no WARN, 1 SKIP    parent test PASS 
      all SKIP                                         parent test SKIP 
      all PASS                                         parent test PASS 
 
      Precedence order is: 
 
      FAIL => CHECK => WAIVED => WARN => PASS => SKIP 
 
      1 corner case: 
 
      If all items SKIP, parent is SKIP. So if total number of ITEMS and 
      total number if SKIP match, then parent is SKIP. 
 
      SKIP is a lower priority than PASS as it reflects a preemptive cut 
      of a PASS condition (nothing was actually checked/done) whereas PASS 
      implies actual work to QA.

<hr /><i>matt added on 2013-04-24 23:21:44 UTC:</i><br />
<verbatim> 
      The following rules need to apply to megatest for status for 
      parent tests with items 
 
      1 FAIL item                                      parent test FAIL 
      no FAIL, 1 CHECK                                 parent test CHECK 
      no FAIL, no CHECK, 1 WAIVED                      parent test WAIVED 
      no FAIL, no CHECK, no WAIVED, 1 WARN             parent test WARN 
      no FAIL, no CHECK, no WAIVED, no WARN, 1 SKIP    parent test PASS 
      all SKIP                                         parent test SKIP 
      all PASS                                         parent test PASS 
 
      Precedence order is: 
 
      FAIL => CHECK => WAIVED => WARN => PASS => SKIP 
 
      1 corner case: 
 
      If all items SKIP, parent is SKIP. So if total number of ITEMS and 
      total number if SKIP match, then parent is SKIP. 
 
      SKIP is a lower priority than PASS as it reflects a preemptive cut 
      of a PASS condition (nothing was actually checked/done) whereas PASS 
      implies actual work to QA. 
</verbatim>

<verbatim> 
o	Error loading runconfigs that uses #{getenv USER} definition 
	Failing to locate targets defined in unloaded runconfigs 
o	–show-config => dumps (STDOUT) contents of megatest.config post-processing 
o	–show-runconfig [–target <target>] => same but for runconfigs 
o	–load <script> => scheme script (exits after execution) 
o	–target <target> :runname <runname> -testpatt <testpatt> -waive <comment> -state COMPLETED –status FAIL 
o	Reset FAIL -> NOT_STARTED 
o	–target <target> :runname <runname> -set-state-status NOT_STARTED,n/a :state COMPLETED :status FAIL –testpatt % 
	make fail_reset/fail_rerun 
</verbatim>

<hr /><i>matt added on 2013-03-07 00:07:35 UTC:</i><br />
Exact line in runconfigs.config is: 
<verbatim> 
./runconfigs.config:[include ./configs/#{getenv USER}.runconfigs.config] 
</verbatim>

[ ] megatest -generate-test 
	[ ] generate test issue - scripts vs. a command line. Look for |, space, quotes, / as signature of  
# Add additional steps here. Format is "stepname script" 
[ezsteps] 
getrepo fsl repo get `echo $TECH|sed 's/^p12/fdk/'` oalibs 
co_rel (cd oalibs;fsl co $RELEASE) 
convert convertall.sh 
Error: (open-output-file) cannot open file - No such file or directory: "../tests/csv_gen/fsl repo get `echo $TECH|sed 's/^p12/fdk/'` oalibs.sh" 
 
        Call history: 
 
        g210219 
        genexample.scm:240: print 
        genexample.scm:241: print 
        genexample.scm:242: get-environment-variable 
        genexample.scm:242: print 
        genexample.scm:243: get-environment-variable 
        genexample.scm:243: print 
        genexample.scm:244: print 
        genexample.scm:245: print 
        genexample.scm:246: print 
        g245251 
        genexample.scm:251: conc 
        genexample.scm:251: with-output-to-file 
        genexample.scm:253: print 
        genexample.scm:254: conc 
        genexample.scm:254: with-output-to-file

Remove runs on test_mt_vars and item 5 will not be deleted