20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
|
A word on Bisecting
~~~~~~~~~~~~~~~~~~~
Bisecting is a debug strategy intended to speed up finding the root
cause.
["graphviz", "bisecting.png"]
----------------------------------------------------------------------
include::bisecting.dot[]
----------------------------------------------------------------------
It is common to start debugging where the problem was observed and
then work back. However by inspecting the output at stage "C" in the
example above you would potentially save a lot of debug effort, this
is similar to the feature in source control tools like git and fossil
called biseceting.
To know where to look for problems you will want to look at the log files at various stages in the execution process.
["graphviz", "megatest-test-stages.png"]
----------------------------------------------------------------------
include::megatest-test-stages.dot[]
----------------------------------------------------------------------
.How to check variable values at each stage
[width="80%",cols="<,2m,2m",frame="topbot",options="header"]
|======================
|Stage | How to inspect | Watch for
|A: post config processing | megatest -show-config -target your/target | #f (failed var processing)
|B: post runconfig | megatest -show-runconfig -target your/target |
|C: processing testconfigs | inspect output from "megatest -run ..." | Messages indicating issues process configs, dependency problems.
|D: process testconfig for test launch | inspect output from megatest runner | Zero items (items expansion yielded no items)
|E,F: launching test | start test xterm, look at mt_launch.log | Did your batch system accept the job? Has the job landed on a machine?
|G: starting test | look at your batch systems logs for the process | Did the megatest -execute process start and run?
|H,H1,H2: step exectution | look at <stepname>.log, <stepname>.html and your own internal logs | Do you have sufficiently tight logpro rules? You must always have a "required" rule!
|======================
Examining The Environment
~~~~~~~~~~~~~~~~~~~~~~~~~
Test Control Panel - xterm
^^^^^^^^^^^^^^^^^^^^^^^^^^
From the dashboard click on a test PASS/FAIL button. This brings up a
test control panel. Aproximately near the center left of the window
there is a button "Start Xterm". Push this to get an xterm with the
full context and environment loaded for that test. You can run scripts
or ezsteps by copying from the testconfig (hint, load up the
testconfig in a separate text editor window). This is the easiest
way to debug your tests.
During Config File Processing
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
It is often helpful to know the content of variables in various
contexts as Megatest does the actions needed to run your tests. A handy technique is to force the startup of an xterm in the context being examined.
|
>
>
>
|
|
|
|
|
>
>
>
>
>
>
>
>
>
|
|
|
>
|
>
|
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
|
A word on Bisecting
~~~~~~~~~~~~~~~~~~~
Bisecting is a debug strategy intended to speed up finding the root
cause.
.A complex process with a problem found in stage "E"
["graphviz", "bisecting.png"]
----------------------------------------------------------------------
include::bisecting.dot[]
----------------------------------------------------------------------
It is common to start debugging where the problem was observed and
then work back. However by inspecting the output at stage "C" in the
example above you would potentially save a lot of debug effort, this
is similar to the feature in source control tools like git and fossil
called biseceting.
To know where to look for problems you will want to look at the log files at various stages in the execution process.
.A simplified diagram of the stages Megatest goes through to run a test.
["graphviz", "megatest-test-stages.png"]
----------------------------------------------------------------------
include::megatest-test-stages.dot[]
----------------------------------------------------------------------
.How to check variable values and inspect logs at each stage
[width="80%",cols="<,2m,2m",frame="topbot",options="header"]
|======================
|Stage | How to inspect | Watch for
|A: post config processing | megatest -show-config -target your/target | #f (failed var processing)
|B: post runconfig | megatest -show-runconfig -target your/target |
|C: processing testconfigs | inspect output from "megatest -run ..." | Messages indicating issues process configs, dependency problems.
|D: process testconfig for test launch | inspect output from megatest runner | Zero items (items expansion yielded no items)
|E,F: launching test | start test xterm, look at mt_launch.log | Did your batch system accept the job? Has the job landed on a machine?
|G: starting test | look at your batch systems logs for the process | Did the megatest -execute process start and run?
|H,H1,H2: step exectution | look at <stepname>.log, <stepname>.html and your own internal logs | Do you have sufficiently tight logpro rules? You must always have a "required" rule!
|======================
Examining The Test Logs and Environment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Test Control Panel - xterm
^^^^^^^^^^^^^^^^^^^^^^^^^^
From the dashboard click on a test PASS/FAIL button. This brings up a
test control panel. Aproximately near the center left of the window
there is a button "Start Xterm". Push this to get an xterm with the
full context and environment loaded for that test. You can run scripts
or ezsteps by copying from the testconfig (hint, load up the
testconfig in a separate text editor window).
With more recent versions of Megatest you can step through your test
from the test control panel. Click on the cell labeled "rerun this
step" to only rerun the step or click on "restart from here" to rerun
that step and downstream steps.
NOTE 1: visual feedback can take some time, give it a few seconds and
you will see the step change color to blue as it starts running.
NOTE 2: steping through only works if you are using ezsteps.
Config File Processing
^^^^^^^^^^^^^^^^^^^^^^
As described above it is often helpful to know the content of
variables in various contexts as Megatest works through the actions
needed to run your tests. A handy technique is to force the startup of
an xterm in the context being examined.
For example, if an item list is not being generated as expected you
can inject the startup of an xterm as if it were an item:
.Original items table
-----------------
[items]
|