14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
|
//
// You should have received a copy of the GNU General Public License
// along with Megatest. If not, see <http://www.gnu.org/licenses/>.
Debugging
---------
A word on Bisecting
~~~~~~~~~~~~~~~~~~~
Bisecting is a debug strategy intended to speed up finding the root
cause.
.A complex process with a problem found in stage "E"
["graphviz", "bisecting.png"]
----------------------------------------------------------------------
include::bisecting.dot[]
----------------------------------------------------------------------
It is common to start debugging where the problem was observed and
then work back. However by inspecting the output at stage "C" in the
example above you would potentially save a lot of debug effort, this
is similar to the feature in source control tools like git and fossil
called biseceting.
To know where to look for problems you will want to look at the log files at various stages in the execution process.
.A simplified diagram of the stages Megatest goes through to run a test.
["graphviz", "megatest-test-stages.png"]
----------------------------------------------------------------------
include::megatest-test-stages.dot[]
----------------------------------------------------------------------
.How to check variable values and inspect logs at each stage
[width="80%",cols="<,2m,2m",frame="topbot",options="header"]
|======================
|Stage | How to inspect | Watch for
|A: post config processing | megatest -show-config -target your/target | #f (failed var processing)
|B: post runconfig | megatest -show-runconfig -target your/target |
|C: processing testconfigs | inspect output from "megatest -run ..." | Messages indicating issues process configs, dependency problems.
|D: process testconfig for test launch | inspect output from megatest runner | Zero items (items expansion yielded no items)
|E,F: launching test | start test xterm, look at mt_launch.log | Did your batch system accept the job? Has the job landed on a machine?
|G: starting test | look at your batch systems logs for the process | Did the megatest -execute process start and run?
|H,H1,H2: step exectution | look at <stepname>.log, <stepname>.html and your own internal logs | Do you have sufficiently tight logpro rules? You must always have a "required" rule!
|======================
|
|
|
|
<
>
>
|
>
|
>
|
>
>
|
>
>
>
>
|
>
>
>
>
>
>
>
<
<
<
<
>
>
|
>
>
>
>
|
>
>
>
>
<
<
<
|
<
>
>
>
>
>
<
<
<
<
|
|
|
>
>
|
>
>
>
|
>
>
|
>
>
>
>
>
|
>
>
>
>
>
|
>
>
|
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
|
//
// You should have received a copy of the GNU General Public License
// along with Megatest. If not, see <http://www.gnu.org/licenses/>.
Debugging
---------
Well Written Tests
~~~~~~~~~~~~~~~~~~
Test Design and Surfacing Errors
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Design your tests to surface errors. Ensure that all logs are
processed by logpro (or a custom log processing tool) and can be
reached by a mouse click or two from the test control panel.
To illustrate, here is a set of scripts with nested calls where script1.sh calls script2.sh which calls script3.sh which finally calls the Cadence EDA tool virtuoso:
.script1.sh
..............................
#!/bin/bash
code ...
script2.sh some parameters > script2.log
more code ...
..............................
.script2.sh
..............................
#!/bin/bash
code ...
script3.sh some more parameters > script3.log
more code ...
..............................
.script3.sh
..............................
#!/bin/bash
code ...
virtuoso params and switches ...
more code ...
..............................
The log files script2.log, script3.log and the log output from
virtuoso are not accessible from the test control panel. It would be
much better for future users of your automation to use steps more
fully. One easy option would be to post process the logs in downstream
additional steps:
.testconfig
..............................
[ezsteps]
step1 script1.sh
step2 cat script2.log
step3 cat script3.log
[logpro]
step1 ;; some logpro rules
(expect:required in "LogFileBody" > 0 "Expect this output" #/something expected/)
step2 ;; some logpro rules for script2.sh
step3 ;; some logpro rules for script3.sh
[scripts]
script1.sh #!/bin/bash
code ...
...
..............................
With the above testconfig the logs for every critical part of the
automation are fully surfaced and rules can be created to flag errors,
warnings, aborts and to ignore false errors. A user of your automation
will be able to see the important error with two mouse clicks from the
runs view.
An even better would be to eliminate the nesting if possible. As a
general statement with layers - less is usually more. By flattening
the automation into a sequence of steps you can use the test control
panel to re-run a step with a single click or from the test xterm run
only the errant step from the command line.
The message here is make debugging and maintenace easy for future
users (and yourself) by keeping clicks-to-error in mind.
Examining The Test Logs and Environment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Test Control Panel - xterm
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
75
76
77
78
79
80
81
82
83
84
85
86
87
88
|
step" to only rerun the step or click on "restart from here" to rerun
that step and downstream steps.
NOTE 1: visual feedback can take some time, give it a few seconds and
you will see the step change color to blue as it starts running.
NOTE 2: steping through only works if you are using ezsteps.
Config File Processing
^^^^^^^^^^^^^^^^^^^^^^
As described above it is often helpful to know the content of
variables in various contexts as Megatest works through the actions
needed to run your tests. A handy technique is to force the startup of
|
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
|
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
|
step" to only rerun the step or click on "restart from here" to rerun
that step and downstream steps.
NOTE 1: visual feedback can take some time, give it a few seconds and
you will see the step change color to blue as it starts running.
NOTE 2: steping through only works if you are using ezsteps.
A word on Bisecting
~~~~~~~~~~~~~~~~~~~
Bisecting is a debug strategy intended to speed up finding the root
cause of some bug.
.A complex process with a problem found in stage "E"
["graphviz", "bisecting.png"]
----------------------------------------------------------------------
include::bisecting.dot[]
----------------------------------------------------------------------
It is common to start debugging where the problem was observed and
then work back. However by inspecting the output at stage "C" in the
example above you would potentially save a lot of debug effort, this
is similar to the feature in source control tools like git and fossil
called biseceting.
Tough Bugs
~~~~~~~~~~
Most bugs in Megatest based automation will be in the scripts called
in your test steps and if you utilize the good design practice
described above should be fairly easy for you to reproduce, isolate
and find.
Some bugs however will come from subtle and hard to detect
interactions between Megatest and your OS and Unix environment. This
includes things like constructed variables that are legal in one
context (e.g. tcsh) but illegal in another context (e.g. bash),
variables that come from your login scripts and access and permissions
issues (e.g. a script that silently fails due to no access to needed
data). Other bugs might be due to Megatest itself.
To isolate bugs like this you may need to look at the log files at
various stages in the execution process of your run and tests.
.A simplified diagram of the stages Megatest goes through to run a test.
["graphviz", "megatest-test-stages.png"]
----------------------------------------------------------------------
include::megatest-test-stages.dot[]
----------------------------------------------------------------------
.How to check variable values and inspect logs at each stage
[width="80%",cols="<,2m,2m",frame="topbot",options="header"]
|======================
|Stage | How to inspect | Watch for
|A: post config processing | megatest -show-config -target your/target | #f (failed var processing)
|B: post runconfig | megatest -show-runconfig -target your/target | Add -debug 0,9 to see which file your settings come from
|C: processing testconfigs | inspect output from "megatest -run ..." | Messages indicating issues process configs, dependency problems
|D: process testconfig for test launch | inspect output from megatest runner | Zero items (items expansion yielded no items)
|E,F: launching test | start test xterm, look at mt_launch.log | Did your batch system accept the job? Has the job landed on a machine?
|G: starting test | look at your batch systems logs for the process | Did the megatest -execute process start and run?
|H,H1,H2: step exectution | look at <stepname>.log, <stepname>.html and your own internal logs | Do you have sufficiently tight logpro rules? You must always have a "required" rule!
|======================
Bisecting megatest.csh/sh
^^^^^^^^^^^^^^^^^^^^^^^^^
Sometimes finding the environment variable that is causing the problem
can be very difficult. Bisection can be applied.
Edit the megatest.csh or megatest.sh file and comment out 50% per
round, source in fresh xterm and run the test.
This idea can also be applied to your .cshrc, .bashrc, .aliases and
other similar files.
csh and -f
^^^^^^^^^^
A common issue when tcsh or csh shells are used for scripting is to
forget or choose to not use -f in your #! line.
.Not good
..............................
#!/bin/tcsh
...
..............................
.Good
..............................
#!/bin/tcsh -f
...
..............................
Config File Processing
^^^^^^^^^^^^^^^^^^^^^^
As described above it is often helpful to know the content of
variables in various contexts as Megatest works through the actions
needed to run your tests. A handy technique is to force the startup of
|