Megatest

Check-in [fa512c78ed]
Login
Overview
Comment:Fixed -m, added conditional waiver propagation
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | development
Files: files | file ages | folders
SHA1: fa512c78ed216a70589360f567b77759d30ce312
User & Date: mrwellan on 2013-04-08 14:08:48
Other Links: branch diff | manifest | tags
Context
2013-04-08
15:11
Updated version check-in: cc739baf17 user: mrwellan tags: development
14:18
Version 1.5405: smart waiver propagation, fix broken -m message for -test-status and -step check-in: b3506d6c89 user: mrwellan tags: trunk, v1.5405
14:08
Fixed -m, added conditional waiver propagation check-in: fa512c78ed user: mrwellan tags: development
2013-03-28
18:44
Slight improvement to waiton expansion that supports [system ...] and the like check-in: d98fb0961e user: mrwellan tags: development
Changes

Modified configf.scm from [05823356b4] to [c9fe6d3ae6].

237
238
239
240
241
242
243


244
245
246
247
248
249
250
	    #f
	    (let ((match (assoc var sectdat)))
	      (if match ;; (and match (list? match)(> (length match) 1))
		  (cadr match)
		  #f))
	    ))
      #f))



(define (configf:section-vars cfgdat section)
  (let ((sectdat (hash-table-ref/default cfgdat section '())))
    (if (null? sectdat)
	'()
	(map car sectdat))))








>
>







237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
	    #f
	    (let ((match (assoc var sectdat)))
	      (if match ;; (and match (list? match)(> (length match) 1))
		  (cadr match)
		  #f))
	    ))
      #f))

(define configf:lookup config-lookup)

(define (configf:section-vars cfgdat section)
  (let ((sectdat (hash-table-ref/default cfgdat section '())))
    (if (null? sectdat)
	'()
	(map car sectdat))))

Modified docs/manual/megatest_manual.txt from [2feb2da4e0] to [5c6fd3ea7c].

59
60
61
62
63
64
65




















66
67
68
69
70
71
72
the distributed compute platform in use. A template script is provided
which can launch jobs on local and remote Linux hosts. Currently
megatest uses the network filesystem to call home to your master
sqlite3 database.

include::getting_started.txt[] include::writing_tests.txt[]
include::reference.txt[]





















[appendix]
Example Appendix
================
One or more optional appendixes go here at section level zero.

Appendix Sub-section







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
the distributed compute platform in use. A template script is provided
which can launch jobs on local and remote Linux hosts. Currently
megatest uses the network filesystem to call home to your master
sqlite3 database.

include::getting_started.txt[] include::writing_tests.txt[]
include::reference.txt[]


=========================================================
# logpro_file  input_glob
# matching file(s) will be diff'd with previous run and logpro applied
# if PASS or WARN result from logpro then WAIVER state is set
#
[waivers]
waiver_1 logpro lookittmp.log

[waiver_rules]

# This builtin rule is the default if there is no <waivername>.logpro file
# diff   diff %file1% %file2%

# This builtin rule is applied if a <waivername>.logpro file exists
# logpro diff %file1% %file2% | logpro %waivername%.logpro %waivername%.html
=========================================================



[appendix]
Example Appendix
================
One or more optional appendixes go here at section level zero.

Appendix Sub-section

Modified docs/megatest-training.odp from [d231c12bb1] to [093164d585].

cannot compute difference between binary files

Modified megatest-version.scm from [af9292025d] to [82fd80ecd9].

1
2
3
4
5
6
7
;; Always use two digit decimal
;; 1.01, 1.02...1.10,1.11 ... 1.99,2.00..

(declare (unit megatest-version))

(define megatest-version 1.5402)






|

1
2
3
4
5
6
7
;; Always use two digit decimal
;; 1.01, 1.02...1.10,1.11 ... 1.99,2.00..

(declare (unit megatest-version))

(define megatest-version 1.5404)

Modified megatest.scm from [1ccef434db] to [0c1c2447d3].

28
29
30
31
32
33
34








35
36
37
38
39
40
41

(define *db* #f) ;; this is only for the repl, do not use in general!!!!

(include "common_records.scm")
(include "key_records.scm")
(include "db_records.scm")
(include "megatest-fossil-hash.scm")









(define help (conc "
Megatest, documentation at http://chiselapp.com/user/kiatoa/repository/megatest
  version " megatest-version "
  license GPL, Copyright Matt Welland 2006-2012

Usage: megatest [options]







>
>
>
>
>
>
>
>







28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

(define *db* #f) ;; this is only for the repl, do not use in general!!!!

(include "common_records.scm")
(include "key_records.scm")
(include "db_records.scm")
(include "megatest-fossil-hash.scm")

;; (use trace)
;; (trace db:teststep-set-status!
;;        tests:test-set-status!
;;        cdb:test-set-status-state
;;        cdb:client-call
;;        tests:check-waiver-eligibility)
       

(define help (conc "
Megatest, documentation at http://chiselapp.com/user/kiatoa/repository/megatest
  version " megatest-version "
  license GPL, Copyright Matt Welland 2006-2012

Usage: megatest [options]

Modified tests.scm from [ab3c2465d2] to [40dd9ff166].

9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
;;  PURPOSE.
;;======================================================================

;;======================================================================
;; Tests
;;======================================================================

(use sqlite3 srfi-1 posix regex regex-case srfi-69 dot-locking tcp)
(import (prefix sqlite3 sqlite3:))

(declare (unit tests))
(declare (uses db))
(declare (uses common))
(declare (uses items))
(declare (uses runconfig))







|







9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
;;  PURPOSE.
;;======================================================================

;;======================================================================
;; Tests
;;======================================================================

(use sqlite3 srfi-1 posix regex regex-case srfi-69 dot-locking tcp directory-utils)
(import (prefix sqlite3 sqlite3:))

(declare (unit tests))
(declare (uses db))
(declare (uses common))
(declare (uses items))
(declare (uses runconfig))
183
184
185
186
187
188
189

























































190
191
192
193
194
195
196
197
198
199
200
201
202





203
204


205
206
207
208
209
210
211


212
213
214
215


216

217
218
219
220
221
222
223
224
225
226
227
228
				    (> (db:test-get-event_time testdat)(db:test-get-event_time stored-test))))
			   ;; this test is younger, store it in the hash
			   (hash-table-set! tests-hash full-testname testdat))))
		   results)
		  (if (null? tal)
		      (map cdr (hash-table->alist tests-hash)) ;; return a list of the most recent tests
		      (loop (car tal)(cdr tal))))))))))


























































;; Do not rpc this one, do the underlying calls!!!
(define (tests:test-set-status! test-id state status comment dat)
  (debug:print-info 4 "tests:test-set-status! test-id=" test-id ", state=" state ", status=" status ", dat=" dat)
  (let* ((db          #f)
	 (real-status status)
	 (otherdat    (if dat dat (make-hash-table)))
	 (testdat     (cdb:get-test-info-by-id *runremote* test-id))
	 (run-id      (db:test-get-run_id testdat))
	 (test-name   (db:test-get-testname   testdat))
	 (item-path   (db:test-get-item-path testdat))
	 ;; before proceeding we must find out if the previous test (where all keys matched except runname)
	 ;; was WAIVED if this test is FAIL





	 (waived   (if (equal? status "FAIL")
		       (let ((prev-test (open-run-close test:get-previous-test-run-record db run-id test-name item-path)))


			 (if prev-test ;; true if we found a previous test in this run series
			     (let ((prev-status (db:test-get-status   prev-test))
				   (prev-state  (db:test-get-state    prev-test))
				   (prev-comment (db:test-get-comment prev-test)))
			       (debug:print 4 "prev-status " prev-status ", prev-state " prev-state ", prev-comment " prev-comment)
			       (if (and (equal? prev-state  "COMPLETED")
					(equal? prev-status "WAIVED"))


				   prev-comment ;; waived is either the comment or #f
				   #f))
			     #f))
		       #f)))


    (if waived (set! real-status "WAIVED"))

    (debug:print 4 "real-status " real-status ", waived " waived ", status " status)

    ;; update the primary record IF state AND status are defined
    (if (and state status)
	(cdb:test-set-status-state *runremote* test-id real-status state #f))
    
    ;; if status is "AUTO" then call rollup (note, this one modifies data in test
    ;; run area, it does remote calls under the hood.
    (if (and test-id state status (equal? status "AUTO")) 
	(db:test-data-rollup #f test-id status))

    ;; add metadata (need to do this way to avoid SQL injection issues)







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>













>
>
>
>
>
|
|
>
>
|
|
|
|
|
|
|
>
>
|
|
|

>
>
|
>




|







183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
				    (> (db:test-get-event_time testdat)(db:test-get-event_time stored-test))))
			   ;; this test is younger, store it in the hash
			   (hash-table-set! tests-hash full-testname testdat))))
		   results)
		  (if (null? tal)
		      (map cdr (hash-table->alist tests-hash)) ;; return a list of the most recent tests
		      (loop (car tal)(cdr tal))))))))))

;; Check for waiver eligibility
;;
(define (tests:check-waiver-eligibility testdat prev-testdat)
  (let* ((testconfig  (tests:get-testconfig (db:test-get-testname testdat) #f))
	 (test-rundir (db:test-get-rundir testdat))
	 (prev-rundir (db:test-get-rundir prev-testdat))
	 (waivers     (configf:section-vars testconfig "waivers"))
	 (waiver-rx   (regexp "^(\\S+)\\s+(.*)$"))
	 (diff-rule   "diff %file1% %file2%")
	 (logpro-rule "diff %file1% %file2% | logpro %waivername%.logpro %waivername%.html"))
    (push-directory test-rundir)
    (let ((result (if (null? waivers)
		      #f
		      (let loop ((hed (car waivers))
				 (tal (cdr waivers)))
			(debug:print 0 "INFO: Applying waiver rule \"" hed "\"")
			(let* ((waiver      (configf:lookup testconfig "waivers" hed))
			       (wparts      (if waiver (string-match waiver-rx waiver) #f))
			       (waiver-rule (if wparts (cadr wparts)  #f))
			       (waiver-glob (if wparts (caddr wparts) #f))
			       (logpro-file (if waiver
						(let ((fname (conc hed ".logpro")))
						  (if (file-exists? fname)
						      fname 
						      (begin
							(debug:print 0 "INFO: No logpro file " fname " falling back to diff")
							#f)))
						#f))
			       ;; if rule by name of waiver-rule is found in testconfig - use it
			       ;; else if waivername.logpro exists use logpro-rule
			       ;; else default to diff-rule
			       (rule-string (let ((rule (configf:lookup testconfig "waiver_rules" waiver-rule)))
					      (if rule
						  rule
						  (if logpro-file
						      logpro-rule
						      (begin
							(debug:print 0 "INFO: No logpro file " logpro-file " found, using diff rule")
							diff-rule)))))
			       ;; (string-substitute "%file1%" "foofoo.txt" "This is %file1% and so is this %file1%." #t)
			       (processed-cmd (string-substitute 
					       "%file1%" (conc test-rundir "/" waiver-glob)
					       (string-substitute
						"%file2%" (conc prev-rundir "/" waiver-glob)
						(string-substitute
						 "%waivername%" hed rule-string #t) #t) #t))
			       (res            #f))
			  (debug:print 0 "INFO: waiver command is \"" processed-cmd "\"")
			  (if (eq? (system processed-cmd) 0)
			      (if (null? tal)
				  #t
				  (loop (car tal)(cdr tal)))
			      #f))))))
      (pop-directory)
      result)))


;; Do not rpc this one, do the underlying calls!!!
(define (tests:test-set-status! test-id state status comment dat)
  (debug:print-info 4 "tests:test-set-status! test-id=" test-id ", state=" state ", status=" status ", dat=" dat)
  (let* ((db          #f)
	 (real-status status)
	 (otherdat    (if dat dat (make-hash-table)))
	 (testdat     (cdb:get-test-info-by-id *runremote* test-id))
	 (run-id      (db:test-get-run_id testdat))
	 (test-name   (db:test-get-testname   testdat))
	 (item-path   (db:test-get-item-path testdat))
	 ;; before proceeding we must find out if the previous test (where all keys matched except runname)
	 ;; was WAIVED if this test is FAIL

	 ;; NOTES:
	 ;;  1. Is the call to test:get-previous-run-record remotified?
	 ;;  2. Add test for testconfig waiver propagation control here
	 ;;
	 (prev-test   (if (equal? status "FAIL")
			  (open-run-close test:get-previous-test-run-record db run-id test-name item-path)
			  #f))
	 (waived   (if prev-test
		       (if prev-test ;; true if we found a previous test in this run series
			   (let ((prev-status  (db:test-get-status  prev-test))
				 (prev-state   (db:test-get-state   prev-test))
				 (prev-comment (db:test-get-comment prev-test)))
			     (debug:print 4 "prev-status " prev-status ", prev-state " prev-state ", prev-comment " prev-comment)
			     (if (and (equal? prev-state  "COMPLETED")
				      (equal? prev-status "WAIVED"))
				 (if comment
				     comment
				     prev-comment) ;; waived is either the comment or #f
				 #f))
			   #f)
		       #f)))
    (if (and waived 
	     (tests:check-waiver-eligibility testdat prev-test))
	(set! real-status "WAIVED"))

    (debug:print 4 "real-status " real-status ", waived " waived ", status " status)

    ;; update the primary record IF state AND status are defined
    (if (and state status)
	(cdb:test-set-status-state *runremote* test-id real-status state (if waived waived comment)))
    
    ;; if status is "AUTO" then call rollup (note, this one modifies data in test
    ;; run area, it does remote calls under the hood.
    (if (and test-id state status (equal? status "AUTO")) 
	(db:test-data-rollup #f test-id status))

    ;; add metadata (need to do this way to avoid SQL injection issues)

Modified tests/fullrun/tests/ezlog_fail/testconfig from [39388ec16f] to [da8d5b9ba5].

1
2
3
4
5
6















7
8
9
10
11
12
13
[setup]

[ezsteps]
lookittmp   ls /tmp
lookithome  ls /home
















[test_meta]
author matt
owner  bob
description This test runs two ezstep, the first of which is expected to fail using a simple logpro file.

tags first,single
reviewed 09/10/2011, by Matt






>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
[setup]

[ezsteps]
lookittmp   ls /tmp
lookithome  ls /home

# logpro_file  input_glob
# matching file(s) will be diff'd with previous run and logpro applied
# if PASS or WARN result from logpro then WAIVER state is set
#
[waivers]
waiver_1 logpro lookittmp.log

[waiver_rules]

# This builtin rule is the default if there is no <waivername>.logpro file
# diff   diff %file1% %file2%

# This builtin rule is applied if a <waivername>.logpro file exists
# logpro diff %file1% %file2% | logpro %waivername%.logpro %waivername%.html

[test_meta]
author matt
owner  bob
description This test runs two ezstep, the first of which is expected to fail using a simple logpro file.

tags first,single
reviewed 09/10/2011, by Matt

Added tests/fullrun/tests/ezlog_fail/waiver_1.logpro version [f9358fd909].



>
1
(expect:warning  in "Body"  = 0 "Any warning" #/WARNING/)

Modified utils/mk_wrapper from [af950f4763] to [d96f80e4dc].

1
2
3
4
5
6
7
8
9
10
11
12
13
#!/bin/bash

prefix=$1
cmd=$2

echo "#!/bin/bash"
if [ "$LD_LIBRARY_PATH" != "" ];then
  echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH"
fi

fullcmd="$prefix/bin/$cmd"

echo "$fullcmd \$*"












|
1
2
3
4
5
6
7
8
9
10
11
12
13
#!/bin/bash

prefix=$1
cmd=$2

echo "#!/bin/bash"
if [ "$LD_LIBRARY_PATH" != "" ];then
  echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH"
fi

fullcmd="$prefix/bin/$cmd"

echo "$fullcmd \"\$@\""