Overview
| Comment: | Added auto propagation of env. vars in steps |
|---|---|
| Downloads: | Tarball | ZIP archive | SQL archive |
| Timelines: | family | ancestors | descendants | both | v1.65 |
| Files: | files | file ages | folders |
| SHA1: |
21d4a4c30b7d4a9e2dd50e5d35b8e231 |
| User & Date: | mrwellan on 2020-10-26 22:12:21 |
| Other Links: | branch diff | manifest | tags |
Context
|
2020-10-27
| ||
| 22:43 | Updated ezsteps propagation and documentation check-in: b8cc18503a user: matt tags: v1.65 | |
|
2020-10-26
| ||
| 22:12 | Added auto propagation of env. vars in steps check-in: 21d4a4c30b user: mrwellan tags: v1.65 | |
|
2020-10-25
| ||
| 22:54 | Removed gratuitious call to get-count-tests-running-for-run-id and some indenting fixes. check-in: 8085b19fd6 user: matt tags: v1.65 | |
Changes
Modified docs/manual/megatest_manual.html from [00d3df112f] to [70c989fffc].
| ︙ | ︙ | |||
3026 3027 3028 3029 3030 3031 3032 3033 3034 3035 3036 3037 3038 3039 |
</div></div>
<div class="paragraph"><p>To transfer the environment to the next step you can do the following:</p></div>
<div class="listingblock">
<div class="title">Propagate environment to next step</div>
<div class="content monospaced">
<pre>$MT_MEGATEST -env2file .ezsteps/${stepname}</pre>
</div></div>
</div>
<div class="sect2">
<h3 id="_scripts">Scripts</h3>
<div class="listingblock">
<div class="title">Specifying scripts inline (best used for only simple scripts)</div>
<div class="content monospaced">
<pre>[scripts]
| > > > > > > > > > > > > > > > > > > > > | 3026 3027 3028 3029 3030 3031 3032 3033 3034 3035 3036 3037 3038 3039 3040 3041 3042 3043 3044 3045 3046 3047 3048 3049 3050 3051 3052 3053 3054 3055 3056 3057 3058 3059 |
</div></div>
<div class="paragraph"><p>To transfer the environment to the next step you can do the following:</p></div>
<div class="listingblock">
<div class="title">Propagate environment to next step</div>
<div class="content monospaced">
<pre>$MT_MEGATEST -env2file .ezsteps/${stepname}</pre>
</div></div>
<div class="sect3">
<h4 id="_automatic_environment_propagation_with_ezsteps">Automatic environment propagation with Ezsteps</h4>
<div class="paragraph"><p>Simply give your step name an extension of .sh or .csh. The environment variables will be propagated.</p></div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<img src="/nfs/pdx/disks/ice.disk.002/icfadm/pkgs/asciidoc/8.6.7/images/icons/note.png" alt="Note">
</td>
<td class="content">aliases and variables with strange whitespace or characters will not propagate correctly. Put in a ticket on the <a href="http://www.kiatoa.com/fossils/megatest">http://www.kiatoa.com/fossils/megatest</a> site if you need support for a specific strange character combination.</td>
</tr></table>
</div>
<div class="listingblock">
<div class="title">Example of auto propagation</div>
<div class="content monospaced">
<pre>[ezsteps]
step1.sh export SOMEVAR=$(ps -def | wc -l);ls /tmp
# The next step will get the value of $SOMEVAR from step1.sh
step2.sh echo $SOMEVAR</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="_scripts">Scripts</h3>
<div class="listingblock">
<div class="title">Specifying scripts inline (best used for only simple scripts)</div>
<div class="content monospaced">
<pre>[scripts]
|
| ︙ | ︙ | |||
3497 3498 3499 3500 3501 3502 3503 | </div> </div> </div> <div id="footnotes"><hr></div> <div id="footer"> <div id="footer-text"> Version 1.5<br> | | | 3517 3518 3519 3520 3521 3522 3523 3524 3525 3526 3527 3528 | </div> </div> </div> <div id="footnotes"><hr></div> <div id="footer"> <div id="footer-text"> Version 1.5<br> Last updated 2020-10-13 21:24:33 PDT </div> </div> </body> </html> |
Modified docs/manual/reference.txt from [530f6e150c] to [a5ee87815b].
| ︙ | ︙ | |||
740 741 742 743 744 745 746 747 748 749 750 751 752 753 |
To transfer the environment to the next step you can do the following:
.Propagate environment to next step
----------------------------
$MT_MEGATEST -env2file .ezsteps/${stepname}
----------------------------
Scripts
~~~~~~~
.Specifying scripts inline (best used for only simple scripts)
----------------------------
[scripts]
| > > > > > > > > > > > > > > > | 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 |
To transfer the environment to the next step you can do the following:
.Propagate environment to next step
----------------------------
$MT_MEGATEST -env2file .ezsteps/${stepname}
----------------------------
Automatic environment propagation with Ezsteps
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Simply give your step name an extension of .sh or .csh. The environment variables will be propagated.
NOTE: aliases and variables with strange whitespace or characters will not propagate correctly. Put in a ticket on the http://www.kiatoa.com/fossils/megatest site if you need support for a specific strange character combination.
.Example of auto propagation
---------------------------
[ezsteps]
step1.sh export SOMEVAR=$(ps -def | wc -l);ls /tmp
# The next step will get the value of $SOMEVAR from step1.sh
step2.sh echo $SOMEVAR
---------------------------
Scripts
~~~~~~~
.Specifying scripts inline (best used for only simple scripts)
----------------------------
[scripts]
|
| ︙ | ︙ |
Modified env.scm from [028e47144f] to [c7d61e935d].
| ︙ | ︙ | |||
19 20 21 22 23 24 25 | ;;====================================================================== (declare (unit env)) (use sql-de-lite) ;; srfi-1 posix regex regex-case srfi-69 srfi-18 call-with-environment-variables) (define (env:open-db fname) | > > | | | | | | | | 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 |
;;======================================================================
(declare (unit env))
(use sql-de-lite) ;; srfi-1 posix regex regex-case srfi-69 srfi-18 call-with-environment-variables)
(define (env:open-db fname)
(let* ((db-exists (if (equal? fname ":memory:")
#f
(common:file-exists? fname)))
(db (open-database fname)))
(if (not db-exists)
(begin
(exec (sql db "CREATE TABLE IF NOT EXISTS envvars (
id INTEGER PRIMARY KEY,
context TEXT NOT NULL,
var TEXT NOT NULL,
val TEXT NOT NULL,
CONSTRAINT envvars_constraint UNIQUE (context,var))"))))
(set-busy-handler! db (busy-timeout 10000))
db))
;; save vars in given context, this is NOT incremental by default
;;
(define (env:save-env-vars db context #!key (incremental #f)(vardat #f))
(with-transaction
|
| ︙ | ︙ | |||
75 76 77 78 79 80 81 82 83 84 85 86 87 88 |
(let ((sep (cadr (assoc var paths))))
(env:merge-path-envvar sep (hash-table-ref result var) val))
val)))))
(sql db "SELECT var,val FROM envvars WHERE context=?")
context))
contexts)
result))
;; get list of removed variables between two contexts
;;
(define (env:get-removed db contexta contextb)
(let ((result (make-hash-table)))
(query
(for-each-row
| > > > > > > > > > > > > > > > > > > > > > > > | 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 |
(let ((sep (cadr (assoc var paths))))
(env:merge-path-envvar sep (hash-table-ref result var) val))
val)))))
(sql db "SELECT var,val FROM envvars WHERE context=?")
context))
contexts)
result))
;; envdelta: a-b (start=a, end=b, get the delta)
;; ofile: #f = write to stdout, else write to file with string name
;;
(define (env:envdelta db envdelta ofile)
(let ((match (string-split envdelta "-")));; (string-match "([a-z0-9_]+)=([a-z0-9_\\-,]+)" envdelta)))
(if (not (null? match))
(let* ((parts match) ;; (string-split equn "-"))
(minuend (car parts))
(subtraend (cadr parts))
(added (env:get-added db minuend subtraend))
(removed (env:get-removed db minuend subtraend))
(changed (env:get-changed db minuend subtraend)))
;; (pp (hash-table->alist added))
;; (pp (hash-table->alist removed))
;; (pp (hash-table->alist changed))
(if (args:get-arg "-o")
(with-output-to-file
(args:get-arg "-o")
(lambda ()
(env:print added removed changed)))
(env:print added removed changed)))
#f)))
;; get list of removed variables between two contexts
;;
(define (env:get-removed db contexta contextb)
(let ((result (make-hash-table)))
(query
(for-each-row
|
| ︙ | ︙ |
Modified ezsteps.scm from [ef12da0318] to [d79baadfc3].
| ︙ | ︙ | |||
31 32 33 34 35 36 37 38 | ;; (declare (uses filedb)) (include "common_records.scm") (include "key_records.scm") (include "db_records.scm") (include "run_records.scm") | > > > > > > > > > > > > > > > > > > > > | > > > > > > > < | | > | 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 |
;; (declare (uses filedb))
(include "common_records.scm")
(include "key_records.scm")
(include "db_records.scm")
(include "run_records.scm")
(define (ezsteps:step-name->mode stepname)
(match (string-search "\\.([^\\.]+)$" stepname)
((_ ext) (string->symbol ext))
(else #f)))
(define (ezsteps:create-step-script envdbf stepname prevstepname mode cmd)
(let* ((shebang (case mode
((sh) "/bin/sh")
((csh) "/bin/csh")
(else "/bin/bash")))
(sourcef (conc ".ezsteps/" prevstepname))
(scriptn stepname)) ;; remember the name already has an extension .sh, .csh etc.
(with-output-to-file scriptn
(lambda ()
;; the shebang line
(print "#!" shebang "\n")
;; first setup the source of the previous changes
(if (file-exists? sourcef)
(print "source " sourcef))
;; save the env at start
(print "megatest -envcap "stepname"_start "envdbf)
;; run the command
(print cmd "\necode=$?")
;; save the env at end
(print "megatest -envcap "stepname"_end "envdbf)
;; write the delta
(print "megatest -envdelta "stepname"_start-"stepname"_end -dumpmode bash -o .ezsteps/"stepname" "envdbf)
(print "exit $ecode")))
(system (conc "chmod a+x " scriptn))))
;;
(define (launch:runstep ezstep run-id test-id exit-info m tal testconfig all-steps-dat prevstepname envdbf)
(let* ((stepname (car ezstep)) ;; do stuff to run the step
(stepmode (ezsteps:step-name->mode stepname))
(stepinfo (cadr ezstep))
;; (let ((info (cadr ezstep)))
;; (if (proc? info) "" info)))
;; (stepproc (let ((info (cadr ezstep)))
;; (if (proc? info) info #f)))
(stepparts (string-match (regexp "^(\\{([^\\}\\{]*)\\}\\s*|)(.*)$") stepinfo))
(stepparams (if (and (list? stepparts)
|
| ︙ | ︙ | |||
78 79 80 81 82 83 84 |
(lambda ()
(print ";; logpro file extracted from testconfig\n"
";;")
(print tconfig-logpro)))
(set! logpro-used #t)))
;; NB// can safely assume we are in test-area directory
| | > | > > > > > | | 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 |
(lambda ()
(print ";; logpro file extracted from testconfig\n"
";;")
(print tconfig-logpro)))
(set! logpro-used #t)))
;; NB// can safely assume we are in test-area directory
(debug:print 4 *default-log-port* "ezsteps:\n stepname: " stepname " stepinfo: " stepinfo
" stepparts: " stepparts
" stepparams: " stepparams " stepcmd: " stepcmd)
(if stepmode (ezsteps:create-step-script envdbf stepname prevstepname stepmode stepcmd))
;; ;; first source the previous environment
;; (let ((prev-env (conc ".ezsteps/" prevstep (if (string-search (regexp "csh")
;; (get-environment-variable "SHELL")) ".csh" ".sh"))))
;; (if (and prevstep (common:file-exists? prev-env))
;; (set! script (conc script "source " prev-env))))
;; call the command using mt_ezstep
;; (set! script (conc "mt_ezstep " stepname " " (if prevstep prevstep "x") " " stepcmd))
(debug:print 4 *default-log-port* "script: " script)
(rmt:teststep-set-status! run-id test-id stepname "start" "-" #f #f)
;; now launch the actual process
(call-with-environment-variables
(list (cons "PATH" (conc (get-environment-variable "PATH") ":.")))
(lambda () ;; (process-run "/bin/bash" "-c" "exec ls -l /tmp/foobar > /tmp/delme-more.log 2>&1")
(let* ((cmd (conc (if stepmode
stepname
stepcmd)
" > " stepname ".log 2>&1")) ;; >outfile 2>&1
(pid #f))
(let ((proc (lambda ()
(set! pid (process-run "/bin/bash" (list "-c" cmd))))))
(if subrun
(begin
(debug:print-info 0 *default-log-port* "Running without MT_.* environment variables.")
(common:propogate-mt-vars-to-subrun proc '("MT_TARGET" "MT_LINKTREE" "MT_RUNNAME")))
|
| ︙ | ︙ |
Modified launch.scm from [2c7bba3a8e] to [740cbb909d].
| ︙ | ︙ | |||
166 167 168 169 170 171 172 |
(set! ezsteps #t) ;; set the needed flag
(set! ezstepslst
(append (or ezstepslst '())
(list (list "subrun" (conc "{subrun=true} " mt-cmd)))))))
;; process the ezsteps
(if ezsteps
| > | > > | > | 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 |
(set! ezsteps #t) ;; set the needed flag
(set! ezstepslst
(append (or ezstepslst '())
(list (list "subrun" (conc "{subrun=true} " mt-cmd)))))))
;; process the ezsteps
(if ezsteps
(let* ((envdbf (conc "/tmp/."(current-user-name)"-"(current-process-id)"-"run-id"-"test-id".db"))
(all-steps-dat (make-hash-table))) ;; keep all the info around as stepname ==> alist;
;;; where 'params is the params list (add other
;;; stuff as needed)
(if (not (common:file-exists? ".ezsteps"))(create-directory ".ezsteps"))
;; if ezsteps was defined then we are sure to have at least one step but check anyway
(if (not (> (length ezstepslst) 0))
(debug:print-error 0 *default-log-port* "ezsteps defined but ezstepslst is zero length")
(let loop ((ezstep (car ezstepslst))
(tal (cdr ezstepslst))
(prevstep #f))
(debug:print-info 0 *default-log-port* "Processing ezstep \"" (string-intersperse ezstep " ") "\"")
;; check exit-info (vector-ref exit-info 1)
(if (launch:einf-exit-status exit-info) ;; (vector-ref exit-info 1)
(let* ((logpro-used (launch:runstep ezstep run-id test-id exit-info m
tal testconfig all-steps-dat prevstep envdbf))
(stepname (car ezstep))
(stepparms (hash-table-ref all-steps-dat stepname)))
(setenv "MT_STEP_NAME" stepname)
(pp (hash-table->alist all-steps-dat))
;; if logpro-used read in the stepname.dat file
(if (and logpro-used (common:file-exists? (conc stepname ".dat")))
(launch:load-logpro-dat run-id test-id stepname))
|
| ︙ | ︙ |
Modified megatest.scm from [9752b7fe00] to [5ccef97439].
| ︙ | ︙ | |||
866 867 868 869 870 871 872 |
(let ((envcap (args:get-arg "-envcap")))
(if envcap
(let* ((db (env:open-db (if (null? remargs) "envdat.db" (car remargs)))))
(env:save-env-vars db envcap)
(env:close-database db)
(set! *didsomething* #t))))
| | > > > | 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 |
(let ((envcap (args:get-arg "-envcap")))
(if envcap
(let* ((db (env:open-db (if (null? remargs) "envdat.db" (car remargs)))))
(env:save-env-vars db envcap)
(env:close-database db)
(set! *didsomething* #t))))
;; delta "language" will eventually be res=a+b-c but for now it is just res=a-b
;;
;; db file can be stuck on the end of the command line:
;; megatest -envdelta start-end -dumpmode bash -o .ezsteps/step5.sh /tmp/myfile.db
;;
(let ((envdelta (args:get-arg "-envdelta")))
(if envdelta
(let ((match (string-split envdelta "-")));; (string-match "([a-z0-9_]+)=([a-z0-9_\\-,]+)" envdelta)))
(if (not (null? match))
(let* ((db (env:open-db (if (null? remargs) "envdat.db" (car remargs))))
;; (resctx (cadr match))
|
| ︙ | ︙ |