Overview
Comment: | Start of moving rollup off the server |
---|---|
Downloads: | Tarball | ZIP archive | SQL archive |
Timelines: | family | ancestors | descendants | both | v1.80-servload |
Files: | files | file ages | folders |
SHA1: |
95c5f92eb5fcd42fa8b1113358df1900 |
User & Date: | matt on 2023-04-18 08:31:56 |
Other Links: | branch diff | manifest | tags |
Context
2023-04-18
| ||
20:52 | wip check-in: 4c125c180a user: matt tags: v1.80-servload | |
08:31 | Start of moving rollup off the server check-in: 95c5f92eb5 user: matt tags: v1.80-servload | |
2023-04-17
| ||
23:12 | Caching extended to couple more calls. Impact seems dramatic. However note that likely not all scenarios for clearing the cache are covered. check-in: 4126673c03 user: matt tags: v1.80-servload | |
Changes
Modified TODO from [fa3d981ca6] to [ff11150ce4].
︙ | |||
54 55 56 57 58 59 60 61 | 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | + + + + + + + + + + + + + + + + + + + + | . Re-work the dbstruct data structure? .. Move main.db to global? .. [ run-id.db inmemdb last-mod last-read last-sync inuse ] . Re-work all queries to use run-id to dereference server . Open main.db directly in calls to -runtests etc. No need to talk remote? . remove common:faux-lock db:get-test-info-by-id db:get-test-state-status-by-id db:get-test-info - do a get id by name/item-path cache the id- use test id plus run id to get from cache need to do db:get-test-info-db look at html gen for items - rollup needs deduplication nonoverlap ;; cache write these with transaction db:teststep-set-status! db:test-set-top-process-id ;; called a lot, maybe from rollup? db:get-all-state-status-counts-for-test ;; load to move from server to client tests:summarize-items ;; appears to be on client tests:summarize-tests |
Modified db.scm from [78fbf3122c] to [e4a21bbcbc].
︙ | |||
2313 2314 2315 2316 2317 2318 2319 | 2313 2314 2315 2316 2317 2318 2319 2320 2321 2322 2323 2324 2325 2326 2327 2328 2329 2330 2331 2332 2333 2334 2335 2336 2337 2338 2339 2340 2341 2342 2343 2344 2345 2346 2347 2348 2349 | - - - + + + + | ;; ;; speed up for common cases with a little logic ;; ;; NB// Ultimately this will be deprecated in deference to mt:test-set-state-status-by-id ;; ;; NOTE: run-id is not used ;; ;; (define (db:test-set-state-status dbstruct run-id test-id newstate newstatus newcomment) |
︙ | |||
3124 3125 3126 3127 3128 3129 3130 | 3125 3126 3127 3128 3129 3130 3131 3132 3133 3134 3135 3136 3137 3138 | - - - - - - - - - - - - - - - | (begin (debug:print-error 0 *default-log-port* "reception failed. Received \"" msg "\" but cannot translate it.") (print-call-chain (current-error-port)) msg))) ;; crude reply for when things go awry ((zmq nmsg)(with-input-from-string msg (lambda ()(deserialize)))) (else msg))) ;; rpc |
︙ | |||
3183 3184 3185 3186 3187 3188 3189 | 3169 3170 3171 3172 3173 3174 3175 3176 3177 3178 3179 3180 3181 3182 3183 3184 | - + + | (if (not (equal? item-path "")) ;; only roll up IF incoming test is an item (let* ((state-status-counts (db:get-all-state-status-counts-for-test db run-id test-name item-path state status)) ;; item-path is used to exclude current state/status of THIS test (state-statuses (db:roll-up-rules state-status-counts state status)) (newstate (car state-statuses)) (newstatus (cadr state-statuses))) (set! new-state-eh newstate) (set! new-status-eh newstatus) |
︙ |
Modified launch.scm from [7e82dfb83e] to [aa8485d496].
︙ | |||
739 740 741 742 743 744 745 | 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 | - + + | ;; NO NEED TO CALL set-state-status-and-roll-up-items HERE, THIS IS DONE IN set-state-status-and-roll-up-items called by tests:test-set-status! ) ) ;; for automated creation of the rollup html file this is a good place... (if (not (equal? item-path "")) |
︙ |
Modified mt.scm from [9ff41cb92d] to [5d7251e2a1].
︙ | |||
188 189 190 191 192 193 194 | 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 | - + - + | (debug:print-info 0 *default-log-port* "TRIGGERED on " trigger ", running command " fullcmd " output at " (get-environment-variable "NBFAKE_LOG")) (process-run fullcmd) (if prev-nbfake-log (setenv "NBFAKE_LOG" prev-nbfake-log) (unsetenv "NBFAKE_LOG")) )) |
︙ | |||
260 261 262 263 264 265 266 | 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 | - - - - - - - - - - | (define (mt:test-set-state-status-by-id run-id test-id newstate newstatus newcomment) (if (not (and run-id test-id)) (begin (debug:print-error 0 *default-log-port* "bad data handed to mt:test-set-state-status-by-id, run-id=" run-id ", test-id=" test-id ", newstate=" newstate) (print-call-chain (current-error-port)) #f) (begin |
︙ | |||
292 293 294 295 296 297 298 | 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 | - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + | ;; (mt:process-triggers run-id test-id new-state new-status) #t);) ;;(mt:test-set-state-status-by-id run-id test-id new-state new-status new-comment))) (define (mt:test-set-state-status-by-testname-unless-completed run-id test-name item-path new-state new-status new-comment) (let ((test-id (rmt:get-test-id run-id test-name item-path))) (mt:test-set-state-status-by-id-unless-completed run-id test-id new-state new-status new-comment))) |
︙ |
Modified rmt.scm from [0af3ea0170] to [5890df38f8].
︙ | |||
441 442 443 444 445 446 447 448 449 450 451 452 453 454 | 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 | + | (assert (number? run-id) "FATAL: Run id required.") (rmt:send-receive 'get-count-tests-running-in-jobgroup run-id (list run-id jobgroup))) (define (rmt:set-state-status-and-roll-up-run run-id state status) (assert (number? run-id) "FATAL: Run id required.") (rmt:send-receive 'set-state-status-and-roll-up-run run-id (list run-id state status))) ;; run on client version of set-state-status-and-roll-up-run (define (rmt:update-pass-fail-counts run-id test-name) (assert (number? run-id) "FATAL: Run id required.") (rmt:general-call 'update-pass-fail-counts run-id test-name test-name test-name)) (define (rmt:top-test-set-per-pf-counts run-id test-name) (assert (number? run-id) "FATAL: Run id required.") |
︙ |
Modified rmtmod.scm from [c4f748fe17] to [ac1e77f103].
︙ | |||
175 176 177 178 179 180 181 | 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | (assert (number? run-id) "FATAL: Run id required.") (rmtmod:send-receive 'synchash-get run-id (list run-id proc synckey keynum params))) (define (rmt:get-tests-for-run-mindata run-id testpatt states status not-in) (assert (number? run-id) "FATAL: Run id required.") (rmtmod:send-receive 'get-tests-for-run-mindata run-id (list run-id testpatt states status not-in))) |