Megatest

Check-in [fd3c06195d]
Login
Overview
Comment:Tell calling client to wait and try again if server is overloaded (in terms of parallel api calls over 25)
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | multi-server-hack
Files: files | file ages | folders
SHA1: fd3c06195dae25aaf4cdee9be9eac1f172ac926a
User & Date: matt on 2017-03-24 14:52:47
Other Links: branch diff | manifest | tags
Context
2017-03-24
15:18
Reduced threshold for server pushback on clients to 20 api calls/parallel. Added couple more calls to check if server is running before starting a new one. check-in: 37e149d637 user: matt tags: multi-server-hack
14:52
Tell calling client to wait and try again if server is overloaded (in terms of parallel api calls over 25) check-in: fd3c06195d user: matt tags: multi-server-hack
13:51
Show connection stats every 60 seconds. Remove stat of megatest.db from rmt:send-receive, it was happening on every call. check-in: 6baac6187e user: matt tags: multi-server-hack
Changes

Modified api.scm from [d0edc7e79b] to [b54cac363d].

123
124
125
126
127
128
129


130
131
132
133
134
135
136
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138







+
+







     (debug:print 0 *default-log-port* "WARNING: api:execute-requests received an exception from peer")
     (print-call-chain (current-error-port))
     (debug:print 0 *default-log-port* " message: "  ((condition-property-accessor 'exn 'message) exn))       
     (vector #f (vector exn call-chain dat))) ;; return some stuff for debug if an exception happens
   (cond
    ((not (vector? dat))                    ;; it is an error to not receive a vector
     (vector #f #f "remote must be called with a vector")       )
    ((> *api-process-request-count* 25)
     (vector #f 'overloaded))
    (else  
     (let* ((cmd-in (vector-ref dat 0))
            (cmd    (if (symbol? cmd-in)
                        cmd-in
                        (string->symbol cmd-in)))
            (params (vector-ref dat 1))
            (start-t (current-milliseconds))

Modified rmt.scm from [4b028f3c38] to [5974ce4c87].

204
205
206
207
208
209
210





211
212
213
214
215
216
217
218








219
220
221
222
223
224
225
204
205
206
207
208
209
210
211
212
213
214
215








216
217
218
219
220
221
222
223
224
225
226
227
228
229
230







+
+
+
+
+
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+







	      ((http)
	       (mutex-unlock! *rmt-mutex*)
	       res)
	      (else
	       (debug:print 0 *default-log-port* "ERROR: transport " (remote-transport runremote) " is unknown")
	       (mutex-unlock! *rmt-mutex*)
	       (exit 1)))
	    (if (eq? res 'overloaded)
		(let ((wait-delay (+ attemptnum (* attemptnum 10))))
		  (debug:print 0 *default-log-port* "WARNING: server is overloaded. Delaying " wait-delay " seconds and trying call again.")
		  (thread-sleep! wait-delay)
		  (rmt:send-receive cmd rid params attemptnum: (+ attemptnum 1)))
	    (begin
	      (debug:print 0 *default-log-port* "WARNING: communication failed. Trying again, try num: " attemptnum)
	      (remote-conndat-set!    runremote #f)
	      (remote-server-url-set! runremote #f)
              (debug:print-info 12 *default-log-port* "rmt:send-receive, case  9.1")
	      (mutex-unlock! *rmt-mutex*)
	      (server:start-and-wait *toppath*)
	      (rmt:send-receive cmd rid params attemptnum: (+ attemptnum 1)))))))))
		(begin
		  (debug:print 0 *default-log-port* "WARNING: communication failed. Trying again, try num: " attemptnum)
		  (remote-conndat-set!    runremote #f)
		  (remote-server-url-set! runremote #f)
		  (debug:print-info 12 *default-log-port* "rmt:send-receive, case  9.1")
		  (mutex-unlock! *rmt-mutex*)
		  (server:start-and-wait *toppath*)
		  (rmt:send-receive cmd rid params attemptnum: (+ attemptnum 1))))))))))

;; (define (rmt:update-db-stats run-id rawcmd params duration)
;;   (mutex-lock! *db-stats-mutex*)
;;   (handle-exceptions
;;    exn
;;    (begin
;;      (debug:print 0 *default-log-port* "WARNING: stats collection failed in update-db-stats")