The Megatest Users Manual

Matt Welland

version 1.0,

April 2012

Table of Contents

- +version 1.5, +June 2020 +

Table of Contents

Preface

@@ -782,11 +782,11 @@

This book is organised as three sub-books; getting started, writing tests and reference.

License

    Copyright 2006-2017, Matthew Welland.
+    Copyright 2006-2020, Matthew Welland.
 
     This document is part of Megatest.
 
     Megatest is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
@@ -809,11 +809,11 @@
 The Megatest project was started for two reasons, the first was an
 immediate and pressing need for a generalized tool to manage a suite
 of regression tests and the second was the fact that I had written or
 maintained several such tools at different companies over the years. I
 thought a single open source tool, flexible enough to meet the needs
-of any team doing continuous integrating and or running a complex
+of any team doing continuous integration and or running a complex
 suite of tests for release qualification would solve some problems for
 me and for others.
 
 
 -- Matt Welland, original author of the Megatest tool suite.
@@ -955,20 +955,552 @@

TODO / Road Map

Note: This road-map is a wish list and not a formal plan. Items are in +rough priority but are subject to change. Development is driven by +user requests, developer "itch" and bug reports. Please contact +matt@kiatoa.com with requests or bug reports. Requests from inside +Intel generally take priority.

Dashboard and runs

+
+Multi-area dashboard view +
+

Tests Support

+
+Add variable $MT_RUNPATH = $MT_LINKTREE/$MT_TARGET/$MT_RUNNAME +
+
+
+Improve [script], especially indent handling +
+

Scalability

+
+Overflow database methodology - combine the best of the v1.63 + multi-db approach and the current db-in-tmp approach (currently + slowness can be seen when number of tests in a db goes over 50-100k, + with the overflow db it will be able to handle 1000’s of runs with + 50-100k tests per run). High priority - goal is to complete this by + 20Q3. +
+

Mtutils/CI

+
+Enable mtutil calls from dashboard (for remote control) +
+
+
+Logs browser (esp. for surfacing mtutil related activities) +
+
+
+Embed ftfplan for distributed automation, completed activities trigger QA runs which trigger deployment etc. +
+
+
+Jenkins junit XML support [DONE] +
+
+
+Add output flushing in teamcity support +
+

Build system

+
+./configure ⇒ ubuntu, sles11, sles12, rh7 [WIP] +
+
+
+Switch to using simple runs query everywhere +
+
+
+Add end_time to runs and add a rollup call that sets state, status and end_time +
+

Code refactoring/quality/performance

+
+Switch to scsh-process pipeline management for job execution/control +
+
+
+Use call-with-environment-variables where possible. +
+

Migration to inmem db and or overflow db

+
+Re-work the dbstruct data structure? +
+
1. +
  +[ run-id.db inmemdb last-mod last-read last-sync inuse ] +
  +
+

Some ideas for Megatest 2.0

+
+Aggressive megatest.config and runconfig.config caching. +
+
1. +
  +Cache the configs in $MT_RUNPATH +
  +
2. +
  +Following invocations of –run, -rerun* will calculate the new config but only overwrite the cached file IF changed +
  +
+
+
+If the cached file changes ALL existing tests go from COMPLETED → STALE, I’m not sure what to do about RUNNING tests +
+
+
+!VARS in runconfigs are not exported to the environment. They are accessed via rget as if the ! was not there. +
+
+
+Per test copy commands (example is incomplete). +
+

[testcopy]
+%/iind% unison SRC DEST
+% cp –r SRC DEST

Add ability to move runs to other Areas (overlaps with overflow db system)

+
+allow shrinking megatest.db data by moving runs to an alternate + Megatest area with same keys. +
+
+
+add param -destination [area|path]. when specified runs are copied to new + area and removed from local db. +
+
+
+the data move would involve these steps +
+
1. +
  +copy the run data to destination area megatest.db +
  +
2. +
  +mark the run records as deleted, do not remove the run data on disk +
  +
+
+
+accessing the data would be by running dashboard in the satellite area +
+
+
+future versions of Megatest dashboard should support displaying areas in a + merged way. +
+
+
+some new controls would be supported in the config +
+
1. +
  +[setup] ⇒ allow-runs [no|yes] ⇐= used to disallow runs +
  +
2. +
  +[setup] ⇒ auto-migrate=[areaname|path] ⇐= used to automatically + migrate data to a satellite area. +
  +
+

Eliminate ties to homehost (part of overflow db system)

+
+Server creates captain pkt +
+
+
+Create a lock in the db +
+
+
+Relinquish db when done +
+

Tasks - better management of run manager processes etc.

+
+adjutant queries tasks table for next action [Migrate into mtutil] +
+
1. +
  +Task table used for tracking runner process [Replaced by mtutil] +
  +
2. +
  +Task table used for jobs to run [Replaced by mtutil] +
  +
3. +
  +Task table used for queueing runner actions (remove runs, + cleanRunExecute, etc) [Replaced by mtutil] +
  +
+
+
+adjutant (server/task dispatch/execution manager) +
+

Stale propagation

+
+Mark dependent tests for clean/rerun -rerun-downstream +
+
+
+On run start check for defunct tests in RUNNING, LAUNCHED or REMOTEHOSTSTART and correct or notify +
+
+
+Fix: refresh of gui sometimes fails on last item (race condition?) +
+

Bin list

+
+Rerun step and or subsequent steps from gui [DONE?] +
+
+
+Refresh test area files from gui +
+
+
+Clean and re-run button +
+
+
+Clean up STATE and STATUS handling. +
+
1. +
  +Dashboard and Test control panel are reverse order - choose and fix +
  +
2. +
  +Move seldom used states and status to drop down selector +
  +
+
+
+Access test control panel when clicking on Run Summary tests +
+
+
+Feature: -generate-index-tree +
+
+
+Change specifing of state and status to use STATE1/STATUS1,STATE2/STATUS2 +
+
+
+rest api available for use with Perl, Ruby etc. scripts +
+
+
+megatest.config setup entries for: +
+
1. +
  +run launching (e.g. /bin/sh %CMD% > /dev/null) +
  +
2. +
  +browser "konqueror %FNAME% +
  +
+
+
+refdb: Add export of csv, json and sexp +
+
+
+Convert to using call-with-environment-variables where possible. Should allow handling of parallel runs in same process. +
+
+
+Re-work text interface wizards. Several bugs on record. Possibly convert to gui based. +
+
+
+Add to testconfig requirements section; launchlimiter scriptname, calls scriptname to check if ok to launch test +
+
+
+Refactor Run Summary view, currently very clumsy +
+
+
+Add option to show steps in Run Summary view +
+
+
+Refactor guis for resizeablity +
+
+
+Add filters to Run Summary view and Run Control view +
+
+
+Add to megatest.config or testconfig; rerunok STATE/STATUS,STATE/STATUS… +
+
+
+Launch gates for diskspace; /path/one>1G,/path/two>200M,/tmp>5G,#{scheme toppath}>1G +
+
+
+Tool tips +
+
+
+Filters on Run Summary, Summary and Run Control panel +
+
+
+Built in log viewer (partially implemented) +
+
+
+Refactor the test control panel + Help and documentation +
+
+
+Complete the user manual (I’ve been working on this lately). +
+
+
+Online help in the gui + Streamlined install +
+
+
+Deployed or static build +
+
+
+Added option to compile IUP (needed for VMs) +
+
+
+Server side run launching +
+
+
+Wizards for creating tests, regression areas (current ones are text only and limited). +
+
+
+Fully functional built in web service (currently you can browse runs but it is very simplistic). +
+
+
+Gui panels for editing megatest.config and runconfigs.config +
+
+
+Fully isolated tests (no use of NFS to see regression area files) +
+
+
+Windows version +
+

Installation

Dependencies

Chicken scheme and a number of "eggs" are required for building -Megatest. See the script installall.sh in the utils directory of the -source distribution for an automated way to install everything -needed for building Megatest on Linux.

+Megatest. In the v1.66 and beyond assistance to create the build +system is built into the Makefile.

Installation steps (overview)

./configure
+make chicken
+setup.sh make -j install

Or install the needed build system manually:

+
+Chicken scheme from http://call-cc.org +
+
+
+IUP from http://webserver2.tecgraf.puc-rio.br/iup/ +
+
+
+CD from http://webserver2.tecgraf.puc-rio.br/cd/ +
+
+
+IM from https://webserver2.tecgraf.puc-rio.br/im/ +
+
+
+ffcall from http://webserver2.tecgraf.puc-rio.br/iup/ +
+
+
+Nanomsg from https://nanomsg.org/ (NOTE: Plan is to eliminate nanomsg dependency). +
+
+
+Needed eggs (look at the eggs lists in the Makefile) +
+

Then follow these steps:

Installation steps (self-built chicken scheme build system)

./configure
+make -j install

Getting Started

@@ -2837,385 +3369,10 @@

Road Map

Note 1: This road-map continues to evolve and subject to change without notice.

Here is a smattering of ideas for Megatest 2.0

-
-Add variable $MT_RUNPATH = $MT_LINKTREE/$MT_TARGET/$MT_RUNNAME -
-

[db]
-api legacy|new

-
-One big lesson from the 1.63-1.65 generation was that the main.db, 1.db … model was really good at scaling. I’d like to combine that model with the current also-very-good model. Obviously this is a disruptive change. I think making the old model the default and the new model an option for at least one generation would be fair. -
-
-
-Rigorous megatest.config and runconfig.config caching. -
-
1. -
  -Cache the configs in $MT_RUNPATH -
  -
2. -
  -Following invocations of –run, -rerun* will calculate the new config but only overwrite the cached file IF changed -
  -
-
-
-If the cached file changes ALL existing tests go from COMPLETED → STALE, I’m not sure what to do about RUNNING tests -
-
-
-!VARS in runconfigs are not exported to the environment. They are accessed via rget as if the ! was not there. -
-
-
-Per test copy commands (crude example below is not correct). -
-

[testcopy]
-%/iind% unison SRC DEST
-% cp –r SRC DEST

-
-Test management via pkts (optional?) -
-
1. -
  -Control pkt types: run, kill, rerunclean, clean, archive, status? -
  -
2. -
  -Status pkt types: ack, step, status_change -
  -
-
-
-Add nanomsg as a transport option -. -
-

RFC M01: Add ability to move runs to other Areas

Purpose: allow shrinking megatest.db data by moving runs to an alternate - Megatest area with same keys.

Method: extend db sync to take a different megatest area as a destination.

Design:

-
-add param -destination [area|path]. when specified runs are copied to new - area and removed from local db. -
-
-
-the data move would involve these steps -
-
1. -
  -copy the run data to destination area megatest.db -
  -
2. -
  -mark the run records as deleted, do not remove the run data on disk -
  -
-
-
-accessing the data would be by running dashboard in the satellite area -
-
-
-future versions of Megatest dashboard should support displaying areas in a - merged way. -
-
-
-some new controls would be supported in the config -
-
1. -
  -[setup] ⇒ allow-runs [no|yes] ⇐= used to disallow runs -
  -
2. -
  -[setup] ⇒ auto-migrate=[areaname|path] ⇐= used to automatically - migrate data to a satellite area. -
  -
-

Branch: This work is taking place on branch v1.65-reduce-records

RFC M02: Move data into completed-runs.db

Purpose: shrink megatest.db data to enable lower load and higher performance.

Method: add a completed-runs.db and automatically move runs data from megatest.db to that db

Design:

-
-completed-runs.db is a full megatest database with complete schema -
-
-
-the data move would involve these steps -
-
1. -
  -copy the run data to completed-runs.db -
  -
2. -
  -remove the run data, first from /tmp/…/megatest.db and /tmp/…/megatest_ref.db, followed by megatest.db -
  -
-
-
-accessing the data would be unchanged for most operations. -
-
-
-a mode -full-db will be added which when specified would attach the completed-runs.db to megatest.db before doing the query -
-
-
-mechanisms for moving runs to/from the megatest.db would be added -
-
1. -
  --reduce-records ⇒ move runs to completed-runs.db -
  -
2. -
  --restore-records ⇒ move runs from completed-runs.db to megatest.db -
  -
-

Branch: This work is taking place on branch v1.65-reduce-records

RFC M03: Automatic homehost migrations

Purpose: Automatically migrate homehost.

Method: Check that there are no tests running, launched or remotehoststart in past ½ hour then if not on homehost migrate the db to current host

Design:

-
-Check that the system is quiescent, i.e. that there are no runs in flight or recently run -
-
-
-Create a lock -
-
-
-Migrate the /tmp cache db to the current host -
-
-
-Update the .homehost file -
-
-
-Remove the lock -
-

Branch: This work not yet started

Architecture Refactor

Goals

-
-Reduce load on the file system. Sqlite3 files on network filesystem can be - a burden. [DONE] -
-
-
-Reduce number of servers and frequency of start/stop. This is mostly an - issue of clutter but also a reduction in "moving parts". [DONE] -
-
-
-Coalesce activities to a single home host where possible. Give the user - feedback that they have started the dashboard on a host other than the - home host. [DONE] -
-
-
-Reduce number of processes involved in managing running tests. -
-

Changes Needed

-
-ACID compliant db will be on /tmp and synced to megatest.db with a five - second max delay. [DONE] -
-
-
-Read/writes to db for processes on homehost will go direct to /tmp - megatest.db file. [DONE] -
-
-
-Read/wites fron non-homehost processes will go through one server. Bulk - reads (e.g. for dashboard or list-runs) will be cached on the current host - in /tmp and synced from the home megatest.db in the testsuite area. [DONE] -
-
-
-Db syncs rely on the target db file timestame minus some margin. [DONE] -
-
-
-Since bulk reads do not use the server we can switch to simple RPC for the - network transport. [DONE] -
-
-
-Test running manager process extended to manage multiple running tests. -
-

Current Items

ww05 - migrate to inmem-db

-
-Switch to inmem db with fast sync to on disk db’s [DONE] -
-
-
-Server polls tasks table for next action -
-
1. -
  -Task table used for tracking runner process [Replaced by mtutil] -
  -
2. -
  -Task table used for jobs to run [Replaced by mtutil] -
  -
3. -
  -Task table used for queueing runner actions (remove runs, - cleanRunExecute, etc) [Replaced by mtutil] -
  -
-

shifting, note that the preceding blank line is needed.

The Megatest Users Manual

Preface

TODO / Road Map

Installation

Dependencies

Getting Started

Third Party plugins

Team City

Jenkins - JUnit Realtime Test Reporter

Road Map

Here is a smattering of ideas for Megatest 2.0

RFC M01: Add ability to move runs to other Areas

RFC M02: Move data into completed-runs.db

RFC M03: Automatic homehost migrations

Architecture Refactor

Goals

Changes Needed

Current Items

ww05 - migrate to inmem-db

Index