Preface
@@ -786,11 +786,11 @@This book is organised as three sub-books; getting started, writing tests and reference.
Copyright 2006-2017, Matthew Welland.
+ Copyright 2006-2020, Matthew Welland.
This document is part of Megatest.
Megatest is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
@@ -813,11 +813,11 @@
The Megatest project was started for two reasons, the first was an
immediate and pressing need for a generalized tool to manage a suite
of regression tests and the second was the fact that I had written or
maintained several such tools at different companies over the years. I
thought a single open source tool, flexible enough to meet the needs
-of any team doing continuous integrating and or running a complex
+of any team doing continuous integration and or running a complex
suite of tests for release qualification would solve some problems for
me and for others.
-- Matt Welland, original author of the Megatest tool suite.
@@ -959,20 +959,552 @@
TODO / Road Map
+Note: This road-map is a wish list and not a formal plan. Items are in +rough priority but are subject to change. Development is driven by +user requests, developer "itch" and bug reports. Please contact +matt@kiatoa.com with requests or bug reports. Requests from inside +Intel generally take priority.
Dashboard and runs
-
+
-
+
+Multi-area dashboard view +
+
+
Tests Support
-
+
-
+
+Add variable $MT_RUNPATH = $MT_LINKTREE/$MT_TARGET/$MT_RUNNAME +
+
+ -
+
+Improve [script], especially indent handling +
+
+
Scalability
-
+
-
+
+Overflow database methodology - combine the best of the v1.63 + multi-db approach and the current db-in-tmp approach (currently + slowness can be seen when number of tests in a db goes over 50-100k, + with the overflow db it will be able to handle 1000’s of runs with + 50-100k tests per run). High priority - goal is to complete this by + 20Q3. +
+
+
Mtutils/CI
-
+
-
+
+Enable mtutil calls from dashboard (for remote control) +
+
+ -
+
+Logs browser (esp. for surfacing mtutil related activities) +
+
+ -
+
+Embed ftfplan for distributed automation, completed activities trigger QA runs which trigger deployment etc. +
+
+ -
+
+Jenkins junit XML support [DONE] +
+
+ -
+
+Add output flushing in teamcity support +
+
+
Build system
-
+
-
+
+./configure ⇒ ubuntu, sles11, sles12, rh7 [WIP] +
+
+ -
+
+Switch to using simple runs query everywhere +
+
+ -
+
+Add end_time to runs and add a rollup call that sets state, status and end_time +
+
+
Code refactoring/quality/performance
-
+
-
+
+Switch to scsh-process pipeline management for job execution/control +
+
+ -
+
+Use call-with-environment-variables where possible. +
+
+
Migration to inmem db and or overflow db
-
+
-
+
+Re-work the dbstruct data structure? +
++-
+
-
+
+[ run-id.db inmemdb last-mod last-read last-sync inuse ] +
+
+
+ -
+
Some ideas for Megatest 2.0
-
+
-
+
+Aggressive megatest.config and runconfig.config caching. +
++-
+
-
+
+Cache the configs in $MT_RUNPATH +
+
+ -
+
+Following invocations of –run, -rerun* will calculate the new config but only overwrite the cached file IF changed +
+
+
+ -
+
-
+
+If the cached file changes ALL existing tests go from COMPLETED → STALE, I’m not sure what to do about RUNNING tests +
+
+ -
+
+!VARS in runconfigs are not exported to the environment. They are accessed via rget as if the ! was not there. +
+
+ -
+
+Per test copy commands (example is incomplete). +
+
+
[testcopy] +%/iind% unison SRC DEST +% cp –r SRC DEST+
Add ability to move runs to other Areas (overlaps with overflow db system)
-
+
-
+
+allow shrinking megatest.db data by moving runs to an alternate + Megatest area with same keys. +
+
+ -
+
+add param -destination [area|path]. when specified runs are copied to new + area and removed from local db. +
+
+ -
+
+the data move would involve these steps +
++-
+
-
+
+copy the run data to destination area megatest.db +
+
+ -
+
+mark the run records as deleted, do not remove the run data on disk +
+
+
+ -
+
-
+
+accessing the data would be by running dashboard in the satellite area +
+
+ -
+
+future versions of Megatest dashboard should support displaying areas in a + merged way. +
+
+ -
+
+some new controls would be supported in the config +
++-
+
-
+
+[setup] ⇒ allow-runs [no|yes] ⇐= used to disallow runs +
+
+ -
+
+[setup] ⇒ auto-migrate=[areaname|path] ⇐= used to automatically + migrate data to a satellite area. +
+
+
+ -
+
Eliminate ties to homehost (part of overflow db system)
-
+
-
+
+Server creates captain pkt +
+
+ -
+
+Create a lock in the db +
+
+ -
+
+Relinquish db when done +
+
+
Tasks - better management of run manager processes etc.
-
+
-
+
+adjutant queries tasks table for next action [Migrate into mtutil] +
++-
+
-
+
+Task table used for tracking runner process [Replaced by mtutil] +
+
+ -
+
+Task table used for jobs to run [Replaced by mtutil] +
+
+ -
+
+Task table used for queueing runner actions (remove runs, + cleanRunExecute, etc) [Replaced by mtutil] +
+
+
+ -
+
-
+
+adjutant (server/task dispatch/execution manager) +
+
+
Stale propagation
-
+
-
+
+Mark dependent tests for clean/rerun -rerun-downstream +
+
+ -
+
+On run start check for defunct tests in RUNNING, LAUNCHED or REMOTEHOSTSTART and correct or notify +
+
+ -
+
+Fix: refresh of gui sometimes fails on last item (race condition?) +
+
+
Bin list
-
+
-
+
+Rerun step and or subsequent steps from gui [DONE?] +
+
+ -
+
+Refresh test area files from gui +
+
+ -
+
+Clean and re-run button +
+
+ -
+
+Clean up STATE and STATUS handling. +
++-
+
-
+
+Dashboard and Test control panel are reverse order - choose and fix +
+
+ -
+
+Move seldom used states and status to drop down selector +
+
+
+ -
+
-
+
+Access test control panel when clicking on Run Summary tests +
+
+ -
+
+Feature: -generate-index-tree +
+
+ -
+
+Change specifing of state and status to use STATE1/STATUS1,STATE2/STATUS2 +
+
+ -
+
+rest api available for use with Perl, Ruby etc. scripts +
+
+ -
+
+megatest.config setup entries for: +
++-
+
-
+
+run launching (e.g. /bin/sh %CMD% > /dev/null) +
+
+ -
+
+browser "konqueror %FNAME% +
+
+
+ -
+
-
+
+refdb: Add export of csv, json and sexp +
+
+ -
+
+Convert to using call-with-environment-variables where possible. Should allow handling of parallel runs in same process. +
+
+ -
+
+Re-work text interface wizards. Several bugs on record. Possibly convert to gui based. +
+
+ -
+
+Add to testconfig requirements section; launchlimiter scriptname, calls scriptname to check if ok to launch test +
+
+ -
+
+Refactor Run Summary view, currently very clumsy +
+
+ -
+
+Add option to show steps in Run Summary view +
+
+ -
+
+Refactor guis for resizeablity +
+
+ -
+
+Add filters to Run Summary view and Run Control view +
+
+ -
+
+Add to megatest.config or testconfig; rerunok STATE/STATUS,STATE/STATUS… +
+
+ -
+
+Launch gates for diskspace; /path/one>1G,/path/two>200M,/tmp>5G,#{scheme toppath}>1G +
+
+ -
+
+Tool tips +
+
+ -
+
+Filters on Run Summary, Summary and Run Control panel +
+
+ -
+
+Built in log viewer (partially implemented) +
+
+ -
+
+Refactor the test control panel + Help and documentation +
+
+ -
+
+Complete the user manual (I’ve been working on this lately). +
+
+ -
+
+Online help in the gui + Streamlined install +
+
+ -
+
+Deployed or static build +
+
+ -
+
+Added option to compile IUP (needed for VMs) +
+
+ -
+
+Server side run launching +
+
+ -
+
+Wizards for creating tests, regression areas (current ones are text only and limited). +
+
+ -
+
+Fully functional built in web service (currently you can browse runs but it is very simplistic). +
+
+ -
+
+Gui panels for editing megatest.config and runconfigs.config +
+
+ -
+
+Fully isolated tests (no use of NFS to see regression area files) +
+
+ -
+
+Windows version +
+
+
Installation
Dependencies
Chicken scheme and a number of "eggs" are required for building -Megatest. See the script installall.sh in the utils directory of the -source distribution for an automated way to install everything -needed for building Megatest on Linux.
./configure +make chicken +setup.sh make -j install+
Or install the needed build system manually:
-
+
-
+
+Chicken scheme from http://call-cc.org +
+
+ -
+
+IUP from http://webserver2.tecgraf.puc-rio.br/iup/ +
+
+ -
+
+CD from http://webserver2.tecgraf.puc-rio.br/cd/ +
+
+ -
+
+IM from https://webserver2.tecgraf.puc-rio.br/im/ +
+
+ -
+
+ffcall from http://webserver2.tecgraf.puc-rio.br/iup/ +
+
+ -
+
+Nanomsg from https://nanomsg.org/ (NOTE: Plan is to eliminate nanomsg dependency). +
+
+ -
+
+Needed eggs (look at the eggs lists in the Makefile) +
+
+
Then follow these steps:
./configure +make -j install+
+