Index: docs/manual/Makefile ================================================================== --- docs/manual/Makefile +++ docs/manual/Makefile @@ -1,5 +1,7 @@ -megatest_manual.html : megatest_manual.txt getting_started.txt writing_tests.txt reference.txt +megatest_manual.html : megatest_manual.txt getting_started.txt writing_tests.txt reference.txt ../plan.txt asciidoc megatest_manual.txt dos2unix megatest_manual.html +clean: + rm -f megatest_manual.html Index: docs/manual/megatest_manual.html ================================================================== --- docs/manual/megatest_manual.html +++ docs/manual/megatest_manual.html @@ -750,21 +750,22 @@

This book is organised as three sub-books; getting started, writing tests and reference.

Why Megatest?

The Megatest project was started for two reasons, the first was an immediate and pressing need for a generalized tool to manage a suite -of regression tests and the second was the fact that I had written or -maintained several such tools at different companies over the years -and it seemed a good thing to have a single open source tool, flexible -enough to meet the needs of any team doing continuous integrating and -or running a complex suite of tests for release qualification.

+of regression tests and the second was the fact that the author had +written or maintained several such tools at different companies over +the years and it seemed a good thing to have a single open source +tool, flexible enough to meet the needs of any team doing continuous +integrating and or running a complex suite of tests for release +qualification.

Megatest Design Philosophy

Megatest is intended to provide the minimum needed resources to make -writing a suite of tests and implementing continuous build for -software, design engineering or process control (via owlfs for +writing a suite of tests and tasks for implementing continuous build +for software, design engineering or process control (via owlfs for example) without being specialized for any specific problem space. Megatest in of itself does not know what constitutes a PASS or FAIL of a test. In most cases megatest is best used in conjunction with logpro or a similar tool to parse, analyze and decide on the test outcome.

@@ -778,10 +779,319 @@ which can launch jobs on local and remote Linux hosts. Currently megatest uses the network filesystem to call home to your master sqlite3 database.

+ +

Road Map

+

Note: This road-map is tentative and subject to change without notice.

+
+

ww32

+
    +
  1. +

    +Rerun step and or subsequent steps from gui +

    +
  2. +
  3. +

    +Refresh test area files from gui +

    +
  4. +
  5. +

    +Clean and re-run button +

    +
  6. +
  7. +

    +Clean up STATE and STATUS handling. +

    +
      +
    1. +

      +Dashboard and Test control panel are reverse order - choose and fix +

      +
    2. +
    3. +

      +Move seldom used states and status to drop down selector +

      +
    4. +
    +
  8. +
  9. +

    +Access test control panel when clicking on Run Summary tests +

    +
  10. +
  11. +

    +Feature: -generate-index-tree +

    +
  12. +
  13. +

    +Change specifing of state and status to use STATE1/STATUS1,STATE2/STATUS2 +

    +
  14. +
+
+
+

ww33

+
    +
  1. +

    +http api available for use with Perl, Ruby etc. scripts +

    +
  2. +
  3. +

    +megatest.config setup entries for: +

    +
      +
    1. +

      +run launching (e.g. /bin/sh %CMD% > /dev/null) +

      +
    2. +
    3. +

      +browser "konqueror %FNAME% +

      +
    4. +
    +
  4. +
+
+
+

ww34

+
    +
  1. +

    +Mark dependent tests for clean/rerun -rerun-downstream +

    +
  2. +
  3. +

    +On run start check for defunct tests in RUNNING, LAUNCHED or REMOTEHOSTSTART and correct or notify +

    +
  4. +
  5. +

    +Fix: refresh of gui sometimes fails on last item (race condition?) +

    +
  6. +
+
+
+

ww35

+
    +
  1. +

    +refdb: Add export of csv, json and sexp +

    +
  2. +
  3. +

    +Convert to using call-with-environment-variables where possible. Should allow handling of parallel runs in same process. +

    +
  4. +
  5. +

    +Re-work text interface wizards. Several bugs on record. Possibly convert to gui based. +

    +
  6. +
  7. +

    +Add to testconfig requirements section; launchlimiter scriptname, calls scriptname to check if ok to launch test +

    +
  8. +
  9. +

    +Refactor Run Summary view, currently very clumsy +

    +
  10. +
  11. +

    +Add option to show steps in Run Summary view +

    +
  12. +
+
+
+

ww36

+
    +
  1. +

    +Refactor guis for resizeablity +

    +
  2. +
  3. +

    +Add filters to Run Summary view and Run Control view +

    +
  4. +
  5. +

    +Add to megatest.config or testconfig; rerunok STATE/STATUS,STATE/STATUS… +

    +
  6. +
+
+
+

Bin List

+
    +
  1. +

    +Quality improvements +

    +
      +
    1. +

      +Server stutters occasionally +

      +
    2. +
    3. +

      +Large number of items or tests still has some issues. +

      +
    4. +
    5. +

      +Code refactoring +

      +
    6. +
    7. +

      +Replace remote process with true API using json (supports Web app also) +

      +
    8. +
    +
  2. +
  3. +

    +Streamline the gui +

    +
      +
    1. +

      +Everything resizable +

      +
    2. +
    3. +

      +Less clutter +

      +
    4. +
    5. +

      +Tool tips +

      +
    6. +
    7. +

      +Filters on Run Summary, Summary and Run Control panel +

      +
    8. +
    9. +

      +Built in log viewer (partially implemented) +

      +
    10. +
    11. +

      +Refactor the test control panel +

      +
    12. +
    +
  4. +
  5. +

    +Help and documentation +

    +
      +
    1. +

      +Complete the user manual (I’ve been working on this lately). +

      +
    2. +
    3. +

      +Online help in the gui +

      +
    4. +
    +
  6. +
  7. +

    +Streamlined install +

    +
      +
    1. +

      +Deployed version (download a location independent ready to run binary bundle) +

      +
    2. +
    3. +

      +Install Makefile (in progress, needed for Mike to install on VMs) +

      +
    4. +
    5. +

      +Added option to compile IUP (needed for VMs) +

      +
    6. +
    +
  8. +
  9. +

    +Server side run launching +

    +
  10. +
  11. +

    +Support for re-running, cleaning etc. of individual steps (ezsteps makes this very easy to implement). +

    +
  12. +
  13. +

    +Launch process needs built in daemonizing (easy to do, just need to test it thoroughly). +

    +
  14. +
  15. +

    +Wizards for creating tests, regression areas (current ones are text only and limited). +

    +
  16. +
  17. +

    +Fully functional built in web service (currently you can browse runs but it is very simplistic). +

    +
  18. +
  19. +

    +Wildcards in runconfigs: e.g. [p1271/9/%/%] +

    +
  20. +
  21. +

    +Gui panels for editing megatest.config and runconfigs.config +

    +
  22. +
  23. +

    +Fully isolated tests (no use of NFS to see regression area files) +

    +
  24. +
  25. +

    +Windows version +

    +
  26. +

Getting Started

Getting started with Megatest
@@ -869,17 +1179,25 @@

An example link to anchor at start of the first sub-section.

An example link to a bibliography entry [taoup].

-

Reference

+

Writing Tests

The First Chapter of the Second Part

Chapters grouped into book parts are at level 1 and can contain sub-sections.

+
+

Reference

+
+

The First Chapter of the Second Part

+
+

Chapters grouped into book parts are at level 1 and can contain +sub-sections.

+

The testconfig File

@@ -1084,10 +1402,10 @@

Index: docs/manual/megatest_manual.txt ================================================================== --- docs/manual/megatest_manual.txt +++ docs/manual/megatest_manual.txt @@ -13,22 +13,23 @@ Why Megatest? ~~~~~~~~~~~~~ The Megatest project was started for two reasons, the first was an immediate and pressing need for a generalized tool to manage a suite -of regression tests and the second was the fact that I had written or -maintained several such tools at different companies over the years -and it seemed a good thing to have a single open source tool, flexible -enough to meet the needs of any team doing continuous integrating and -or running a complex suite of tests for release qualification. +of regression tests and the second was the fact that the author had +written or maintained several such tools at different companies over +the years and it seemed a good thing to have a single open source +tool, flexible enough to meet the needs of any team doing continuous +integrating and or running a complex suite of tests for release +qualification. Megatest Design Philosophy ~~~~~~~~~~~~~~~~~~~~~~~~~~ Megatest is intended to provide the minimum needed resources to make -writing a suite of tests and implementing continuous build for -software, design engineering or process control (via owlfs for +writing a suite of tests and tasks for implementing continuous build +for software, design engineering or process control (via owlfs for example) without being specialized for any specific problem space. Megatest in of itself does not know what constitutes a PASS or FAIL of a test. In most cases megatest is best used in conjunction with logpro or a similar tool to parse, analyze and decide on the test outcome. @@ -42,11 +43,13 @@ the distributed compute platform in use. A template script is provided which can launch jobs on local and remote Linux hosts. Currently megatest uses the network filesystem to call home to your master sqlite3 database. -include::getting_started.txt[] include::writing_tests.txt[] +include::../plan.txt[] +include::getting_started.txt[] +include::writing_tests.txt[] include::reference.txt[] [appendix] Example Appendix ================ Index: docs/plan.txt ================================================================== --- docs/plan.txt +++ docs/plan.txt @@ -1,7 +1,58 @@ Road Map ======== + +Note: This road-map is tentative and subject to change without notice. + +ww32 +~~~~ + +. Rerun step and or subsequent steps from gui +. Refresh test area files from gui +. Clean and re-run button +. Clean up STATE and STATUS handling. +.. Dashboard and Test control panel are reverse order - choose and fix +.. Move seldom used states and status to drop down selector +. Access test control panel when clicking on Run Summary tests +. Feature: -generate-index-tree +. Change specifing of state and status to use STATE1/STATUS1,STATE2/STATUS2 + +ww33 +~~~~ + +. http api available for use with Perl, Ruby etc. scripts +. megatest.config setup entries for: +.. run launching (e.g. /bin/sh %CMD% > /dev/null) +.. browser "konqueror %FNAME% + +ww34 +~~~~ + +. Mark dependent tests for clean/rerun -rerun-downstream +. On run start check for defunct tests in RUNNING, LAUNCHED or REMOTEHOSTSTART and correct or notify +. Fix: refresh of gui sometimes fails on last item (race condition?) + +ww35 +~~~~ + +. refdb: Add export of csv, json and sexp +. Convert to using call-with-environment-variables where possible. Should allow handling of parallel runs in same process. +. Re-work text interface wizards. Several bugs on record. Possibly convert to gui based. +. Add to testconfig requirements section; launchlimiter scriptname, calls scriptname to check if ok to launch test +. Refactor Run Summary view, currently very clumsy +. Add option to show steps in Run Summary view + +ww36 +~~~~ + +. Refactor guis for resizeablity +. Add filters to Run Summary view and Run Control view +. Add to megatest.config or testconfig; rerunok STATE/STATUS,STATE/STATUS... +. Launch gates for diskspace; /path/one>1G,/path/two>200M,/tmp>5G + +Bin List +~~~~~~~~ . Quality improvements .. Server stutters occasionally .. Large number of items or tests still has some issues. .. Code refactoring @@ -28,17 +79,5 @@ . Wildcards in runconfigs: e.g. [p1271/9/%/%] . Gui panels for editing megatest.config and runconfigs.config . Fully isolated tests (no use of NFS to see regression area files) . Windows version -Move test specific db to test dir -================================= - -. Create teststats.db -. Redirect test run stats to teststats.db -. Redirect test steps data to teststats.db -. Redirect test_data to teststats.db -. Direct dboard to get stats from teststats.db -. Redirect kill requests to teststats.db -. Kill requests need to kill all processes in the tree -. Roll up overall stats to megatest.db every five minutes or when test done -. Add any necessary tests