Logprocessor

Purpose

With logpro you can write command files to rigorously analyze log files. For example you may have a complex simulation with many steps, compilation, simulation, analysis, clean up etc. You want to know in an automated way if the simulation ran. Manually reading the log file(s) once or twice searching for errors, warnings and messages is fine but by the third or forth time you want to automate it. Logpro can help.

Download

Deploy packages compiled for Ubuntu12 and Sles10 are here:

64 bit Sles10

32 bit Ubuntu

To install put the files somewhere and symlink the logpro file to your bin.

Usage

Running logpro. EXAMPLE: Run the program "runsim" passing the output through logpro using the cmdfile.logpro log analysis command file. The annotated log file is written to "run.html" and run-logro.log.

% runsim | logpro cmdfile.logpro run.html > run-logpro.log

In a Makefile:

simlog.html : cpu.v
       verilog cpu.v | logpro cpu.logpro simlog.html

The benefit in using logpro calls in a Makefile is that the exit code can be trusted to reflect the pass/fail status correctly. See the ./doc/tip/example.logpro (which operates on ./doc/tip/example.log and produces ./doc/tip/example.html) in the source repository)

Exit Codes

All ok	0
ERROR	1
WARNING	2
CHECK	3

Installation

Command file details

Your command file is written in scheme but don't let that scare you. Only a handful of single line stanzas are needed to write a rock solid log analyzer. Here is an example command file and the logpro output from running against this example log file

More examples can be seen in the example.logpro file in the source directory http://www.kiatoa.com/fossils/logpro/doc/tip/example.logpro

Syntax

( trigger "trigger name" #/trigger pattern/ )
( section "start trigger" "end trigger" )
( expect [in|not-in] "section name" [> or < or =] integer  #/regular expression/ )

Sections

You can break your log file into sections to make applying the rules very specific. For example you may want to only apply a set of rules in the compile section of the log file related to compiler failures.

Example: Create a section called "Init" that only applies between a line containing the string "This is a header" and a blank line. The lines match on regular expressions delimited by #/ and /.

(trigger "Init"     #/This is a header/)
(trigger "InitEnd"  #/^\s*$/)
(section "Init" "Init" "InitEnd")

You almost certainly want a section which covers the entire log file, here we call it "body". We cheat and do not create a trigger for the end of the section since we want that section valid for the entire log file:

(trigger "Body"     #/^.*$/) ;; anything starts the body
(section "Body"     "Body" "EndBody")

Some sections may occur multiple times in a file. Use trigger-with-limit for those. Use a large number if you don't know how many times the trigger may be hit.

(trigger-with-limit "Blah5"    100 #/^begin Blah5/)
(trigger-with-limit "Blah5End" 100 #/^end Blah5/)
(section "Blah5" "Blah5" "Blah5End")

Required Expects

There will be some lines in your log file that, if missing, indicate a problem. Create required expects for those:

;; If we don't get a header then we know something went badly wrong
(expect:required in "Init"  = 1 "Header"      #/This is a header/)

Catch and ignore errors

The rules are applied in the order they are specified in the command file. Once a rule is hit in the log file no other rules are tested. You can use this to create a catch all and then add exceptions. In this example ERROR 244 is false and should be ignored:

(expect:ignore   in "Body" = 1 "FALSE ERROR" #/ERROR 244/)
(expect:error    in "Body"  = 0 "ERROR BLAH"  #/error/i) ;; disallow any errors

Expire rules

Any rule can have a date set beyond which it will no longer be applied. This is intended to be used on ignore rules that are only intended to be temporarily applied. The date format is MM/DD/YYYY:

(expect:ignore in "Body" < 99 "These are real but I want to ignore them for now" #/ERROR 1234/ expires: "02/04/2011")

This kind of rule is best written as "< N" where N is more than you are ever likely to get of these that you want to ignore.

Values

Extract values from your logfile and apply simple rules for pass/fail checks.

Syntax is:

(expect:value   in section  value  tol  comment_string regex [hook: hookname])

Example, extract a number following a string "Measured voltage output: " and compare it with 1.9 +/- 0.1. Call the hook "freq"

(expect:value    in "LogFileBody" 1.9 0.1 "Output voltage" #/Measured voltage output:\s*([\d\.\+\-e]+)v/ hook: "freq")

Example, extract two numbers from a string, the second number must be greater than 1.9:

(expect:value    in "LogFileBody" 1.9 >   "Time voltage" #/out: (\d+)\s+(\d+)/ matchnum: 2)

More Values Techniques

Hooks

Create a hook to handle the information gathered on a frequency measurement:

(hook:add "freq"    "echo \"Value hook activated: expected=#{expected}, measured=#{measured}, tolerance=#{tolerance}, message=#{message}\"")