pegasus-statistics(1) [debian man page]

PEGASUS-STATISTICS(1)													     PEGASUS-STATISTICS(1)

NAME

       pegasus-statistics - A tool to generate statistics about the workflow run.

SYNOPSIS

       pegasus-statistics [-h|--help]
			  [-o|--output dir]
			  [-c|--conf propfile]
			  [-p|--statistics-level level]
			  [-t|--time-filter filter]
			  [-i|--ignore-db-inconsistency]
			  [-v|--verbose]
			  [-q|--quiet]
			  [submitdir]

DESCRIPTION

       pegasus-statistics generates statistics about the workflow run like total jobs/tasks/sub workflows ran , how many succeeded/failed etc. It
       generates job instance statistics like run time, condor queue delay etc. It generates invocation statistics information grouped by
       transformation name. It also generates job instance and invocation statistics information grouped by time and host.

OPTIONS

       -h, --help
	   Prints a usage summary with all the available command-line options.

       -o dir, --output dir
	   Writes the output to the given directory.

       -c propfile, --conf propfile
	   The properties file to use. This option overrides all other property files.

       -s level, --statistics-level level
	   Specifies the statistics information to generate. Valid levels are: all, summary, wf_stats, jb_stats, tf_stats, and ti_stats. Default
	   is summary. The output generated by pegasus-statistics is based on the the level set:

	   o	all: generates all the statistics information.

	   o	summary: generates the workflow statistics summary. In the case of a hierarchical workflow the summary is across all sub
	       workflows.

	   o	wf_stats: generates the workflow statistics information of each individual workflow. In case of a hierarchical workflow the
	       workflow statistics are created for each sub workflow.

	   o	jb_stats: generates the job statistics information of each individual workflow. In case of hierarchical workflow the job
	       statistics is created for each sub workflows.

	   o	tf_stats: generates the invocation statistics information of each individual workflow grouped by transformation name .In case of
	       hierarchical workflow the transformation statistics is created for each sub workflows.

	   o	ti_stats: generates the job instance and invocation statistics like total count and runtime grouped by time and host.

       -t filter, --time-filter filter
	   Specifies the time filter to group the time statistics. Valid filter values are: month, week, day, hour. Default is day.

       -i, --ignore-db-inconsistency
	   Turn off the the check for database consistency.

       -v, --verbose
	   Increases the log level. If omitted, the default level will be set to WARNING. When this option is given, the log level is changed to
	   INFO. If this option is repeated, the log level will be changed to DEBUG.

       -q, --quiet
	   Decreases the log level. If omitted, the default level will be set to WARNING. When this option is given, the log level is changed to
	   ERROR.

EXAMPLE

       Runs pegasus-statistics and writes the output to the given directory:

	   $ pegasus-statistics  -o /scratch/statistics /scratch/grid-setup/run0001

AUTHORS

       Prasanth Thomas

       Pegasus Team http://pegasus.isi.edu

								    05/24/2012						     PEGASUS-STATISTICS(1)

PEGASUS-EXITCODE(1)													       PEGASUS-EXITCODE(1)

NAME

       pegasus-exitcode - Checks the stdout/stderr files of a workflow job for any indication that an error occurred in the job. This script is
       intended to be invoked automatically by DAGMan as the POST script of a job.

SYNOPSIS

       pegasus-exitcode [-h][-t n][-r rv][-n] job.out

DESCRIPTION

       pegasus-exitcode is a utility that examines the STDOUT of a job to determine if the job failed, and renames the STDOUT and STDERR files of
       a job to preserve them in case the job is retried.

       Pegasus uses pegasus-exitcode as the DAGMan postscript for all jobs submitted via Globus GRAM. This tool exists as a workaround to a known
       problem with Globus where the exitcodes of GRAM jobs are not returned. This is a problem because Pegasus uses the exitcode of a job to
       determine if the job failed or not.

       In order to get around the exitcode problem, Pegasus wraps all GRAM jobs with Kickstart, which records the exitcode of the job in an XML
       invocation record, which it writes to the job's STDOUT. The STDOUT is transferred from the execution host back to the submit host when the
       job terminates. After the job terminates, DAGMan runs the job's postscript, which Pegasus sets to be pegasus-exitcode. pegasus-exitcode
       looks at the invocation record generated by kickstart to see if the job succeeded or failed. If the invocation record indicates a failure,
       then pegasus-exitcode returns a non-zero result, which indicates to DAGMan that the job has failed. If the invocation record indicates that
       the job succeeded, then pegasus-exitcode returns 0, which tells DAGMan that the job succeeded.

       pegasus-exitcode performs several checks to determine whether a job failed or not. These checks include:

	1. Is STDOUT empty? If it is empty, then the job failed.

	2. Are there any <status> tags with a non-zero value? If there are, then the job failed. Note that, if this is a clustered job, there
	   could be multiple <status> tags, one for each task. If any of them are non-zero, then the job failed.

	3. Is there at least one <status> tag with a zero value? There must be at least one successful invocation or the job has failed.

       In addition, pegasus-exitcode allows the caller to specify the exitcode returned by Condor using the --return argument. This can be passed
       to pegasus-exitcode in a DAGMan post script by using the $RETURN variable. If this value is non-zero, then pegasus-exitcode returns a
       non-zero result before performing any other checks. For GRAM jobs, the value of $RETURN will always be 0 regardless of whether the job
       failed or not.

       Also, pegasus-exitcode allows the caller to specify the number of successful tasks it should see using the --tasks argument. If
       pegasus-exitcode does not see N successful tasks, where N is set by --tasks, then it will return a non-zero result. The default value is 1.
       This can be used to detect failures in clustered jobs where, for any number of reasons, invocation records do not get generated for all the
       tasks in the clustered job.

       In addition to checking the success/failure of a job, pegasus-exitcode also renames the STDOUT and STDERR files of the job so that if the
       job is retried, the STDOUT and STDERR of the previous run are not lost. It does this by appending a sequence number to the end of the
       files. For example, if the STDOUT file is called "job.out", then the first time the job is run pegasus-exitcode will rename the file
       "job.out.000". If the job is run again, then pegasus-exitcode sees that "job.out.000" already exists and renames the file "job.out.001". It
       will continue to rename the file by incrementing the sequence number every time the job is executed.

OPTIONS

       -h, --help
	   Prints a usage summary with all the available command-line options.

       -t n, --tasks n
	   Number of tasks expected. If less than n tasks succeeded, then pegasus-exitcode will fail with a non-zero return value. This is used in
	   cases where we may not get a Kickstart invocation record for some tasks. Normally Seqexec will detect failed Kickstart invocations and
	   fail accordingly.

       -r rv, --return rv
	   Return value reported by DAGMan. This can be specified in the DAG using the $RETURN variable. If this is non-zero, then
	   pegasus-exitcode immediately fails with a non-zero return value itself.

       -n, --no-rename
	   Don't rename job.out and job.err to .out.XXX and .err.XXX. This option is used primarily for testing.

AUTHORS

       Gideon Juve <juve@usc.edu>

       Pegasus Team http://pegasus.isi.edu

								    05/24/2012						       PEGASUS-EXITCODE(1)