
[//000000001]: # (bench \- Benchmarking/Performance tools)
[//000000002]: # (Generated from file 'bench\.man' by tcllib/doctools with format 'markdown')
[//000000003]: # (Copyright &copy; 2007\-2008 Andreas Kupries <andreas\_kupries@users\.sourceforge\.net>)
[//000000004]: # (bench\(n\) 0\.6 tcllib "Benchmarking/Performance tools")

<hr> [ <a href="../../../../toc.md">Main Table Of Contents</a> &#124; <a
href="../../../toc.md">Table Of Contents</a> &#124; <a
href="../../../../index.md">Keyword Index</a> &#124; <a
href="../../../../toc0.md">Categories</a> &#124; <a
href="../../../../toc1.md">Modules</a> &#124; <a
href="../../../../toc2.md">Applications</a> ] <hr>

# NAME

bench \- bench \- Processing benchmark suites

# <a name='toc'></a>Table Of Contents

  - [Table Of Contents](#toc)

  - [Synopsis](#synopsis)

  - [Description](#section1)

  - [PUBLIC API](#section2)

      - [Benchmark execution](#subsection1)

      - [Result manipulation](#subsection2)

      - [Result format](#subsection3)

  - [Bugs, Ideas, Feedback](#section3)

  - [See Also](#seealso)

  - [Keywords](#keywords)

  - [Category](#category)

  - [Copyright](#copyright)

# <a name='synopsis'></a>SYNOPSIS

package require Tcl 8\.5 9  
package require bench ?0\.6?  

[__::bench::locate__ *pattern* *paths*](#1)  
[__::bench::run__ ?*option value*\.\.\.? *interp\_list* *file*\.\.\.](#2)  
[__::bench::versions__ *interp\_list*](#3)  
[__::bench::del__ *bench\_result* *column*](#4)  
[__::bench::edit__ *bench\_result* *column* *newvalue*](#5)  
[__::bench::merge__ *bench\_result*\.\.\.](#6)  
[__::bench::norm__ *bench\_result* *column*](#7)  
[__::bench::out::raw__ *bench\_result*](#8)  

# <a name='description'></a>DESCRIPTION

This package provides commands for the execution of benchmarks written in the
bench language, and for the processing of results generated by such execution\.

A reader interested in the bench language itself should start with the *[bench
language introduction](bench\_lang\_intro\.md)* and proceed from there to the
formal *[bench language specification](bench\_lang\_spec\.md)*\.

# <a name='section2'></a>PUBLIC API

## <a name='subsection1'></a>Benchmark execution

  - <a name='1'></a>__::bench::locate__ *pattern* *paths*

    This command locates Tcl interpreters and returns a list containing their
    paths\. It searches them in the list of *paths* specified by the caller,
    using the glob *pattern*\.

    The command resolves soft links to find the actual executables matching the
    pattern\. Note that only interpreters which are marked as executable and are
    actually executable on the current platform are put into the result\.

  - <a name='2'></a>__::bench::run__ ?*option value*\.\.\.? *interp\_list* *file*\.\.\.

    This command executes the benchmarks declared in the set of files, once per
    Tcl interpreter specified via the *interp\_list*, and per the configuration
    specified by the options, and then returns the accumulated timing results\.
    The format of this result is described in section [Result
    format](#subsection3)\.

    It is assumed that the contents of the files are written in the bench
    language\.

    The available options are

      * __\-errors__ *flag*

        The argument is a boolean value\. If set errors in benchmarks are
        propagated to the command, aborting benchmark execution\. Otherwise they
        are recorded in the timing result via a special result code\. The default
        is to propagate and abort\.

      * __\-threads__ *n*

        The argument is a non\-negative integer value declaring the number of
        threads to use while executing the benchmarks\. The default value is
        __0__, to not use threads\.

      * __\-match__ *pattern*

        The argument is a glob pattern\. Only benchmarks whose description
        matches the pattern are executed\. The default is the empty string, to
        execute all patterns\.

      * __\-rmatch__ *pattern*

        The argument is a regular expression pattern\. Only benchmarks whose
        description matches the pattern are executed\. The default is the empty
        string, to execute all patterns\.

      * __\-iters__ *n*

        The argument is positive integer number, the maximal number of
        iterations for any benchmark\. The default is __1000__\. Individual
        benchmarks can override this\.

      * __\-pkgdir__ *path*

        The argument is a path to an existing, readable directory\. Multiple
        paths can be specified, simply use the option multiple times, each time
        with one of the paths to use\.

        If no paths were specified the system will behave as before\. If one or
        more paths are specified, say __N__, each of the specified
        interpreters will be invoked __N__ times, with one of the specified
        paths\. The chosen path is put into the interpreters' __auto\_path__,
        thus allowing it to find specific versions of a package\.

        In this way the use of __\-pkgdir__ allows the user to benchmark
        several different versions of a package, against one or more
        interpreters\.

        *Note:* The empty string is allowed as a path and causes the system to
        run the specified interpreters with an unmodified __auto\_path__\. In
        case the package in question is available there as well\.

  - <a name='3'></a>__::bench::versions__ *interp\_list*

    This command takes a list of Tcl interpreters, identified by their path, and
    returns a dictionary mapping from the interpreters to their versions\.
    Interpreters which are not actually executable, or fail when interrogated,
    are not put into the result\. I\.e the result may contain less interpreters
    than there in the input list\.

    The command uses builtin command __info patchlevel__ to determine the
    version of each interpreter\.

## <a name='subsection2'></a>Result manipulation

  - <a name='4'></a>__::bench::del__ *bench\_result* *column*

    This command removes a column, i\.e\. all benchmark results for a specific Tcl
    interpreter, from the specified benchmark result and returns the modified
    result\.

    The benchmark results are in the format described in section [Result
    format](#subsection3)\.

    The column is identified by an integer number\.

  - <a name='5'></a>__::bench::edit__ *bench\_result* *column* *newvalue*

    This command renames a column in the specified benchmark result and returns
    the modified result\. This means that the path of the Tcl interpreter in the
    identified column is changed to an arbitrary string\.

    The benchmark results are in the format described in section [Result
    format](#subsection3)\.

    The column is identified by an integer number\.

  - <a name='6'></a>__::bench::merge__ *bench\_result*\.\.\.

    This commands takes one or more benchmark results, merges them into one big
    result, and returns that as its result\.

    All benchmark results are in the format described in section [Result
    format](#subsection3)\.

  - <a name='7'></a>__::bench::norm__ *bench\_result* *column*

    This command normalizes the timing results in the specified benchmark result
    and returns the modified result\. This means that the cell values are not
    times anymore, but factors showing how much faster or slower the execution
    was relative to the baseline\.

    The baseline against which the command normalizes are the timing results in
    the chosen column\. This means that after the normalization the values in
    this column are all __1__, as these benchmarks are neither faster nor
    slower than the baseline\.

    A factor less than __1__ indicates a benchmark which was faster than the
    baseline, whereas a factor greater than __1__ indicates a slower
    execution\.

    The benchmark results are in the format described in section [Result
    format](#subsection3)\.

    The column is identified by an integer number\.

  - <a name='8'></a>__::bench::out::raw__ *bench\_result*

    This command formats the specified benchmark result for output to a file,
    socket, etc\. This specific command does no formatting at all, it passes the
    input through unchanged\.

    For other formatting styles see the packages
    __[bench::out::text](bench\_wtext\.md)__ and
    __[bench::out::csv](bench\_wcsv\.md)__ which provide commands to
    format benchmark results for human consumption, or as CSV data importable by
    spread sheets, respectively\.

    Complementary, to read benchmark results from files, sockets etc\. look for
    the package __[bench::in](bench\_read\.md)__ and the commands provided
    by it\.

## <a name='subsection3'></a>Result format

After the execution of a set of benchmarks the raw result returned by this
package is a Tcl dictionary containing all the relevant information\. The
dictionary is a compact representation, i\.e\. serialization, of a 2\-dimensional
table which has Tcl interpreters as columns and benchmarks as rows\. The cells of
the table contain the timing results\. The Tcl interpreters / columns are
identified by their paths\. The benchmarks / rows are identified by their
description\.

The possible keys are all valid Tcl lists of two or three elements and have one
of the following forms:

  - \{interp \*\}

    The set of keys matching this glob pattern capture the information about all
    the Tcl interpreters used to run the benchmarks\. The second element of the
    key is the path to the interpreter\.

    The associated value is the version of the Tcl interpreter\.

  - \{desc \*\}

    The set of keys matching this glob pattern capture the information about all
    the benchmarks found in the executed benchmark suite\. The second element of
    the key is the description of the benchmark, which has to be unique\.

    The associated value is irrelevant, and set to the empty string\.

  - \{usec \* \*\}

    The set of keys matching this glob pattern capture the performance
    information, i\.e\. timing results\. The second element of the key is the
    description of the benchmark, the third element the path of the Tcl
    interpreter which was used to run it\.

    The associated value is either one of several special result codes, or the
    time it took to execute the benchmark, in microseconds\. The possible special
    result codes are

      * ERR

        Benchmark could not be executed, failed with a Tcl error\.

      * BAD\_RES

        The benchmark could be executed, however the result from its body did
        not match the declared expectations\.

# <a name='section3'></a>Bugs, Ideas, Feedback

This document, and the package it describes, will undoubtedly contain bugs and
other problems\. Please report such in the category *bench* of the [Tcllib
Trackers](http://core\.tcl\.tk/tcllib/reportlist)\. Please also report any ideas
for enhancements you may have for either package and/or documentation\.

When proposing code changes, please provide *unified diffs*, i\.e the output of
__diff \-u__\.

Note further that *attachments* are strongly preferred over inlined patches\.
Attachments can be made by going to the __Edit__ form of the ticket
immediately after its creation, and then using the left\-most button in the
secondary navigation bar\.

# <a name='seealso'></a>SEE ALSO

[bench\_intro](bench\_intro\.md), [bench\_lang\_intro](bench\_lang\_intro\.md),
[bench\_lang\_spec](bench\_lang\_spec\.md), bench\_read, bench\_wcsv, bench\_wtext

# <a name='keywords'></a>KEYWORDS

[benchmark](\.\./\.\./\.\./\.\./index\.md\#benchmark),
[merging](\.\./\.\./\.\./\.\./index\.md\#merging),
[normalization](\.\./\.\./\.\./\.\./index\.md\#normalization),
[performance](\.\./\.\./\.\./\.\./index\.md\#performance),
[testing](\.\./\.\./\.\./\.\./index\.md\#testing)

# <a name='category'></a>CATEGORY

Benchmark tools

# <a name='copyright'></a>COPYRIGHT

Copyright &copy; 2007\-2008 Andreas Kupries <andreas\_kupries@users\.sourceforge\.net>
