A benchmark is the combination of a test case, test function, and a data tag.


------------
Definition: BMCONTEXT = <benchmark> <metric> <platform> <host>

Note: In general, comparisons are meaningful only within the same
BMCONTEXT. This assumption will be enforced in the user interface described
below.


------------
The xml file (produced by QTestLib) contains the benchmark (text case, test
function, and data tag), metric, and result (value and iterations).

When used as a standalone term, a branch refers to the combination of a Git
repository and a Git branch.


------------
Definition: SNAPSHOT = -timestamp <secs since 1/1 1970> | -sha1 <sha1>

A -timestamp value t has the following semantics:

   t = -1: the latest result ('last' may be used as an alias)
   t >= 0: the earliest result that is not earlier than t ('first' may be used as an alias for 0)

Note: The -sha1 option requires exact matching and is thus less powerful than -timestamp.


----------------------------------
Use case 1: Add new results:

    ./bmc put results <xml file> <platform> <host> <git repo> [<git dir>]

    Notes:

        - <xml file> must (essentially) be of the same format as the one being dumped
          by QTestLib.

        - <git dir> must refer to a directory accessible from the local
          file system and the git command needs to be installed (i.e. the program
          needs to run the git command from <git dir>).
          By default, <git dir> is the current directory.

        - The new results are assumed to have been generated using a product
          built from a clean checkout of the current head commit of the current git branch
          in <git dir>.

        - The sha1 to be stored in the database is assumed to be the current head commit in
          <git dir> (extracted running 'git log -n 1 HEAD --pretty=format:%H').

        - The timestamp to be stored in the database is the Unix timestamp of the committer
          date (extracted running 'git log -n 1 HEAD --pretty=format:%ct').

        - <git repo> must be passed explicitly since it cannot in general
          be derived from <git dir> (e.g. in the case of a local, standalone
          repository).

    Example:

        ./bmc put results results.xml linux-g++ barbarella qt master $QTDIR

----------------------------------
Use case 2: List available snapshots for a given branch:

    ./bmc get snapshots BMCONTEXT <branch>

    Example:

        ./bmc list -snapshots tst_QWidget updateOpaque 10x10x1 walltime \
            linux-g++ barbarella qt master

----------------------------------
Use case 3: List available benchmarks (subject to optional restrictions):

    ./bmc get benchmarks [-metric <metric>] [-platform <platform>] [-host <host>] \
        [-branch <branch>]

    Example 1:

        ./bmc get benchmarks

    Example 2:

        ./bmc get benchmarks -platform linux-g++

    Example 3:

        ./bmc get benchmarks -platform linux-g++ -host barbarella -metric walltime \
            -branch qt master

----------------------------------
Use case 4: List available metrics (subject to optional restrictions):

    ./bmc get metrics [-benchmark <test case> <test function> <data tag>] \
        [-platform <platform>] [-host <host>] [-branch <branch>]

    Example 1:

        ./bmc get metrics

    Example 2:

        ./bmc get metrics -benchmark tst_QWidget updateOpaque 10x10x1 \
            -platform linux-g++ -host barbarella -branch qt master

----------------------------------
Use case 5: List available platforms (subject to optional restrictions):

    ./bmc get platforms [-benchmark <test case> <test function> <data tag>] \
        [-metric <metric>] [-host <host>] [-branch <branch>]

----------------------------------
Use case 6: List available hosts (subject to optional restrictions):

    ./bmc get hosts [-benchmark <test case> <test function> <data tag>] \
        [-metric <metric>] [-platform <platform>] [-branch <branch>]

----------------------------------
Use case 7: List available branches (subject to optional restrictions):

    ./bmc get platforms [-benchmark <test case> <test function> <data tag>] \
        [-metric <metric>] [-platform <platform>] [-host <host>]

----------------------------------
Use case 8: List the raw results within a particular historical range.

    ./bmc get history BMCONTEXT <branch> SNAPSHOT1 SNAPSHOT2

    Example 1:

        ./bmc get history tst_QWidget updateOpaque 10x10x1 walltime \
            linux-g++ barbarella qt master -timestamp first -timestamp last

    Example 2 (1250146917 corresponds to the date Thu Aug 13 09:01:57 2009 +0200):

        ./bmc get history tst_QWidget updateOpaque 10x10x1 walltime \
            linux-g++ barbarella qt master -timestamp 1250146917 -timestamp last

    Example 3:

        ./bmc get history tst_QWidget updateOpaque 10x10x1 walltime \
            linux-g++ barbarella qt master -timestamp first \
            -sha1 3bad2f608813968d85c2ea51c1bd75dca7f7e22a

----------------------------------
Use case 9: Compute trend values within a particular historical range
            (to detect perfomance regressions etc.).

    ./bmc get trend BMCONTEXT <branch> SNAPSHOT1 SNAPSHOT2