--------------------------------------------------------------------------------------------
OVERVIEW

This repository contains BM - a set of tools for storing and analyzing
benchmark results produced by QTestLib (or strictly anything that outputs the
same XML format as QTestLib).

A server (bmserver) maintains a database of benchmark results. A simple
stream-based protocol is defined for storing results and retrieving results
and/or context (meta) data associated with the results.

A client (bmclient) takes input from command-line, standard input, and/or
files, and generates output to standard output. The bmclient may either be
used directly by a user from a console, or launched from another program such
as bmrunner (see below) or a web server. When launched from a web server, the
bmclient acts as a CGI program and outputs JSON data to be forwarded to the
JavsScript program in the web client.

A benchmark runner (bmrunner) executes a given set of QTestLib-based benchmark
programs and passes the results of each one to a bmsrever via a bmclient.


--------------------------------------------------------------------------------------------
CONTENTS OF THIS DIRECTORY

    README            - This file.

    src/              - Source code:
        src/bm/           - Common code (C++/Qt).
        src/bmclient/     - The bmclient (C++/Qt).
        src/bmserver/     - The bmserver (C++/Qt).
        src/bmrunner/     - A program for running QTestLib-based benchmark
                            programs and passing the results to a bmserver (C++/Qt).
        src/addtestdata/  - A program for passing artificial benchmark results
                            to a bmserver (C++/Qt).
        src/bmweb/        - Web client code (HTML and JavaScript).

    doc/              - Various documentation, including:
        doc/bmclient.html - The bmclient (including command-line args and JSON output).
        doc/bmserver.html - The bmserver.
        doc/bmproto.html  - The XML-based client-server protocol.
        doc/bmrunner.html - The bmrunner.


--------------------------------------------------------------------------------------------
DEPLOYMENT AND INSTALLATION

The computers that form the BM system play different roles (some may play more
than one). Below is a brief description of the typical installation- and
deployment steps required for each role.

Role 1: Running the bmserver.

    Step 1: Build and install the bmserver (a Qt program).
    Step 2: Start the bmserver, passing a SQLite database file to be either
            created (with no data) or reopened.
    Step 3: Arrange for the bmserver to be automatically started at boot time
            if desired (on Linux, this typically involves updating files under
            /etc/init.d/).

Role 2: Generating new benchmark results and passing them to the bmserver.

    Step 1: Use git to fetch the latest version of the product to be
            benchmarked (typically a particular version of Qt).
    Step 2: Build and install the product.
    Step 3: Build and install the benchmark programs (typically QTestLib
            applications linked with the latest product version).
    Step 4: Build and install the bmclient (a Qt program).
    Step 5: Build and install the bmrunner (a Qt program).
    Step 6: Execute the bmrunner, passing the location of the bmserver, the
            directory of the benchmark programs etc. (see separate
            documentation for bmrunner). Note: The bmrunner assumes that the
            bmclient can be launched directly, so for example on Linux, the
            PATH and LD_LIBRARY_PATH may need to be modified first. The
            bmclient in turn assumes that the git command is available
            (ditto).

    Note: It is strictly not necessary to repeat steps 4 and 5 every time new
          results are to be generated, but in a fully automated setup it could
          be convenient to rebuild/reinstall even these two programs.

Role 3: Running the web server.

    Step 1: Build and install the bmclient (a Qt program).
    Step 2: Install the bmweb directory.
    Step 3: Build and install the web server (see separate documentation for
            the web server).
    Step 4: Install a script named 'bmcclientwrapper' that typically looks
            like this on Linux:

            #!/bin/sh
            LD_LIBRARY_PATH=<location of any shared libraries required by the bmclient> \
            BMSERVER=<location of the bmserver (host:port)> \
            <bmclient executable>

    Step 5: Configure the web server to 1) allow access to the bmweb directory,
            and 2) to allow launching 'bmclientwrapper' as a CGI script (see
            separate documentation for the web server).
    Step 6: Start the web server.
    Step 7: Arrange for the web server to be automatically started at boot time
            if desired.

    Note: The BM system has been tested with Apache 2.2.11 on Linux.

Role 4: Viewing results in a web client.

    Step 1: Build, install, and run the web client.
    Step 2: Open the URL to the BM site (i.e. referring to the bmweb directory on
            the web server host.)
    Step 3: Enable cookies for the BM site to store certain settings between
            sessions.

    Note: The BM system has been tested with Firefox 3.5.3 on Linux. Currently,
          the use of the <canvas> tag for client-side graphics (in particular
          the history plot) limits the usability of certain browsers, like
          konqueror.