BM Protocol

Overview

The BM protocol is a simple protocol that assumes a reliable byte channel (like TCP) between a client and a BM server. Requests are sent from the client to the server. Each request is followed by a reply sent from the server to the client.

General Message Structure

Each request or reply message consists of a header followed by a data part. The header consists of a single four-byte quint32 value (in big endian, i.e. most significant byte first) that holds the size of the data part (in bytes). The data part is encoded in XML as specified in the table below. Note: Any reply message may optionally contain an error attribute in its <reply> tag. If non-empty, the value of this attribute is an error message that describes the reason why the request could not be successfully processed.

Name Description Request message Reply message
PutResults Adds new results to the database. 
<request type="PutResults">
    <context platform="..." host="..."
        gitRepo="..." gitBranch="..."
        timestamp="..." sha1="..."
        testCase="..." />
    <testFunction name="...">
        <dataTag name="..." metric="..."
            value"..." />
        ...
    </testFunction>
    ...
</request>
<reply type="PutResults" />
GetMetrics Lists available metrics. Additional context components (platform, host, or git repo/branch) may be specified as non-empty values to restrict the output to metrics for which at least one result exists in combination with those values. If all additional entity values are empty, the output will include all available metrics, even those that haven't been associated with any result yet. 
<request type="GetMetrics">
    <context platform="..."
        host="..." gitRepo="..."
        gitBranch="..." />
</request>

<reply type="GetMetrics">
    <metric name="..." />
    <metric name="..." />
    ...
</reply>
GetPlatforms Behaves similarly to GetMetrics (see this). 
<request type="GetPlatforms">
    <context metric="..."
        host="..." gitRepo="..."
        gitBranch="..." />
</request>

<reply type="GetPlatforms">
    <platform name="..." />
    <platform name="..." />
    ...
</reply>
GetHosts Behaves similarly to GetMetrics (see this). 
<request type="GetHosts">
    <context metric="..."
        platform="..." gitRepo="..."
        gitBranch="..." />
</request>

<reply type="GetHosts">
    <host name="..." />
    <host name="..." />
    ...
</reply>
GetBranches Behaves similarly to GetMetrics (see this). 
<request type="GetBranches">
    <context metric="..."
        platform="..." host="..." />
</request>

<reply type="GetBranches">
    <branch gitRepo="..." gitBranch="..." />
    <branch gitRepo="..." gitBranch="..." />
    ...
</reply>
GetHistory Lists statistics and history for a specific benchmark.

Note: The value first for timestamp1 specifies the first available snapshot, while the value last for timestamp2 specifies the last available snapshot. Otherwise, timestamp1 refers to the earliest snapshot that is not earlier than the specified value, while timestamp2 refers to the latest snapshot that is not later than the specified value.

Notes:
  • The <lastDiff> tag is omitted altogether if no last difference exists (within the specified range).
  • A non-negative value for the index attribute in the <lastDiff> tag refers to the index (0 = first) of the last difference in the list of <result> tags. (A negative value means that the list of <result> tags doesn't include the last difference.)
  • The <lastValue> tag is omitted altogether if no last value exists (i.e. the specified range is empty).
  • The <regression> tag is omitted altogether if the regression cannot be computed.
  • The id attribute in the <result> tag is the primary key that identifies the result in the database.
 
<request type="GetHistory">
    <context testCase="..." testFunction="..."
        dataTag="..." metric="..."
        platform="..." host="..."
        gitRepo="..." gitBranch="..."
        timestamp1="..." timestamp2="..."
        diffTolerance="..." stabTolerance="..."
        maxSize="..." />
</request>

<reply type="GetHistory">
    <lastDiff index="..." stable="..."
        timestamp="..." sha1="..."
        value="..." />
    <lastValue stable="..." />
    <regression a="..." b="..." />
    <result timestamp="..." sha1="..."
        value="..." id="..." />
    <result timestamp="..." sha1="..."
        value="..." id="..." />
    ...
</reply>
GetHistory2 Lists statistics and history for a specific benchmark in two branches.
2 B COMPLETED 
<request type="GetHistory2">
    <context testCase="..." testFunction="..."
        dataTag="..." metric="..."
        platform="..." host="..."
        gitRepo1="..." gitBranch1="..."
        gitRepo2="..." gitBranch2="..."
        timestamp1="..." timestamp2="..."
        diffTolerance="..." stabTolerance="..."
        maxSize="..." />
</request>

<reply type="GetHistory2">
    <lastDiff1 index="..." stable="..."
        timestamp="..." sha1="..."
        value="..." />
    <lastValue1 stable="..." />
    <regression1 a="..." b="..." />
    <result1 timestamp="..." sha1="..."
        value="..." />
    <result1 timestamp="..." sha1="..."
        value="..." />
    ...
    <lastDiff2 index="..." stable="..."
        timestamp="..." sha1="..."
        value="..." />
    <lastValue2 stable="..." />
    <regression2 a="..." b="..." />
    <result2 timestamp="..." sha1="..."
        value="..." />
    <result2 timestamp="..." sha1="..."
        value="..." />
    ...
</reply>
GetRankedBenchmarks Lists the benchmarks with the most important last difference values.

The snapshot range (timestamp1/2) is defined as for the GetHistory request (see this).

For each benchmark in the reply message, each of the <lastValue>, <lastDiff>, and <regression> is included only if the value exists (or can be computed in the case of the regression). 
<request type="GetRankedBenchmarks">
    <context metric="..."
        platform="..." host="..."
        gitRepo="..." gitBranch="..."
        timestamp1="..." timestamp2="..."
        diffTolerance="..." stabTolerance="..."
        ranking="..." scope="..."
        maxSize="..."
        testCaseFilter="..."
        testFunctionFilter="..."
        dataTagFilter="..." />
</request>

<reply type="GetRankedBenchmarks">
    <benchmark testCase="..."
        testFunction="..." dataTag="...">
        <lastValue timestamp="..."
            sha1="..." value="..."
            stable="..." />
        <lastDiff timestamp="..."
            sha1="..." value="..."
            stable="..." />
        <regression a="..." b="..." />
    </benchmark>
    ...
</reply>
GetRankedBenchmarks2 Lists the benchmarks with the most important last value differences for two branches. 
<request type="GetRankedBenchmarks2">
    <context metric="..."
        platform="..." host="..."
        gitRepo1="..." gitBranch1="..."
        gitRepo2="..." gitBranch2="..."
        diffTolerance="..." stabTolerance="..."
        ranking="..." scope="..."
        maxSize="..." 
        testCaseFilter="..."
        testFunctionFilter="..."
        dataTagFilter="..." />
</request>

<reply type="GetRankedBenchmarks2">
    <benchmark testCase="..."
        testFunction="..." dataTag="...">
        <lastValue1 timestamp="..."
            sha1="..." value="..."
            stable="..." />
        <lastValue2 timestamp="..."
            sha1="..." value="..."
            stable="..." />
    </benchmark>
    ...
</reply>
GetStats Computes basic statistics for the benchmarks within a certain context. 
<request type="GetStats">
    <context metric="..."
        platform="..." host="..."
        gitRepo="..." gitBranch="..."
        timestamp1="..." timestamp2="..."
        diffTolerance="..." stabTolerance="..." />
</request>

<reply type="GetStats">
    <stats nBenchmarks="..."
        nLastDiffs="..."
        nStableLastDiffs="..."
        minStableLastDiff="..."
        avgStableLastDiff="..."
        maxStableLastDiff="..." />
</reply>
GetContexts Lists available contexts, i.e. combinations of metric, platform, host, and branch. Note: the attribute values in a context element are zero-based indexes of entries in the respective component lists. 
<request type="GetContexts" />

<reply type="GetContexts">
    <metric name="..." />
    <metric name="..." />
    ...
    <platform name="..." />
    <platform name="..." />
    ...
    <host name="..." />
    <host name="..." />
    ...
    <branch gitRepo="..." gitBranch="..." />
    <branch gitRepo="..." gitBranch="..." />
    ...
    <context
        metric="..." platform="..."
        host="..." branch="..." />
    <context
        metric="..." platform="..."
        host="..." branch="..." />
    ...
</reply>