blob: ce5367d4a06b086ac4bbf9659844022dfe2c5d6a (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
|
A benchmark is the combination of a test case, test function, and a data tag.
------------
Definition: BMCONTEXT = <benchmark> <metric> <platform> <host>
Note: In general, comparisons are meaningful only within the same
BMCONTEXT. This assumption will be enforced in the user interface described
below.
------------
The xml file (produced by QTestLib) contains the benchmark (text case, test
function, and data tag), metric, and result (value and iterations).
When used as a standalone term, a branch refers to the combination of a Git
repository and a Git branch.
------------
Definition: SNAPSHOT = -timestamp <secs since 1/1 1970> | -sha1 <sha1>
A -timestamp value t has the following semantics:
t = -1: the latest result ('last' may be used as an alias)
t >= 0: the earliest result that is not earlier than t ('first' may be used as an alias for 0)
Note: The -sha1 option requires exact matching and is thus less powerful than -timestamp.
----------------------------------
Use case 1: Add new results:
./bmc put results <xml file> <platform> <host> <git repo> [<git dir>]
Notes:
- <xml file> must (essentially) be of the same format as the one being dumped
by QTestLib.
- <git dir> must refer to a directory accessible from the local
file system and the git command needs to be installed (i.e. the program
needs to run the git command from <git dir>).
By default, <git dir> is the current directory.
- The new results are assumed to have been generated using a product
built from a clean checkout of the current head commit of the current git branch
in <git dir>.
- The sha1 to be stored in the database is assumed to be the current head commit in
<git dir> (extracted running 'git log -n 1 HEAD --pretty=format:%H').
- The timestamp to be stored in the database is the Unix timestamp of the committer
date (extracted running 'git log -n 1 HEAD --pretty=format:%ct').
- <git repo> must be passed explicitly since it cannot in general
be derived from <git dir> (e.g. in the case of a local, standalone
repository).
Example:
./bmc put results results.xml linux-g++ barbarella qt master $QTDIR
----------------------------------
Use case 2: List available snapshots for a given branch:
./bmc get snapshots BMCONTEXT <branch>
Example:
./bmc list -snapshots tst_QWidget updateOpaque 10x10x1 walltime \
linux-g++ barbarella qt master
----------------------------------
Use case 3: List available benchmarks (subject to optional restrictions):
./bmc get benchmarks [-metric <metric>] [-platform <platform>] [-host <host>] \
[-branch <branch>]
Example 1:
./bmc get benchmarks
Example 2:
./bmc get benchmarks -platform linux-g++
Example 3:
./bmc get benchmarks -platform linux-g++ -host barbarella -metric walltime \
-branch qt master
----------------------------------
Use case 4: List available metrics (subject to optional restrictions):
./bmc get metrics [-benchmark <test case> <test function> <data tag>] \
[-platform <platform>] [-host <host>] [-branch <branch>]
Example 1:
./bmc get metrics
Example 2:
./bmc get metrics -benchmark tst_QWidget updateOpaque 10x10x1 \
-platform linux-g++ -host barbarella -branch qt master
----------------------------------
Use case 5: List available platforms (subject to optional restrictions):
./bmc get platforms [-benchmark <test case> <test function> <data tag>] \
[-metric <metric>] [-host <host>] [-branch <branch>]
----------------------------------
Use case 6: List available hosts (subject to optional restrictions):
./bmc get hosts [-benchmark <test case> <test function> <data tag>] \
[-metric <metric>] [-platform <platform>] [-branch <branch>]
----------------------------------
Use case 7: List available branches (subject to optional restrictions):
./bmc get platforms [-benchmark <test case> <test function> <data tag>] \
[-metric <metric>] [-platform <platform>] [-host <host>]
----------------------------------
Use case 8: List the raw results within a particular historical range.
./bmc get history BMCONTEXT <branch> SNAPSHOT1 SNAPSHOT2
Example 1:
./bmc get history tst_QWidget updateOpaque 10x10x1 walltime \
linux-g++ barbarella qt master -timestamp first -timestamp last
Example 2 (1250146917 corresponds to the date Thu Aug 13 09:01:57 2009 +0200):
./bmc get history tst_QWidget updateOpaque 10x10x1 walltime \
linux-g++ barbarella qt master -timestamp 1250146917 -timestamp last
Example 3:
./bmc get history tst_QWidget updateOpaque 10x10x1 walltime \
linux-g++ barbarella qt master -timestamp first \
-sha1 3bad2f608813968d85c2ea51c1bd75dca7f7e22a
----------------------------------
Use case 9: Compute trend values within a particular historical range
(to detect perfomance regressions etc.).
./bmc get trend BMCONTEXT <branch> SNAPSHOT1 SNAPSHOT2
|
