BLOCKBENCH: A Framework for Analyzing Private

Blockchains

Tien Tuan Anh Dinh‡ Ji Wang‡ Gang Chen§ Rui Liu‡ Beng Chin Ooi‡ Kian-Lee Tan‡
‡ §
National University of Singapore Zhejiang University
‡
{dinhtta, wangji, liur, ooibc, tankl}@comp.nus.edu.sg §
[email protected]
arXiv:1703.04057v1 [cs.DB] 12 Mar 2017

ABSTRACT order of transactions. However, traditional database sys-

Blockchain technologies are taking the world by storm. Pub- tems work in a trusted environment and employ well known
lic blockchains, such as Bitcoin and Ethereum, enable secure concurrency control techniques [36, 48, 8] to order transac-
peer-to-peer applications like crypto-currency or smart con- tions. Blockchain’s key advantage is that it does not assume
tracts. Their security and performance are well studied. nodes trust each other and therefore is designed to achieve
This paper concerns recent private blockchain systems de- Byzantine fault tolerance.
signed with stronger security (trust) assumption and perfor- In the original design, Bitcoin’s blockchain stores coins
mance requirement. These systems target and aim to dis- as the system states shared by all participants. For this
rupt applications which have so far been implemented on top simple application, Bitcoin nodes implement a simple repli-
of database systems, for example banking, finance and trad- cated state machine model which simply moves coins from
ing applications. Multiple platforms for private blockchains one address to another. Since then, blockchain has grown
are being actively developed and fine tuned. However, there rapidly to support user-defined states and Turing complete
is a clear lack of a systematic framework with which different state machine models. Ethereum [2] is a well-known ex-
systems can be analyzed and compared against each other. ample which enables any decentralized, replicated applica-
Such a framework can be used to assess blockchains’ viabil- tions known as smart contracts. More importantly, interest
ity as another distributed data processing platform, while from the industry has started to drive development of new
helping developers to identify bottlenecks and accordingly blockchain platforms that are designed for private settings
improve their platforms. in which participants are authenticated. Blockchain systems
In this paper, we first describe Blockbench, the first in such environments are called private (or permissioned), as
evaluation framework for analyzing private blockchains. It opposed to the early systems operating in public (or permis-
serves as a fair means of comparison for different platforms sionless) environments where anyone can join and leave. Ap-
and enables deeper understanding of different system de- plications for security trading and settlement [44], asset and
sign choices. Any private blockchain can be integrated to finance management [39, 40], banking and insurance [29] are
Blockbench via simple APIs and benchmarked against being built and evaluated. These applications are currently
workloads that are based on real and synthetic smart con- supported by enterprise-grade database systems like Oracle
tracts. Blockbench measures overall and component-wise and MySQL, but blockchain has the potential to disrupt
performance in terms of throughput, latency, scalability and this status quo because it incurs lower infrastructure and
fault-tolerance. Next, we use Blockbench to conduct com- human costs [29]. In particular, blockchain’s immutability
prehensive evaluation of three major private blockchains: and transparency help reduce human errors and the need
Ethereum, Parity and Hyperledger Fabric. The results demon- for manual intervention due to conflicting data. Blockchain
strate that these systems are still far from displacing current can help streamline business processes by removing dupli-
database systems in traditional data processing workloads. cate efforts in data governance. Goldman Sachs estimated 6
Furthermore, there are gaps in performance among the three billion saving in current capital market [29], and J.P. Mor-
systems which are attributed to the design choices at differ- gan forecast that blockchains will start to replace currently
ent layers of the blockchain’s software stack. redundant infrastructure by 2020 [40].
Given this trend in employing blockchain in settings where
database technologies have established dominance, one ques-
1. INTRODUCTION tion to ask is to what extent can blockchain handle data
Blockchain technologies are gaining massive momentum processing workload. Another question is which platform
in the last few years, largely due to the success of Bitcoin to choose from many that are available today, because even
crypto-currency [41]. A blockchain, also called distributed though blockchain is an open protocol, different platforms
ledger, is essentially an append-only data structure main- exist in silo. In this work, we develop a benchmarking
tained by a set of nodes which do not fully trust each other. framework called Blockbench to address both questions.
All nodes in a blockchain network agree on an ordered set Blockbench is the first benchmark for studying and com-
of blocks, each containing multiple transactions, thus the paring performance of permissioned blockchains. Although
blockchain can be viewed as a log of ordered transactions. nodes in a permissioned blockchain still do not trust each
In a database context, blockchain can be viewed as a solu- other, their identities are authenticated, which allows the
tion to the distributed transaction management problems: system to use more efficient protocols for tolerating Byzan-
nodes keep replicas of the data and agree on an execution
tine failure than in public settings. We do not focus on • We present the first benchmark for understanding and
public blockchains because their performance (and trade- comparing the performance of permissioned blockchain
offs against security guarantee) is relatively well studied [27, systems.
38, 15, 9]. Our framework is not only useful for application
developers to assess blockchain’s potentials in meeting the • We conduct a comprehensive evaluation of Ethereum,
application need, but also offers insights for platform devel- Parity and Hyperledger. Our empirical results present
opers: helping them to identify and improve on the perfor- concrete evidence of blockchain’s limitations in han-
mance bottlenecks. dling data processing workloads, and reveal bottle-
We face three challenges in developing Blockbench. First, necks in the three systems. The results serve as a
a blockchain system comprises many parts and we observe baseline for further development of blockchain tech-
that a wide variety of design choices are made among differ- nologies.
ent platforms at almost every single detail. In Blockbench,
we divide the blockchain architecture into three modular lay- In the next section, we discuss blockchain systems in more
ers and focus our study on them: the consensus layer, data detail. Section 3 describes Blockbench design and imple-
model and execution layer. Second, there are many differ- mentation. Section 4 presents our comparative performance
ent choices of platforms, but not all of them have reached studies of three systems. We discuss lessons learned from
a mature design, implementation and an established user the results in Section 5 and related work in Section 6, and
base. For this, we start by designing Blockbench based we conclude in Section 7.
on three most mature platforms within our consideration,
namely Ethereum [2], Parity [22] and Hyperledger [31], and 2. PRIVATE BLOCKCHAINS
then generalize to support future platforms. All three plat-
A typical blockchain system consists of multiple nodes
forms support smart contracts and can be deployed in a
which do not fully trust each other. Some nodes exhibit
private environment. Third, there is lack of a database-
Byzantine behavior, but the majority is honest. Together,
oriented workloads for blockchain. Although the real Eth-
the nodes maintain a set of shared, global states and perform
ereum transactions and contracts can be found on the pub-
transactions modifying the states. Blockchain is a special
lic blockchain, it is unclear if such workload is sufficiently
data structure which maintains the states and the historical
representative to assess blockchain’s general data processing
transactions. All nodes in the system agree on the transac-
capabilities. To address this challenge, we treat blockchain
tions and their order as stored on the blockchain. Because of
as a key-value storage coupled with an engine which can
this, blockchain is often referred to as a distributed ledger.
realize both transactional and analytical functionality via
Blockchain transactions. A transaction in a blockchain
smart contracts. We then design and run both transaction
is the same as in traditional database: a sequence of opera-
and analytics workloads based on real and synthetic data.
tions applied on some states. As such, a blockchain transac-
Blockbench is a flexible and extensible framework that
tion requires the same ACID semantics. The key difference
provides a number of workloads, and comes with Ethereum,
is the failure model under consideration. Current transac-
Parity and Hyperledger as backends. Workloads are transaction-
tional, distributed databases [46, 14] employ classic concur-
oriented currently and designed to macro-benchmark and
rency control techniques such as two-phase commit to ensure
micro-benchmark blockchain for supporting database-like ap-
ACID. They can achieve high performance, because of the
plications. Specifically, the current macro-benchmark in-
simple failure model, i.e. crash failure. In contrast, the
cludes a key-value (YCSB), an OLTP (Smallbank) workload
original blockchain design considers a much hostile environ-
and a number of real Ethereum smart contract workloads.
ment in which nodes are Byzantine and they are free to join
For each of the consensus, data and execution layer, there
and leave. Under this model, the overhead of concurrency
is at least a micro-benchmark workload to measure its per-
control is much higher [11].
formance in isolation. For example, for the execution layer,
Bitcoin. In Bitcoin [41], the states are digital coins (crypto-
Blockbench provides two workloads that stress test the
currency) available in the network. A Bitcoin transaction
smart contract I/O and computation speed. New workloads
moves coins from one set of addresses to another set of ad-
and blockchains can be easily integrated via a simple set of
dresses. Each node broadcasts a set of transactions it wants
APIs. Blockbench quantifies the performance of a back-
to perform. Special nodes called miners collect transactions
end system in several dimensions: throughput, latency, scal-
into blocks, check for their validity, and start a consensus
ability and fault tolerance. It supports security evaluation
protocol to append the blocks onto the blockchain. Figure 1
by simulating network-level attacks. Using Blockbench,
shows the blockchain data structure, in which each block is
we conduct an in-depth comparison of the three blockchain
linked to its predecessor via a cryptographic pointer, all the
systems on two macro benchmark and four micro bench-
way back to the first (genesis) block. Bitcoin uses proof-of-
mark workloads. The results show that blockchain systems’
work (PoW) for consensus: only a miner which has success-
performance is limited, far below what is expected of a state-
fully solved a computationally hard puzzle (finding the right
of-the-art database system (such as H-Store). Hyperledger
nonce for the block header) can append to the blockchain.
consistently outperforms the other two systems across seven
PoW is tolerant of Byzantine failure, but it is probabilistic
benchmarks. But it fails to scale beyond 16 nodes. Our
in nature: it is possible that two blocks are appended at
evaluation shows that the consensus protocols account for
the same time, creating a fork in the blockchain. Bitcoin
the performance gap at the application layer for Ethereum
resolves this by only considering a block as confirmed after
and Hyperledger. We also identify a processing bottleneck
it is followed by a number of blocks (typically six blocks).
in Parity. Finally, our evaluation also reveals bottlenecks in
This probabilistic guarantee causes both security and per-
the execution and data layer of Ethereum and Parity.
formance issues: attacks have been demonstrated by an ad-
In summary, our contributions are:
versary controlling only 25% of the nodes [26], and Bitcoin
 Asset Securities contract Doubler{
Application Crypto-currency Asset Securities ...
Crypto-currency management settlement struct Partitipant {
management settlement
address etherAddress;
Blockchain uint amount;
block t block t+1
}
... Block header Block header ... Partitipant[] public participants;
Transaction Contract Transaction Contract
unit public balance = 0;
roothash roothash roothash roothash ...
function enter(){
...
balance+= msg.value;
Smart contract Smart contract
...
if (balance > 2*participants[payoutIdx].amount){
Code Code
input, output input, output transactionAmount = ...
participants[payoutIdx].
State State
storage storage etherAddress.send(transactionAmount);
...
Hardware
}
}
CPU Network
...
Storage
}

Figure 1: Blockchain software stack on a fully validating Figure 2: An example of smart contract, written in Solidity
node. A non-validating node stores only the block head- language, for a pyramid scheme on Ethereum.
ers. Different blockchain platforms offer different interface
between the blockchain and application layer. YCSB,
Blockchain Contracts Application Smallbank, BLOCKBENCH
layers etc. workloads

transaction throughput remains very low (7 transactions per Compilers, VM, CPU-Heavy
Dockers, etc. Execution Engine
second [15]).
Ethereum. Due to simple transaction semantics, Bitcoin Blocks Analytics,
nodes execute a very simple state machine pre-built into the Transactions, Data Model IO-Heavy
Indexing, etc.
protocol. Ethereum [2] extends Bitcoin to support user-
defined and Turing complete state machines. In particular,
PoW, PoS,
Ethereum blockchain lets the user define any complex com- PBFT, etc.
Consensus Commits

putations in the form of smart contracts. Once deployed,

the smart contract is executed on all Ethereum nodes as a Figure 3: Abstraction layers in blockchain, and the corre-
replicated state machine. Beside the shared states of the sponding workloads in Blockbench.
blockchains (crypto-currency, for example), each smart con-
tract has access to its own states. Figure 1 shows the soft-
ware stack in a typical Ethereum node: a fully validating settings wherein nodes are authenticated. Although PoW
node contains the entire history of the blockchain, whereas is still useful in such permissioned environments, as in the
a non-validating node stores only the block headers. One case of Ethereum, there are more efficient and determinis-
key difference with Bitcoin is that smart contract states are tic solutions where node identities are known. Distributed
maintained as well as normal transactions. In fact, a smart fault-tolerant consensus in such a closed settings is a well
contract is identified by a unique address which has its own studied topic in distributed systems. Zab [33], Raft [42],
money balance (in Ether), and upon retrieving a transaction Paxos [35], PBFT [11] are popular protocols that are in ac-
to its address, it executes the contract’s logics. Ethereum tive use today. Recent permissioned blockchains either use
comes with an execution engine, called Ethereum Virtual existing PBFT, as in Hyperledger [31], or develop their own
Machine (EVM), to execute smart contracts. Figure 2 shows variants, as in Parity [22], Ripple [44] and ErisDB [5]. Most
a snippet of popular contract running on Ethereum, which of these systems support smart contracts, though in different
implements a pyramid scheme: users send money to this languages, with different APIs and execution engines (see a
contract which is used to pay interests to early participants. more comprehensive comparison in the Appendix). As a
This contract has its own states, namely the list of partici- result, permissioned blockchains can execute complex appli-
pants, and exports a function called enter. A user invokes cation more efficiently than PoW-based blockchains, while
this contract by sending his money through a transaction, being Byzantine fault tolerant. These properties and the
which is accessed by the smart contract as msg.sender and commercial interests from major banking and financial insti-
msg.amount. tutions have bestowed on private blockchains the potentials
Private blockchain. Ethereum uses the same consensus to disrupt the current practice in data management.
protocol as Bitcoin does, though with different parameters.
In fact, 90% of public blockchain systems employ variants of
the proof-of-work protocol. PoW is non-deterministic and
3. Blockbench DESIGN
computationally expensive, both rendering it unsuitable for This section discusses blockchain’s common layers of ab-
applications such as banking and finance which must han- stractions and the benchmarking workloads.
dle a lot of transactions in a deterministic manner. Recent
blockchain systems, e.g., Hyperledger, consider restricted
3.1 Blockchain Layers nerable to 25% attacks [26]). However, PBFT assumes that
There are many choices of blockchains: over 200 Bitcoin node identities are known, therefore it can only work in the
variants, Ethereum and other permissioned blockchains. To permissioned settings. Additionally, the protocol is unlikely
meaningfully compare them, we identify four abstraction to be able to scale to the network size of Ethereum, because
layers found in all of these systems (Figure 3) and design of its communication overhead.
our workloads to target these layers. The consensus layer In between, there are various hybrid designs that combine
contains protocols via which a block is considered appended both scalability of PoW and safety property of PBFT [43].
to the blockchain. The data layer contains the structure, For example, Bitcoin-NG [25] decouples consensus from trans-
content and operations on the blockchain data. The execu- action validation by using PoW for leader election who can
tion layer includes details of the runtime environment sup- then append more than one block at a time. Similarly, Byz-
port blockchain operations. Finally, the application layer coin [34] and Elastico [37] leverage PoW to determine ran-
includes classes of blockchain applications. In a related dom, smaller consensus groups which run PBFT. Another
work, Croman et. al. [15] proposed to divide blockchain into example is the Tendermint protocol, adopted by ErisDB [5],
several planes: network, consensus, storage, view and side which combines proof-of-stake (PoS) and PBFT. Unlike PoW,
plane. While similar to our four layers, the plane abstrac- PoS selects a node which can append a block by its invest-
tions were geared towards crypto-currency applications and ment (or stake) in the system, therefore avoid expending
did not take into account the execution of smart contracts. CPU resources. Parity [22] implements a simplified version
Our layers model more accurately the real implementations of PoS called Proof of Authority (or PoA). In this protocol,
of private blockchains. We now discuss these layers in turn. a set of authorities are pre-determined and each authority
is assigned a fixed time slot within which it can generate
3.1.1 Consensus blocks. PoA makes a strong assumption that the author-
ities are trusted, and therefore is only suitable for private
The role of the consensus layer is to get all nodes in the
deployment.
system to agree on the blockchain content. That is, if a node
appends (or commits) a block, the other nodes also append
the same block to their copy of the blockchain. Protocols for 3.1.2 Data model
reaching consensus in the crash-failure model play a key role In Bitcoin, transactions are first class citizens: they are
in distributed databases, wherein nodes agree on a global system states representing digital coins in the network. Pri-
transaction order. Blockchain systems, on the other hand, vate blockchains depart from this model, by focusing on
employ a spectrum of Byzantine fault-tolerant protocols [49]. accounts. One immediate benefit is simplicity, especially
At one extreme, Ethereum, like Bitcoin, uses proof-of- for applications involving crypto-currencies. For instance,
work whose difficulty is agreed upon and adjusted gradually transferring money from one user to another in Bitcoin in-
to achieve a rate of (currently) one block per 14s (Bitcoin’s volves searching for transactions belonging to the sender,
difficulty achieves a rate of one block per 10m). In essence, then marking some of them as spent, whereas it is easily
proof-of-work selects at each round a random node which done in Ethereum by updating two accounts in one trans-
can append a block, where the probability of being selected action. An account in Ethereum has a balance as its state,
is determined by the node’s total computing power. This and is updated upon receiving a transaction. A special type
simple scheme works against Sybil attack [20] - a common of account, called smart contract, contains executable code
attack in open, decentralized environments in which the ad- and private states (Figure 1). When receiving a transaction,
versary can acquire multiple identities. However, it con- in addition to updating its balance, the contract’s code is in-
sumes a lot of energy and computing power, as nodes spend voked with arguments specified in the transaction. The code
their CPU cycles solving puzzles instead of doing otherwise can read the states of other non-contract accounts, and it
useful works. Worse still, it does not guarantee safety: two can send new transactions during execution. Parity adopts
nodes may both be selected to append to the blockchain, and the same data model as in Ethereum. In Hyperledger, there
both blocks can be accepted. This causes fork in the block- is only one type of account called chaincode which is the
chain, and most PoW-based systems add additional rules, same as Ethereum’s contract. Chaincode can only access its
for example, only blocks on the longest chain are considered private storage and they are isolated from each other.
accepted. Ethereum, in particular, adopts a PoW variant A block contains a list of transactions, and a list of smart
called GHOST [45] which accepts blocks in heavy branches. contracts executed as well as their latest states. Each block
In any case, a block can be confirmed as part of the block- is identified by the cryptographic hash of its content, and
chain only with some high probability. linked to the previous block’s identity. In Parity, the entire
At the other extreme, Hyperledger uses the classic PBFT block content is kept in memory. In Ethereum and Hy-
protocol, which is communication bound: O(N 2 ) where N perledger, the content is organized in a two layered data
is the number of nodes. PBFT can tolerate fewer than N3 structure. The states are stored in a disk-based key-value
failures, and works in three phases in which nodes broadcast storage (LevelDB[4] in Ethereum and RocksDB[6] in Hyper-
messages to each other. First, the pre-prepare phase selects ledger), and organized in a hash tree whose root is included
a leader which chooses a value to commit. Next, the prepare in the block header. Ethereum caches the states in memory,
phase broadcasts the value to be validated. Finally, the com- while Hyperledger outsources its data management entirely
mit phase waits for more than two third of the nodes to con- to the storage engine. Only states affected by the block’s
firm before announcing that the value is committed. PBFT transactions are recorded in the root hash. The hash tree
has been shown to achieve liveness and safety properties in for transaction list is a classic Merkle tree, as the list is
a partially asynchronous model [11], thus, unlike PoW, once not large. On the other hand, different Merkle tree vari-
the block is appended it is confirmed immediately. It can ants are used for the state tree. Ethereum and Parity em-
tolerate more failures than PoW (which is shown to be vul- ploy Patricia-Merkle tree that supports efficient update and
search operations. Hyperledger implements Bucket-Merkle WorkloadClient WorkloadClient
tree which uses a hash function to group states into a list of ...
buckets from which a Merkle tree is built.
Block headers and the key-value storage together maintain
all the historical transactions and states of the blockchain.
For validating and executing transactions, a blockchain node
needs only a few recent blocks (or just the latest block for Configuration Driver StatsCollector
PBFT-based systems). However, the node also interacts via
some RPC-like mechanisms with light-weight clients who do
not have the entire blockchain. Such external interfaces en- IBlockchainConnector
able building of third-party applications on top of block-
chain. Current systems support a minimum set of queries Ethereum Parity Hyperledger ErisDB ...
including getting blocks and transactions based on their IDs.
Ethereum and Parity expose a more comprehensive set of
APIs via JSON-RPC, supporting queries of account states Figure 4: Blockbench software stack. New workloads
at specific blocks and of other block statistics. are added by implementing IWorkloadConnector inter-
face. New blockchain backends are added by implement-
3.1.3 Execution layer ing IBlockchainConnector. Current backends include Eth-
ereum, Parity and Hyperledger.
A contract (or chaincode) is executed in a runtime envi-
ronment. One requirement is that the execution must be
fast, because there are multiple contracts and transactions
in one block and they must all be verified by the node. An- zation (DAO) is the most active application in Ethereum,
other is that the execution must be deterministic, ideally creating communities for crowd funding, exchange, invest-
the same at all nodes. Deterministic execution avoid unnec- ment, or any other decentralized activities. A DAO manages
essary inconsistency in transaction input and output which funds contributed by participants and gives its users voting
leads to blocks being aborted. In both PoW and PBFT, power proportional to their contributions. Parity’s main ap-
aborting transactions wastes computing resources. plication is the wallet application that manages Ether. As
Ethereum develops its own machine language (bytecode) major banks are now considering adopting crypto-currency,
and a virtual machine (called EVM) for executing the code, some fintech companies are building applications that take
which is also adopted by Parity. EVM is optimized for crypto-currency to mediate financial transactions, for ex-
Ethereum-specific operations. For example, every code in- ample, in currency exchange market [44]. Other examples
struction executed in Ethereum costs a certain amount of include applying the currency and smart contracts for more
gas, and the total cost must be properly tracked and charged transparent and cost-effective asset management [39, 40].
to the transaction’s sender. Furthermore, the code must Some applications propose to build on blockchain’s im-
keep track of intermediate states and reverse them if the mutability and transparency for better application work-
execution runs out of gas. Hyperledger, in contrast, does flows in which humans are the bottlenecks. For example,
not consider these semantics in its design, so it simply sup- security settlements and insurance processes can be sped up
ports running of compiled machine codes inside Docker im- by storing data on the blockchain [29]. Another example is
ages. Specifically, chaincodes are deployed as Docker im- sharing economy applications, such as AirBnB, which can
ages interacting with Hyperledger’s backend via pre-defined use blockchain to evaluate reputation and trust in a decen-
interfaces. One advantage of Hyperledger’s environment tralized settings, because historical activities of any users
is that it supports multiple high-level programming lan- are available and immutable. This also extends to Inter-
guages such as Go and Java, as opposed to Ethereum’s net of Things settings, where devices need to establish trust
own language. In terms of development environment, Hy- among each other [3].
perledger exposes only simple key-value operations, namely
putState and getState. This is restricted, because any 3.2 Blockbench Implementation
contract states must be mapped into key-value tuples. In Figure 4 illustrates the current Blockbench’s implemen-
contrast, Ethereum and Parity support a richer set of data tation. To evaluate a blockchain system, the first step is
types such as map, array and composite structures. These to integrate the blockchain into the framework’s backend by
high-level data types in Ethereum and Parity make it easier implementing IBlockchainConnector interface. The inter-
and faster to develop new contracts. face contains operations for deploying application, invoking
it by sending a transaction, and for querying the blockchain’s
3.1.4 Application layer states. Ethereum, Parity and Hyperledger are current back-
Many applications are being proposed for blockchain, lever- ends supported by Blockbench, while ErisDB integration
aging the latter’s two key properties. First, data in the is under development. A user can use one of the existing
blockchain is immutable and transparent to the participants, workloads (discussed next) to evaluate the blockchain, or
meaning that once a record is appended, it can never be implement a new workload using the IWorkloadConnector
changed. Second, it is resilient to dishonest and malicious interface (we assume that the smart contract handling the
participants. Even in permissioned settings, participants workload’s logic is already implemented and deployed on the
can be mutually distrustful. The most popular application, blockchain). This interface essentially wraps the workload’s
however, is still crypto-currency. Ethereum has its own cur- operations into transactions to be sent to the blockchain.
rency (Ether) and a majority of smart contracts running on Specifically, it has a getNextTransaction method which re-
it are currency related. Decentralized Autonomous Organi- turns a new blockchain transaction. Blockbench’s core
component is the Driver which takes as input a workload, Smart contracts Description
user-defined configuration (number of operations, number YCSB Key-value store
of clients, threads, etc.), executes it on the blockchain and Smallbank OLTP workload
EtherId Name registrar contract
outputs running statistics.
Doubler Ponzi scheme
Asynchronous Driver. One challenge in implement-
WavesPresale Crowd sale
ing the Driver is that current blockchain systems are asyn- VersionKVStore Keep state’s versions (Hyperledger only)
chronous services, meaning that transactions submitted to IOHeavy Read and write a lot of data
the systems are processed at a later time. This is in con- CPUHeavy Sort a large array
trast to databases, especially transactional databases, in DoNothing Simple contract, do nothing
which operations are synchronous, i.e. they block until the
systems finish processing. When a transaction is submit- Table 1: Summary of smart contracts implemented in
ted, Ethereum, Parity and Hyperledger return a transac- Blockbench. Each contract has one Solidity version for
tion ID which can be used for checking the transaction sta- Parity and Ethereum, and one Golang version for Hyper-
tus at a later time. Such asynchronous semantics could ledger.
result in better performance, but it forces the Driver to
periodically poll for status of the submitted requests. In
particular, Driver maintains a queue of outstanding trans- ior caused by an attacker. The attacker can be a compro-
actions that have not been confirmed. New transaction mised node or rouge participant within the system. Under
IDs are added to the queue by worker threads. A polling this threat model, security of a blockchain is defined as the
thread periodically invokes getLatestBlock(h) method in safety property of the underlying consensus protocol. In par-
the IBlockchainConnector interface, which returns a list of ticular, security means that the non-Byzantine nodes have
new confirmed blocks on the blockchain from a given height the same blockchain data. Violation of the safety property
h. Ethereum and Parity consider a block as confirmed if it is leads to forks in the blockchain. Classic Byzantine tolerant
at least confirmationLength blocks from the current block- protocols such as PBFT are proven to ensure safety for a
chain’s tip, whereas Hyperledger confirms a block as soon certain number of failures, thus security is guaranteed. On
as it appears on the blockchain. The Driver then extracts the other hand, in PoW systems like Bitcoin or Ethereum,
transaction lists from the confirmed blocks’ content and re- forks can occur due to network delays causing two nodes
moves matching ones in the local queue. getLatestBlock(h) to mine the same blocks. While such accidental forks can
can be implemented in all three systems by first requesting be quickly resolved, forks engineered by the attackers can
for the blockchain’s current tip t, then requesting the con- be used for double spending and selfish mining. In the for-
tent of all blocks in the range (h, t]. ErisDB provides a mer, the attacker sends a transaction to a block in the fork,
publish/subscribe interface that could simplify the imple- waits for it to be accepted by the users, then sends a con-
mentation of this function. flicting transaction to another block in the main branch. In
3.3 Evaluation Metrics the latter, by withholding blocks and maintaining a private,
long fork, the attacker disrupts the incentives for mining and
The output statistics of running a workload with different forces other participants to join the attacker’s coalition. By
configurations can be used to evaluate the blockchain against compromising 25% of the nodes, the attacker can control the
three performance metrics. entire network’s block generation [26].
• Throughput: measured as the number of successful In this work we quantify security as the number of blocks
transactions per second. A workload can be configured in the forks. Such blocks, called orphan or stale blocks, rep-
with multiple clients and threads per clients to saturate resent the window of vulnerability in which the attacker can
the blockchain throughput. perform double spending or selfish mining. To manipulate
forks, the key strategy is to isolate a group of nodes, i.e.
• Latency: measured as the response time per transac- to partition the network. For example, eclipse attack [30]
tion. Driver implements blocking transaction, i.e. it exploits the application-level protocol to surround the tar-
waits for one transaction to finish before starting an- geted nodes with ones under the attacker’s control. At the
other. network level, BGP hijacking [7] requires controlling as few
as 900 prefixes to isolate 50% of the Bitcoin’s total min-
• Scalability: measured as the changes in throughput ing power. Blockbench implements a simulation of these
and latency when increasing number of nodes and num- attacks by partitioning the network for a given duration.
ber of concurrent workloads. In particular, during partition Blockbench runtime drops
• Fault tolerance: measured as how the throughput and network traffic between any two nodes in the two partitions.
latency change during node failure. Although block- Security is then measured by the ratio between the total
chain systems are tolerant against Byzantine failure, it number of blocks included in the main branch and the total
is not possible to simulate all Byzantine behaviors. In number of blocks confirmed by the users. The lower the ra-
Blockbench we simulate three failure modes: crash tio, the less vulnerable the system is from double spending
failure in which a node simply stops, network delay for selfish mining.
in which we inject arbitrary delays into messages, and
random response in which we corrupt the messages ex-
3.4 Workloads
changed among the nodes. We divide the workloads into two major categories: macro
benchmark for evaluating performance of the application
Security metrics. A special case of Byzantine failures layer, and micro benchmark for testing the lower layers. We
that is important to blockchain systems is malicious behav- have implemented the smart contracts for all workloads for
Ethereum, Parity and Hyperledger, whose details are sum- chain as a whole. As discussed early in this section, a block-
marized in Table 1. Ethereum and Parity use the same exe- chain system comprises multiple layers, and each layer may
cution model, therefore they share the same smart contract have different impact on the overall performance. We design
implementations. several workloads to stress the layers in order to understand
their individual performance.
DoNothing. This contract accepts transaction as input
3.4.1 Macro benchmark workloads and simply returns. In other words, it involves minimal
We port two popular database benchmark workloads into number of operations at the execution layer and data model
Blockbench, and three other real workloads found in the layer, thus the overall performance will be mainly deter-
Ethereum blockchain. mined by the consensus layer. Previous works on perfor-
Key-value storage. We implement a simple smart con- mance of blockchain consensus protocol [34, 43] use time
tract which functions as a key-value storage. The WorkloadClient to consensus to measure its performance. In Blockbench,
is based on the YCSB driver [13]. It preloads each store with this metric is directly reflected in the transaction latency.
a number of records, and supports requests with different ra- Analytics. This workload considers the performance of
tios of read and write operations. YCSB is widely used for blockchain system in answering analytical queries about the
evaluating NoSQL databases. historical data. Similar to an OLAP benchmark, this work-
OLTP (Smallbank). Unlike YCSB which does not con- load evaluates how the system implements scan-like and ag-
sider transactions, Smallbank [10] is a popular benchmark gregate queries, which are determined by its data model.
for OLTP workload. It consists of three tables and four ba- Specifically, we implement two queries for extracting statis-
sic procedures simulating basic operations on bank accounts. tics from the blockchain data:
We implement it as a smart contract which simply transfers Q1: Compute the total transaction values committed be-
money from one account to another. tween block i and block j.
EtherId. This is a popular contract that implements a
domain name registrar. It supports creation, modification Q2: Compute the largest transaction value involving a given
and ownership transfer of domain names. A user can re- state (account) between block i and block j.
quest an existing domain by paying a certain amount to the In ClientWorkload, we pre-load the blockhain with trans-
current domain’s owner. This contract has been written for actions carrying integer values (representing money trans-
Ethereum blockchain, and can be ported to Parity without ferring) and the states with integer values. For Ethereum,
change. In Hyperledger, we create two different key-value both queries can be implemented via JSON-RPC APIs that
namespaces in the contract: one for storing the domain name return transaction details and account balances at a specific
data structures, and another for users’ account balances. In block. For Hyperledger, however, the second query must
domain creation, the contract simply inserts domain value be implemented via a chaincode (VersionKVStore), because
into the first name space, using the domain name as the the system does not have APIs to query historical states.
key. For ownership transfer, it checks the second namespace IOHeavy. Current blockchain systems rely on key-value
if the requester has sufficient fund before updating the first storage to persist blockchain transactions and states. Each
namespace. To simulate real workloads, the contract con- storage system may perform differently under different work-
tains a function to pre-allocate user accounts with certain loads [50]. This workload is designed to evaluate the IO per-
balances. formance by invoking a contract that performs a large num-
Doubler. This is a contract that implements a pyramid ber of random writes and random reads to the contract’s
scheme. As shown in Figure 2, participants send money states. The I/O bandwidth can be estimated via the ob-
to this contract, and get rewards as more people join the served transaction latency.
scheme. In addition to the list of participants and their con- CPUHeavy. This workload measures the efficiency of
tributions, the contract needs to keep the index of the next the execution layer for computationally heavy tasks. EVM
payout and updates the balance accordingly after paying may be fast at executing Ethereum specific operations, but
early participants. Similar to EthereId, this contract has it is unclear how it performs on general tasks for which ma-
already been written for Ethereum, and can be ported to chine native codes may be more efficient. We deploy a smart
Parity directly. To implement it in Hyperledger, we need to contract which initializes a large array, and runs the quick
translate the list operations into key-value semantics, mak- sort algorithm over it. The execution layer performance can
ing the chaincode more bulky than the Ethereum counter- then be measured by the observed transaction latency.
part.
WavesPresale. This contract supports digital token sales.
It maintains two states: the total number of tokens sold so 4. PERFORMANCE BENCHMARK
far, and the list of previous sale transactions. It supports We selected Ethereum, Parity and Hyperledger for our
operations to add a new sale, to transfer ownership of a pre- study, as they occupy different positions in the blockchain
vious sale, and to query a specific sale records. Ethereum design space, and also for their codebase maturity. We eval-
and Parity support composite structure data types, making uate the three systems using both macro and micro bench-
it straightforward to implement the application logic. In mark workloads described in the previous section1 . Our
contrast, in Hyperledger, we have to translate this structure main findings are:
into key-value semantics by using separate key-value names-
paces. • Hyperledger performs consistently better than Eth-
ereum and Parity across the benchmarks. But it fails
3.4.2 Micro benchmark workloads to scale up to more than 16 nodes.
1
The previous workloads test the performance of block- We have released Blockbench for public use [1].
 • Ethereum and Parity are more resilient to node fail- PoW and PBFT (indeed, we observe that Parity has the
ures, but they are vulnerable to security attacks that same CPU utilization and lower network utilization than
forks the blockchain. Hyperledger). Figure 5[b,c] shows that Parity’s throughput
and latency remains constant with increasing transaction
• The main bottlenecks in Hyperledger and Ethereum rates (beyond 40 tx/s). To understand its performance fur-
are the consensus protocols, but for Parity the bottle- ther, we measure the queue of pending transactions at the
neck is caused by transaction signing. client. Figure 6 compares the queue sizes before and af-
ter the systems reach their peak throughput. With only 8
• Ethereum and Parity incur large overhead in terms of
tx/s, the queues for Ethereum and Hyperledger remain at
memory and disk usage. Their execution engine is also
roughly constant sizes, but Parity’s queue size increases as
less efficient than that of Hyperledger.
time passes. More interestingly, under high loads (512 tx/s
• Hyperledger’s data model is low level, but its flexibility per client), Parity’s queue is always smaller than Ethereum’s
enables customized optimization for analytical queries and Hyperledger’s. This behavior indicates that Parity pro-
of the blockchain data. cesses transactions at a constant rate, and that it enforces a
maximum client request rate at around 80 tx/s. As a conse-
We used the popular Go implementation of Ethereum, quence, Parity achieves both lower throughput and latency
geth v1.4.18, the Parity release v1.6.0 and the Hyperledger than other systems.
Fabric release v0.6.0-preview. We set up a private testnet Another observation is that there are differences between
for Ethereum and Parity by defining a genesis block and di- YCSB and Smallbank workloads in Hyperledger and Eth-
rectly adding peers to the miner network. For Ethereum, we ereum. There is a drop of 10% in throughput and 20%
manually tuned the difficulty variable in the genesis block increase in latency. Since executing a Smallbank smart con-
to ensure that miners do not diverge in large networks. For tract is more expensive than executing a YCSB contract
Parity, we set the stepDuration variable to 1. In both Eth- (there are more reading and writing to the blockchain’s states),
ereum and Parity, confirmationLength is set to 5 seconds. the results suggest that there are non-negligible costs in the
The default batch size in Hyperledger is 500. execution layer of blockchains.
The experiments were run on a 48-node commodity clus- At its peak throughput, Hyperledger generates 3.1 blocks
ter. Each node has an E5-1650 3.5GHz CPU, 32GB RAM, per second and achieves the overall throughput of 1273 tx/s.
2TB hard drive, running Ubuntu 14.04 Trusty, and con- We remark that this throughput is far lower than what an in-
nected to the other nodes via 1GB switch. The results below memory database system can deliver (see Appendix B). As
are averaged over 5 independent runs. For Ethereum, we re- the throughput is a function of the block sizes and block gen-
served 8 cores out of the available 12 cores per machine, so eration rate, we measured the effect of increasing the block
that the periodical polls from the client’s driver process do sizes in the three systems. The results (see Appendix B)
not interfere with the mining process (which is CPU inten- demonstrate that with bigger block sizes, the block genera-
sive). tion rate decreases proportionally, thus the overall through-
put does not improve.
4.1 Macro benchmarks
This section discusses the performance of the blockchain 4.1.2 Scalability
systems at the application layer, by running them with the We fixed the client request rate and increased both the
YCSB and Smallbank benchmarks over multiple nodes. number of clients and the number of servers. Figure 7 il-
lustrates how well the three systems scale to handle larger
4.1.1 Throughput and latency YCSB workloads (the results for Smallbank are similar and
We measured peak performance of the three systems with included in Appendix B). Parity’s performance remains con-
8 servers and 8 concurrent clients over the period of 5 min- stant as the network size and offered load increase, due to
utes. Each client sends transactions to a server with a re- the constant transaction processing rate at the servers. In-
quest rate varying from 8 tx/s to 1024 tx/s. Figure 5 shows terestingly, while Ethereum’s throughput and latency de-
the throughput and latency at peak, and how these metrics grade almost linearly beyond 8 servers, Hyperledger stops
change with varying transaction rates. working beyond 16 servers.
We observe that in terms of throughput, Hyperledger out- To understand why Hyperledger failed to scale beyond
performs other systems in both benchmarks. Specifically, it 16 servers and 16 clients, we examined the system’s logs
has up to 5.5x and 28x higher throughput than Ethereum and found that the nodes were repeatedly trying and failing
and Parity respectively. Parity has the lowest latency and to reach consensus on new views which contain batches of
Ethereum has the highest. The gap between Hyperledger transactions. In fact, the servers were in different views and
and Ethereum is because of the difference in consensus pro- consequently were receiving conflicting view change mes-
tocol: one is based on PBFT while the other is based on sages from the rest of the network. Further investigation
PoW. We measured CPU and network utilization during the reveals that conflicting views occurred because the consen-
experiments, and observe that Hyperledger is communica- sus messages are rejected by other peers on account of the
tion bound whereas Ethereum is CPU bound (see Appendix message channel being full. As messages are dropped, the
B). At 8 servers, communication cost in broadcasting mes- views start to diverge and lead to unreachable consensus.
sages is much cheaper than block mining whose difficulty is In fact, we also observe that as time passes, client requests
set at roughly 2.5s per block. took longer to return (see Appendix B), suggesting that the
The performance gap between Parity and Hyperledger is servers were over saturated in processing network messages.
not because of the consensus protocol, as we expect Parity’s We note, however, that the original PBFT protocol guaran-
PoA protocol to be simpler and more efficient than both tees both liveness and safety, thus Hyperledger’s failure to
 Throughput Latency Throughput Latency
Ethereum Hyperledger Ethereum Hyperledger
Ethereum Ethereum Parity Parity
3
104 Parity 10 Parity
Hyperledger Hyperledger 103 102
92 114
1273 1122 102 51

second
38

#tx/s
3

second
10
#tx/s

284 101 102

255
4
101
3
102
45 46 100
101
0 200 400 600 8001000 1000 200 400 600 8001000
101 YCSB Smallbank 10-1 YCSB Smallbank #request/s #request/s
(a) Peak performance (b) Performance with varying request rates

Figure 5: Blockchain performance with 8 clients and 8 servers.

1400 Throughput 180 Latency

Ethereum
1000 Queue length, 8 tx/s 60000 Queue length, 512 tx/s 1200 160 Parity
Ethereum Ethereum 140 Hyperledger
Parity Parity 1000
800 50000 120
Hyperledger Hyperledger
800 Ethereum 100

second
#tx/s
40000 Parity
600 Hyperledger 80
600
#requests

#requests

60
30000 400
40
400
20000 200 20
08 12 16 20 24 28 32 0 8 12 16 20 24 28 32
200 10000 #nodes #nodes

00 50 100 150 200 250 300 00 50 100 150 200 250 300 Figure 8: Performance scalability (with 8 clients).
time (second) time (second)

Figure 6: Client’s request queue, for request rates of 8 tx/s scale beyond 16 servers is due to the implementation of the
and 512 tx/s. protocol. In fact, in the latest codebase (which was updated
after we have finished our benchmark), the PBFT compo-
nent was replaced by another implementation. We plan to
evaluate this new version in the future work.
The results so far indicate that scaling both the number of
104 Throughput Latency clients and number of servers degrades the performance and
Ethereum Ethereum even causes Hyperledger to fail. We next examined the costs
Parity 103 Parity of increasing the number of servers alone while fixing the
Hyperledger Hyperledger number of clients. Figure 8 shows that the performance be-
103 102
comes worse as there are more servers, meaning that the sys-
tems incur some network overheads. Because Hyperledger
second
#tx/s

is communication bound, having more servers means more

101 messages being exchanged and higher overheads. For Eth-
2
10 ereum, even though it is computation bound, it still con-
sumes a modest amount of network resources for propagat-
100 ing transactions and blocks to other nodes. Furthermore,
with larger network, the difficulty is increased to account for
101 124 8 12162024283210-1 124 8 121620242832 the longer propagation delays. We observe that to prevent
the network from diverging, the difficulty level increases at
#nodes #nodes higher rate than the number of nodes. Thus, one reason
Figure 7: Performance scalability (with the same number of for Ethereum’s throughput degradation is due to network
clients and servers). sizes. Another reason is that in our settings, 8 clients send
requests to only 8 servers, but these servers do not always
broadcast transactions to each other (they keep mining on
 1800 #Transactions commited 104 Execution time Peak memory usage
1600 Ethereum-12 Parity-12 Hyperledger-12 Ethereum Ethereum
Ethereum-16 Parity-16 Hyperledger-16 Parity Parity
1400 103
Hyperledger 232.78 108 Hyperledger
1200 2
1000 10 79.61
22,819
#txs

24.04 13,090

second
800 101 10.52 107

MB
600 3.01 1.94 4,150
400 2,078
100 1,353
0.33 106
200 0.19 718
473
-1
00 10 376
50 100 150 200 250 300 350 400
time (second) x x
10-2 1M 10M 100M 105 1M 10M 100M
input size input size
Figure 9: Failing 4 nodes at 250th second (fixed 8 clients)
for 12 and 16 servers. X-12 and X-16 mean running 12 and
Figure 11: CPUHeavy workload, ‘X’ indicates Out-of-
16 servers using blockchain X respectively.
Memory error.

500 #Blocks generated of the original2 . Figure 10 compares the vulnerability of

Ethereum-bc Parity-bc Hyperledger-bc
Ethereum-total Parity-total Hyperledger-total the three systems running with 8 clients and 8 servers. Re-
400 call that vulnerability is measured as the differences in the
300 number of total blocks and the number of blocks on the
#blocks

main branch (Section 3.3), we refer to this as ∆. Both Eth-

200 ereum and Parity blockchains fork at 100th seconds, and ∆
increases as time passes. For the attack duration, upto 30%
100 of the blocks are generated in the forked branch, meaning
that the systems are highly exposed to double spending or
00 50 100 150 200 250 300 350 400 selfish mining attacks. When the partition heals, the nodes
time (second) come to consensus on the main branch and discard the forked
blocks. As a consequence, ∆ stops increasing shortly after
Figure 10: Blockchain forks caused by attacks that parti- 250th second. Hyperledger, in stark contrast, has no fork
tions the network in half at 100th second and lasts for 150 which is as expected because its consensus protocol is proven
seconds. X-total means the total number of blocks generated to guaranteed safety. We note, however, that Hyperledger
in blockchain X, X-bc means the total number of blocks that takes longer than the other two systems to recover from the
reach consensus in blockchain X. attacks (about 50 seconds more). This is because of the syn-
chronization protocol executed after the partitioned nodes
reconnect.
their own transaction pool). As a result, the network mining 4.2 Micro benchmarks
capability is not fully utilized. This section discusses the performance of the blockchain
system at execution, data and consensus layers by evaluat-
4.1.3 Fault tolerance and security ing them with micro benchmark workloads. For the first
To evaluate how resilient the systems are to failures by two layers, the workloads were run using one client and one
crashing, we ran the systems with 8 clients for over 5 min- server. For the consensus layer, we used 8 clients and 8
utes, during which we killed off 4 servers at 250th second. servers.
Figure 9 shows that Ethereum is nearly unaffected by the
change, suggesting that the failed servers do not contributing 4.2.1 Execution layer
significantly to the mining process. In Parity, each node gen- We deployed the CPUHeavy smart contract that is ini-
erates blocks at a constant rate, thus failing 4 nodes means tialized with an integer array of a given size. The array is
the remaining nodes are given more time to generate more initialized in descending order. We invoked the contract to
blocks, therefore the overall throughput is unaffected. In sort the array using quicksort algorithm, and measured the
contrast, the throughput drops considerably in Hyperledger. execution time and server’s peak memory usage. The results
For 12 servers, Hyperledger stops generating blocks after the for varying input sizes are shown in Figure 11. Although
failure, which is as expected because the PBFT can only tol- Ethereum and Parity use the same execution engine, i.e.
erate fewer than 4 failures in a 12-server network. With 16 EVM, Parity’s implementation is more optimized, therefore
servers, the system still generated blocks but at a lower rate, it is more computation and memory efficient. An interest-
which were caused by the remaining servers having to sta- ing finding is that Ethereum incurs large memory overhead.
bilize the network after the failures by synchronizing their In sorting 10M elements, it uses 22GB of memory, as com-
views. pared to 473MB used by Hyperledger. Ethereum runs out of
We next simulated the attack that renders the blockchain 2
We note that partitioning a N -node network in half does
vulnerable to double spending. The attack, described in not mean there are N/2 Byzantine nodes. In fact, Byzan-
Section 3.3, partitioned the network at 100th second and tine tolerance protocols do not count network adversary as
lasted for 150 seconds. We set the partition size to be half Byzantine failure
memory when sorting more than 10M elements. In Hyper- 5. DISCUSSION
ledger, the smart contract is compiled and runs directly on Understanding blockchain systems. Our framework is
the native machine within Docker environment, thus it does designed to provide better understanding of the performance
not have the overheads associated with executing high-level and design of different private blockchain systems. As more
EVM byte code. As the result, Hyperledger is much more and more blockchain systems are being proposed, each of-
efficient in term of speed and memory usage. Finally, we fering different sets of feature, Blockbench’s main value
note that all three systems fail to make use of the multi-core is that it narrows down the design space into four distinct
architecture, i.e. they execute the contracts using only one abstraction layers. Our survey of current blockchain sys-
core. tems (see Appendix A) show that the four layers are suffi-
cient to capture the key characteristics of these systems. By
4.2.2 Data model benchmarking these layers, one can gain insights into the de-
IO Heavy. We deployed the IOHeavy smart contract sign trade-offs and performance bottlenecks. In this paper,
that performs a number of read and write operations of for example, by running the IOHeavy workload we identify
key-value tuples. We used 20-byte keys and 100-byte val- that Parity trades performance for scalability by keeping
ues. Figure 12 reports the throughput and disk usage for states in memory. Another example is the trade-off in data
these operations. Ethereum and Parity use the same data model made by Hyperledger. On the one hand, the sim-
model and internal index structure, therefore they incur sim- ple key-value model means some analytical queries cannot
ilar space overheads. Both use an order of magnitude more be directly supported. On the other hand, it enables opti-
storage space than Hyperledger which employs a simple key- mization that helps answering the queries more efficiently.
value data model. Parity holds all the state information in Finally, we identify that the bottleneck in Parity is not due
memory, so it has better I/O performance but fails to han- to the consensus protocol, but due to the server’s transac-
dle large data (capped by over 3M states under our hard- tion signing. We argue that such insights are not easy to
ware settings). On the contrary, Ethereum only caches only extract without a systematic analysis framework.
parts of the state in memory (using LRU for eviction policy), Usability of blockchain. Our experience in working with
therefore it can handle more data than Parity at the cost of the three blockchain systems confirms the belief that in its
throughput. Hyperledger leverages RocksDB to manage its current state blockchain are not yet ready for mass usage.
states, which makes it more efficient at scale. Both their designs and codebases are still being refined con-
Analytic Queries. We implemented the analytics work- stantly, and there are no other established applications be-
load by initializing the three systems with over 120, 000 ac- side crypto-currency. Of the three systems, Ethereum is
counts with a fixed balance. We then pre-loaded them with more mature both in terms of its codebase, user base and de-
100, 000 blocks, each contains 3 transactions on average. veloper community. Another usability issue we encountered
The transaction transfers a value from one random account is in porting smart contracts from one system to another, be-
to another random account. Due to Parity’s overheads in cause of their distinct programming models (see Section 3).
signing transactions when there are many accounts, we con- This is likely to be exacerbated as more blockchain platforms
sidered transactions using only 1024 accounts. We then exe- are being proposed [44, 16].
cuted the two queries described in Section 3.4 and measured Bringing database designs into blockchain. The chal-
their latencies. Figure 13 shows that the performance for Q1 lenge in scaling blockchain by improving its consensus proto-
is similar, whereas Q2 sees a significant gap between Hyper- cols is being addressed in many recent works [34, 37]. How-
ledger and the rest. We note that the main bottleneck for ever, as we demonstrated in the previous section, there are
both Q1 and Q2 is the number of network (RPC) requests other performance bottlenecks. We propose four approaches
sent by the client. For Q1, the client sends the same num- in applying design principles from database systems to im-
ber of requests to all systems, therefore their performance prove blockchain.
are similar. On the other hand, for Q2 the client sends one Decouple storage, execution engine and consensus layer
RPC per block to Ethereum and Parity, but only one RPC from each other, then optimize and scale them independently.
to Hyperledger because of our customized smart contract For instance, current systems employ generic key-value stor-
implementation (see Appendix C). This saving in network age, which may not be best suited to the unique data struc-
roundtrip time translates to over 10x improvement in Q2 ture and operations in blockchain. UStore [19] demonstrates
latency. that a storage designed around the blockchain data struc-
ture is able to achieve better performance than existing im-
4.2.3 Consensus plementations.
We deployed the DoNothing smart contract that accepts Embrace new hardware primitives. Many data processing
a transaction and returns immediately. We measured the systems are taking advantage of new hardware to boost their
throughput of this workload and compare against that of performance [47, 50, 21]. For blockchain, using trusted hard-
YCSB and Smallbank. The differences compared to other ware, the underlying Byzantine fault tolerance protocols can
workloads, shown in Figure 13[c] is indicative of the cost be modified to incur fewer network messages [12]. Systems
of consensus protocol versus the rest of the software stack. like Parity and Ethereum can take advantage of multi-core
In particular, for Ethereum we observe 10% increases in CPUs and large memory to improve contract execution and
throughput as compared to YCSB, which means that execu- I/O performance.
tion of the YCSB transaction accounts for the 10% overhead. Sharding. Blockchain is essentially a replicated state ma-
We observe no differences among these workloads in Parity, chine system, in which each node maintains the same data.
because the bottleneck in Parity is due to transaction sign- As such, blockchains are fundamentally different to database
ing (even empty transactions still need to be signed), not systems such as H-Store in which the data is partitioned (or
due to consensus or transaction execution. sharded) across the nodes. Sharding helps reduce the com-
 Average throughput (write) Average throughput (read) Disk usage
104 Ethereum Parity Hyperledger Ethereum Parity Hyperledger 105 Ethereum Parity Hyperledger
5618 5411 5123 4852 4527
10 4 9329 8785 9234 8758 8974 8700 8670 8605 30,221
12,804 12,104
1329 1181 1087 104
second

second
103 5,459 5,045 4,865

MB
377 359 337 2019 2,337 2,086 2,477
1631
1282 1,283
103 103 675
102 69 360
512
0.8M 1.6M 3.2M
x
6.4M
x
12.8M 0.8M 1.6M 3.2M
x
6.4M 12.8M
x 0.8M 1.6M 3.2M 6.4M
x x
12.8M
# tuples # tuples # tuples

(a) Write (b) Read (c) Disk usage

Figure 12: IOHeavy workload, ‘X’ indicates Out-of-Memory error.

Latency 13.314 101 Latency 104 Transaction througput

101 Ethreum 8.9018.465
Ethereum 4.907 SmallBank
3.472
Parity Parity YCSB
Hyperledger Hyperledger DoNothing 112212731285
1.374
100 103
100 0.9150.984 0.595
0.427
0.533
second

second

256 284 328

#tx/s
0.168
0.1290.135 -1 0.107 0.091
10 -1 10 0.076 102
0.0320.034
0.038 0.033 0.031 45 45 46
0.025 0.024 0.023
0.0200.0200.019 0.019 0.020
-2 -2
10 1 10 100 1,000 10,000 10 1 10 100 1,000 10,000
# blocks scanned # blocks scanned 101 Ethereum Parity Hyperledger

(a) Analytics workload (Q1) (b) Analytics workload (Q2) (c) DoNothing workload

Figure 13: Analytics and DoNothing workloads.

putation cost and can make transaction processing faster. There are many standard frameworks for benchmarking
The main challenge with sharding is to ensure consistency database systems. OLTP-Bench [18] contains standard work-
among multiple shards. However, existing consistency pro- loads such as TPC-C for transactional systems. YCSB [13]
tocols used in database systems do not work under Byzan- contains key-value workloads. HiBench [32] and BigBench [28]
tine failure. Nevertheless, their designs can offer insights feature big-data analytics workloads for MapReduce-like sys-
into realizing a more scalable sharding protocol for block- tems. Blockbench shares the same high-level design as
chain. Recent work [37] has demonstrated the feasibility these frameworks, but its workloads and main driver are
of sharding the consensus protocol, making important steps designed specifically for blockchain systems.
towards partitioning the entire blockchain.
Support declarative language. Having a set of high-level 7. CONCLUSION
operations that can be composed in a declarative manner
In this paper we proposed the first benchmarking frame-
makes it easy to define complex smart contracts. It also
work, called Blockbench, for evaluating private blockchain
opens up opportunities for low-level optimizations that speed
systems. Blockbench contains workloads for measuring
up contract execution.
the data processing performance, and workloads for under-
standing the performance at different layers of the block-
6. RELATED WORK chain. Using Blockbench, we conducted comprehensive
Performance studies of blockchain systems have so far analysis of three major blockchain systems, namely Eth-
been restricted to public blockchains. For example, [17, 15] ereum, Parity and Hyperledger with two macro benchmarks
analyze the effect of block sizes and network propagation and four micro benchmarks. The results showed that current
time on the overall throughputs. Recent proposals for im- blockchains are not well suited for large scale data process-
proving Bitcoin performance [27, 34, 37, 25, 43] have mainly ing workloads. We demonstrated several bottlenecks and
focused on the consensus layer, in which analytical models design trade-offs at different layers of the software stack.
or network simulations are used to validate the new designs.
Various aspects of Ethereum, such as their block process-
ing time (for syncing with other nodes) and transactions
processing time, have also been benchmarked [24, 23]. Our
analysis using Blockbench differs from these works in that
it is the first to evaluate private blockchains systems at scale
against database workloads. Furthermore, it compares two
different systems and analyzes how their designs affect the
overall performances. Future extensions of Blockbench
would enable more comparative evaluations of the key com-
ponents in blockchain.
Acknowledgment M. Zhang. UStore: a distributed storage with rich
We would like to thank the anonymous reviewers for their semantics. https://arxiv.org/pdf/1702.02799.pdf.
comments and suggestions that help us improve the paper. [20] J. Douceur. The sybil attack. In IPTPS, 2002.
Special thanks to Hao Zhang, Loi Luu, the developers from [21] A. Dragojevic, D. Narayanan, E. B. Nightingale,
Ethereum, Parity and Hyperledger projects for helping us M. Renzelmann, A. Shamis, A. Badam, and
with the experiment setup. This work is funded by the Na- M. Castro. No compromises: distributed transactions
tional Research Foundation, Prime Minister’s Office, Sin- with consistency, availability and performance. In
gapore, under its Competitive Research Programme (CRP SOSP, 2015.
Award No. NRF-CRP8-2011-08). [22] Ethcore. Parity: next generation ethereum browser.
https://ethcore.io/parity.html.
8. REFERENCES [23] Ethcore. Performance analysis.
[1] BlockBench: private blockchains benchmarking. https://blog.ethcore.io/performance-analysis/.
https://github.com/ooibc88/blockbench. [24] Ethereum. Ethereum benchmarks.
[2] Ethereum blockchain app platform. https://github.com/ethereum/wiki/wiki/Benchmarks.
https://www.ethereum.org/. [25] I. Eyal, A. E. Gencer, E. G. Sirer, and R. van Renesse.
[3] Ibm watson iot. Bitcoin-ng: A scalable blockchain protocol. In NSDI,
http://www.ibm.com/internet-of-things. 2016.
[4] Leveldb. https://leveldb.org. [26] I. Eyal and E. G. Sirer. Majority is not enough:
[5] Monax: The ecosystem application platform. Bitcoin mining is vulnerable. In Fiancial
https://monax.io. Cryptography, 2014.
[6] Rocksdb. https://rocksdb.org. [27] A. Gervais, G. O. Karame, K. Wust, V. Glykantizis,
[7] M. Apostolaki, A. Zohar, and L. Vanbever. Hijacking H. Ritzdorf, and S. Capkun. On the security and
bitcoin: Large-scale network attacks on performance of proof of work blockchains.
crypto-currencies. https://arxiv.org/abs/1605.07524, https://eprint.iacr.org/2016/555.pdf.
2016. [28] A. Ghazal, T. Rabl, M. Hu, F. Raab, M. Poess,
[8] P. Bailis, A. Fekete, M. J. Franklin, A. Ghodsi, J. M. A. Crolotte, and H.-A. Jacobsen. Bigbench: towards
Hellerstein, and I. Stoica. Coordination avoidance in an industry standard benchmark for big data
database systems. In VLDB. analytics. In SIGMOD, 2013.
[9] J. Bonneau, A. Miller, J. Clark, A. Narayanan, J. A. [29] G. S. Group. Blockchain: putting theory into practice,
Kroll, and E. W. Felten. Sok: Research perspectives 2016.
and challenges for bitcoin and crypto-currencies. In [30] E. Heilman, A. Kendler, A. Zohar, and S. Goldberg.
2015 IEEE Symposium on Security and Privacy, pages Eclipse attacks on Bitcoin’s peer-to-peer network. In
104–121. IEEE, 2015. USENIX Security, 2015.
[10] M. Cahill, U. Rohm, and A. D. Fekete. Serializable [31] Hyperledger. Blockchain technologies for business.
isolation for snapshot databases. In SIGMOD, 2008. https://www.hyperledger.org.
[11] M. Castro and B. Liskov. Practical byzantine fault [32] Intel. Hibench suite.
tolerance. In Proceedings of the third symposium on https://github.com/intel-hadoop/HiBench.
Operating systems design and implementation, pages [33] F. P. Junqueira, B. C. Reed, and M. Serafini. Zab:
173–186. USENIX Association, 1999. high-performance broadcast for primary-backup
[12] B.-G. Chun, P. Maniatis, S. Shenker, and systems. In Dependable Systems and Networks, 2011.
J. Kubiatowicz. Attested append-only memory: [34] E. Kokoris-Kogias, P. Jovanovic, N. Gailly, I. Khoffi,
Making adversaries stick to their word. In SOSP, 2007. L. Gasser, and B. Ford. Enhancing bitcoin security
[13] B. F. Cooper, A. Silberstein, E. Tam, and performance with strong consistency via collective
R. Ramakrishnan, and R. Sears. Benchmarking cloud signing. In USENIX Security, 2016.
serving systems with ycsb. In SoCC, 2010. [35] L. Lamport. Paxos made simple. SIGACT News, 2001.
[14] J. C. Corbett and J. D. et al. Spanner: Google’s [36] Q. Lin, P. Chang, G. Chen, B. C. Ooi, K.-L. Tan, and
globally-distributed database. In OSDI, 2012. Z. Wang. Towards a non-2pc transaction management
[15] K. Croman, C. Decker, I. Eyal, A. E. Gencer, A. Juels, in distrubted database systems. In SIGMOD, 2016.
A. Kosba, A. Miller, P. Saxena, E. Shi, and E. Gün. [37] L. Luu, V. Narayanan, C. Zhang, K. Baweija,
On scaling decentralized blockchains. In Proc. 3rd S. Gilbert, and P. Saxena. A secure sharding protocol
Workshop on Bitcoin and Blockchain Research, 2016. for open blockchains. In CCS, 2016.
[16] Crypti. A decentralized application platform. [38] L. Luu, J. Teutsch, R. Kulkarni, and P. Saxena.
https://crypti.me. Demystifying Incentives in the Consensus Computer.
[17] C. Decker and R. Wattenhofer. Information CCS ’15, pages 706–719, 2015.
propagation in bitcoin network. In P2P, 2013. [39] Melonport. Blockchain software for asset management.
[18] D. E. Difallah, A. Pavlo, C. Curino, and http://melonport.com.
P. Cudre-Mauroux. Oltp-bench: An extensible testbed [40] J. Morgan and O. Wyman. Unlocking economic
for benchmarking relational databases. In VLDB, advantage with blockchain. a guide for asset
2013. managers., 2016.
[19] A. Dinh, J. Wang, S. Wang, W.-N. Chin, Q. Lin, [41] S. Nakamoto. Bitcoin: A peer-to-peer electronic cash
B. C. Ooi, P. Ruan, K.-L. Tan, Z. Xie, H. Zhang, and system, 2008.
 Throughput vs. HStore used in Ethereum, Dfinity and Parity, while Eris-DB only
Ethereum 142702 supports Solidity. Hyperledger, Stellar, Corda and Sawtooth
105 Parity Lake exploit various mature programming languages, such
Hyperledger 21596 as Python, Java, Golang, etc. ScalableBFT and Tezos even
104 H-Store develop their own smart contract languages. Most block-
chain platforms’ data models are account-based. Two ex-
#tx/s

1273 1122
103 ceptions in the table are Ripple and Corda. Their data
284 255 models are similar to Bitcoin’s unspent transaction outputs
102 45 46
(UTXO) which represents the coins in the network.
Each platform offers different consensus protocols. Hy-
perledger implements PBFT in the version we evaluated,
101 YCSB Smallbank while Ethereum implements a variation of PoW (Proof-of-
Work). Eris-DB builds on top of Tendermint protocol but
Figure 14: Performance of the three blockchain systems ver- only works in the latest version (v 0.12). Ripple and Tezos
sus H-Store. deploy Proof-of-Stake (PoS) schemes (the one in Ripple is
referred to Ripple Consensus Ledger) where the next block
is created based on accounts’ wealth, i.e., the stake. Parity
[42] D. Ongaro and J. Ousterhout. In search of an takes another consensus protocol, Proof-of-Authority (PoA),
understandable consensus algorithm. In USENIX which holds a predefined set of ”authorities” to create new
ATC, 2014. blocks in a fixed time slot and secure the blockchain network.
[43] R. Pass and E. Shi. Hybrid consensus: efficient Sawtooth Lake uses Proof-of-Elapsed-Time (PoET) as its
consensus in the permissionless model. consensus protocol, which in nature is a lottery algorithm
https://eprint.iacr.org/2016/917.pdf. and decides the creator of block arbitrarily. Stellar develops
[44] Ripple. Ripple. https://ripple.com. its own mechanism, Stellar Consensus Protocol, which is a
[45] Y. Sompolinsky and A. Zohar. Accelerating bitcoin’s construction for decentralized Byzantine agreement. There
transaction processing: fast money grows on trees, not is no source code that helps determine which consensus pro-
chains. Cryptology ePrint Archive, Report 2013/881, tocol Dfinity uses, but its documents suggest that a Block-
2013. https://eprint.iacr.org/2013/881.pdf. chain Nervous System will govern the whole platform via a
voting mechanism based on neurons that interact with each
[46] M. Stonebraker, S. Madden, D. J. Abadi,
other and are controlled by users.
S. Harizopoulos, N. Hachem, and P. Helland. The end
of and architectural era (it’s time for a complete
rewrite). In VLDB, 2007.
[47] K.-L. Tan, Q. Cai, B. C. Ooi, W.-F. Wong, C. Yao, B. MACRO BENCHMARKS
and H. Zhang. In-memory databases: Challenges and We compared the performance of the three blockchain sys-
opportunities from software and hardware tems against a popular in-memory database system, namely
perspectives. SIGMOD Records, 44(2), 2015. H-Store, using the YCSB and Smallbank workload. We ran
[48] A. Thomson, T. Diamond, S. chun Weng, K. Ren, H-Store’s own benchmark driver and set the transaction rate
P. Shao, and D. J. Abadi. Calvin: fast distributed at 100,000 tx/s. Figure 14 shows at least an order of mag-
transaction for partitioned database systems. In nitude gap in throughput and two order of magnitude in la-
SIGMOD, 2012. tency. Specifically, H-Store achieves over 140K tx/s through-
[49] M. Vukolic. The quest for scalable blockchain fabric: put while maintaining sub-millisecond latency. The gap in
proof-of-work vs. bft replication. In Open Problems in performance is due to the cost of consensus protocols. For
Network Security - iNetSec, 2015. YCSB, for example, H-Store requires almost no coordination
[50] H. Zhang, G. Chen, B. C. Ooi, K.-L. Tan, and among peers, whereas Ethereum and Hyperledger suffer the
M. Zhang. In-memory big data management and overhead of PoW and PBFT.
processing: a survey. TKDE, 2015. An interesting observation is the overhead of Smallbank.
Recall that Smallbank is a more complex transactional work-
APPENDIX load than YCSB, in which multiple keys are updated in a
single transaction. Smallbank is simple but is representative
of the large class of transactional workloads such as TPC-C.
A. SURVEY OF BLOCKCHAIN PLATFORMS We observe that in H-Store, Smallbank achieves 6.6x lower
We compare eleven promising blockchain platforms in Ta- throughput and 4x higher latency than YCSB, which indi-
ble 2. We can see that all but Ripple support smart con- cates the cost of distributed transaction management proto-
tracts. Ethereum, Eris-DB, Dfinity and Parity execute the col, because H-Store is a sharded database. In contrast, the
contracts using Ehtereum Virtual Machine (EVM), whereas blockchain suffers modest degradation in performance: 10%
Corda runs them in Java Virtual Machine (JVM). Hyper- in throughput and 20% in latency. This is because each node
ledger, Stellar and Tezos employ Docker images, ScalableBFT in blockchain maintains the entire state (replicated state ma-
takes Haskell execution environment, and Sawtooth Lake chine), thus there is no overhead in coordinating distributed
launches contracts on top of Trusted Execution Environment transactions as the data is not partitioned.
(TEE) such as Intel Software Guard Extensions (SGX). These The results demonstrate that blockchain performs poorly
platforms also support different languages to develop smart at data processing tasks currently handled by database sys-
contracts. For example, Solidity, Serpent and LLL are mainly tems. However, we stress that blockchains and databases
 Table 2: Comparison of blockchain platforms

Smart
Smart contract
Application contract Data model Consensus
language
execution
Hyperledger Smart contract Dockers Golang, Java Account-based PBFT
Smart contract,
Ethereum Crypto- EVM Solidity, Serpent, LLL Account-based Ethash (PoW)
currency
Eris-DB Smart contract EVM Solidity Account-based Tendermint (BFT)
Crypto- Ripple Consensus
Ripple - - UTXO-based
currency Ledger (PoS)
Haskell
ScalableBFT Smart contract Pact Account-based ScalableBFT
Execution
JavaScript, Golang, Stellar Consensus
Stellar Smart contract Dockers Account-based
Java, Ruby, Python, C# Protocol
Blockchain
Dfinity Smart contract EVM Solidity, Serpent, LLL Account-based
Nervous System
Parity Smart contract EVM Solidity, Serpent, LLL Account-based Proof of Authority
Smart contract,
Tezos Contract Script
Tezos Crypto- Dockers Account-based Proof of Stake
Language
currency
Corda Smart contract JVM Kotlin, Java UTXO-based Raft
Sawtooth Proof of Elapsed
Smart contract TEE Python Account-based
Lake Time

90 CPU utilization only crash failures. Other features which are optional in
80
70 most database systems are cryptographic signatures on ev-
60 Ethereum
50 Parity ery single transaction, and wide-area fully replicated state
40
%

30 Hyperledger machines. Although databases are designed without security

20
10 features and tolerance to Byzantine failures, we remark that
00 20 40 60 80 100
time (second) the gap remains too high for blockchains to be disruptive
100 Network utilization to incumbent database systems. Nevertheless, the popular-
80 Ethereum ity of blockchain is a clear indication that there is a need
60 Parity
for a Byzantine tolerant data processing systems which can
Mbps

Hyperledger
40 accommodate a large number of users.
20
00 Figure 15 shows the effect of varying block sizes in the
20 40 60 80 100
time (second) overall throughput. While it is straightforward to set the
block size in Hyperledger by configuring the batchSize vari-
Figure 16: Resource utilization. able, there is no direct way to specify the same in Ethereum.
An Ethereum miner uses gasLimit value to restrict the over-
all cost in constructing a block, thus we tuned this value to
Block generation rate simulate different sizes. In Parity, gasLimit is not applica-
101 ble to local transaction and it has no effect on the block size.
Small 5.20 Instead, we observe that the block size can be controlled by
Medium 3.10 tuning stepDuration value, which essentially decides how
Large 1.75 much time a validator can use to build a block. In the exper-
#block/s

100 1.00 iments, medium size refers to the default settings, whereas
large and small refer to 2x and 0.5x of the default size. The
0.56
results show that increases in block sizes lead to proportional
0.34 0.28 decreases in block generation rate, meaning that the overall
0.22 throughput does not improve.
0.12 Figure 16 compares CPU and network utilization of the
10-1 Ethereum Parity Hyperledger three systems over the period of 100 seconds. It is easy to
see that Ethereum is CPU bound, as it fully utilizes 8 CPU
cores. Hyperledger, on the other hand, uses CPU sparingly
Figure 15: Block generation rate.
and spends the rest of the time on network communication.
Parity, in contrast, has lower resource footprints than other
are designed with different goals and assumptions. Specif- two systems. For Ethereum and Hyperledger, the pattern
ically, the protocols for Byzantine failure tolerance are an is the direct consequence of the consensus protocol: PoW is
overkill for traditional database settings where there are CPU bound whereas PBFT is communication bound.
 104 Throughput Latency 80000 Queue length
Ethereum
Parity 103
Ethereum
Parity 70000 Ethereum
Hyperledger Hyperledger 60000 Parity
103 50000 Hyperledger

#requests
102
40000

second
#tx/s

101 30000
102 20000
100 10000
00 50 100 150 200 250 300 350
101 12 4 8 12 16 20 24 28 32 10-1 12 4 8 12 16 20 24 28 32 time (second)
#nodes #nodes

Figure 19: Scalability with Smallbank benchmark. Figure 18: Queue length at the client.

type account_t struct {

Balance int Figure 17 shows the latency distribution. Ethereum has
CommitBlock int
} both higher latency and higher variance, because PoW is
type transaction_t { a randomized process which means the duration between
From string blocks are unpredictable. Parity has the lowest variance
To string because the server restricts the client request rate at 80 tx/s.
Val int Figure 18 illustrates the request queue at the client for
} the settings of 20 servers and 20 clients. The queue behav-
func Invoke_SendValue(from_account string,
to_account string, value int) { ior of Ethereum reflects the normal case, i.e. the queue grew
var pending_list []transaction_t and shrank depending on how fast the transactions are com-
pending_list = decode(GetState("pending_list")) mitted. Hyperledger failed to generate blocks in this case,
var new_txn transaction_t therefore the queue never shrank. However, there are du-
new_txn = transaction_t { rations in which the queue size remains constant. Further-
from_account, to_account, value more, at the beginning, the queue in Hyperledger is smaller
}
pending_list = append(pending_list, new_txn) than that in Ethereum, even though the clients are send-
PutState(’pending_list’, encode(pending_list)) ing at the same rate. This suggests there is a bottleneck in
} processing network requests at the Hyperledger servers.
func Query_BlockTransactionList(block_number int) Figure 19 illustrates the scalability of the three systems
[]transaction_t { using the Smallbank benchmark. We observe similar pat-
return decode(GetState("block:"+block_number)) terns to the YCSB benchmark (Figure 7), except that Hy-
}
func Query_AccountBlockRange(account string, perledger failed to scale beyond 8 nodes instead of 16.
start_block int, end_block int)
[]account_t { C. ANLAYTICS SMART CONTRACT
version := decode(GetState(account+":latest"))
var ret []account_t Figure 20 shows the implementation of the smart con-
while true { tract method that answer Q2 of the analytics workload. To
var acc account_t support historical data lookup, we append a counter to the
acc = decode(GetState(account+":"+version)) key of each account. To fetch a specific version of an ac-
if acc.CommitBlock >= start_block && count, we use key account:version. We store the latest
acc.CommitBlock < end_block { version of the account using key account:latest, and keep
ret = append(ret, acc)
} else if acc.CommitBlock < start_block { a CommitBlock in the data field for every version which in-
break; dicates in which block the balance of this version is com-
} mitted. To answer query that fetches a list of balance of a
version -= 1 given account within a given block range, the method scans
} all versions of this account and returns the balance values
return ret that are committed within the given block range. Ethereum
} and Parity provide JSON-PRC APIs getBalance(account,
block) to query information of an account at a given block
Figure 20: Code snippet from the VersionKVStore smart number. This API fetches only one version of the account
contract for analytics workload (Q1 and Q2). per HTTP roundtrip, so it is less efficient than pushing the
query logic to server side.

1.0 Latency distribution

0.8
0.6
cdf

0.4 Ethereum-YCSB
Ethereum-SB
Parity-YCSB
0.2 Parity-SB
Hyperledger-YCSB
Hyperledger-SB
0.0
100 101 102 103
time (second)

Figure 17: Latency distribution.