BLOCKBENCH: A Framework for Analyzing Private

Blockchains

Tien Tuan Anh Dinh‡ Ji Wang‡ Gang Chen§ Rui Liu‡ Beng Chin Ooi‡ Kian-Lee Tan‡
‡ §
National University of Singapore Zhejiang University
‡
{dinhtta, wangji, liur, ooibc, tankl}@comp.nus.edu.sg §
[email protected]

ABSTRACT 1. INTRODUCTION
Blockchain technologies are taking the world by storm. Pub- Blockchain technologies are gaining massive momentum
lic blockchains, such as Bitcoin and Ethereum, enable secure in the last few years, largely due to the success of Bitcoin
peer-to-peer applications like crypto-currency or smart con- crypto-currency [41]. A blockchain, also called distributed
tracts. Their security and performance are well studied. ledger, is essentially an append-only data structure main-
This paper concerns recent private blockchain systems de- tained by a set of nodes which do not fully trust each other.
signed with stronger security (trust) assumption and perfor- All nodes in a blockchain network agree on an ordered set
mance requirement. These systems target and aim to dis- of blocks, each containing multiple transactions, thus the
rupt applications which have so far been implemented on top blockchain can be viewed as a log of ordered transactions.
of database systems, for example banking, finance and trad- In a database context, blockchain can be viewed as a solu-
ing applications. Multiple platforms for private blockchains tion to the distributed transaction management problems:
are being actively developed and fine tuned. However, there nodes keep replicas of the data and agree on an execution
is a clear lack of a systematic framework with which different order of transactions. However, traditional database sys-
systems can be analyzed and compared against each other. tems work in a trusted environment and employ well known
Such a framework can be used to assess blockchains’ viabil- concurrency control techniques [36, 48, 8] to order transac-
ity as another distributed data processing platform, while tions. Blockchain’s key advantage is that it does not assume
helping developers to identify bottlenecks and accordingly nodes trust each other and therefore is designed to achieve
improve their platforms. Byzantine fault tolerance.
In this paper, we first describe BLOCKBENCH, the first In the original design, Bitcoin’s blockchain stores coins
evaluation framework for analyzing private blockchains. It as the system states shared by all participants. For this
serves as a fair means of comparison for different platforms simple application, Bitcoin nodes implement a simple repli-
and enables deeper understanding of different system de- cated state machine model which simply moves coins from
sign choices. Any private blockchain can be integrated to one address to another. Since then, blockchain has grown
BLOCKBENCH via simple APIs and benchmarked against rapidly to support user-defined states and Turing complete
workloads that are based on real and synthetic smart con- state machine models. Ethereum [2] is a well-known ex-
tracts. BLOCKBENCH measures overall and component- ample which enables any decentralized, replicated applica-
wise performance in terms of throughput, latency, scala- tions known as smart contracts. More importantly, interest
bility and fault-tolerance. Next, we use BLOCKBENCH from the industry has started to drive development of new
to conduct comprehensive evaluation of three major private blockchain platforms that are designed for private settings
blockchains: Ethereum, Parity and Hyperledger Fabric. The in which participants are authenticated. Blockchain systems
results demonstrate that these systems are still far from dis- in such environments are called private (or permissioned), as
placing current database systems in traditional data process- opposed to the early systems operating in public (or permis-
ing workloads. Furthermore, there are gaps in performance sionless) environments where anyone can join and leave. Ap-
among the three systems which are attributed to the design plications for security trading and settlement [44], asset and
choices at different layers of the blockchain’s software stack. finance management [39, 40], banking and insurance [29] are
We have released BLOCKBENCH for public use. being built and evaluated. These applications are currently
supported by enterprise-grade database systems like Oracle
and MySQL, but blockchain has the potential to disrupt
this status quo because it incurs lower infrastructure and
human costs [29]. In particular, blockchain’s immutability
Permission to make digital or hard copies of all or part of this work for personal or and transparency help reduce human errors and the need
classroom use is granted without fee provided that copies are not made or distributed for manual intervention due to conflicting data. Blockchain
for profit or commercial advantage and that copies bear this notice and the full cita- can help streamline business processes by removing dupli-
tion on the first page. Copyrights for components of this work owned by others than
ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re- cate efforts in data governance. Goldman Sachs estimated 6
publish, to post on servers or to redistribute to lists, requires prior specific permission billion saving in current capital market [29], and J.P. Mor-
and/or a fee. Request permissions from [email protected]. gan forecast that blockchains will start to replace currently
SIGMOD’17, May 14-19, 2017, Chicago, IL, USA redundant infrastructure by 2020 [40].

c 2017 ACM. ISBN 978-1-4503-4197-4/17/05. . . $15.00 Given this trend in employing blockchain in settings where
DOI: http://dx.doi.org/10.1145/3035918.3064033
database technologies have established dominance, one ques- comparison of the three blockchain systems on two macro
tion to ask is to what extent can blockchain handle data benchmark and four micro benchmark workloads. The re-
processing workload. Another question is which platform sults show that blockchain systems’ performance is limited,
to choose from many that are available today, because even far below what is expected of a state-of-the-art database
though blockchain is an open protocol, different platforms system (such as H-Store). Hyperledger consistently outper-
exist in silo. In this work, we develop a benchmarking forms the other two systems across seven benchmarks. But
framework called BLOCKBENCH to address both ques- it fails to scale beyond 16 nodes. Our evaluation shows that
tions. BLOCKBENCH is the first benchmark for study- the consensus protocols account for the performance gap at
ing and comparing performance of permissioned blockchains. the application layer for Ethereum and Hyperledger. We
Although nodes in a permissioned blockchain still do not also identify a processing bottleneck in Parity. Finally, our
trust each other, their identities are authenticated, which evaluation also reveals bottlenecks in the execution and data
allows the system to use more efficient protocols for tol- layer of Ethereum and Parity.
erating Byzantine failure than in public settings. We do In summary, our contributions are:
not focus on public blockchains because their performance
(and trade-offs against security guarantee) is relatively well • We present the first benchmarking framework for un-
studied [27, 38, 15, 9]. Our framework is not only useful derstanding and comparing the performance of per-
for application developers to assess blockchain’s potentials missioned blockchain systems. We have released the
in meeting the application need, but also offers insights for framework for public use [1].
platform developers: helping them to identify and improve • We conduct a comprehensive evaluation of Ethereum,
on the performance bottlenecks. Parity and Hyperledger. Our empirical results present
We face three challenges in developing BLOCKBENCH. concrete evidence of blockchain’s limitations in han-
First, a blockchain system comprises many parts and we ob- dling data processing workloads, and reveal bottle-
serve that a wide variety of design choices are made among necks in the three systems. The results serve as a
different platforms at almost every single detail. In BLOCK- baseline for further development of blockchain tech-
BENCH, we divide the blockchain architecture into three nologies.
modular layers and focus our study on them: the consen-
sus layer, data model and execution layer. Second, there are In the next section, we discuss blockchain systems in more
many different choices of platforms, but not all of them have detail. Section 3 describes BLOCKBENCH design and im-
reached a mature design, implementation and an established plementation. Section 4 presents our comparative perfor-
user base. For this, we start by designing BLOCKBENCH mance studies of three systems. We discuss lessons learned
based on three most mature platforms within our considera- from the results in Section 5 and related work in Section 6,
tion, namely Ethereum [2], Parity [22] and Hyperledger [31], and we conclude in Section 7.
and then generalize to support future platforms. All three
platforms support smart contracts and can be deployed in 2. PRIVATE BLOCKCHAINS
a private environment. Third, there is lack of a database-
A typical blockchain system consists of multiple nodes
oriented workloads for blockchain. Although the real Eth-
which do not fully trust each other. Some nodes exhibit
ereum transactions and contracts can be found on the pub-
Byzantine behavior, but the majority is honest. Together,
lic blockchain, it is unclear if such workload is sufficiently
the nodes maintain a set of shared, global states and perform
representative to assess blockchain’s general data processing
transactions modifying the states. Blockchain is a special
capabilities. To address this challenge, we treat blockchain
data structure which maintains the states and the historical
as a key-value storage coupled with an engine which can
transactions. All nodes in the system agree on the transac-
realize both transactional and analytical functionality via
tions and their order as stored on the blockchain. Because of
smart contracts. We then design and run both transaction
this, blockchain is often referred to as a distributed ledger.
and analytics workloads based on real and synthetic data.
Blockchain transactions. A transaction in a blockchain
BLOCKBENCH is a flexible and extensible framework
is the same as in traditional database: a sequence of opera-
that provides a number of workloads, and comes with Eth-
tions applied on some states. As such, a blockchain transac-
ereum, Parity and Hyperledger as backends. Workloads
tion requires the same ACID semantics. The key difference
are transaction-oriented currently and designed to macro-
is the failure model under consideration. Current transac-
benchmark and micro-benchmark blockchain for supporting
tional, distributed databases [46, 14] employ classic concur-
database-like applications. Specifically, the current macro-
rency control techniques such as two-phase commit to ensure
benchmark includes a key-value (YCSB), an OLTP (Small-
ACID. They can achieve high performance, because of the
bank) workload and a number of real Ethereum smart con-
simple failure model, i.e. crash failure. In contrast, the
tract workloads. For each of the consensus, data and exe-
original blockchain design considers a much hostile environ-
cution layer, there is at least a micro-benchmark workload
ment in which nodes are Byzantine and they are free to join
to measure its performance in isolation. For example, for
and leave. Under this model, the overhead of concurrency
the execution layer, BLOCKBENCH provides two work-
control is much higher [11].
loads that stress test the smart contract I/O and compu-
Bitcoin. In Bitcoin [41], the states are digital coins (crypto-
tation speed. New workloads and blockchains can be easily
currency) available in the network. A Bitcoin transaction
integrated via a simple set of APIs. BLOCKBENCH quan-
moves coins from one set of addresses to another set of ad-
tifies the performance of a backend system in several dimen-
dresses. Each node broadcasts a set of transactions it wants
sions: throughput, latency, scalability and fault tolerance.
to perform. Special nodes called miners collect transactions
It supports security evaluation by simulating network-level
into blocks, check for their validity, and start a consensus
attacks. Using BLOCKBENCH, we conduct an in-depth
protocol to append the blocks onto the blockchain. Figure 1
 Asset Securities contract Doubler{
Application Crypto-currency Asset Securities ...
Crypto-currency management settlement struct Partitipant {
management settlement
address etherAddress;
Blockchain uint amount;
block t block t+1
}
... Block header Block header ... Partitipant[] public participants;
Transaction Contract Transaction Contract
unit public balance = 0;
roothash roothash roothash roothash ...
function enter(){
...
balance+= msg.value;
Smart contract Smart contract
...
if (balance > 2*participants[payoutIdx].amount){
Code Code
input, output input, output transactionAmount = ...
participants[payoutIdx].
State State
storage storage etherAddress.send(transactionAmount);
...
Hardware
}
}
CPU Network
...
Storage
}

Figure 1: Blockchain software stack on a fully validating Figure 2: An example of smart contract, written in Solidity
node. A non-validating node stores only the block head- language, for a pyramid scheme on Ethereum.
ers. Different blockchain platforms offer different interface
between the blockchain and application layer.
contract which is used to pay interests to early participants.
This contract has its own states, namely the list of partici-
shows the blockchain data structure, in which each block is pants, and exports a function called enter. A user invokes
linked to its predecessor via a cryptographic pointer, all the this contract by sending his money through a transaction,
way back to the first (genesis) block. Bitcoin uses proof-of- which is accessed by the smart contract as msg.sender and
work (PoW) for consensus: only a miner which has success- msg.amount.
fully solved a computationally hard puzzle (finding the right Private blockchain. Ethereum uses the same consensus
nonce for the block header) can append to the blockchain. protocol as Bitcoin does, though with different parameters.
PoW is tolerant of Byzantine failure, but it is probabilistic In fact, 90% of public blockchain systems employ variants of
in nature: it is possible that two blocks are appended at the proof-of-work protocol. PoW is non-deterministic and
the same time, creating a fork in the blockchain. Bitcoin computationally expensive, both rendering it unsuitable for
resolves this by only considering a block as confirmed after applications such as banking and finance which must han-
it is followed by a number of blocks (typically six blocks). dle a lot of transactions in a deterministic manner. Recent
This probabilistic guarantee causes both security and per- blockchain systems, e.g., Hyperledger, consider restricted
formance issues: attacks have been demonstrated by an ad- settings wherein nodes are authenticated. Although PoW
versary controlling only 25% of the nodes [26], and Bitcoin is still useful in such permissioned environments, as in the
transaction throughput remains very low (7 transactions per case of Ethereum, there are more efficient and determinis-
second [15]). tic solutions where node identities are known. Distributed
Ethereum. Due to simple transaction semantics, Bitcoin fault-tolerant consensus in such a closed settings is a well
nodes execute a very simple state machine pre-built into the studied topic in distributed systems. Zab [33], Raft [42],
protocol. Ethereum [2] extends Bitcoin to support user- Paxos [35], PBFT [11] are popular protocols that are in ac-
defined and Turing complete state machines. In particular, tive use today. Recent permissioned blockchains either use
Ethereum blockchain lets the user define any complex com- existing PBFT, as in Hyperledger [31], or develop their own
putations in the form of smart contracts. Once deployed, variants, as in Parity [22], Ripple [44] and ErisDB [5]. Most
the smart contract is executed on all Ethereum nodes as a of these systems support smart contracts, though in different
replicated state machine. Beside the shared states of the languages, with different APIs and execution engines (see a
blockchains (crypto-currency, for example), each smart con- more comprehensive comparison in the Appendix). As a
tract has access to its own states. Figure 1 shows the soft- result, permissioned blockchains can execute complex appli-
ware stack in a typical Ethereum node: a fully validating cation more efficiently than PoW-based blockchains, while
node contains the entire history of the blockchain, whereas being Byzantine fault tolerant. These properties and the
a non-validating node stores only the block headers. One commercial interests from major banking and financial insti-
key difference with Bitcoin is that smart contract states are tutions have bestowed on private blockchains the potentials
maintained as well as normal transactions. In fact, a smart to disrupt the current practice in data management.
contract is identified by a unique address which has its own
money balance (in Ether), and upon retrieving a transaction 3. BLOCKBENCH DESIGN
to its address, it executes the contract’s logics. Ethereum This section discusses blockchain’s common layers of ab-
comes with an execution engine, called Ethereum Virtual stractions and the benchmarking workloads.
Machine (EVM), to execute smart contracts. Figure 2 shows
a snippet of popular contract running on Ethereum, which
implements a pyramid scheme: users send money to this
 YCSB,
Blockchain Contracts BLOCKBENCH In any case, a block can be confirmed as part of the block-
Application Smallbank,
layers etc. workloads chain only with some high probability.
At the other extreme, Hyperledger uses the classic PBFT
Compilers, VM, CPU-Heavy
Dockers, etc. Execution Engine protocol, which is communication bound: O(N 2 ) where N
is the number of nodes. PBFT can tolerate fewer than N3
Blocks Analytics,
failures, and works in three phases in which nodes broadcast
Transactions, Data Model IO-Heavy messages to each other. First, the pre-prepare phase selects
Indexing, etc.
a leader which chooses a value to commit. Next, the prepare
PoW, PoS, phase broadcasts the value to be validated. Finally, the com-
PBFT, etc.
Consensus Commits
mit phase waits for more than two third of the nodes to con-
firm before announcing that the value is committed. PBFT
Figure 3: Abstraction layers in blockchain, and the corre- has been shown to achieve liveness and safety properties in
sponding workloads in BLOCKBENCH. a partially asynchronous model [11], thus, unlike PoW, once
the block is appended it is confirmed immediately. It can
tolerate more failures than PoW (which is shown to be vul-
3.1 Blockchain Layers nerable to 25% attacks [26]). However, PBFT assumes that
There are many choices of blockchains: over 200 Bitcoin node identities are known, therefore it can only work in the
variants, Ethereum and other permissioned blockchains. To permissioned settings. Additionally, the protocol is unlikely
meaningfully compare them, we identify four abstraction to be able to scale to the network size of Ethereum, because
layers found in all of these systems (Figure 3) and design of its communication overhead.
our workloads to target these layers. The consensus layer In between, there are various hybrid designs that combine
contains protocols via which a block is considered appended both scalability of PoW and safety property of PBFT [43].
to the blockchain. The data layer contains the structure, For example, Bitcoin-NG [25] decouples consensus from trans-
content and operations on the blockchain data. The execu- action validation by using PoW for leader election who can
tion layer includes details of the runtime environment sup- then append more than one block at a time. Similarly, Byz-
port blockchain operations. Finally, the application layer coin [34] and Elastico [37] leverage PoW to determine ran-
includes classes of blockchain applications. In a related dom, smaller consensus groups which run PBFT. Another
work, Croman et. al. [15] proposed to divide blockchain into example is the Tendermint protocol, adopted by ErisDB [5],
several planes: network, consensus, storage, view and side which combines proof-of-stake (PoS) and PBFT. Unlike PoW,
plane. While similar to our four layers, the plane abstrac- PoS selects a node which can append a block by its invest-
tions were geared towards crypto-currency applications and ment (or stake) in the system, therefore avoid expending
did not take into account the execution of smart contracts. CPU resources. Parity [22] implements a simplified version
Our layers model more accurately the real implementations of PoS called Proof of Authority (or PoA). In this protocol,
of private blockchains. We now discuss these layers in turn. a set of authorities are pre-determined and each authority
is assigned a fixed time slot within which it can generate
3.1.1 Consensus blocks. PoA makes a strong assumption that the author-
ities are trusted, and therefore is only suitable for private
The role of the consensus layer is to get all nodes in the
deployment.
system to agree on the blockchain content. That is, if a node
appends (or commits) a block, the other nodes also append
the same block to their copy of the blockchain. Protocols for 3.1.2 Data model
reaching consensus in the crash-failure model play a key role In Bitcoin, transactions are first class citizens: they are
in distributed databases, wherein nodes agree on a global system states representing digital coins in the network. Pri-
transaction order. Blockchain systems, on the other hand, vate blockchains depart from this model, by focusing on
employ a spectrum of Byzantine fault-tolerant protocols [50]. accounts. One immediate benefit is simplicity, especially
At one extreme, Ethereum, like Bitcoin, uses proof-of- for applications involving crypto-currencies. For instance,
work whose difficulty is agreed upon and adjusted gradually transferring money from one user to another in Bitcoin in-
to achieve a rate of (currently) one block per 14s (Bitcoin’s volves searching for transactions belonging to the sender,
difficulty achieves a rate of one block per 10m). In essence, then marking some of them as spent, whereas it is easily
proof-of-work selects at each round a random node which done in Ethereum by updating two accounts in one trans-
can append a block, where the probability of being selected action. An account in Ethereum has a balance as its state,
is determined by the node’s total computing power. This and is updated upon receiving a transaction. A special type
simple scheme works against Sybil attack [20, 49] - a com- of account, called smart contract, contains executable code
mon attack in open, decentralized environments in which the and private states (Figure 1). When receiving a transaction,
adversary can acquire multiple identities. However, it con- in addition to updating its balance, the contract’s code is in-
sumes a lot of energy and computing power, as nodes spend voked with arguments specified in the transaction. The code
their CPU cycles solving puzzles instead of doing otherwise can read the states of other non-contract accounts, and it
useful works. Worse still, it does not guarantee safety: two can send new transactions during execution. Parity adopts
nodes may both be selected to append to the blockchain, and the same data model as in Ethereum. In Hyperledger, there
both blocks can be accepted. This causes fork in the block- is only one type of account called chaincode which is the
chain, and most PoW-based systems add additional rules, same as Ethereum’s contract. Chaincode can only access its
for example, only blocks on the longest chain are considered private storage and they are isolated from each other.
accepted. Ethereum, in particular, adopts a PoW variant A block contains a list of transactions, and a list of smart
called GHOST [45] which accepts blocks in heavy branches. contracts executed as well as their latest states. Each block
is identified by the cryptographic hash of its content, and WorkloadClient WorkloadClient
linked to the previous block’s identity. In Parity, the entire ...
block content is kept in memory. In Ethereum and Hy-
perledger, the content is organized in a two layered data
structure. The states are stored in a disk-based key-value
storage (LevelDB[4] in Ethereum and RocksDB[6] in Hyper-
ledger), and organized in a hash tree whose root is included Configuration Driver StatsCollector
in the block header. Ethereum caches the states in memory,
while Hyperledger outsources its data management entirely
to the storage engine. Only states affected by the block’s IBlockchainConnector
transactions are recorded in the root hash. The hash tree
for transaction list is a classic Merkle tree, as the list is Ethereum Parity Hyperledger ErisDB ...
not large. On the other hand, different Merkle tree vari-
ants are used for the state tree. Ethereum and Parity em-
ploy Patricia-Merkle tree that supports efficient update and Figure 4: BLOCKBENCH software stack. New work-
search operations. Hyperledger implements Bucket-Merkle loads are added by implementing IWorkloadConnector in-
tree which uses a hash function to group states into a list of terface. New blockchain backends are added by implement-
buckets from which a Merkle tree is built. ing IBlockchainConnector. Current backends include Eth-
Block headers and the key-value storage together maintain ereum, Parity and Hyperledger.
all the historical transactions and states of the blockchain.
For validating and executing transactions, a blockchain node
needs only a few recent blocks (or just the latest block for types such as map, array and composite structures. These
PBFT-based systems). However, the node also interacts via high-level data types in Ethereum and Parity make it easier
some RPC-like mechanisms with light-weight clients who do and faster to develop new contracts.
not have the entire blockchain. Such external interfaces en-
able building of third-party applications on top of block- 3.1.4 Application layer
chain. Current systems support a minimum set of queries Many applications are being proposed for blockchain, lever-
including getting blocks and transactions based on their IDs. aging the latter’s two key properties. First, data in the
Ethereum and Parity expose a more comprehensive set of blockchain is immutable and transparent to the participants,
APIs via JSON-RPC, supporting queries of account states meaning that once a record is appended, it can never be
at specific blocks and of other block statistics. changed. Second, it is resilient to dishonest and malicious
participants. Even in permissioned settings, participants
3.1.3 Execution layer can be mutually distrustful. The most popular application,
A contract (or chaincode) is executed in a runtime envi- however, is still crypto-currency. Ethereum has its own cur-
ronment. One requirement is that the execution must be rency (Ether) and a majority of smart contracts running on
fast, because there are multiple contracts and transactions it are currency related. Decentralized Autonomous Organi-
in one block and they must all be verified by the node. An- zation (DAO) is the most active application in Ethereum,
other is that the execution must be deterministic, ideally creating communities for crowd funding, exchange, invest-
the same at all nodes. Deterministic execution avoid unnec- ment, or any other decentralized activities. A DAO manages
essary inconsistency in transaction input and output which funds contributed by participants and gives its users voting
leads to blocks being aborted. In both PoW and PBFT, power proportional to their contributions. Parity’s main ap-
aborting transactions wastes computing resources. plication is the wallet application that manages Ether. As
Ethereum develops its own machine language (bytecode) major banks are now considering adopting crypto-currency,
and a virtual machine (called EVM) for executing the code, some fintech companies are building applications that take
which is also adopted by Parity. EVM is optimized for crypto-currency to mediate financial transactions, for ex-
Ethereum-specific operations. For example, every code in- ample, in currency exchange market [44]. Other examples
struction executed in Ethereum costs a certain amount of include applying the currency and smart contracts for more
gas, and the total cost must be properly tracked and charged transparent and cost-effective asset management [39, 40].
to the transaction’s sender. Furthermore, the code must Some applications propose to build on blockchain’s im-
keep track of intermediate states and reverse them if the mutability and transparency for better application work-
execution runs out of gas. Hyperledger, in contrast, does flows in which humans are the bottlenecks. For example,
not consider these semantics in its design, so it simply sup- security settlements and insurance processes can be sped up
ports running of compiled machine codes inside Docker im- by storing data on the blockchain [29]. Another example is
ages. Specifically, chaincodes are deployed as Docker im- sharing economy applications, such as AirBnB, which can
ages interacting with Hyperledger’s backend via pre-defined use blockchain to evaluate reputation and trust in a decen-
interfaces. One advantage of Hyperledger’s environment tralized settings, because historical activities of any users
is that it supports multiple high-level programming lan- are available and immutable. This also extends to Inter-
guages such as Go and Java, as opposed to Ethereum’s net of Things settings, where devices need to establish trust
own language. In terms of development environment, Hy- among each other [3].
perledger exposes only simple key-value operations, namely
putState and getState. This is restricted, because any 3.2 BLOCKBENCH Implementation
contract states must be mapped into key-value tuples. In Figure 4 illustrates the current BLOCKBENCH’s imple-
contrast, Ethereum and Parity support a richer set of data mentation. To evaluate a blockchain system, the first step is
to integrate the blockchain into the framework’s backend by • Scalability: measured as the changes in throughput
implementing IBlockchainConnector interface. The inter- and latency when increasing number of nodes and num-
face contains operations for deploying application, invoking ber of concurrent workloads.
it by sending a transaction, and for querying the blockchain’s
states. Ethereum, Parity and Hyperledger are current back- • Fault tolerance: measured as how the throughput and
ends supported by BLOCKBENCH, while ErisDB integra- latency change during node failure. Although block-
tion is under development. A user can use one of the exist- chain systems are tolerant against Byzantine failure,
ing workloads (discussed next) to evaluate the blockchain, or it is not possible to simulate all Byzantine behaviors.
implement a new workload using the IWorkloadConnector In BLOCKBENCH we simulate three failure modes:
interface (we assume that the smart contract handling the crash failure in which a node simply stops, network de-
workload’s logic is already implemented and deployed on the lay in which we inject arbitrary delays into messages,
blockchain). This interface essentially wraps the workload’s and random response in which we corrupt the messages
operations into transactions to be sent to the blockchain. exchanged among the nodes.
Specifically, it has a getNextTransaction method which re-
Security metrics. A special case of Byzantine failures
turns a new blockchain transaction. BLOCKBENCH’s core
that is important to blockchain systems is malicious behav-
component is the Driver which takes as input a workload,
ior caused by an attacker. The attacker can be a compro-
user-defined configuration (number of operations, number
mised node or rouge participant within the system. Under
of clients, threads, etc.), executes it on the blockchain and
this threat model, security of a blockchain is defined as the
outputs running statistics.
safety property of the underlying consensus protocol. In par-
Asynchronous Driver. One challenge in implement-
ticular, security means that the non-Byzantine nodes have
ing the Driver is that current blockchain systems are asyn-
the same blockchain data. Violation of the safety property
chronous services, meaning that transactions submitted to
leads to forks in the blockchain. Classic Byzantine tolerant
the systems are processed at a later time. This is in con-
protocols such as PBFT are proven to ensure safety for a
trast to databases, especially transactional databases, in
certain number of failures, thus security is guaranteed. On
which operations are synchronous, i.e. they block until the
the other hand, in PoW systems like Bitcoin or Ethereum,
systems finish processing. When a transaction is submit-
forks can occur due to network delays causing two nodes
ted, Ethereum, Parity and Hyperledger return a transac-
to mine the same blocks. While such accidental forks can
tion ID which can be used for checking the transaction sta-
be quickly resolved, forks engineered by the attackers can
tus at a later time. Such asynchronous semantics could
be used for double spending and selfish mining. In the for-
result in better performance, but it forces the Driver to
mer, the attacker sends a transaction to a block in the fork,
periodically poll for status of the submitted requests. In
waits for it to be accepted by the users, then sends a con-
particular, Driver maintains a queue of outstanding trans-
flicting transaction to another block in the main branch. In
actions that have not been confirmed. New transaction
the latter, by withholding blocks and maintaining a private,
IDs are added to the queue by worker threads. A polling
long fork, the attacker disrupts the incentives for mining and
thread periodically invokes getLatestBlock(h) method in
forces other participants to join the attacker’s coalition. By
the IBlockchainConnector interface, which returns a list of
compromising 25% of the nodes, the attacker can control the
new confirmed blocks on the blockchain from a given height
entire network’s block generation [26].
h. Ethereum and Parity consider a block as confirmed if it is
In this work we quantify security as the number of blocks
at least confirmationLength blocks from the current block-
in the forks. Such blocks, called orphan or stale blocks, rep-
chain’s tip, whereas Hyperledger confirms a block as soon
resent the window of vulnerability in which the attacker can
as it appears on the blockchain. The Driver then extracts
perform double spending or selfish mining. To manipulate
transaction lists from the confirmed blocks’ content and re-
forks, the key strategy is to isolate a group of nodes, i.e.
moves matching ones in the local queue. getLatestBlock(h)
to partition the network. For example, eclipse attack [30]
can be implemented in all three systems by first requesting
exploits the application-level protocol to surround the tar-
for the blockchain’s current tip t, then requesting the con-
geted nodes with ones under the attacker’s control. At the
tent of all blocks in the range (h, t]. ErisDB provides a
network level, BGP hijacking [7] requires controlling as few
publish/subscribe interface that could simplify the imple-
as 900 prefixes to isolate 50% of the Bitcoin’s total mining
mentation of this function.
power. BLOCKBENCH implements a simulation of these
attacks by partitioning the network for a given duration. In
3.3 Evaluation Metrics particular, during partition BLOCKBENCH runtime drops
The output statistics of running a workload with different network traffic between any two nodes in the two partitions.
configurations can be used to evaluate the blockchain against Security is then measured by the ratio between the total
three performance metrics. number of blocks included in the main branch and the total
number of blocks confirmed by the users. The lower the ra-
• Throughput: measured as the number of successful tio, the less vulnerable the system is from double spending
transactions per second. A workload can be configured for selfish mining.
with multiple clients and threads per clients to saturate
the blockchain throughput. 3.4 Workloads
We divide the workloads into two major categories: macro
• Latency: measured as the response time per transac- benchmark for evaluating performance of the application
tion. Driver implements blocking transaction, i.e. it layer, and micro benchmark for testing the lower layers. We
waits for one transaction to finish before starting an- have implemented the smart contracts for all workloads for
other. Ethereum, Parity and Hyperledger, whose details are sum-
 Smart contracts Description part.
YCSB Key-value store WavesPresale. This contract supports digital token sales.
Smallbank OLTP workload It maintains two states: the total number of tokens sold so
EtherId Name registrar contract
far, and the list of previous sale transactions. It supports
Doubler Ponzi scheme
operations to add a new sale, to transfer ownership of a pre-
WavesPresale Crowd sale
VersionKVStore Keep state’s versions (Hyperledger only)
vious sale, and to query a specific sale records. Ethereum
IOHeavy Read and write a lot of data and Parity support composite structure data types, making
CPUHeavy Sort a large array it straightforward to implement the application logic. In
DoNothing Simple contract, do nothing contrast, in Hyperledger, we have to translate this structure
into key-value semantics by using separate key-value names-
Table 1: Summary of smart contracts implemented in paces.
BLOCKBENCH. Each contract has one Solidity version for
3.4.2 Micro benchmark workloads
Parity and Ethereum, and one Golang version for Hyper-
ledger. The previous workloads test the performance of block-
chain as a whole. As discussed early in this section, a block-
chain system comprises multiple layers, and each layer may
marized in Table 1. Ethereum and Parity use the same exe- have different impact on the overall performance. We design
cution model, therefore they share the same smart contract several workloads to stress the layers in order to understand
implementations. their individual performance.
DoNothing. This contract accepts transaction as input
3.4.1 Macro benchmark workloads and simply returns. In other words, it involves minimal
number of operations at the execution layer and data model
We port two popular database benchmark workloads into
layer, thus the overall performance will be mainly deter-
BLOCKBENCH, and three other real workloads found in
mined by the consensus layer. Previous works on perfor-
the Ethereum blockchain.
mance of blockchain consensus protocol [34, 43] use time to
Key-value storage. We implement a simple smart con-
consensus to measure its performance. In BLOCKBENCH,
tract which functions as a key-value storage. The
this metric is directly reflected in the transaction latency.
WorkloadClient is based on the YCSB driver [13]. It preloads
Analytics. This workload considers the performance of
each store with a number of records, and supports requests
blockchain system in answering analytical queries about the
with different ratios of read and write operations. YCSB is
historical data. Similar to an OLAP benchmark, this work-
widely used for evaluating NoSQL databases.
load evaluates how the system implements scan-like and ag-
OLTP (Smallbank). Unlike YCSB which does not con-
gregate queries, which are determined by its data model.
sider transactions, Smallbank [10] is a popular benchmark
Specifically, we implement two queries for extracting statis-
for OLTP workload. It consists of three tables and four ba-
tics from the blockchain data:
sic procedures simulating basic operations on bank accounts.
We implement it as a smart contract which simply transfers Q1: Compute the total transaction values committed be-
money from one account to another. tween block i and block j.
EtherId. This is a popular contract that implements a Q2: Compute the largest transaction value involving a given
domain name registrar. It supports creation, modification state (account) between block i and block j.
and ownership transfer of domain names. A user can re-
quest an existing domain by paying a certain amount to the In ClientWorkload, we pre-load the blockhain with trans-
current domain’s owner. This contract has been written for actions carrying integer values (representing money trans-
Ethereum blockchain, and can be ported to Parity without ferring) and the states with integer values. For Ethereum,
change. In Hyperledger, we create two different key-value both queries can be implemented via JSON-RPC APIs that
namespaces in the contract: one for storing the domain name return transaction details and account balances at a specific
data structures, and another for users’ account balances. In block. For Hyperledger, however, the second query must
domain creation, the contract simply inserts domain value be implemented via a chaincode (VersionKVStore), because
into the first name space, using the domain name as the the system does not have APIs to query historical states.
key. For ownership transfer, it checks the second namespace IOHeavy. Current blockchain systems rely on key-value
if the requester has sufficient fund before updating the first storage to persist blockchain transactions and states. Each
namespace. To simulate real workloads, the contract con- storage system may perform differently under different work-
tains a function to pre-allocate user accounts with certain loads [51]. This workload is designed to evaluate the IO per-
balances. formance by invoking a contract that performs a large num-
Doubler. This is a contract that implements a pyramid ber of random writes and random reads to the contract’s
scheme. As shown in Figure 2, participants send money states. The I/O bandwidth can be estimated via the ob-
to this contract, and get rewards as more people join the served transaction latency.
scheme. In addition to the list of participants and their con- CPUHeavy. This workload measures the efficiency of
tributions, the contract needs to keep the index of the next the execution layer for computationally heavy tasks. EVM
payout and updates the balance accordingly after paying may be fast at executing Ethereum specific operations, but
early participants. Similar to EthereId, this contract has it is unclear how it performs on general tasks for which ma-
already been written for Ethereum, and can be ported to chine native codes may be more efficient. We deploy a smart
Parity directly. To implement it in Hyperledger, we need to contract which initializes a large array, and runs the quick
translate the list operations into key-value semantics, mak- sort algorithm over it. The execution layer performance can
ing the chaincode more bulky than the Ethereum counter- then be measured by the observed transaction latency.
4. PERFORMANCE BENCHMARK 1000 Queue length, 8 tx/s 60000 Queue length, 512 tx/s
We selected Ethereum, Parity and Hyperledger for our Ethereum Ethereum
study, as they occupy different positions in the blockchain Parity 50000 Parity
800 Hyperledger Hyperledger
design space, and also for their codebase maturity. We eval-
uate the three systems using both macro and micro bench- 40000
mark workloads described in the previous section. 600

#requests

#requests
30000
• Hyperledger performs consistently better than Eth- 400
ereum and Parity across the benchmarks. But it fails 20000
to scale up to more than 16 nodes.
200 10000
• Ethereum and Parity are more resilient to node fail-
ures, but they are vulnerable to security attacks that 00 00
forks the blockchain. 50 100 150 200 250 300 50 100 150 200 250 300
time (second) time (second)
• The main bottlenecks in Hyperledger and Ethereum
are the consensus protocols, but for Parity the bottle- Figure 6: Client’s request queue, for request rates of 8 tx/s
neck is caused by transaction signing. and 512 tx/s.

• Ethereum and Parity incur large overhead in terms of

memory and disk usage. Their execution engine is also
less efficient than that of Hyperledger. and Ethereum is because of the difference in consensus pro-
tocol: one is based on PBFT while the other is based on
• Hyperledger’s data model is low level, but its flexibility PoW. We measured CPU and network utilization during the
enables customized optimization for analytical queries experiments, and observe that Hyperledger is communica-
of the blockchain data. tion bound whereas Ethereum is CPU bound (see Appendix
B). At 8 servers, communication cost in broadcasting mes-
We used the popular Go implementation of Ethereum, sages is much cheaper than block mining whose difficulty is
geth v1.4.18, the Parity release v1.6.0 and the Hyperledger set at roughly 2.5s per block.
Fabric release v0.6.0-preview. We set up a private testnet The performance gap between Parity and Hyperledger is
for Ethereum and Parity by defining a genesis block and di- not because of the consensus protocol, as we expect Parity’s
rectly adding peers to the miner network. For Ethereum, we PoA protocol to be simpler and more efficient than both
manually tuned the difficulty variable in the genesis block PoW and PBFT (indeed, we observe that Parity has the
to ensure that miners do not diverge in large networks. For same CPU utilization and lower network utilization than
Parity, we set the stepDuration variable to 1. In both Eth- Hyperledger). Figure 5[b,c] shows that Parity’s throughput
ereum and Parity, confirmationLength is set to 5 seconds. and latency remains constant with increasing transaction
The default batch size in Hyperledger is 500. rates (beyond 40 tx/s). To understand its performance fur-
The experiments were run on a 48-node commodity clus- ther, we measure the queue of pending transactions at the
ter. Each node has an E5-1650 3.5GHz CPU, 32GB RAM, client. Figure 6 compares the queue sizes before and af-
2TB hard drive, running Ubuntu 14.04 Trusty, and con- ter the systems reach their peak throughput. With only 8
nected to the other nodes via 1GB switch. The results below tx/s, the queues for Ethereum and Hyperledger remain at
are averaged over 5 independent runs. For Ethereum, we re- roughly constant sizes, but Parity’s queue size increases as
served 8 cores out of the available 12 cores per machine, so time passes. More interestingly, under high loads (512 tx/s
that the periodical polls from the client’s driver process do per client), Parity’s queue is always smaller than Ethereum’s
not interfere with the mining process (which is CPU inten- and Hyperledger’s. This behavior indicates that Parity pro-
sive). cesses transactions at a constant rate, and that it enforces a
maximum client request rate at around 80 tx/s. As a conse-
4.1 Macro benchmarks quence, Parity achieves both lower throughput and latency
This section discusses the performance of the blockchain than other systems.
systems at the application layer, by running them with the Another observation is that there are differences between
YCSB and Smallbank benchmarks over multiple nodes. YCSB and Smallbank workloads in Hyperledger and Eth-
ereum. There is a drop of 10% in throughput and 20%
4.1.1 Throughput and latency increase in latency. Since executing a Smallbank smart con-
We measured peak performance of the three systems with tract is more expensive than executing a YCSB contract
8 servers and 8 concurrent clients over the period of 5 min- (there are more reading and writing to the blockchain’s states),
utes. Each client sends transactions to a server with a re- the results suggest that there are non-negligible costs in the
quest rate varying from 8 tx/s to 1024 tx/s. Figure 5 shows execution layer of blockchains.
the throughput and latency at peak, and how these metrics At its peak throughput, Hyperledger generates 3.1 blocks
change with varying transaction rates. per second and achieves the overall throughput of 1273 tx/s.
We observe that in terms of throughput, Hyperledger out- We remark that this throughput is far lower than what an in-
performs other systems in both benchmarks. Specifically, it memory database system can deliver (see Appendix B). As
has up to 5.5x and 28x higher throughput than Ethereum the throughput is a function of the block sizes and block gen-
and Parity respectively. Parity has the lowest latency and eration rate, we measured the effect of increasing the block
Ethereum has the highest. The gap between Hyperledger sizes in the three systems. The results (see Appendix B)
 Throughput Latency Throughput Latency
Ethereum Hyperledger Ethereum Hyperledger
Ethereum Ethereum Parity Parity
3
104 Parity 10 Parity
Hyperledger Hyperledger 103 102
92 114
1273 1122 102 51

second
38

#tx/s
3

second
10
#tx/s

284 101 102

255
4
101
3
102
45 46 100
101
0 200 400 600 8001000 1000 200 400 600 8001000
101 YCSB Smallbank 10-1 YCSB Smallbank #request/s #request/s
(a) Peak performance (b) Performance with varying request rates

Figure 5: Blockchain performance with 8 clients and 8 servers.

Throughput Latency 1400 Throughput 180 Latency

104 Ethereum
Ethereum Ethereum 1200 160 Parity
Parity 103 Parity 140 Hyperledger
Hyperledger Hyperledger 1000
120
103 10 2 800 Ethereum 100

second
#tx/s
Parity
second
#tx/s

600 Hyperledger 80
1
10 400
60
102 40
200
100 20
08 12 16 20 24 28 32 0 8 12 16 20 24 28 32
#nodes #nodes
1 -1
10 124 8 12162024283210 124 8 121620242832
#nodes #nodes Figure 8: Performance scalability (with 8 clients).
Figure 7: Performance scalability (with the same number of
clients and servers).
sus messages are rejected by other peers on account of the
message channel being full. As messages are dropped, the
demonstrate that with bigger block sizes, the block genera- views start to diverge and lead to unreachable consensus.
tion rate decreases proportionally, thus the overall through- In fact, we also observe that as time passes, client requests
put does not improve. took longer to return (see Appendix B), suggesting that the
servers were over saturated in processing network messages.
4.1.2 Scalability We note, however, that the original PBFT protocol guaran-
We fixed the client request rate and increased both the tees both liveness and safety, thus Hyperledger’s failure to
number of clients and the number of servers. Figure 7 il- scale beyond 16 servers is due to the implementation of the
lustrates how well the three systems scale to handle larger protocol. In fact, in the latest codebase (which was updated
YCSB workloads (the results for Smallbank are similar and after we have finished our benchmark), the PBFT compo-
included in Appendix B). Parity’s performance remains con- nent was replaced by another implementation. We plan to
stant as the network size and offered load increase, due to evaluate this new version in the future work.
the constant transaction processing rate at the servers. In- The results so far indicate that scaling both the number of
terestingly, while Ethereum’s throughput and latency de- clients and number of servers degrades the performance and
grade almost linearly beyond 8 servers, Hyperledger stops even causes Hyperledger to fail. We next examined the costs
working beyond 16 servers. of increasing the number of servers alone while fixing the
To understand why Hyperledger failed to scale beyond number of clients. Figure 8 shows that the performance be-
16 servers and 16 clients, we examined the system’s logs comes worse as there are more servers, meaning that the sys-
and found that the nodes were repeatedly trying and failing tems incur some network overheads. Because Hyperledger
to reach consensus on new views which contain batches of is communication bound, having more servers means more
transactions. In fact, the servers were in different views and messages being exchanged and higher overheads. For Eth-
consequently were receiving conflicting view change mes- ereum, even though it is computation bound, it still con-
sages from the rest of the network. Further investigation sumes a modest amount of network resources for propagat-
reveals that conflicting views occurred because the consen- ing transactions and blocks to other nodes. Furthermore,
 1800 #Transactions commited 104 Execution time Peak memory usage
1600 Ethereum-12 Parity-12 Hyperledger-12 Ethereum Ethereum
Ethereum-16 Parity-16 Hyperledger-16 Parity Parity
1400 103
Hyperledger 232.78 108 Hyperledger
1200 2
1000 10 79.61
22,819
#txs

24.04 13,090

second
800 101 10.52 107

MB
600 3.01 1.94 4,150
400 2,078
100 1,353
0.33 106
200 0.19 718
473
-1
00 10 376
50 100 150 200 250 300 350 400
time (second) x x
10-2 1M 10M 100M 105 1M 10M 100M
input size input size
Figure 9: Failing 4 nodes at 250th second (fixed 8 clients)
for 12 and 16 servers. X-12 and X-16 mean running 12 and
Figure 11: CPUHeavy workload, ‘X’ indicates Out-of-
16 servers using blockchain X respectively.
Memory error.

500 #Blocks generated

Ethereum-bc Parity-bc Hyperledger-bc which were caused by the remaining servers having to sta-
Ethereum-total Parity-total Hyperledger-total
400 bilize the network after the failures by synchronizing their
views.
300 We next simulated the attack that renders the blockchain
#blocks

vulnerable to double spending. The attack, described in

200 Section 3.3, partitioned the network at 100th second and
lasted for 150 seconds. We set the partition size to be half
100 of the original1 . Figure 10 compares the vulnerability of
00 the three systems running with 8 clients and 8 servers. Re-
50 100 150 200 250 300 350 400 call that vulnerability is measured as the differences in the
time (second)
number of total blocks and the number of blocks on the
main branch (Section 3.3), we refer to this as ∆. Both Eth-
Figure 10: Blockchain forks caused by attacks that parti- ereum and Parity blockchains fork at 100th seconds, and ∆
tions the network in half at 100th second and lasts for 150 increases as time passes. For the attack duration, upto 30%
seconds. X-total means the total number of blocks generated of the blocks are generated in the forked branch, meaning
in blockchain X, X-bc means the total number of blocks that that the systems are highly exposed to double spending or
reach consensus in blockchain X. selfish mining attacks. When the partition heals, the nodes
come to consensus on the main branch and discard the forked
blocks. As a consequence, ∆ stops increasing shortly after
with larger network, the difficulty is increased to account for 250th second. Hyperledger, in stark contrast, has no fork
the longer propagation delays. We observe that to prevent which is as expected because its consensus protocol is proven
the network from diverging, the difficulty level increases at to guaranteed safety. We note, however, that Hyperledger
higher rate than the number of nodes. Thus, one reason takes longer than the other two systems to recover from the
for Ethereum’s throughput degradation is due to network attacks (about 50 seconds more). This is because of the syn-
sizes. Another reason is that in our settings, 8 clients send chronization protocol executed after the partitioned nodes
requests to only 8 servers, but these servers do not always reconnect.
broadcast transactions to each other (they keep mining on
their own transaction pool). As a result, the network mining 4.2 Micro benchmarks
capability is not fully utilized. This section discusses the performance of the blockchain
system at execution, data and consensus layers by evaluat-
4.1.3 Fault tolerance and security ing them with micro benchmark workloads. For the first
To evaluate how resilient the systems are to failures by two layers, the workloads were run using one client and one
crashing, we ran the systems with 8 clients for over 5 min- server. For the consensus layer, we used 8 clients and 8
utes, during which we killed off 4 servers at 250th second. servers.
Figure 9 shows that Ethereum is nearly unaffected by the
change, suggesting that the failed servers do not contributing 4.2.1 Execution layer
significantly to the mining process. In Parity, each node gen- We deployed the CPUHeavy smart contract that is ini-
erates blocks at a constant rate, thus failing 4 nodes means tialized with an integer array of a given size. The array is
the remaining nodes are given more time to generate more initialized in descending order. We invoked the contract to
blocks, therefore the overall throughput is unaffected. In sort the array using quicksort algorithm, and measured the
contrast, the throughput drops considerably in Hyperledger. execution time and server’s peak memory usage. The results
For 12 servers, Hyperledger stops generating blocks after the 1
We note that partitioning a N -node network in half does
failure, which is as expected because the PBFT can only tol- not mean there are N/2 Byzantine nodes. In fact, Byzan-
erate fewer than 4 failures in a 12-server network. With 16 tine tolerance protocols do not count network adversary as
servers, the system still generated blocks but at a lower rate, Byzantine failure
for varying input sizes are shown in Figure 11. Although In particular, for Ethereum we observe 10% increases in
Ethereum and Parity use the same execution engine, i.e. throughput as compared to YCSB, which means that execu-
EVM, Parity’s implementation is more optimized, therefore tion of the YCSB transaction accounts for the 10% overhead.
it is more computation and memory efficient. An interest- We observe no differences among these workloads in Parity,
ing finding is that Ethereum incurs large memory overhead. because the bottleneck in Parity is due to transaction sign-
In sorting 10M elements, it uses 22GB of memory, as com- ing (even empty transactions still need to be signed), not
pared to 473MB used by Hyperledger. Ethereum runs out of due to consensus or transaction execution.
memory when sorting more than 10M elements. In Hyper-
ledger, the smart contract is compiled and runs directly on 5. DISCUSSION
the native machine within Docker environment, thus it does Understanding blockchain systems. Our framework is
not have the overheads associated with executing high-level designed to provide better understanding of the performance
EVM byte code. As the result, Hyperledger is much more and design of different private blockchain systems. As more
efficient in term of speed and memory usage. Finally, we and more blockchain systems are being proposed, each of-
note that all three systems fail to make use of the multi-core fering different sets of feature, BLOCKBENCH’s main value
architecture, i.e. they execute the contracts using only one is that it narrows down the design space into four distinct
core. abstraction layers. Our survey of current blockchain sys-
tems (see Appendix A) shows that the four layers are suffi-
4.2.2 Data model cient to capture the key characteristics of these systems. By
IO Heavy. We deployed the IOHeavy smart contract benchmarking these layers, one can gain insights into the de-
that performs a number of read and write operations of sign trade-offs and performance bottlenecks. In this paper,
key-value tuples. We used 20-byte keys and 100-byte val- for example, by running the IOHeavy workload we identify
ues. Figure 12 reports the throughput and disk usage for that Parity trades performance for scalability by keeping
these operations. Ethereum and Parity use the same data states in memory. Another example is the trade-off in data
model and internal index structure, therefore they incur sim- model made by Hyperledger. On the one hand, the sim-
ilar space overheads. Both use an order of magnitude more ple key-value model means some analytical queries cannot
storage space than Hyperledger which employs a simple key- be directly supported. On the other hand, it enables opti-
value data model. Parity holds all the state information in mization that helps answering the queries more efficiently.
memory, so it has better I/O performance but fails to han- Finally, we identify that the bottleneck in Parity is not due
dle large data (capped by over 3M states under our hard- to the consensus protocol, but due to the server’s transac-
ware settings). On the contrary, Ethereum only caches only tion signing. We argue that such insights are not easy to
parts of the state in memory (using LRU for eviction policy), extract without a systematic analysis framework.
therefore it can handle more data than Parity at the cost of
throughput. Hyperledger leverages RocksDB to manage its Usability of blockchain. Our experience in working with
states, which makes it more efficient at scale. the three blockchain systems confirms the belief that in its
Analytic Queries. We implemented the analytics work- current state blockchain are not yet ready for mass usage.
load by initializing the three systems with over 120, 000 ac- Both their designs and codebases are still being refined con-
counts with a fixed balance. We then pre-loaded them with stantly, and there are no other established applications be-
100, 000 blocks, each contains 3 transactions on average. side crypto-currency. Of the three systems, Ethereum is
The transaction transfers a value from one random account more mature both in terms of its codebase, user base and de-
to another random account. Due to Parity’s overheads in veloper community. Another usability issue we encountered
signing transactions when there are many accounts, we con- is in porting smart contracts from one system to another, be-
sidered transactions using only 1024 accounts. We then exe- cause of their distinct programming models (see Section 3).
cuted the two queries described in Section 3.4 and measured This is likely to be exacerbated as more blockchain platforms
their latencies. Figure 13 shows that the performance for Q1 are being proposed [44, 16].
is similar, whereas Q2 sees a significant gap between Hyper- Bringing database designs into blockchain. The chal-
ledger and the rest. We note that the main bottleneck for lenge in scaling blockchain by improving its consensus proto-
both Q1 and Q2 is the number of network (RPC) requests cols is being addressed in many recent works [34, 37]. How-
sent by the client. For Q1, the client sends the same num- ever, as we demonstrated in the previous section, there are
ber of requests to all systems, therefore their performance other performance bottlenecks. We propose four approaches
are similar. On the other hand, for Q2 the client sends one in applying design principles from database systems to im-
RPC per block to Ethereum and Parity, but only one RPC prove blockchain.
to Hyperledger because of our customized smart contract Decouple storage, execution engine and consensus layer
implementation (see Appendix C). This saving in network from each other, then optimize and scale them independently.
roundtrip time translates to over 10x improvement in Q2 For instance, current systems employ generic key-value stor-
latency. age, which may not be best suited to the unique data struc-
ture and operations in blockchain. UStore [19] demonstrates
4.2.3 Consensus that a storage designed around the blockchain data struc-
We deployed the DoNothing smart contract that accepts ture is able to achieve better performance than existing im-
a transaction and returns immediately. We measured the plementations.
throughput of this workload and compare against that of Embrace new hardware primitives. Many data processing
YCSB and Smallbank. The differences compared to other systems are taking advantage of new hardware to boost their
workloads, shown in Figure 13[c] is indicative of the cost performance [47, 51, 21]. For blockchain, using trusted hard-
of consensus protocol versus the rest of the software stack. ware, the underlying Byzantine fault tolerance protocols can
 Average throughput (write) Average throughput (read) Disk usage
104 Ethereum Parity Hyperledger Ethereum Parity Hyperledger 105 Ethereum Parity Hyperledger
5618 5411 5123 4852 4527
10 4 9329 8785 9234 8758 8974 8700 8670 8605 30,221
12,804 12,104
1329 1181 1087 104
second

second
103 5,459 5,045 4,865

MB
377 359 337 2019 2,337 2,086 2,477
1631
1282 1,283
103 103 675
102 69 360
512
0.8M 1.6M 3.2M
x
6.4M
x
12.8M 0.8M 1.6M 3.2M
x
6.4M 12.8M
x 0.8M 1.6M 3.2M 6.4M
x x
12.8M
# tuples # tuples # tuples

(a) Write (b) Read (c) Disk usage

Figure 12: IOHeavy workload, ‘X’ indicates Out-of-Memory error.

Latency 13.314 101 Latency 104 Transaction througput

101 Ethreum 8.9018.465
Ethereum 4.907 SmallBank
3.472
Parity Parity YCSB
Hyperledger Hyperledger DoNothing 112212731285
1.374
100 103
100 0.9150.984 0.595
0.427
0.533
second

second

256 284 328

#tx/s
0.168
0.1290.135 -1 0.107 0.091
10 -1 10 0.076 102
0.0320.034
0.038 0.033 0.031 45 45 46
0.025 0.024 0.023
0.0200.0200.019 0.019 0.020
-2 -2
10 1 10 100 1,000 10,000 10 1 10 100 1,000 10,000
# blocks scanned # blocks scanned 101 Ethereum Parity Hyperledger

(a) Analytics workload (Q1) (b) Analytics workload (Q2) (c) DoNothing workload

Figure 13: Analytics and DoNothing workloads.

be modified to incur fewer network messages [12]. Systems time (for syncing with other nodes) and transactions pro-
like Parity and Ethereum can take advantage of multi-core cessing time, have also been benchmarked [24, 23]. Our anal-
CPUs and large memory to improve contract execution and ysis using BLOCKBENCH differs from these works in that
I/O performance. it is the first to evaluate private blockchains systems at scale
Sharding. Blockchain is essentially a replicated state ma- against database workloads. Furthermore, it compares two
chine system, in which each node maintains the same data. different systems and analyzes how their designs affect the
As such, blockchains are fundamentally different to database overall performances. Future extensions of BLOCKBENCH
systems such as H-Store in which the data is partitioned (or would enable more comparative evaluations of the key com-
sharded) across the nodes. Sharding helps reduce the com- ponents in blockchain.
putation cost and can make transaction processing faster. There are many standard frameworks for benchmarking
The main challenge with sharding is to ensure consistency database systems. OLTP-Bench [18] contains standard work-
among multiple shards. However, existing consistency pro- loads such as TPC-C for transactional systems. YCSB [13]
tocols used in database systems do not work under Byzan- contains key-value workloads. HiBench [32] and BigBench [28]
tine failure. Nevertheless, their designs can offer insights feature big-data analytics workloads for MapReduce-like sys-
into realizing a more scalable sharding protocol for block- tems. BLOCKBENCH shares the same high-level design as
chain. Recent work [37] has demonstrated the feasibility these frameworks, but its workloads and main driver are
of sharding the consensus protocol, making important steps designed specifically for blockchain systems.
towards partitioning the entire blockchain.
Support declarative language. Having a set of high-level 7. CONCLUSION
operations that can be composed in a declarative manner
In this paper we proposed the first benchmarking frame-
makes it easy to define complex smart contracts. It also
work, called BLOCKBENCH, for evaluating private block-
opens up opportunities for low-level optimizations that speed
chain systems. BLOCKBENCH contains workloads for mea-
up contract execution.
suring the data processing performance, and workloads for
understanding the performance at different layers of the
6. RELATED WORK blockchain. Using BLOCKBENCH, we conducted compre-
Performance studies of blockchain systems have so far hensive analysis of three major blockchain systems, namely
been restricted to public blockchains. For example, [17, 15] Ethereum, Parity and Hyperledger with two macro bench-
analyze the effect of block sizes and network propagation marks and four micro benchmarks. The results showed that
time on the overall throughputs. Recent proposals for im- current blockchains are not well suited for large scale data
proving Bitcoin performance [27, 34, 37, 25, 43] have mainly processing workloads. We demonstrated several bottlenecks
focused on the consensus layer, in which analytical models and design trade-offs at different layers of the software stack.
or network simulations are used to validate the new designs.
Various aspects of Ethereum, such as their block processing
Acknowledgment [19] A. Dinh, J. Wang, S. Wang, W.-N. Chin, Q. Lin,
We would like to thank the anonymous reviewers for their B. C. Ooi, P. Ruan, K.-L. Tan, Z. Xie, H. Zhang, and
comments and suggestions that help us improve the paper. M. Zhang. UStore: a distributed storage with rich
Special thanks to Hao Zhang, Loi Luu, the developers from semantics. https://arxiv.org/pdf/1702.02799.pdf.
Ethereum, Parity and Hyperledger projects for helping us [20] J. Douceur. The sybil attack. In IPTPS, 2002.
with the experiment setup. This work is funded by the Na- [21] A. Dragojevic, D. Narayanan, E. B. Nightingale,
tional Research Foundation, Prime Minister’s Office, Sin- M. Renzelmann, A. Shamis, A. Badam, and
gapore, under its Competitive Research Programme (CRP M. Castro. No compromises: distributed transactions
Award No. NRF-CRP8-2011-08). Gang Chen is supported with consistency, availability and performance. In
by the National Natural Science Foundation of China (Grant SOSP, 2015.
No. 61472348). [22] Ethcore. Parity: next generation ethereum browser.
https://ethcore.io/parity.html.
8. REFERENCES [23] Ethcore. Performance analysis.
[1] BlockBench: private blockchains benchmarking. https://blog.ethcore.io/performance-analysis/.
https://github.com/ooibc88/blockbench. [24] Ethereum. Ethereum benchmarks.
[2] Ethereum blockchain app platform. https://github.com/ethereum/wiki/wiki/Benchmarks.
https://www.ethereum.org/. [25] I. Eyal, A. E. Gencer, E. G. Sirer, and R. van Renesse.
[3] Ibm watson iot. Bitcoin-ng: A scalable blockchain protocol. In NSDI,
http://www.ibm.com/internet-of-things. 2016.
[4] Leveldb. https://leveldb.org. [26] I. Eyal and E. G. Sirer. Majority is not enough:
[5] Monax: The ecosystem application platform. Bitcoin mining is vulnerable. In Fiancial
https://monax.io. Cryptography, 2014.
[6] Rocksdb. https://rocksdb.org. [27] A. Gervais, G. O. Karame, K. Wust, V. Glykantizis,
H. Ritzdorf, and S. Capkun. On the security and
[7] M. Apostolaki, A. Zohar, and L. Vanbever. Hijacking
performance of proof of work blockchains.
bitcoin: Large-scale network attacks on
https://eprint.iacr.org/2016/555.pdf.
crypto-currencies. https://arxiv.org/abs/1605.07524,
2016. [28] A. Ghazal, T. Rabl, M. Hu, F. Raab, M. Poess,
A. Crolotte, and H.-A. Jacobsen. Bigbench: towards
[8] P. Bailis, A. Fekete, M. J. Franklin, A. Ghodsi, J. M.
an industry standard benchmark for big data
Hellerstein, and I. Stoica. Coordination avoidance in
analytics. In SIGMOD, 2013.
database systems. In VLDB, 2014.
[29] G. S. Group. Blockchain: putting theory into practice,
[9] J. Bonneau, A. Miller, J. Clark, A. Narayanan, J. A.
2016.
Kroll, and E. W. Felten. Sok: Research perspectives
and challenges for bitcoin and crypto-currencies. In [30] E. Heilman, A. Kendler, A. Zohar, and S. Goldberg.
2015 IEEE Symposium on Security and Privacy, pages Eclipse attacks on Bitcoin’s peer-to-peer network. In
104–121. IEEE, 2015. USENIX Security, 2015.
[10] M. Cahill, U. Rohm, and A. D. Fekete. Serializable [31] Hyperledger. Blockchain technologies for business.
isolation for snapshot databases. In SIGMOD, 2008. https://www.hyperledger.org.
[11] M. Castro and B. Liskov. Practical byzantine fault [32] Intel. Hibench suite.
tolerance. In Proceedings of the third symposium on https://github.com/intel-hadoop/HiBench.
Operating systems design and implementation, pages [33] F. P. Junqueira, B. C. Reed, and M. Serafini. Zab:
173–186. USENIX Association, 1999. high-performance broadcast for primary-backup
[12] B.-G. Chun, P. Maniatis, S. Shenker, and systems. In Dependable Systems and Networks, 2011.
J. Kubiatowicz. Attested append-only memory: [34] E. Kokoris-Kogias, P. Jovanovic, N. Gailly, I. Khoffi,
Making adversaries stick to their word. In SOSP, 2007. L. Gasser, and B. Ford. Enhancing bitcoin security
[13] B. F. Cooper, A. Silberstein, E. Tam, and performance with strong consistency via collective
R. Ramakrishnan, and R. Sears. Benchmarking cloud signing. In USENIX Security, 2016.
serving systems with ycsb. In SoCC, 2010. [35] L. Lamport. Paxos made simple. SIGACT News, 2001.
[14] J. C. Corbett and J. D. et al. Spanner: Google’s [36] Q. Lin, P. Chang, G. Chen, B. C. Ooi, K.-L. Tan, and
globally-distributed database. In OSDI, 2012. Z. Wang. Towards a non-2pc transaction management
[15] K. Croman, C. Decker, I. Eyal, A. E. Gencer, A. Juels, in distrubted database systems. In SIGMOD, 2016.
A. Kosba, A. Miller, P. Saxena, E. Shi, and E. Gün. [37] L. Luu, V. Narayanan, C. Zhang, K. Baweija,
On scaling decentralized blockchains. In Proc. 3rd S. Gilbert, and P. Saxena. A secure sharding protocol
Workshop on Bitcoin and Blockchain Research, 2016. for open blockchains. In CCS, 2016.
[16] Crypti. A decentralized application platform. [38] L. Luu, J. Teutsch, R. Kulkarni, and P. Saxena.
https://crypti.me. Demystifying Incentives in the Consensus Computer.
[17] C. Decker and R. Wattenhofer. Information CCS ’15, pages 706–719, 2015.
propagation in bitcoin network. In P2P, 2013. [39] Melonport. Blockchain software for asset management.
[18] D. E. Difallah, A. Pavlo, C. Curino, and http://melonport.com.
P. Cudre-Mauroux. Oltp-bench: An extensible testbed [40] J. Morgan and O. Wyman. Unlocking economic
for benchmarking relational databases. In VLDB, advantage with blockchain. a guide for asset
2013. managers., 2016.
 Throughput vs. HStore (TEE) such as Intel Software Guard Extensions (SGX). These
Ethereum 142702 platforms also support different languages to develop smart
105 Parity contracts. For example, Solidity, Serpent and LLL are mainly
Hyperledger 21596 used in Ethereum, Dfinity and Parity, while Eris-DB only
104 H-Store supports Solidity. Hyperledger, Stellar, Corda and Sawtooth
#tx/s

1273 Lake exploit various mature programming languages, such

1122
103 as Python, Java, Golang, etc. ScalableBFT and Tezos even
284 255 develop their own smart contract languages. Most block-
102 45 46 chain platforms’ data models are account-based. Two ex-
ceptions in the table are Ripple and Corda. Their data
101 YCSB Smallbank models are similar to Bitcoin’s unspent transaction outputs
(UTXO) which represents the coins in the network.
Each platform offers different consensus protocols. Hy-
Figure 14: Performance of the three blockchain systems ver- perledger implements PBFT in the version we evaluated,
sus H-Store. while Ethereum implements a variation of PoW (Proof-of-
Work). Eris-DB builds on top of Tendermint protocol but
only works in the latest version (v 0.12). Ripple and Tezos
[41] S. Nakamoto. Bitcoin: A peer-to-peer electronic cash deploy Proof-of-Stake (PoS) schemes (the one in Ripple is
system, 2008. referred to Ripple Consensus Ledger) where the next block
[42] D. Ongaro and J. Ousterhout. In search of an is created based on accounts’ wealth, i.e., the stake. Parity
understandable consensus algorithm. In USENIX takes another consensus protocol, Proof-of-Authority (PoA),
ATC, 2014. which holds a predefined set of ”authorities” to create new
[43] R. Pass and E. Shi. Hybrid consensus: efficient blocks in a fixed time slot and secure the blockchain network.
consensus in the permissionless model. Sawtooth Lake uses Proof-of-Elapsed-Time (PoET) as its
https://eprint.iacr.org/2016/917.pdf. consensus protocol, which in nature is a lottery algorithm
[44] Ripple. Ripple. https://ripple.com. and decides the creator of block arbitrarily. Stellar develops
[45] Y. Sompolinsky and A. Zohar. Accelerating bitcoin’s its own mechanism, Stellar Consensus Protocol, which is a
transaction processing: fast money grows on trees, not construction for decentralized Byzantine agreement. There
chains. Cryptology ePrint Archive, Report 2013/881, is no source code that helps determine which consensus pro-
2013. https://eprint.iacr.org/2013/881.pdf. tocol Dfinity uses, but its documents suggest that a Block-
[46] M. Stonebraker, S. Madden, D. J. Abadi, chain Nervous System will govern the whole platform via a
S. Harizopoulos, N. Hachem, and P. Helland. The end voting mechanism based on neurons that interact with each
of and architectural era (it’s time for a complete other and are controlled by users.
rewrite). In VLDB, 2007.
[47] K.-L. Tan, Q. Cai, B. C. Ooi, W.-F. Wong, C. Yao, B. MACRO BENCHMARKS
and H. Zhang. In-memory databases: Challenges and We compared the performance of the three blockchain sys-
opportunities from software and hardware tems against a popular in-memory database system, namely
perspectives. SIGMOD Records, 44(2), 2015. H-Store, using the YCSB and Smallbank workload. We ran
[48] A. Thomson, T. Diamond, S. chun Weng, K. Ren, H-Store’s own benchmark driver and set the transaction rate
P. Shao, and D. J. Abadi. Calvin: fast distributed at 100,000 tx/s. Figure 14 shows at least an order of mag-
transaction for partitioned database systems. In nitude gap in throughput and two order of magnitude in la-
SIGMOD, 2012. tency. Specifically, H-Store achieves over 140K tx/s through-
[49] Q. H. Vu, M. Lupu, and B. C. Ooi. Peer-to-Peer put while maintaining sub-millisecond latency. The gap in
Computing Principles and Applications. performance is due to the cost of consensus protocols. For
Springer-Verlag, 2009. YCSB, for example, H-Store requires almost no coordination
[50] M. Vukolic. The quest for scalable blockchain fabric: among peers, whereas Ethereum and Hyperledger suffer the
proof-of-work vs. bft replication. In Open Problems in overhead of PoW and PBFT.
Network Security - iNetSec, 2015. An interesting observation is the overhead of Smallbank.
[51] H. Zhang, G. Chen, B. C. Ooi, K.-L. Tan, and Recall that Smallbank is a more complex transactional work-
M. Zhang. In-memory big data management and load than YCSB, in which multiple keys are updated in a
processing: a survey. TKDE, 2015. single transaction. Smallbank is simple but is representative
of the large class of transactional workloads such as TPC-C.
APPENDIX We observe that in H-Store, Smallbank achieves 6.6x lower
throughput and 4x higher latency than YCSB, which indi-
A. SURVEY OF BLOCKCHAIN PLATFORMS cates the cost of distributed transaction management proto-
We compare eleven promising blockchain platforms in Ta- col, because H-Store is a sharded database. In contrast, the
ble 2. We can see that all but Ripple support smart con- blockchain suffers modest degradation in performance: 10%
tracts. Ethereum, Eris-DB, Dfinity and Parity execute the in throughput and 20% in latency. This is because each node
contracts using Ehtereum Virtual Machine (EVM), whereas in blockchain maintains the entire state (replicated state ma-
Corda runs them in Java Virtual Machine (JVM). Hyper- chine), thus there is no overhead in coordinating distributed
ledger, Stellar and Tezos employ Docker images, ScalableBFT transactions as the data is not partitioned.
takes Haskell execution environment, and Sawtooth Lake The results demonstrate that blockchain performs poorly
launches contracts on top of Trusted Execution Environment at data processing tasks currently handled by database sys-
 Table 2: Comparison of blockchain platforms

Smart
Smart contract
Application contract Data model Consensus
language
execution
Hyperledger Smart contract Dockers Golang, Java Account-based PBFT
Smart contract,
Ethereum Crypto- EVM Solidity, Serpent, LLL Account-based Ethash (PoW)
currency
Eris-DB Smart contract EVM Solidity Account-based Tendermint (BFT)
Crypto- Ripple Consensus
Ripple - - UTXO-based
currency Ledger (PoS)
Haskell
ScalableBFT Smart contract Pact Account-based ScalableBFT
Execution
JavaScript, Golang, Stellar Consensus
Stellar Smart contract Dockers Account-based
Java, Ruby, Python, C# Protocol
Blockchain
Dfinity Smart contract EVM Solidity, Serpent, LLL Account-based
Nervous System
Parity Smart contract EVM Solidity, Serpent, LLL Account-based Proof of Authority
Smart contract,
Tezos Contract Script
Tezos Crypto- Dockers Account-based Proof of Stake
Language
currency
Corda Smart contract JVM Kotlin, Java UTXO-based Raft
Sawtooth Proof of Elapsed
Smart contract TEE Python Account-based
Lake Time

90 CPU utilization only crash failures. Other features which are optional in
80
70 most database systems are cryptographic signatures on ev-
60 Ethereum
50 Parity ery single transaction, and wide-area fully replicated state
40
%

30 Hyperledger machines. Although databases are designed without security

20
10 features and tolerance to Byzantine failures, we remark that
00 20 40 60 80 100
time (second) the gap remains too high for blockchains to be disruptive
100 Network utilization to incumbent database systems. Nevertheless, the popular-
80 Ethereum ity of blockchain is a clear indication that there is a need
60 Parity
for a Byzantine tolerant data processing systems which can
Mbps

Hyperledger
40 accommodate a large number of users.
20
00 Figure 15 shows the effect of varying block sizes in the
20 40 60 80 100
time (second) overall throughput. While it is straightforward to set the
block size in Hyperledger by configuring the batchSize vari-
Figure 16: Resource utilization. able, there is no direct way to specify the same in Ethereum.
An Ethereum miner uses gasLimit value to restrict the over-
all cost in constructing a block, thus we tuned this value to
101 Block generation rate simulate different sizes. In Parity, gasLimit is not applica-
ble to local transaction and it has no effect on the block size.
Small 5.20
Instead, we observe that the block size can be controlled by
Medium 3.10
tuning stepDuration value, which essentially decides how
Large 1.75 much time a validator can use to build a block. In the exper-
#block/s

100 1.00 iments, medium size refers to the default settings, whereas
0.56 large and small refer to 2x and 0.5x of the default size. The
0.34 0.28
results show that increases in block sizes lead to proportional
0.22 decreases in block generation rate, meaning that the overall
0.12 throughput does not improve.
10-1 Ethereum Parity Hyperledger Figure 16 compares CPU and network utilization of the
three systems over the period of 100 seconds. It is easy to
see that Ethereum is CPU bound, as it fully utilizes 8 CPU
Figure 15: Block generation rate. cores. Hyperledger, on the other hand, uses CPU sparingly
and spends the rest of the time on network communication.
tems. However, we stress that blockchains and databases Parity, in contrast, has lower resource footprints than other
are designed with different goals and assumptions. Specif- two systems. For Ethereum and Hyperledger, the pattern
ically, the protocols for Byzantine failure tolerance are an is the direct consequence of the consensus protocol: PoW is
overkill for traditional database settings where there are CPU bound whereas PBFT is communication bound.
 1.0 Latency distribution terns to the YCSB benchmark (Figure 7), except that Hy-
perledger failed to scale beyond 8 nodes instead of 16.
0.8
C. ANLAYTICS SMART CONTRACT
0.6
cdf

type account_t struct {

0.4 Ethereum-YCSB
Ethereum-SB Balance int
Parity-YCSB CommitBlock int
0.2 Parity-SB }
Hyperledger-YCSB type transaction_t {
Hyperledger-SB
0.0
100 101 102 103
From string
To string
time (second) Val int
}
Figure 17: Latency distribution. func Invoke_SendValue(from_account string,
to_account string, value int) {
var pending_list []transaction_t
80000 Queue length pending_list = decode(GetState("pending_list"))
70000 Ethereum var new_txn transaction_t
new_txn = transaction_t {
60000 Parity from_account, to_account, value
50000 Hyperledger }
#requests

pending_list = append(pending_list, new_txn)

40000 PutState(’pending_list’, encode(pending_list))
30000 }
20000 func Query_BlockTransactionList(block_number int)
10000 []transaction_t {
return decode(GetState("block:"+block_number))
00 50 100 150 200 250 300 350 }
time (second) func Query_AccountBlockRange(account string,
start_block int, end_block int)
[]account_t {
Figure 18: Queue length at the client. version := decode(GetState(account+":latest"))
var ret []account_t
while true {
Figure 17 shows the latency distribution. Ethereum has var acc account_t
both higher latency and higher variance, because PoW is acc = decode(GetState(account+":"+version))
a randomized process which means the duration between if acc.CommitBlock >= start_block &&
blocks are unpredictable. Parity has the lowest variance acc.CommitBlock < end_block {
because the server restricts the client request rate at 80 tx/s. ret = append(ret, acc)
} else if acc.CommitBlock < start_block {
Figure 18 illustrates the request queue at the client for break;
the settings of 20 servers and 20 clients. The queue behav- }
ior of Ethereum reflects the normal case, i.e. the queue grew version -= 1
and shrank depending on how fast the transactions are com- }
mitted. Hyperledger failed to generate blocks in this case, return ret
therefore the queue never shrank. However, there are du- }
rations in which the queue size remains constant. Further-
more, at the beginning, the queue in Hyperledger is smaller Figure 20: Code snippet from the VersionKVStore smart
than that in Ethereum, even though the clients are send- contract for analytics workload (Q1 and Q2).
ing at the same rate. This suggests there is a bottleneck in
processing network requests at the Hyperledger servers.
Figure 20 shows the implementation of the smart con-
tract method that answer Q2 of the analytics workload. To
104 Throughput Latency support historical data lookup, we append a counter to the
Ethereum Ethereum key of each account. To fetch a specific version of an ac-
Parity 103 Parity count, we use key account:version. We store the latest
Hyperledger Hyperledger
version of the account using key account:latest, and keep
103 102 a CommitBlock in the data field for every version which in-
second
#tx/s

dicates in which block the balance of this version is com-

101 mitted. To answer query that fetches a list of balance of a
102 given account within a given block range, the method scans
100 all versions of this account and returns the balance values
that are committed within the given block range. Ethereum
101 12 4 8 12 16 20 24 28 32 10-1 12 4 8 12 16 20 24 28 32 and Parity provide JSON-PRC APIs getBalance(account,
#nodes #nodes
block) to query information of an account at a given block
number. This API fetches only one version of the account
Figure 19: Scalability with Smallbank benchmark. per HTTP roundtrip, so it is less efficient than pushing the
Figure 19 illustrates the scalability of the three systems query logic to server side.
using the Smallbank benchmark. We observe similar pat-