Blockchain benchmarking framework

Enabling users to deploy, monitor, analyse and compare the behaviour of many different blockchain networks.

The Challenges

Comparing and evaluating the performance of different blockchain implementations.

Understanding their behavior, in the presence of faults or malicious attacks.

Setting up and configuring private deployments of blockchain protocols is complex, and time consuming.

Stress testing the protocols under “close-to-real” conditions is complex.

The Facts

The current blockchain ecosystem is fragmented.

Different implementations of heterogeneous blockchains.

The deployment processes of blockchain protocols:

  • have many dependencies and assume a certain level of technical expertise.
  • are complex and time-consuming.
  • are based on primitive command line deployment /testing scripts.

The Need

Αn open-source, easy to use, holistic, and extendable framework.

A framework that will incorporate technical and non-technical teams.

A framework to deploy, monitor, analyze, compare, and report various blockchain performance activities.

The Solution

A modular framework that aims to:

Automate the deployment processes of various blockchain protocols.

Abstracting the technical details from the end-user.

Analyze, and evaluate the behavior of various consensus proposals under different blockchain network deployments.

Expose the user to a UI, abstracting the complexities and time-consuming configuration processes from the end user.

The BBF Architecture

BBF – User Interface

APIs

BENCHMARKING ENGINE

Network Control & Configuration

Traffic generation

Accounts & wallet management

APIs

MONITORING ENGINE

Monitoring server

Metrics’ exporter

Time series database

Data visualization

APIs

BENCHMARKING ENGINE

Network Control & Configuration

Traffic generation

Accounts & wallet management

APIs

MONITORING ENGINE

Monitoring server

Metrics’ exporter

Time series database

Data visualization

APIs

Blockchains Under Test

XRP LEDGER 

HYPERLEDGER BESU

ethereum

Stellar

The XRPL Case

XRPL EXPERIMENTAL SETUP

  • Nodes: 10 proposer & 8 of them acting as validators
  • Network Topology: Full mesh network
  • Traffic Generation: 1m transactions were submitted to random accounts, exchanging a random amount of XRPs
  • Exposed Metrics: ServerLatency, validation Quorum, loadFactor, Peers, Uptime, serverStatedurationUs, convergeTime and proposers

Supported Networks

Supported Consensus

  • Proof of Work (PoW)
  • Proof of Stake (PoS)
  • XRP Ledger Consensus Protocol
  • Stellar Consensus Protocol
  • Clique

Publications

“Validating the Blockchain Benchmarking Framework through Controlled Deployments of XRPL and Ethereum”, IEEE Access (2024), M. Touloupou et al.

“A Systematic Literature Review Towards a Blockchain Benchmarking Framework.” IEEE Access (2022), M. Touloupou et al.

“Benchmarking Blockchains: The case of XRP Ledger and Beyond.” In Proceedings of the 55th Hawaii International Conference on System Sciences. 2022, M. Touloupou et al.

“Towards a Framework for Understanding the Performance of Blockchains”, BRAINS 2021 Conference, M. Touloupou et al.

Christodoulou, Klitos, Elias Iosif, Antonios Inglezakis, and Marinos Themistocleous. “Consensus crash testing: Exploring ripple’s decentralization degree in adversarial environments.” Future Internet 12, no. 3 (2020): 53.

Medium Article: “The Big Bang of Blockchain Consensus Algorithms: The Case of the XRP Ledger”, K. Christodoulou et al.

Medium Article: “The Big Bang of Blockchain Consensus Algorithms (Part II)”, M. Touloupou et al.

Medium Article: “The Big Bang of Blockchain Consensus Algorithms (Part III)”, M. Touloupou et al.

Acknowledgement

The project is funded by the Ripple’s Impact Fund, an advised fund of Silicon Valley Community Foundation (Grant id: 2018–188546, 2021-244121). The UI module of the project has successfully received funding from the XRP Ledger (XRPL) Developer Program, Wave 2 grants offered by Ripple Labs Inc.