Benchmarking blockchain protocols
What are Blockchain Benchmarking Frameworks?
Blockchain Benchmarking Frameworks (BBFs) comprise of a set of tools designed to assess, compare, and analyse the performance of various blockchain protocols and networks. These benchmarking tools typically offer features such as configurable workloads, metrics visualization, and comparative analysis, enabling users to identify performance bottlenecks, optimize configurations, and make informed decisions when designing and deploying blockchain networks. In essence, such frameworks enable users to deploy blockchain networks in controlled environments, monitor their operations, analyse their behaviour under different scenarios, including normal and stress conditions, and compare their performance across multiple dimensions. BBFs are essential in understanding the efficiency, resilience, scalability, and security of blockchain protocols, offering insights into their operational strengths and weaknesses.
Research Mission
We aim to develop an open-source, easy-to-use, holistic, and extendable BBF that caters to both technical and non-technical users. The proposed framework is designed to facilitate the deployment, monitoring, and analysis of various blockchain networks.
BTC-based Certificates
The Challenges
Comparing and evaluating the performance of different blockchain implementations.
Understanding their behavior, in the presence of faults or malicious attacks.
Setting up and configuring private deployments of blockchain protocols is complex, and time consuming.
Stress testing the protocols under “close-to-real” conditions is complex.
The Facts
The current blockchain ecosystem is fragmented.
Different implementations of heterogeneous blockchains.
The deployment processes of blockchain protocols:
have many dependencies and assume a certain level of technical expertise.
are complex and time-consuming.
are based on primitive command line deployment /testing scripts.
The Need
Αn open-source, easy to use, holistic, and extendable framework.
A framework that will incorporate technical and non-technical teams.
A framework to deploy, monitor, analyze, compare, and report various blockchain performance activities.
The Solution
A modular framework that aims to:
Automate the deployment processes of various blockchain protocols.
Abstracting the technical details from the end-user.
Analyze, and evaluate the behavior of various consensus proposals under different blockchain network deployments.
Expose the user to a UI, abstracting the complexities and time-consuming configuration processes from the end user.
UNIC’s Blockchain Benchmarking Framework
Control and Configuration
This module considers the deployment process of the network (e.g., generation of configurations files, bootstrapping the network, adapting the connectivity between the nodes/validators etc.)
Validators Network
This module enables the dynamic spawn of a blockchain network of n number of nodes that act as validators.
Accounts Management and Traffic Generator
This module is responsible for creating new accounts and injecting traffic (in form of transactions) towards the network participants.
Monitoring Services
This module gathers data and enables different data visualizations by looking at the transactions performed in the network. Furthermore, the module reads data regarding the health of the nodes that participates within then network.
Consensus & UNL Manager
Configure the quorum of validator nodes. For example, in the case of the XRPL, this module provides a web service responsible to serve the Unique Node List of each validator.
Accomplishments
Our team has made significant contributions towards the development of an open-source, modular and standardized Blockchain Benchmarking Framework. Through controlled deployments of blockchain protocols such as the XRP Ledger and Ethereum, we have demonstrated the BBF’s effectiveness in simulating network dynamics, transaction throughput, and resilience. Our research has enabled the simulation of real-world scenarios, including node failures and double-spend attacks, to assess the design decisions and robustness of blockchain protocols. Furthermore, our systematic literature review has contributed to the conceptual assessment of blockchain protocols, identifying gaps in existing research and highlighting the need for comprehensive benchmarking tools. Collectively, the research contributions emerging from this project underscore the significance of the Blockchain Benchmarking Framework in facilitating a thorough evaluation of blockchain protocols, offering insights into their performance, stability, and protocol design.
XRPL Contribution Award: Dr Klitos Christodoulou
received during UBRI Connect 2023
The XRPL (XRP Ledger) Contribution Award recognizes significant contributions to the development, improvement, or promotion of the XRP Ledger ecosystem. The 2023 edition of this award was honoured to Dr Klitos Christodoulou and team for their noteworthy contributions to the XRPL community.
research papers selected for inclusion in the 2023 UBRI Highlight Report
Touloupou, Marios, Klitos Christodoulou, Antonios Inglezakis, Elias Iosif, and Marinos Themistocleous. Benchmarking Blockchains: The case of XRP Ledger and Beyond. In Proceedings of the 55th Hawaii International Conference on System Sciences. 2022
Touloupou, Marios, Marinos Themistocleous, Elias Iosif, and Klitos Christodoulou. A Systematic Literature Review Towards a Blockchain Benchmarking Framework. IEEE Access (2022)
Summary of Contributions
and Research Output
The University of Nicosia has been at the forefront of contributing to the development of a modular and open-source Blockchain Benchmarking Framework.
Theoretical Contributions
We have systematically reviewed existing literature to create a comprehensive taxonomy of blockchain performance studies, highlighting the necessity for and potential structure of effective benchmarking frameworks.
Practical Implementations
By deploying controlled environments supporting a variety of blockchain protocols e.g., XRLP and Ethereum clients, we have validated the BBF’s effectiveness in real-world scenarios, demonstrating its adaptability and efficiency in evaluating different blockchain protocols.
Framework Architecture and Modules
We have proposed and developed a modular architecture for a BBF that includes a set of tools for deployment, wallet and traffic management, as well as, tooling for simulating adversarial conditions within the network e.g., node failures and double spent attacks. This architecture has been illustrated through the deployment of a private XRPL Network, showcasing the framework’s capability to assess blockchain protocols under various test cases and scenarios.
Consensus Protocol Analysis
Our work on consensus crash testing, particularly with Ripple’s consensus protocol, provides critical insights into balancing security and decentralization in blockchain networks, contributing to the broader understanding of consensus mechanisms in adversarial environments.
Through these contributions, the University of Nicosia has significantly advanced the field of blockchain benchmarking, supporting the wider adoption of blockchain technologies. Our efforts aim to ensure that the BBF becomes a reference tool for researchers, developers, and organizations in the blockchain ecosystem, facilitating informed decision-making and innovation in blockchain protocol design and deployment.
Research Publications
Touloupou, Marios, Klitos Christodoulou, and Marinos Themistocleous. Validating the Blockchain Benchmarking Framework through Controlled Deployments of XRPL and Ethereum. IEEE Access (2024).
This paper focuses on validating the efficacy of the Blockchain Benchmarking Framework (BBF), developed for the evaluation of blockchain protocols in a controlled environment. The effectiveness of the BBF’s is demonstrated through its application to the XRP Ledger and Ethereum clients deployed in a private and controlled setting. These deployments enabled the simulation of network dynamics, transaction throughput, and resilience across various scenarios. The research methodology includes testing under normal operational conditions, as well as, under stress scenarios e.g., node failures and double-spend attacks, highlighting the BBF’s capacity to evaluate blockchain protocol stability and robustness accurately. The findings showcase the BBF’s adaptability and efficiency in assessing different blockchain protocols, supporting its potential utility in pre-launch performance analysis and protocol design validation. This paper contributes to the field by providing insights into the preliminary assessment of blockchain clients before their mainnet deployment and the validation of protocol design decisions under varied and synthetic conditions.
Touloupou, Marios, Marinos Themistocleous, Elias Iosif, and Klitos Christodoulou. A Systematic Literature Review Towards a Blockchain Benchmarking Framework. IEEE Access (2022).
In this work, we extend the body of knowledge and contribute towards the assessment of blockchain protocols at the conceptual level. We present a comprehensive taxonomy of selected studies on blockchain performance, focusing on identifying similarities and differences while attempting to identify existing work on simulators and benchmarking frameworks that aim to explore the performance of consensus algorithms adapted for use in blockchain networks.
Touloupou, Marios, Klitos Christodoulou, Antonios Inglezakis, Elias Iosif, and Marinos Themistocleous. Benchmarking Blockchains: The case of XRP Ledger and Beyond. In Proceedings of the 55th Hawaii International Conference on System Sciences. 2022. Access Online:
This paper proposes an architecture for a blockchain benchmarking framework that aims at the deployment and evaluation of different blockchain protocols. Moreover, we introduce a set of modules for testing and evaluating their behavior under different test-cases and scenarios. To illustrate the usefulness of the proposed architecture we demonstrate an instantiation with the deployment of a private XRPL Network.
Touloupou, Marios, Klitos Christodoulou, Antonis Inglezakis, Elias Iosif, and Marinos Themistocleous. Towards a framework for understanding the performance of blockchains. In 2021 3rd Conference on Blockchain Research & Applications for Innovative Networks and Services (BRAINS), pp. 47-48. IEEE, 2021.
The paper points out the existing gap in proper testing and software engineering practices for evaluating blockchain usage and performance comprehensively. To bridge this gap, the paper introduces an architecture for a blockchain benchmarking framework designed to deploy and evaluate different blockchain protocols. The framework comprises of a set of modules for testing and assessing the behavior of blockchain protocols under a range of test scenarios.
Christodoulou, Klitos, Elias Iosif, Antonios Inglezakis, and Marinos Themistocleous. Consensus crash testing: Exploring ripple’s decentralization degree in adversarial environments. Future Internet 12, no. 3 (2020): 53.
The core of the study focuses on the analysis of consensus protocols within both permissioned and permissionless distributed-ledger frameworks. The main contribution of this work deals with observations from experimenting with Ripple’s consensus protocol as it is embodied in the XRP Ledger. The main experimental finding suggests that, when a low percentage of malicious nodes is present, the centralization degree of the network can be significantly relaxed ensuring low convergence times. Those findings are of particular importance when engineering a consensus algorithm that would like to balance security with decentralization.
Open-source code
In addition to our pure academic research efforts, we are contributing well documented source code to the community for experimentation and re-use. Find below a complete list of our updated GitHub repos that provide open-source code to the community.
