SPICE
Irmin is a non-CRUD database engine. It is based on distributed version-control systems. It scales to handle a very high number of operations per second. It supports fully automated conflict handling. The append-only approach used in Irmin allows unique optimisations.
Irmin is already in production at version 3.10. As an append-only database, it delivers top-tier performance. Some feedback has already been gathered, and technical analysis suggests it’s possible to achieve best-in-class performance by bringing support for multicore execution. The goals of the project are threefold: first, confirm users actually need the performance boost enabled by multicore support; second, turn experimental support for multicore into production-grade support; third, measure the performance improvements in the users’ use scenarios. This will result in a new main version of Irmin, codenamed IrminVault.
- Motivation for the project: This proposal contributes directly to the objectives of the TrustChain project by addressing key challenges related to green, scalable, and sustainable DLT). Our focus on enhancing the scalability and energy efficiency of Irmin, a programmable storage framework, aligns with TrustChain’s goals of fostering a more sustainable and user-friendly DLT ecosystem. By integrating Irmin with MirageOS, we provide a novel solution that minimises energy consumption and computational overhead, enabling operators to deploy high-performance systems with significantly reduced environmental impact.
- Generic use case description: The proposed advancements have significant industrial implications, both at the European level and globally. By providing a highly scalable and energy-efficient storage solution, we position IrminVault as a cornerstone technology for DLTs, enabling industries such as finance, supply chain, and healthcare to adopt blockchain technologies more sustainably. This addresses market challenges such as the high cost of blockchain deployment, scalability bottlenecks, and environmental concerns.
-
Essential functionalities:
1. Identify and resolve CPU inefficiencies causing Irmin to be CPU-bound. Use advanced profiling tools such as perf, magic-trace, OCaml5 runtime events, and memtrace to locate bottlenecks. Validate the improvements through performance benchmarks.
2. Ensure Irmin's concurrency safety in multicore environments through rigorous testing. Employ property-based testing under stress tests with multi-threaded workloads.
3. Optimise on-disk and in-memory representation of storage objects by inlining small objects.
4. Enhance in-memory caching mechanisms for improved performance. Validate improvements using benchmarks focused on cache-heavy operations.
5. Turn our existing key/value optimised backend into a production-ready IrminVault backend. Expected Outcome: a public release of the optimised key/value store.
6. Develop an API to make IrminVault accessible from other programming languages and demonstrate how to use IrminVault with different consensus mechanisms.
7. Turn our prototype of a MirageOS backend into something that is ready to run production workload. -
Gap being addressed: Irmin is positioned as a competitive solution compared to systems like BigchainDB, IPFS, and Parity Substrate due to its focus on scalability, energy efficiency, and customizability:
1. Scalability and Performance: BigchainDB achieves around 1000 transactions per second and handles large data volumes. IPFS prioritises data availability and resilience, with performance dependent on network conditions. Irmin aims to match or exceed these capabilities with planned benchmarks validating high transaction throughput and historical data scaling.
2. Energy Efficiency: By integrating with MirageOS, Irmin enables energy-efficient deployments on raw block devices, reducing operational overhead and carbon footprint. Competitors generally lack this focus on streamlined energy use.
3. Customizability: Irmin’s programmable workflows allow tailored solutions for specific use cases, offering flexibility that complements existing architectures. This includes enabling fine-grained optimisations for decentralised systems with specific throughput and energy constraints. -
Expected benefits achieved with the novel technology building blocks:
1. Remove CPU Bottlenecks: A 5-10% reduction in CPU overhead during transaction processing and replay workloads.
2. Multicore Safety Testing: Validate the linearizability of the Irmin API. Inline Small Objects: reduce content object count by 80% on realistic benchmarks.
3. Improve Caching: A 5-10% improvement in cache-heavy operations High-Performance Backend: a public release of the optimised key/value store. Multi-Language API Support: an User Manual for IrminVault.
4. MirageOS Support: A public release of IrminVault that runs on MirageOS -
Potential demonstration scenario:
1. Performance Benchmarks: Metrics from testing IrminVault under real-world conditions. We will reuse the usage scenarios developed when assessing the performance bottlenecks that the users were hitting. We also use industry-established data sets.
2. Energy Consumption Analysis: Detailed energy usage comparisons before and after optimisations.
Team
Carine Morel
Former physicist turned OCaml enthusiast, specializing in multicore programming.
Cuihtlauac Alvarado
Corecursive Tinkerer
Gwenaëlle Lecat
ocaml enthusiast and Breton independentist
Entity
Energy Community Bessarion (ECB)
We help our clients build reliable, secure, and high-performance systems.
Website: tarides.com
Linkedin: https://www.linkedin.com/company/tarides
GitHub: https://github.com/tarides