CommunityNews

CommunityNews

Rethinking Sanakirja, a Rust database engine with fast clones

My last post about Sanakirja sparked a few really constructive discussions, and made me realise that people still cared about the problem of on-disk key-value stores, as unfancy as that problem may sound. This post looks back on some design mistakes I’ve made when I wrote it, and includes benchmarks showing it’s now faster than the fastest equivalent C library.

Why?

A long time ago, Pijul was using LMDB as its backend, with a number of fundamental limitations, including:

  • Being restricted to two datatypes: an array of B+ trees where keys are byte strings and values are either (1) byte strings or (2) B+ trees where keys are bytestrings and values are zero-sized. In Rust terminology, this is equivalent to roughly 500 tables, where a table is either BTreeMap<&[u8], &[u8]> or BTreeMap<&[u8], BTreeMap<&[u8], ()>>.
  • Being written in C, meaning that it is potentially fast, but hard to extend in any nontrivial way. About the “fast” part, my benchmarks show that indeed, it is quite fast — just not as fast as a carefully-designed Rust version.
  • More importantly, I needed to fork tables efficiently, without copying anything. This was especially important back then, when Pijul tables for small repositories often weighed dozens of megabytes. It may be slightly less relevant now, but now that it’s there, there is no reason not to use it.This is implemented with an extra table storing reference counts of each page that is referenced at least twice (in order to avoid infinite recursions, the table itself isn’t clonable, and therefore all its pages are referenced once).

This thread was posted by one of our members via one of our news source trackers.

Where Next?

Popular Backend topics Top

tomekzawada
Greetings from Membrane Framework team! Check out our case study based on our latest projects at Software Mansion. https://blog.swmansi...
New
First poster: bot
Why Zig When There is Already C++, D, and Rust? No hidden control flow No hidden allocations First-class support for no standard library...
New
First poster: bot
Rails Best Practices I. Today I share some of my favorite practices applicable to Ruby on Rails (and to web development on small teams g...
New
AstonJ
Not had time to read it yet but this looks like a good interview… Our friend Yukihiro Matsumoto, creator of the Ruby programming langua...
New
First poster: malloryerik
Everyone outside of tech has heard of JavaScript, Java, Python, Ruby and even .Net, but few if any have heard of F#. However, F# may be o...
New
New
brainlid
We take a deeper dive with Nathan Long into IOLists in Elixir. We cover what they are, how they work, the power they have when concatenat...
New
AstonJ
This was posted on the Elixir Forum and thought it was worth sharing here! I love how the excitement of the author shines through and I ...
New
brainlid
In episode 83 of Thinking Elixir, We talk with Isaac Yonemoto about the Zig language and his Zigler Elixir library. We learn where Zig ca...
New
GoulvenClech
Hi everyone :wave: I’m excited to share an article detailing how we have reorganized our Elixir/Phoenix project’s directory structure. W...
New

Other popular topics Top

AstonJ
Or looking forward to? :nerd_face:
New
PragmaticBookshelf
Rust is an exciting new programming language combining the power of C with memory safety, fearless concurrency, and productivity boosters...
New
AstonJ
I ended up cancelling my Moonlander order as I think it’s just going to be a bit too bulky for me. I think the Planck and the Preonic (o...
New
AstonJ
Seems like a lot of people caught it - just wondered whether any of you did? As far as I know I didn’t, but it wouldn’t surprise me if I...
New
wmnnd
Here’s the story how one of the world’s first production deployments of LiveView came to be - and how trying to improve it almost caused ...
New
Maartz
Hi folks, I don’t know if I saw this here but, here’s a new programming language, called Roc Reminds me a bit of Elm and thus Haskell. ...
New
AstonJ
If you want a quick and easy way to block any website on your Mac using Little Snitch simply… File &gt; New Rule: And select Deny, O...
New
PragmaticBookshelf
Author Spotlight: Karl Stolley @karlstolley Logic! Rhetoric! Prag! Wow, what a combination. In this spotlight, we sit down with Karl ...
New
New
AstonJ
This is a very quick guide, you just need to: Download LM Studio: https://lmstudio.ai/ Click on search Type DeepSeek, then select the o...
New