
CommunityNews
Rethinking Sanakirja, a Rust database engine with fast clones
My last post about Sanakirja sparked a few really constructive discussions, and made me realise that people still cared about the problem of on-disk key-value stores, as unfancy as that problem may sound. This post looks back on some design mistakes I’ve made when I wrote it, and includes benchmarks showing it’s now faster than the fastest equivalent C library.
Why?
A long time ago, Pijul was using LMDB as its backend, with a number of fundamental limitations, including:
- Being restricted to two datatypes: an array of B+ trees where keys are byte strings and values are either (1) byte strings or (2) B+ trees where keys are bytestrings and values are zero-sized. In Rust terminology, this is equivalent to roughly 500 tables, where a table is either
BTreeMap<&[u8], &[u8]>
orBTreeMap<&[u8], BTreeMap<&[u8], ()>>
.- Being written in C, meaning that it is potentially fast, but hard to extend in any nontrivial way. About the “fast” part, my benchmarks show that indeed, it is quite fast — just not as fast as a carefully-designed Rust version.
- More importantly, I needed to fork tables efficiently, without copying anything. This was especially important back then, when Pijul tables for small repositories often weighed dozens of megabytes. It may be slightly less relevant now, but now that it’s there, there is no reason not to use it.This is implemented with an extra table storing reference counts of each page that is referenced at least twice (in order to avoid infinite recursions, the table itself isn’t clonable, and therefore all its pages are referenced once).
This thread was posted by one of our members via one of our news source trackers.
Popular Backend topics

Apparently he decided to live-stream how he’s going to create a semver library.
New

Understanding Partial Moves in Rust.
Partial moves are an interesting but often misunderstood feature of Rust. However, with the right ...
New

Idioms for the D Programming Language
This thread was posted by one of our members via one of our news source trackers.
New

So you’re enjoying using WebSockets with Elixir’s Phoenix Framework, and you want to send some binary messages. Maybe it’s an audio clip,...
New

Post on using UDP multicasting with Elixir to broadcast presence, and listen for peers, on a local network. I have found this approach us...
New

The perspective of an ignorant computer science undergrad
It’s likely that you read the title of this post and thought “what is this guy ...
New

New

In episode 78 of Thinking Elixir, we talk with Chase Granberry about Logflare. We learn why Chase started the company, what Logflare does...
New

Another week, another oldies-but-goldies post…
This one about Test Driven Development.
New

In episode 92 of Thinking Elixir, we talk with Mitchell Hanberg and learn about why he created the alternate Phoenix templating language ...
New
Other popular topics

I’ve been really enjoying obsidian.md:
It is very snappy (even though it is based on Electron). I love that it is all local by defaul...
New

Design and develop sophisticated 2D games that are as much fun to make as they are to play. From particle effects and pathfinding to soci...
New

New

Curious to know which languages and frameworks you’re all thinking about learning next :upside_down_face:
Perhaps if there’s enough peop...
New

My first contact with Erlang was about 2 years ago when I used RabbitMQ, which is written in Erlang, for my job. This made me curious and...
New

Do the test and post your score :nerd_face:
:keyboard:
If possible, please add info such as the keyboard you’re using, the layout (Qw...
New

“Finding the Boundaries” Hero’s Journey with Noel Rappin @noelrappin
Even when you’re ultimately right about what the future ho...
New

“A Mystical Experience” Hero’s Journey with Paolo Perrotta @nusco
Ever wonder how authoring books compares to writing articles?...
New

Build efficient applications that exploit the unique benefits of a pure functional language, learning from an engineer who uses Haskell t...
New

Author Spotlight
Dmitry Zinoviev
@aqsaqal
Today we’re putting our spotlight on Dmitry Zinoviev, author of Data Science Essentials in ...
New
Categories:
Sub Categories:
Popular Portals
- /elixir
- /rust
- /ruby
- /wasm
- /erlang
- /phoenix
- /keyboards
- /rails
- /js
- /python
- /security
- /go
- /swift
- /vim
- /clojure
- /emacs
- /haskell
- /java
- /onivim
- /svelte
- /typescript
- /crystal
- /c-plus-plus
- /kotlin
- /tailwind
- /gleam
- /ocaml
- /react
- /elm
- /flutter
- /vscode
- /ash
- /opensuse
- /centos
- /php
- /deepseek
- /html
- /zig
- /scala
- /textmate
- /lisp
- /sublime-text
- /nixos
- /debian
- /react-native
- /agda
- /kubuntu
- /arch-linux
- /django
- /ubuntu
- /revery
- /manjaro
- /spring
- /diversity
- /nodejs
- /lua
- /slackware
- /julia
- /c
- /neovim