_UPD 2023-06-26 14:30 UTC: this is useless benchmark, see https://github.com/bor…gbackup/borg/issues/7674#issuecomment-1606015655 . I did proper benchmark here: https://github.com/borgbackup/borg/issues/7674#issuecomment-1607612493 ._
Hi. I did many speed comparisons. Here are results:
- casync ( https://0pointer.net/blog/casync-a-tool-for-distributing-file-system-images.html ) (which is inspired by many projects, including borg) decompresses particular realistic data (12 snapshots of 10 GiB VM) x1.5 faster than borg 2.0.0b5 on same chunker params and same compression level (compression ratio is same). But compression speed for borg is 2.3x better
- `borg --chunker-params fixed` has x2.5 slower compression speed than my own _trivial_ single-threaded Rust program for data deduplication on same settings/algorithms (compression ratio is same). Decompression for borg is x4.5 slower
- borg can achieve nearly same compression ratio as zpaq's default settings, while being x2.9 faster in compression. Decompression speed is same. (Algorithms are different)
- casync can achieve nearly same compression ratio as zpaq's default settings, while being x1.6 faster in decompression. Compression speed is same. (Algorithms are different)
"Compression" above means "all steps needed for putting given file (VM image) to deduplicator's storage". "Decompression" means opposite action, i. e. extracting from such storage. And everywhere I say borg I mean borg2.
Now let me give details. I'm writing "something like docker, but for VMs" for Linux. And I need deduplicating storage for this, i. e. something like what borg and casync do. So I did benchmark of various deduplicating solutions (and wrote my own trivial single-threaded Rust program). Test data is sequence of 12 snapshots of same VM. The zeroth snapshot (numbered 00) is VM created after fresh Debian install. Snapshot 01 is the same machine after execution of command `echo deb http://deb.debian.org/debian sid main > /etc/apt/sources.list`. Snapshot 02 is snapshot 01 after command `apt-get update && apt-get install -y debootstrap && debootstrap sid /debian http://deb.debian.org/debian && : > /OLD && : > /debian/NEW && echo systemctl switch-root /debian /bin/bash > /root/.bash_history && apt-get install -y systemd`, etc.
Of course, all this snapshots are very similar, and there are a lot of redundancy here.
Test was performed so: I created fresh deduplicating storage, for example, by `borg2 rcreate ...`. Then I stored snapshot 00 to it using command like `borg2 create ...`. Then 01, etc. Then I extracted them back one by one. I recorded time of each storing and extracting. And size of resulting storage.
The programs under test are these:
- No deduplicating at all ("control group" of this experiment). Just compress each snapshot separately using `zstd -4 -T0` (this command uses all available cores)
- borg2 2.0.0b5. Both `--chunker-params fixed` and `--chunker-params buzhash` was tested. borg is single-threaded, according to docs
- casync 2+20201210-1 (version as reported by debian). casync doesn't consider index files `.caibx` as part of its storage. I. e. raw chunks only counted from casync's point of view. But (for fair comparison with borg) I counted them as part of storage. casync uses same chunking algorithm as borg2, i. e. buzhash. As it seems from docs, casync is single-threaded
- zpaq 7.15+repack-1 (version as reported by debian) on default settings. zpaq stores everything to one file. Thus deleting arbitrary old data is not supported. Still, deduplication is supported. zpaq uses all available cores
- My own trivial deduplication solution written in Rust. The program has 125 lines (but heavily uses libraries from crates.io ). It simply cuts file in same-size chunks (i. e. it is analog of `borg2 --chunker-params fixed`), then deduplicates them and compresses using zstd. Similarly to casync the program manages chunks storage, but doesn't manage index files (i. e. user should manage index files somehow on its own, similarly to casync). But for fair comparison with other solutions index file sizes was counted as part of storage. The program is absolutely trivial and does bare minimum. Also it cheats: it compares chunks using weak murmur3 hash instead of some cryptographic hash (this is possible reason for good speed). But this should not affect decompression speed, because there is no reason to compute hashes when decompressing. The program is single-threaded. Unfortunately, there is no analog of casync's `casync gc` (garbage collector) command, so proper deletion of data is not supported, i. e. chunk storage is effectively append-only (of course, this feature can be added if needed)
The systems for testing was these:
- DigitalOcean (DO) VPS, 2 cores. Linux. Debian bookworm. (I don't remember which file system I used for testing)
- AWS VPS, 96 cores. tmpfs. Linux. Debian bullseye
The same versions of software used for both tests (expect for zstd, i. e. "control group", I'm sorry about that). All software was installed from debian repos (except for my solution, of course).
Okay, so here are some particular facts from this experiment.
Let's compare casync's default chunker settings with exact same borg's chunker settings. Here is result (DO):
```json
{"method":"Casync { chunker_params: \"16384:65536:262144\" }","size":2288245859,"compression":[25018278,14123602,20728147,75336765,74989789,71274134,75145956,73516900,73657489,71545543,72030643,71385677],"decompression":[12319323,9242255,11615280,24205058,27393626,27160242,30992227,32594951,32978333,33110100,32684700,32696767],"total_compression":718752927,"total_decompression":306992867}
{"method":"Borg { chunker_params: \"buzhash,14,18,16,48\", compression: \"zstd,3\" }","size":2149378514,"compression":[17679646,8448045,14671946,21504361,23543654,20046573,25479210,24576887,25604774,23566982,24462718,22118519],"decompression":[12550832,12346506,13107262,49148793,48386676,45512108,49184202,50710626,53943566,54338980,56295807,54046982],"total_compression":251703322,"total_decompression":499572346}
```
As well as I understand from casync's sources, casync uses 48 as its window size, so I set same window size for borg. So, everything is same, chunker is same, compression is same (level 3 is default for `casync make --compression=zstd`). So, we (predictably) get nearly same repo size. But compression speed is way bigger for borg. And decompression speed is way bigger for casync.
Now let's compare `borg2 --chunker-params=fixed` with my dedupper with same chunk size and same compression (AWS):
```json
{"method":"Borg { chunker_params: \"fixed,4194304\", compression: \"zstd,1\" }","size":3045630161,"compression":[10498009,5765099,11638767,11808097,11682813,9746850,14360024,12748675,13860854,12773828,13976675,13142882],"decompression":[8436011,8436552,8773607,34001769,34560878,34688943,35455363,35270733,35649059,35656812,35832813,35752490],"total_compression":142002579,"total_decompression":342515034}
{"method":"MyChunker { block_bytes: 4194304, zstd: 1, hash: \"murmur3\" }","size":3045403863,"compression":[5198602,836222,5851959,7503258,5633901,3456217,6540237,4397998,4985617,3890258,4718666,3864603],"decompression":[2717125,2661899,3047367,6441092,6868591,6854122,7567294,7639675,7984931,7982271,8272383,8156871],"total_compression":56877544,"total_decompression":76193627}
{"method":"MyChunker { block_bytes: 4194304, zstd: 1, hash: \"sha256\" }","size":3045797079,"compression":[10820321,6458880,11505570,35665701,33806750,31593533,34644788,32626207,33124389,32051083,32847354,32049456],"decompression":[2677887,2655985,3093219,6251224,6857964,6813882,7581230,7604525,7907221,7911569,8100640,8103418],"total_compression":327194037,"total_decompression":75558768}
```
So, everything is same. And predictably we got nearly same compressed size. But my dedupper is way faster than borg both in compression and decompression (if we cheat using murmur3 for chunk comparision). Compression is x2.5 faster. And decompression is x4.5 faster. I don't compute hashes during decompression (I think there is no reason to do so), so even in sha256 mode decompression is still x4.5 faster than borg. But with sha256 compression becomes x2.3 slower than borg (it seems I chose slow sha256 implementation from crates.io or maybe the reason is `overflow-checks = true`).
Both casync and borg can achieve compression level similar to zpaq's default settings (if properly configured). But in such case casync and borg become better in compression speed or in decompression speed (DO):
```json
{"method":"Zpaq","size":2020550898,"compression":[66510942,13447238,47685405,80578869,80038181,59823872,80654039,65066410,71060346,61404565,64172466,61875377],"decompression":[18750110,20467299,29366756,36280852,41599404,41990291,48276667,51680639,54273853,54363762,55445757,55403322],"total_compression":752317716,"total_decompression":507898716}
{"method":"Borg { chunker_params: \"buzhash,14,18,16,4095\", compression: \"zstd,4\" }","size":2145319615,"compression":[20378384,8886346,16710992,23303757,24969724,20268377,26531417,25136239,26505187,24139859,25912861,22098249],"decompression":[12205145,12290434,12885718,43323609,44173501,43728450,44725463,46298867,46943885,46165601,48665056,48574784],"total_compression":264841397,"total_decompression":449980518}
{"method":"Casync { chunker_params: \"16384:65536:262144\" }","size":2288245859,"compression":[25018278,14123602,20728147,75336765,74989789,71274134,75145956,73516900,73657489,71545543,72030643,71385677],"decompression":[12319323,9242255,11615280,24205058,27393626,27160242,30992227,32594951,32978333,33110100,32684700,32696767],"total_compression":718752927,"total_decompression":306992867}
```
Conclusions:
- casync compresses slower than borg for same (buzhash) parameters. So, something is going wrong and casync should be fixed
- borg decompresses slower than borg for same (buzhash) parameters. So, again, something is going wrong and borg should be fixed
- My dedupper decompresses 4.5 times faster than borg. So, again, something is really wrong with borg decompression
- zpaq's default settings are really bad for my use case, and casync and borg can easily beat them
If I had blog, I would post everything there. But I don't have a blog, so I post everything here, to borg bug tracker. I hope this report will be useful to borg devs. If you want to write me, but don't want to pollute borg bug tracker, just write me directly to safinaskar@gmail.com .
Okay, now let me show you code and raw data.
Code for my dedupper:
<details>
<summary>Cargo.toml</summary>
```toml
[package]
name = "my-chunker"
version = "0.1.0"
edition = "2021"
[profile.release]
overflow-checks = true
[dependencies]
fastmurmur3 = "0.2.0"
hex = "0.4.3"
libc = "0.2.146"
ring = "0.16.20"
zstd = "0.12.3"
```
</details>
<details><summary>src/main.rs</summary>
```rust
#![feature(fs_try_exists)]
#![feature(file_create_new)]
use std::io::Read;
use std::io::Write;
struct Killed;
// "kill" just means "completely drop this value". I. e. we drop it and then shadow it to make sure we will not be able to access it even if it is Copy. I. e. when you see "kill!", assume it is drop
macro_rules! kill {
($x:ident) => {
#[allow(dropping_references)]
#[allow(dropping_copy_types)]
std::mem::drop($x);
#[allow(unused_variables)]
let $x = Killed;
}
}
#[derive(Clone, Copy)]
enum Hash {
MurMur3,
Sha256,
}
fn calc_hash(hash_type: Hash, data: &[u8]) -> Vec<u8> {
return match hash_type {
Hash::MurMur3 => fastmurmur3::hash(data).to_le_bytes().to_vec(),
Hash::Sha256 => ring::digest::digest(&ring::digest::SHA256, data).as_ref().to_vec(),
};
}
// "[()] = [...]" is needed because of clippy bug: https://github.com/rust-lang/rust-clippy/issues/9048 . So when you see "[()] = [f()]", just assume "() = f()"
fn main() {
if std::env::args().collect::<Vec<_>>()[1] == "add" {
let [_, _, ref o_storage, ref o_block_bytes, ref o_zstd, ref o_hash] = *std::env::args().collect::<Vec<_>>() else { panic!("Usage"); };
let block_bytes: usize = o_block_bytes.strip_prefix("--block-bytes=").unwrap().parse().unwrap();
kill!(o_block_bytes);
if !(4096 <= block_bytes && block_bytes <= 10_000_000_000) {
panic!();
}
let zstd_level: i32 = o_zstd.strip_prefix("--zstd=").unwrap().parse().unwrap();
kill!(o_zstd);
let hash_type = match &**o_hash {
"--strong-hash=murmur3" => Hash::MurMur3,
"--strong-hash=sha256" => Hash::Sha256,
_ => panic!(),
};
kill!(o_hash);
let storage = o_storage.strip_prefix("--chunks=").unwrap();
kill!(o_storage);
let mut input = std::io::stdin().lock();
let mut stdout = std::io::stdout().lock();
let mut buf = vec![0u8; block_bytes];
kill!(block_bytes);
let mut hashs = vec![];
loop {
let buf_size = input.read(&mut buf).unwrap();
if buf_size == 0 {
break;
}
let buf = &buf[..buf_size];
kill!(buf_size);
let hash = calc_hash(hash_type, &buf);
hashs.extend_from_slice(&hash);
let hash_str = hex::encode(&hash);
kill!(hash);
if !std::fs::try_exists(format!("{storage}/{hash_str}")).unwrap() {
let compressed = zstd::bulk::compress(buf, zstd_level).unwrap();
let mut file = std::fs::File::create_new(format!("{storage}/{hash_str}")).unwrap();
[()] = [file.write_all(&u64::try_from(buf.len()).unwrap().to_le_bytes()).unwrap()];
kill!(buf);
[()] = [file.write_all(&compressed).unwrap()];
}
}
[()] = [stdout.write_all(&hashs).unwrap()];
} else if std::env::args().collect::<Vec<_>>()[1] == "extract" {
let [_, _, ref o_storage, ref o_hash] = *std::env::args().collect::<Vec<_>>() else { panic!("Usage"); };
let hash_type = match &**o_hash {
"--strong-hash=murmur3" => Hash::MurMur3,
"--strong-hash=sha256" => Hash::Sha256,
_ => panic!(),
};
kill!(o_hash);
let storage = o_storage.strip_prefix("--chunks=").unwrap();
kill!(o_storage);
let mut input = std::io::stdin().lock();
let mut output = std::io::stdout().lock();
loop {
let mut hash_buf = vec![0u8; match hash_type {
Hash::MurMur3 => 128 / 8,
Hash::Sha256 => 256 / 8,
}];
{
let have_read = input.read(&mut hash_buf).unwrap();
if have_read == 0 {
break;
} else if have_read == hash_buf.len() {
// OK
} else {
panic!();
}
}
let hash_str = hex::encode(&hash_buf);
let mut file = std::fs::File::open(format!("{storage}/{hash_str}")).unwrap();
let mut size = [0u8; 8];
[()] = [file.read_exact(&mut size).unwrap()];
let size = usize::try_from(u64::from_le_bytes(size)).unwrap();
let mut data = vec![];
let _: usize = file.read_to_end(&mut data).unwrap();
kill!(file);
{
let decompressed = zstd::bulk::decompress(&data, size).unwrap();
kill!(data);
assert_eq!(decompressed.len(), size);
kill!(size);
[()] = [output.write_all(&decompressed).unwrap()];
}
}
} else {
panic!("Usage");
}
}
```
</details>
Now program for executing benchmark
<details><summary>Cargo.toml</summary>
```toml
[package]
name = "rust"
version = "0.1.0"
edition = "2021"
[dependencies]
regex = "1.8.4"
serde_json = { version = "1.0.79", features = ["preserve_order"] }
serde = { version = "1.0.136", features = ["derive"] }
```
</details>
<details><summary>src/main.rs</summary>
```rust
#![feature(exit_status_error)]
// "[()] = [...]" is needed because of clippy bug: https://github.com/rust-lang/rust-clippy/issues/9048 . So when you see "[()] = [f()]", just assume "() = f()"
fn run(s: &str) {
[()] = [std::process::Command::new("bash").arg("-ec").arg(s).status().unwrap().exit_ok().unwrap()];
}
fn main() {
const REPO: &str = "/home/user/dedup-bench/fs/repo";
const CMP_ME: &str = "/home/user/dedup-bench/fs/cmp-me";
#[derive(Debug)]
enum Method {
ZStd,
Borg { chunker_params: String, compression: String, },
Casync { chunker_params: String, },
Zpaq,
MyChunker { block_bytes: usize, zstd: i32, hash: String },
}
let range = 0..=11;
// let range = 0..=2;
// let range = 0..=5;
run(&format!("rm -rf {REPO}"));
run(&format!("rm -rf {CMP_ME}"));
#[derive(Debug)]
struct Run {
method: Method,
size: usize,
compression: Vec<std::time::Duration>,
decompression: Vec<std::time::Duration>,
total_compression: std::time::Duration,
total_decompression: std::time::Duration,
}
let mut runs = vec![];
let mut methods = vec![];
{
methods.push(Method::ZStd);
for chunker_params in [
"buzhash,19,23,21,4095", // borg default
"buzhash,10,23,16,4095", // borg alternative
"buzhash,14,18,16,4095", // casync default
"buzhash,19,23,21,48", // borg default (casync's window size)
"buzhash,10,23,16,48", // borg alternative
"buzhash,14,18,16,48", // casync default
"fixed,4096,512",
"fixed,4096",
"fixed,4194304,512",
"fixed,4194304",
] {
for compression in [
"none",
"lz4",
"zstd,1",
"zstd,2",
"zstd,3", // casync default
"zstd,4",
] {
methods.push(Method::Borg { chunker_params: chunker_params.to_owned(), compression: compression.to_owned() });
}
}
for chunker_params in [
format!("{}:{}:{}", 1 << 19, 1 << 21, 1 << 23), // borg default
format!("{}:{}:{}", 1 << 10, 1 << 16, 1 << 23), // borg alternative
format!("{}:{}:{}", 1 << 14, 1 << 16, 1 << 18), // casync default
] {
methods.push(Method::Casync { chunker_params: chunker_params.to_owned(), });
}
methods.push(Method::Zpaq);
for block_bytes in [1 << 15, 1 << 16, 1 << 17, 1 << 18, 1 << 19, 1 << 20, 1 << 21, 1 << 22 /*4194304*/, 1 << 23, 1 << 24, 1 << 25, 1 << 26, 1 << 27, 1 << 28, 1 << 29, 1 << 30] {
//for zstd in [-22, -20, -10, -5, -1, 1, 2, 3, 4, 5, 6] {
for zstd in [1] {
//for hash in ["murmur3", "sha256"] {
for hash in ["murmur3"] {
methods.push(Method::MyChunker { block_bytes, zstd, hash: hash.to_owned() });
}
}
}
}
for method in methods {
run(&format!("mkdir {REPO}"));
run(&format!("mkdir {CMP_ME}"));
// Initializing
match method {
Method::ZStd => {},
Method::Borg { .. } => run(&format!("borg2 rcreate --encryption=none --repo {REPO}")),
Method::Casync { .. } => run(&format!("mkdir {REPO}/index {REPO}/storage.castr")),
Method::Zpaq => {},
Method::MyChunker { .. } => run(&format!("mkdir {REPO}/chunks {REPO}/index")),
}
let mut compression = vec![];
for i in range.clone() {
run(&format!("sync -f {REPO}/; sync -f {CMP_ME}/"));
println!("**** {method:?} {i} compression...");
let now = std::time::Instant::now();
match &method {
Method::ZStd => run(&format!("zstd -4 -T0 < /home/user/dedup-bench/sto/{i:02} > {REPO}/{i:02}.zst")),
Method::Borg { chunker_params, compression } => run(&format!("cd /home/user/dedup-bench/sto; borg2 create --repo {REPO} --chunker-params {chunker_params} --compression {compression} {i:02} {i:02}")),
Method::Casync { chunker_params } => run(&format!("casync make --compression=zstd --chunk-size={chunker_params} --store={REPO}/storage.castr {REPO}/index/{i:02}.caibx /home/user/dedup-bench/sto/{i:02} > /dev/null")),
Method::Zpaq => run(&format!("cd /home/user/dedup-bench/sto; zpaq add {REPO}/repo.zpaq {i:02}")),
Method::MyChunker { block_bytes, zstd, hash } => run(&format!("/my-chunker add --chunks={REPO}/chunks --block-bytes={block_bytes} --zstd={zstd} --strong-hash={hash} < /home/user/dedup-bench/sto/{i:02} > {REPO}/index/{i:02}")),
}
run(&format!("sync -f {REPO}/; sync -f {CMP_ME}/"));
compression.push(now.elapsed());
println!("**** {method:?} {i} compression: {:?}", now.elapsed());
}
let output = std::process::Command::new("bash").arg("-ec").arg(format!("du --bytes --apparent-size -s {REPO}")).output().unwrap();
[()] = [output.status.exit_ok().unwrap()];
assert_eq!(output.stderr.len(), 0);
let size: usize;
{
let x = String::from_utf8(output.stdout).unwrap();
let caps = regex::Regex::new(r##"^(?<s>[0-9]+)\t/[-/a-z]+\n$"##).unwrap().captures_iter(&x).collect::<Vec<_>>();
assert!(caps.len() == 1);
size = caps[0]["s"].parse().unwrap();
}
{
let len = size
.to_string()
.chars()
.collect::<Vec<_>>()
.rchunks(3)
.map(|x|x.into_iter().collect::<String>())
.rev()
.collect::<Vec<_>>()
.join("_");
println!("size: {len}");
}
let mut decompression = vec![];
for i in range.clone() {
run(&format!("sync -f {REPO}/; sync -f {CMP_ME}/"));
println!("**** {method:?} {i} decompression...");
let now = std::time::Instant::now();
match method {
Method::ZStd => run(&format!("zstd -d -T0 < {REPO}/{i:02}.zst > {CMP_ME}/data")),
Method::Borg { .. } => run(&format!("cd {CMP_ME}; borg2 extract --repo {REPO} {i:02} {i:02}; mv -i {i:02} data")),
Method::Casync { .. } => run(&format!("casync extract --store={REPO}/storage.castr {REPO}/index/{i:02}.caibx {CMP_ME}/data")),
Method::Zpaq => run(&format!("cd {CMP_ME}; zpaq extract {REPO}/repo.zpaq {i:02}; mv -i {i:02} data")),
Method::MyChunker { block_bytes: _, zstd: _, ref hash } => run(&format!("/my-chunker extract --chunks={REPO}/chunks --strong-hash={hash} < {REPO}/index/{i:02} > {CMP_ME}/data")),
}
run(&format!("sync -f {REPO}/; sync -f {CMP_ME}/"));
decompression.push(now.elapsed());
println!("**** {method:?} {i} decompression: {:?}", now.elapsed());
run(&format!("cmp /home/user/dedup-bench/sto/{i:02} {CMP_ME}/data"));
run(&format!("rm {CMP_ME}/data"));
}
run(&format!("rm -r {REPO}"));
run(&format!("rm -r {CMP_ME}"));
let total_compression = compression.iter().sum();
let total_decompression = decompression.iter().sum();
runs.push(Run { method, size, compression, decompression, total_compression, total_decompression });
}
for i in &runs {
println!("{:?}", i);
}
#[derive(Debug, serde::Serialize)]
struct JsonRun {
method: String,
size: usize,
compression: Vec<u128>,
decompression: Vec<u128>,
total_compression: u128,
total_decompression: u128,
}
for i in runs {
let Run { method, size, compression, decompression, total_compression, total_decompression } = i;
println!("{}", serde_json::to_string(&JsonRun {
method: format!("{:?}", method),
size,
compression: compression.into_iter().map(|x|x.as_micros()).collect(),
decompression: decompression.into_iter().map(|x|x.as_micros()).collect(),
total_compression: total_compression.as_micros(),
total_decompression: total_decompression.as_micros(),
}).unwrap());
}
}
```
</details>
Now snapshots. All snapshots are raw GPT disk images with single ext4 partition. Snapshot 00 is 2 GiB fresh install of debian sid from https://snapshot.debian.org/archive/debian/20220917T224346Z . Next snapshots are produced by executing the following sequence of commands:
<details>
```text
01 echo deb http://deb.debian.org/debian sid main > /etc/apt/sources.list
02 apt-get update && apt-get install -y debootstrap && debootstrap sid /debian http://deb.debian.org/debian && : > /OLD && : > /debian/NEW && echo systemctl switch-root /debian /bin/bash > /root/.bash_history && apt-get install -y systemd
03 (resize to 10G)
04 apt-get install -y git build-essential
05 echo deb-src http://deb.debian.org/debian sid main >> /etc/apt/sources.list
06 apt-get update && apt-get build-dep -y systemd
07 git clone https://github.com/systemd/systemd
08 cd systemd && git checkout f717d7a40a696b351415976f22a4f498c401de41 && meson setup build -Dmode=release && ninja -C build
09 :> /var/lib/dpkg/info/systemd.prerm
10 apt-get purge -y systemd systemd-sysv && cd systemd && make install
11 echo alive
```
</details>
I. e. snapshot 04 is snapshot 03 plus `apt-get install -y git build-essential`. These snapshots naturally appeared when I attempted to reproduce particular systemd bug, so data is 100% realistic.
The snapshot contain nothing confidential, so if you want, I can send them to you, just write me.
Now full benchmark results. (JSON is in the end.)
DO: https://paste.gg/p/anonymous/063883027e9848d8b7f53a046c4bd3cf
AWS: https://paste.gg/p/anonymous/bd6d25d5906f4c0fb94b78af605fb475
(Unfortunately, I used `du` for calculating storage size, but, as well as I remember, `du` utils from DO and AWS seem to produce slightly different results, i. e. these two `du`'s use different algorithms)
Assume all this code as public domain. If you want me to release any of my mentioned software as proper free software project (say, to crates.io), just write me.
Exact commands for invoking borg and other programs under test:
<details>
```rust
// Initializing
match method {
Method::ZStd => {},
Method::Borg { .. } => run(&format!("borg2 rcreate --encryption=none --repo {REPO}")),
Method::Casync { .. } => run(&format!("mkdir {REPO}/index {REPO}/storage.castr")),
Method::Zpaq => {},
Method::MyChunker { .. } => run(&format!("mkdir {REPO}/chunks {REPO}/index")),
}
...
match &method {
Method::ZStd => run(&format!("zstd -4 -T0 < /home/user/dedup-bench/sto/{i:02} > {REPO}/{i:02}.zst")),
Method::Borg { chunker_params, compression } => run(&format!("cd /home/user/dedup-bench/sto; borg2 create --repo {REPO} --chunker-params {chunker_params} --compression {compression} {i:02} {i:02}")),
Method::Casync { chunker_params } => run(&format!("casync make --compression=zstd --chunk-size={chunker_params} --store={REPO}/storage.castr {REPO}/index/{i:02}.caibx /home/user/dedup-bench/sto/{i:02} > /dev/null")),
Method::Zpaq => run(&format!("cd /home/user/dedup-bench/sto; zpaq add {REPO}/repo.zpaq {i:02}")),
Method::MyChunker { block_bytes, zstd, hash } => run(&format!("/my-chunker add --chunks={REPO}/chunks --block-bytes={block_bytes} --zstd={zstd} --strong-hash={hash} < /home/user/dedup-bench/sto/{i:02} > {REPO}/index/{i:02}")),
}
...
match method {
Method::ZStd => run(&format!("zstd -d -T0 < {REPO}/{i:02}.zst > {CMP_ME}/data")),
Method::Borg { .. } => run(&format!("cd {CMP_ME}; borg2 extract --repo {REPO} {i:02} {i:02}; mv -i {i:02} data")),
Method::Casync { .. } => run(&format!("casync extract --store={REPO}/storage.castr {REPO}/index/{i:02}.caibx {CMP_ME}/data")),
Method::Zpaq => run(&format!("cd {CMP_ME}; zpaq extract {REPO}/repo.zpaq {i:02}; mv -i {i:02} data")),
Method::MyChunker { block_bytes: _, zstd: _, ref hash } => run(&format!("/my-chunker extract --chunks={REPO}/chunks --strong-hash={hash} < {REPO}/index/{i:02} > {CMP_ME}/data")),
}
```
</details>
## Is this a BUG / ISSUE report or a QUESTION?
I think borg is slower than needed. Possible bug
