Experiment: 7c26ea6
rad experiment show 7c26ea66a7721d9a640d79f88ef569e43f7b8697
by ee@did:key:z6MkvsiybCuk1WDZhh2aXDx7NTiZejKgWZygMNsUuXDMNvVQ · Apr 24 19:27 2026
read_to_end into thread-local scratch
Measurements
MetricBaselineThe benchmark measurement of the unmodified codeCandidateThe code with the proposed optimization appliedCandidate/Parent ΔThe performance change from baseline to candidate
duration (ms) primary
wall-clock time to compute the content-id of a 64 MiB directory corpus (64 files x 1 MiB each, deterministic PRNG), walking and BLAKE3-hashing every file and then hashing the iroh Collection root; lower is better
8.080 ms (n=1)8.608 ms (n=1)+6.53% REGRESSED
Annotations
hypothesisread into a thread-local scratch buffer is faster than mmap for warm 1 MiB files
next_action_hintstop messing with the per-file I/O mechanism; look at Collection::from_iter+to_blobs and whether the final blake3 hash over the HashSeq blob can be composed from already-computed blob hashes without reserializing
rollback_reason+6% regression — mmap wins because update_mmap feeds blake3 in aligned streaming chunks while a single bulk update on a 1 MiB slice does not parallelise or cache well
Base CommitThe starting commit before the optimization was applied0494e5ccdd2692ef4b135f1722c0c739833046e7
Candidate CommitThe code with the proposed optimization appliedb49af1944004aee178024fbe6dd5ccc696f8147f
bench_cmdbash ./bench/benchmark.sh
build_cmdcargo build --release -p cc-bench
Schemav5
Diff
~ src/share/cid_utils.rs
@@ -172,19 +172,33 @@ pub fn canonical_walk(dir: &Path) -> Result<Vec<(String, PathBuf)>, io::Error> {
172 /// The resulting CID uses the `blake3-hashseq` codec (0x80). 173 pub fn compute_content_id(dir: &Path) -> Result<Cid, io::Error> { 174 use rayon::prelude::*; 175+ use std::cell::RefCell; 176+ use std::io::Read; 177+ 178+ thread_local! { 179+ static SCRATCH: RefCell<Vec<u8>> = const { RefCell::new(Vec::new()) }; 180+ } 181 182 let walked = canonical_walk(dir)?; 183 178- // Hash files in parallel across rayon workers. Each worker hashes a 179- // whole file serially via mmap; the inter-file parallelism already 180- // saturates cores, so intra-file rayon fan-out would just add dispatch 181- // overhead. 184+ // Hash files in parallel across rayon workers. Each worker reuses a 185+ // thread-local scratch buffer to read the whole file and hashes from 186+ // RAM; this avoids per-file mmap setup/teardown and keeps allocations 187+ // out of the hot path. 188 let entries: Vec<(String, iroh_blobs::Hash)> = walked 189 .into_par_iter() 190 .map(|(name, path)| { 185- let mut hasher = blake3::Hasher::new(); 186- hasher.update_mmap(&path).map_err(io::Error::other)?; 187- Ok((name, hasher.finalize().into())) 191+ SCRATCH.with(|buf| { 192+ let mut buf = buf.borrow_mut(); 193+ buf.clear(); 194+ let mut file = std::fs::File::open(&path)?; 195+ let meta = file.metadata()?; 196+ buf.reserve(meta.len() as usize); 197+ file.read_to_end(&mut buf)?; 198+ let mut hasher = blake3::Hasher::new(); 199+ hasher.update(&buf); 200+ Ok::<(String, iroh_blobs::Hash), io::Error>((name, hasher.finalize().into())) 201+ }) 202 }) 203 .collect::<Result<_, io::Error>>()?; 204
Environment
Archaarch64
OSDarwin 14.6.1
CPUApple M2
Agentpi-autoresearch / pi
Filessrc/share/cid_utils.rs