State of the Storage, 2026

From "Look Ma, No hRAMnds" to Canonical Packs & Partial State — A survey of storage in the XRPL ecosystem

There is much more to XRPL's storage problem than simply "look ma, no hRAMnds." At the heart of the system is a structural tension between an emergent, distributed content-addressed (CAS) sea of key-value pairs—which gives the P2P NodeStore its cockroach-like resilience, since any peer can fetch and hash-verify any individual tree node from any other peer—and immediately useful, ready-made packs.

Treating a hierarchical tree as a flat bag of hash-indexed blobs carries a steep tax in memory residency, random disk I/O, and index write amplification, pushing full history past 42 TB and counting despite historically modest transaction volume.

1. Five Costs of the Current Architecture

In upstream rippled, the interaction between the in-memory SHAMap, the flat key-value NodeStore, and local SQLite secondary indexes drives five operational pain points that engineers across the ecosystem are actively trying to relieve:

Paying for Both RAM Residency and Disk Thrashing: Upstream rippled accumulates the entire state tree in RAM (>27M mainnet nodes per ximinez on #7793, since SHAMapInnerNode holds strong shared_ptr links to faulted children; see #8191), yet still serializes every small tree node out to the NodeStore—paying for heavy memory usage and SSD-wearing disk thrashing at once.
~243 GB Write Amplification and ~20 GB/Day Disk Consumption: In our stock rippled 3.4.1 cold-sync capture, syncing a ~7.0 GB store folder wrote ~243 GB to disk across NuDB (the local CAS NodeStore) and SQLite (not part of the P2P CAS store, but a local secondary index built while syncing through it). Worse, two-backend online_delete store rotation rewrites live state into a fresh NuDB store—paying comparable NuDB index write amplification on every rotation. Because rotation is spiky in both memory and disk I/O and can collide with heavy ledgers, operators stretch rotation intervals out to tens of thousands of ledgers and absorb roughly 20 GB of disk a day.
Slow Cold and Warm Syncs: Even on a warm restart, a node must load and verify the state tree in RAM before getting work done. When mainnet tree verification overflows FullBelowCache's 524,288-entry target mid-walk, cache sweeps evict already-verified subtrees and force massive re-reads from disk (142.8M reads over >120 minutes in the 4,000-IOPS cgroup test in #8547).
No Pure Memory Mode (or Fast Bootstrap for It) in Upstream rippled: Unlike Xahau's xahaud (Xahau/xahaud#728), upstream rippled has no true NodeStore-free in-memory mode, nor a way to prime such a node quickly from a local or shared snapshot.
"The Great Orderbook Walk": Building the in-memory orderbook index requires a full state-tree walk—a pragmatic "just ship it" shortcut originally, running at first sync and then every ~28 h (25,600 ledgers; the next release turns it back on). Measured on M3 (mainnet, 2026-10-19, one run each), it scans 20 M state entries (~5 GiB read from disk) to find ~213 k book directories and AMM sides; only 18% of entries need decoding and the rest are skipped from their first 3 bytes.

Denis Angell (@dangell7) at XRPLF has opened several upstream PRs targeting these pressure points. On the memory and tree-residency side, #7965 and #7966 replace advisory [node_size] presets with byte-budgeted [memory_limit] cache targets, while #8191 adds an opt-in shed {enable, run, min_depth} command and [shed_cold_subtrees] sweep hook to drop clean resident SHAMap subtrees below a configurable depth (running only if the gate was already enabled before the request, to avoid freeing nodes under an unguarded descent).

On the rotation and restart side, #8145 amortizes online_delete cold-state rewrites across a finite N-generation ring (copying unmutated cold data once every N rotations), and #8547 holds FullBelowCache entries until the first validated ledger on restart—cutting the 4,000-IOPS warm restart from 142.8M reads (>120 min) to 57.0M reads (49 min), with a 5.1 GB vs. 4.0 GB memory peak.

2. All-In on Memory: Xahau's `rwdb` and the `SHAMap` Canonicalization Race

Even when you try to keep everything in RAM to maximize validator performance and eliminate disk thrashing, legacy storage assumptions get in the way. At Xahau, they built an rwdb backend that tried to use memory for everything—including an in-memory relational database interface (understandable given our own xrpld measurements, where SQLite's transaction.db writes ~1 MB/s just to maintain four indexes over ~90 KB of transactions). Yet originally, alongside holding the full SHAMap state tree in RAM, rwdb was also running—and rotating!—an in-memory NodeStore directly behind it.

Why suffer the waste of keeping a full tree in memory and duplicating/rotating it in an in-memory NodeStore right behind it? Because disabling the backing store exposed sporadic SHAMapMissingNode crashes—driven especially by a subtle canonicalization race bug. When local tree construction raced against syncing a faster peer's version, SHAMap::canonicalize in TreeNodeCache operated as a pick rather than a merge: the inner node that won the cache slot discarded the losing copy along with any child shared_ptr linkage the loser had populated. With no backing NodeStore to re-fetch those unlinked children by hash, later traversals hit SHAMapMissingNode.

Merging child linkage onto the canonical winner—together with a sliding ledger_history retention window, disabled FullBelowCache, and peer fallbacks in Xahau/xahaud#728 (fix: NULL RWDB and mutex locks)—enabled wiring [node_db] type=rwdb to a no-op NullFactory backend. We shared those canonicalization findings with Ripple as a suspected contributing interaction when missing-node symptoms surfaced during online_delete rotation (#7763, #7793, #7870; see @sublimator's comment and working notes gist:d89356a and gist:6da8c77—artifacts of that era rather than canonical truth).

3. The Other Extreme: Ruthless RAM Budgets, Page Cache, and Bootstrap vs. Steady State

Chasing sub-100 MB memory numbers on PocketXRP.com (watching xrpld sit under 100 MB in top -o mem) isn't a parlor trick: when you are ruthless about targeting a tiny RAM and page-cache budget, you start accounting for everything—exposing memory creep that 32–64 GB of RAM would hide, and forcing the core tree walks and on-disk representations themselves to be fast because you cannot lean on a giant cache to mask slow hash lookups, per-node allocations, or full leaf deserialization.

Measured on M3 (mainnet, 2026-10-19, one run each), the order-book walk now takes 39 s and 20 s CPU, and the order-book index it builds costs about 11 MiB. Before, it took 61–65 s and 52–54 s CPU, peaking at 521–707 MiB and leaving 224–256 MiB behind (not mimalloc's purge delay, tested three ways).

Why the old walk left memory behind: a memory page holds blocks of one size; the old walk made millions of short-lived cache entries and built the order-book sets among them out of same-size blocks that stay, so most pages kept a few and couldn't be returned—skipping the book sets in a diagnostic run removed it (pages holding 64-byte blocks: 201.6 → 1.7 MiB). The rest of the old walk's leftover, about 44 MiB, was the tree cache's hash tables keeping their buckets sized for the walk's two million entries; a sweep now gives them back (5.76 M buckets → 18.8 k after the same walk).

At the same time, process footprint is only part of the picture: walking a tree via cold disk reads is expensive even with mmap (fewer system calls, but the working set still requires RAM in the OS page cache and mappings—which is exactly why core on-disk tree traversal has to be as lean and direct as possible). And testing under hard memory limits proved that bootstrapping/indexing and steady-state ledger processing have completely different resource profiles:

Why Cold Sync Needed ~2 GB Including Page Cache: PocketXRP could sync the state tree to full in 1 GB, but building the final hash table to index the nodes in the store—performed only after the tree was built, relying in the interim on temporary staging structures—generated so many uncached random writes (normally smoothed over by the OS page cache) that it ground to a halt without ~2 GB including OS page cache. Doing that index build in even less memory is an open area of research (assuming it's even possible without steep trade-offs), though in practice there are bigger fish to fry.
2 GB Cold Sync vs. 1 GB Steady-State Processing: Because testing inside a VM can allow the host's cache to make "cold" guest reads artificially warm, we ran inside a 2 GB Linux cgroup over a custom userspace FUSE filesystem that injected read latency—a harsher, pessimistic environment. It was able to cold-sync inside the 2 GB cgroup, and reusing that same store, chug along processing ledgers inside a 1 GB cgroup.

4. "Why Not Both?": KeyDom, Hybrid Trees, and the 42 TB Mega-Index

Prototyping v2 of Xahau's catalogue (CATL) packs—using an on-disk, structurally shared tree log with relative byte-offset pointers to child nodes—led directly to the KeyDom prototype storage engine, where inner nodes link directly into an append-only data file for a measured 1.8× speedup in tree walking. Crucially, KeyDom is more than a drop-in NuDB replacement: it is a tree-aware NuDB paired with a SHAMap updated to operate as a hybrid tree, supporting transient walks and Packed(Inner|Leaf) nodes—thin in-memory pointers to store byte offsets. Those thin offset handles are only kept at the active frontier, which stays small: they live in memory mainly during the mutable SHAMap window when a ledger's transactions apply tree updates (insert/delete/update), or during the "pounce" stage of syncing when the node sprints to catch the live tip after getting most of the way by streaming directly to disk.

To make on-disk and hybrid trees fast without losing P2P verifiability, we don't have to choose between file offsets and cryptographic hashes—as with many things, "why not both?" In a partial pack, parent inner nodes can use direct file offsets for fast access to co-located children while leaving external or cold branches as 256-bit hash placeholders for the storage engine's resolver (fetched on demand from another local store or peers).

Crucially, history, network syncing, and live transaction processing can all share the same pack data structures. With full history at 42 TB and counting (carrying the entire tree in history alongside many dormant accounts) and the old Shards feature removed in rippled 2.3.0, operators currently have no middle ground between 42 TB or a tiny recent window (Xahau's stationary pinned store for catalogue imports maintains 2–3 concurrent indexes and imports slowly). Instead of every node burning CPU and disk building identical indexes from scratch, full and partial history should be retrievable like little Lego blocks over BitTorrent or P2P:

Community NuDB Walkers → Simple Leaf Packs: Open-source tools that walk cloned NuDB data can convert stores into simple, well-compressed, shareable leaf-only packs (like Xahau's CATL v1) as a foundation for richer pack and shard formats.
Network-Signed Canonical Packs as a State Coordination Layer & mmap Bootstrap: Packs and their indexes could be shared in a canonical form—and even indexed or attested on-chain by core validators or similar committees—so a top-level hash authenticates the pack while a low-memory streaming pass validates the trees and headers within. Instead of every node picking an arbitrary online_delete window in the dark, on-chain pack attestations can act as a state coordination layer: while an interval has no canonical pack recorded on-chain yet, the network keeps the live P2P "cockroach CAS" window open, and once a pack is attested on-chain, nodes have both a coordinated checkpoint to download, mount, and mmap to prime network sync and a natural watermark for trimming the volatile CAS window.
Perfectly Hashed Packs, Staged Mega-Index Probing, and Ledger-Interval Hints: Cold packs/snapshots that ship already perfectly hashed can be joined together and imported into a global mega-index spanning the packs—avoiding the multi-index lookup weakness I believe the original discarded sharding design had (not that shards were killed because of indexing, and from memory there was some kind of master index for the SQLite DBs). When you slide in a new pack brick, while the mega-index updates in the background (itself an area for research), the storage resolver can check multiple indexes on misses—probing the most relevant stores first using ledger-index hints since packs cover ledger intervals. Better still, for most workloads outside P2P "cockroach CAS" single-node serving, a ledger-based index to the account root hash is all you really need to incorporate a pack into a local mega-index quickly.

5. Eye to the Horizon: Configurable Node Modes and Hot, Cold, and Dormant State

In the discussion on #7793, Ripple engineer ximinez noted that aside from memory-only nodes, we shouldn't hold the entire ledger in memory all the time (citing >27M mainnet nodes when a typical ledger needs only the touched few thousand leaf nodes and their inner nodes). Agreed! As sublimator replied, tree residency should be an explicit, configurable policy across distinct node operating modes:

Low-Memory / On-Disk Mode: Optimized for minimal RAM at the cost of disk writes (or paired with mmap-backed packs).
Memory-Saving-Writes Mode: Using memory to save those disk writes, while using local or network-shared snapshots/packs to cold-start quickly.
"In-Memory Except Very Cold Nodes" Hybrid Mode: A middle ground between a full-memory node and a fully on-disk node—keeping active state in RAM while leaving "very cold" nodes on disk—which is why pinning down what "cold" is, exactly, is so valuable.

If you keep your eye to the horizon, the whole point of this work is to lessen the need to keep the entire tree—and its many dormant accounts—in history and in RAM. That requires asking how big the tree can get, how big Ripple wants it to get, and what separates hot, cold, and dormant state: