Keep writes fast with the default WAL

Compare logging paths

AsyncCompressed and AsyncShared let the caller continue before background logging finishes. An abrupt process exit can lose queued records; normal disposal drains them.

With the built-in provider, transactional factories promote Sync and SyncCompressed to Shared, and AsyncCompressed to AsyncShared. Compression settings, explicit shared modes, and None are retained. Custom providers and provider factories keep their configured modes and define their own recovery guarantees.

Acknowledge durable writes

Ordinary record appends do not require a disk sync per write. When ordinary-tree writes must survive a power loss before you acknowledge them, await SyncWALAsync once per request or batch, after all of its writes:

tree.Upsert(orderKey, order);
tree.Upsert(stockKey, stock);
tree.Upsert(auditKey, audit);
await tree.SyncWALAsync();
// All three writes survive a power loss. Acknowledge the request.

For transactions, CommitDurableAsync, TryCommitDurableAsync, and ExecuteDurableAsync sync the commit record before making its writes visible to readers. Ordinary Commit() publishes first; a later SyncWALAsync() makes those earlier commits durable without undoing their earlier visibility. These guarantees require WAL to be enabled. See transaction durability.

What a disk sync costs

A disk sync (fsync on Linux, FlushFileBuffers on Windows) returns only after the storage device confirms that the data is on stable media. That takes roughly 0.1–1 ms on an NVMe drive, a few milliseconds on a SATA SSD, and around 10 ms on a hard disk. A writer that waits for a sync after every write cannot start its next write until that sync finishes, so its throughput is capped by the sync latency: with 1 ms syncs, it can do at most 1,000 writes per second.

Measured on one machine (Windows, NVMe SSD, 10,000 inserts, AsyncCompressed):

Syncing every write made a single writer about 900 times slower. Concurrent waiters share syncs, which recovers part of the cost, but the device latency still sets the pace.

How much can be lost without a sync

Compressed modes automatically sync each dirty WAL at their background flush points. Writes that have not reached a completed sync can still be lost:

Automatic syncs seal the tail into checksummed blocks in the main file and sync that file once. That immutable prefix remains protected across later tail rollovers, even if the tail checkpoint is lost. Unchanged WALs skip repeated disk syncs. Full-block rollovers themselves do not fsync, and background intervals are scheduling settings rather than a guaranteed persistence deadline. Await SyncWALAsync to wait for the complete write prefix across all segments before acknowledging it.

When you need it

The decision depends on how likely a failure is to interrupt a recent write, what losing that update would mean, and the cost of waiting on every save. An explicit SyncWALAsync() makes the application wait for a particular persistence boundary.

For example, consider the following hypothetical server. These failure rates and exposure times are assumptions for the calculation, not measured industry rates or ZoneTree guarantees. In particular, the 100 ms power-loss exposure below cannot be inferred from the background writer's polling interval:

For failures independent of write timing, the chance of overlap is approximately the failure frequency multiplied by the exposed time. These figures use a 365-day year and a constant-rate failure model. A failure triggered by the request itself does not fit that independence assumption. An application crash only threatens writes still pending inside the process; it does not discard data already in the OS file cache.

Consider a user updating a display name, biography, or notification preference. The application may accept the small chance that a recent edit must be repeated after a failure. In that case, making every save wait for disk synchronization imposes a cost on every update to reduce a risk the product can tolerate. The profile is still persistent data; normal background persistence can be an appropriate choice for it.

Write volume also matters. At 100 writes per second, a 100 ms exposure represents about 10 recent writes at the time of a failure. A small probability for one write can therefore still mean several affected updates on a busy server. Whether that is acceptable depends on the consequences: repeating a profile edit is different from losing a confirmed ledger entry or a message whose sender has deleted its copy.

Use SyncWALAsync() when those consequences require an acknowledgment to wait for disk synchronization. It reduces the risk from pending local writes; it does not eliminate network failures, hardware failures, or every other way an operation can fail. One wait after all writes in a request or batch can cover that acknowledgment without paying for a separate wait after each record.

How the sync works

The task completes when the WAL records of every write that completed before the call are synced to the disk, including writes in segments that are still closing or waiting to merge. Concurrent calls share disk syncs, so many writers waiting together cost about one sync.

If a WAL write or disk sync fails, the task faults with WriteAheadLogSyncFailedException. A shared journal keeps reporting the failure and cannot publish a new checkpoint over it; reopen the tree to recover. A transaction whose durable sync fails stays InDoubt and invisible in that instance, and further writing commits are rejected. Its commit record may still be found during recovery. A cancellation token on SyncWALAsync cancels only the wait, not the sync.

With shared WALs, moving a memory segment forward freezes its logical journal handle and gives the new segment another handle into the same ordered journal. Rotation does not wait for a WAL sync. Recovery validates records in global journal order and stops at the first incomplete suffix across all chunks. Transactions recover one prefix of commits, including every synced commit; incomplete transactions remain invisible. Missing required chunks or broken sequence continuity are corruption.

Ordinary trees can still use the per-segment modes Sync, SyncCompressed, and AsyncCompressed. Their unsynced records can persist independently across segments; operation indexes order surviving records but do not restore missing ones. The built-in transactional factories use shared WALs to avoid that limitation.

Shared WAL syncs also persist new journal file names with a directory sync on platforms that require it. Merge publication persists the replacement disk files' directory entries before publishing metadata or reclaiming old segments and WAL chunks. Metadata snapshots persist their rename before clearing the preceding metadata WAL. Custom file providers must supply equivalent namespace durability through SyncDirectory.

Shared journal chunks

Shared and AsyncShared use the same compressed-block stream and version-1 tail format as the private compressed WALs. One active physical chunk serves all memory segments. SharedModeOptions.MaximumChunkSize targets 64 MiB of uncompressed records by default. Records can span compression blocks but never split between chunks; a single larger record can exceed the chunk target.

Physical rotation seals the completed chunk, copies its remaining tail into its main file, and releases its handles and mapping without a disk sync. Later syncs flush unsynced closed main files one at a time and skip sealed chunks already made durable. A continuously busy async writer therefore does not retain every old chunk's open handles.

Only whole obsolete chunks at the beginning of the journal are reclaimed. The durable manifest advances the replay start before those files are deleted. A chunk containing records still needed by a live segment remains retained.

Configure only what you need

To state the default explicitly in a shared factory configuration:

using ZoneTree;
using ZoneTree.Options;

using var tree = new ZoneTreeFactory<int, string>()
    .SetDataDirectory("data/explicit-wal")
    .ConfigureWriteAheadLogOptions(options =>
    {
        options.WriteAheadLogMode = WriteAheadLogMode.AsyncCompressed;
    })
    .OpenOrCreate();
using var maintainer = tree.CreateMaintainer();

tree.Upsert(42, "Logged in the background");

For codec defaults and block-size controls, see compression.

Existing private WALs keep their detected logging mode and recorded compression format; their filenames identify the mode. Existing shared journals keep the mode in their manifest. Preserve the full WAL directory when moving a database. When selecting shared logging for a database with legacy per-segment WALs, the built-in provider migrates each live WAL and tail into the shared journal and syncs the replacement before deleting the legacy files. Released WALs from 2.0.8 and earlier remain supported.

Compressed blocks and the tail file

Compressed WALs have a main file of committed blocks and a companion .tail file for the unfinished block. Keep both when making an offline copy.

A write to the stream reaches the operating system before it returns, so it survives a process crash. With the local file stream provider, the active tail is mapped into memory, and appending within it is a memory copy into the operating system's file cache. The prepared tail allocation holds an 8-byte header and one compression block, rounded up to a 512-byte boundary, followed by two 512-byte checkpoint sectors. A 16-byte format descriptor occupies the end of the reserved space. With the default 256 KiB block size, this totals 263,680 bytes. Other file stream providers write the payload and updated length header through their streams.

A sync seals the current tail into the main file and syncs that file once. Each new block has a 32-byte header containing its index, stored and logical lengths, a checksum of the stored payload, and a checksum of the header. Recovery validates the blocks themselves, so the tail checkpoint does not need a second disk sync. It is written after the main-file sync as a recovery hint: any copy that persists identifies a durable prefix, but recovery also works when that hint is absent or older than the latest sync. Explicit and automatic syncs use the same path, and later appends reuse the tail without overwriting the durable main prefix.

WALs from 2.0.8 and earlier remain readable, including their variable-size tails. Before a legacy tail is reused, its content is sealed into a new checksummed main-file block and synced. The original main-file blocks remain readable, while subsequent appends use the new block format. The tail allocation keeps format version 1.

A recovered unfinished tail can also hold the only durable copy of records saved by a clean close. Its first rollover syncs the main-file copy before reusing that tail. This protection applies once to the recovered tail; subsequent ordinary rollovers do not issue disk syncs.

Automatic sync scheduling:

The async writer's default EmptyQueuePollInterval is 100 ms. These intervals control scheduling, not a maximum persistence delay. For transactional trees, AutoSyncInterval schedules explicit sync requests even while the queue stays busy and advances MVCC's durable bookkeeping. It defaults to zero. The WAL's own background job can sync bytes without updating that MVCC timestamp.

Close the tree cleanly

Stop accepting application work, finish active operations and scans, then dispose the maintainer and tree. Declaring the maintainer after the tree, as above, gives the right reverse disposal order.

A normal close does not require merging logged records first. Maintainer waits coordinate tracked merge work; tree disposal finishes its WAL work. See maintenance for lifecycle details.

Async logging may serialize a key or value after the write call returns. Treat stored objects and backing buffers as read-only, as described in value mutability.

Use None for an intentional rebuild policy

With None, metadata and merged disk records remain, but unmerged writes cannot be recovered after closing. For a controlled handoff without WAL, finish the required merge with writers coordinated, as shown in file stream providers.