Replication

KVNode streams changes from a parent node to child nodes via an NDJSON-based replication protocol.

How Replication Works

1. A client writes a key-value pair to a node.
2. The receiving node persists the write in its WAL and applies it to the in-memory store.
3. If the node has a parent configured, it forwards the write upstream via the /v1/replicate endpoint.
4. If the node has children, it broadcasts the write to all connected replicas over the replication stream.
5. Each replica applies the change to its own store, maintaining an eventually consistent copy.

Replication Protocol

The replication endpoint (POST /v1/replicate) uses NDJSON (Newline-Delimited JSON) as the wire format. The protocol supports these command types:

Example: Replication Stream

1{"type":"BASELINE_BEGIN","collection":"my-collection","snapshotSeq":1}
2{"type":"SET","collection":"my-collection","seq":1,"key":"user:alice","value":{"role":"admin"},"version":1,"timestamp":1700000000}
3{"type":"BASELINE_END","collection":"my-collection","snapshotSeq":1}
4{"type":"SET","collection":"my-collection","seq":2,"key":"user:bob","value":{"role":"user"},"version":1,"timestamp":1700000001}
5{"type":"PING","timestamp":1700000010}

Node Status

The /v1/status endpoint exposes the replication topology:

1curl "https://api.hola.cloud/v1/status" \
2  -H "apikey: your-api-key" \
3  -H "secret: your-api-secret"

Response:

 1{
 2  "node": "node-1",
 3  "role": "primary",
 4  "parent": "",
 5  "collections": {
 6    "my-collection": {
 7      "keys": 42,
 8      "lastSeq": 128,
 9      "walSizeBytes": 65536
10    }
11  },
12  "children": 2,
13  "uptimeSeconds": 86400,
14  "replication": {
15    "enabled": true,
16    "parent_connected": false
17  }
18}

Node Metrics

Monitor performance with the /v1/metrics endpoint:

1curl "https://api.hola.cloud/v1/metrics" \
2  -H "apikey: your-api-key" \
3  -H "secret: your-api-secret"

Response:

1{
2  "writes_total": 15000,
3  "reads_total": 82000,
4  "replication_commands_sent_total": 15000,
5  "replication_commands_received_total": 0,
6  "children_connected": 2,
7  "parent_connected": false
8}

Readiness Checks

The /readyz endpoint verifies that a node is ready to serve traffic. For replicas, this means the parent connection must be established and the WAL fully replayed. For primaries, readiness is immediate after startup.

1curl "https://api.hola.cloud/readyz"

Response when not ready:

1{"ok":false,"node":"node-2","role":"replica","ready":false,"checks":{"wal_replayed":true,"parent_connected":false}}

Pluggable WAL Backends

KVNode supports multiple WAL backends, selectable at startup:

Each backend implements the same WAL interface, so switching between them requires no application-level changes.