Stream the objects of a gzipped JSON array in batches

The bulk is a JSON array of more than a million objects; parsing it in one call holds the whole decoded tree in memory, which is what exhausted the stress-test runners. This reader walks the file in fixed-size byte chunks and emits the array in batches of batch_size complete top-level objects, each as its own JSON array text, so the caller parses one batch at a time. Object boundaries are found with a vectorised scan: unescaped quotes delimit string literals, braces outside them change the depth, and an object ends where the depth returns to zero. Nothing beyond n_max objects is read, so a sample of a 200 MB bulk costs a few megabytes. The batch size bounds the transient the JSON decoder needs: 5,000 objects keep it near 1 GB where 25,000 took 1.7 GB, at the same speed.

Usage

read_json_objects(
  path,
  emit,
  n_max = Inf,
  batch_size = 5000L,
  chunk_size = 16777216L
)

Arguments

path

Path to a gzipped (or plain) JSON array file.

emit

Function called with the JSON text of each batch.

n_max

Maximum number of objects to read (default: all).

batch_size

Objects per batch.

chunk_size

Bytes read from the file per step.

Value

The number of objects emitted, invisibly.