Skip to main content

Token & Record Streaming

Juneau exposes a public low-level streaming API that complements the POJO databind path (Serializer.write(...) / Parser.read(...)) with cursor-based push and pull primitives. Two surfaces, layered:

Surfaces at a glance​

SurfaceReturnsMethodsWhen to use
RecordReader / RecordWriterone whole-value record per callcanRead() / read(Class), write(Object)Format-neutral whole-value access; works on every Juneau format.
TokenReader / TokenWriterone fine-grained event per callnext(), startObject() / fieldName(...) / string(...) / etc.Structural events for formats whose wire encoding is already token-shaped (JSON family, CBOR, MsgPack).

TokenReader extends RecordReader and TokenWriter extends RecordWriter, so FULL-capability formats inherit record-stream semantics for free. PARTIAL-capability formats expose only the record surface — calling structural methods on them is a compile error (the type system enforces "no nested structure visible at the cursor").

Capability role interfaces​

A format declares what streaming surfaces it supports by implementing role interfaces in org.apache.juneau.marshall.stream. Capability is detected with a plain instanceof check — there is no capability enum:

RoleGranted methodImplemented by
RecordReadable / RecordWritablereadRecords(input) / writeRecords(output)every streaming format
TokenReadable / TokenWritablereadTokens(input) / writeTokens(output)json, json5, jsonl, cbor, msgpack
ArrayRecordReadable / ArrayRecordWritablereadArrayRecords(input) / writeArrayRecords(output)json, json5, cbor, msgpack, bson, parquet, yaml (readers + writers); xml, html (readers only — no array-record writer)

TokenReadable extends RecordReadable (and TokenWritable extends RecordWritable), so a format that exposes the token cursor automatically gets the record cursor for free: its readRecords(...) default returns the readTokens(...) cursor. Formats that only expose the record surface implement RecordReadable/RecordWritable alone — there is no readTokens(...) method to call, so structural access is a compile-time impossibility.

A format exposes the token cursor only when its wire encoding admits a clean structural mapping to the normalized token vocabulary. For example, jsonl is TokenReadable because each line is itself a complete JSON value that maps directly onto the token model, whereas csv and sse are RecordReadable-only because a CSV row and an SSE event carry no nested structure the cursor could surface. Writer-only formats (jsonschema, jcs, soap-xml) have no parser, so they implement only the *Writable roles.

Negotiate a parser/serializer at runtime with instanceof, and use the cursor-quality signal isRecordStreaming() to tell a true O(1) streaming cursor from a buffered one:

boolean json = JsonParser.DEFAULT instanceof TokenReadable;    // true  (FULL)
boolean yaml = YamlParser.DEFAULT instanceof TokenReadable;    // false (record-only)
boolean rec  = YamlParser.DEFAULT instanceof RecordReadable;   // true

if (JsonParser.DEFAULT instanceof RecordReadable p) {
    boolean streaming = p.isRecordStreaming();                 // true = O(1) streaming, false = buffered
}

Per-format capability matrix​

The table below is the authoritative, code-derived summary of what each format supports. Reading columns reflect the roles implemented by the format's Parser; writing columns reflect the roles implemented by its Serializer. The Streaming column is the cursor-quality signal (isRecordStreaming() / isArrayRecordStreaming() / cursor-level isStreaming()).

FormatTierRec readTok readArr readRec writeTok writeArr writeCursor streaming
jsonFULL✓✓✓✓✓✓O(1)
json5FULL✓✓✓✓✓✓O(1)
jsonlFULL✓✓✓ ¹✓✓✓ ¹O(1)
cborFULL✓✓✓✓✓✓O(1)
msgpackFULL✓✓✓ ²✓✓✓ ²O(1) record/token; array-record buffered ²
jsonschemaFULL (writer-only)———✓✓✓ ³buffered ³
jcsFULL (writer-only)———✓✓✓ ³buffered ³
bsonPARTIAL✓✗✓✓✗✓buffered
parquetPARTIAL✓✗✓✓✗✓buffered
yamlPARTIAL✓✗✓✓✗✓buffered
xmlPARTIAL✓✗✓✓✗✗ ⁴buffered
htmlPARTIAL✓✗✓✓✗✗ ⁴buffered
csvPARTIAL✓✗✗✓✗✗buffered ⁵
ssePARTIAL✓✗✗✓✗✗buffered
uonPARTIAL✓✗✗✓✗✗buffered
protoPARTIAL✓✗✗✓✗✗buffered
tomlPARTIAL✓✗✗✓✗✗buffered
iniPARTIAL✓✗✗✓✗✗buffered
hoconPARTIAL✓✗✗✓✗✗buffered
hjsonPARTIAL✓✗✗✓✗✗buffered
markdownPARTIAL✓✗✗✓✗✗buffered
plaintextPARTIAL✓✗✗✓✗✗buffered

Legend

  • ✓ / ✗ — the role interface is / isn't implemented (capability is detected with instanceof); — = no parser exists (writer-only format).
  • Rec / Tok / Arr — the record (RecordReadable/RecordWritable), token (TokenReadable/TokenWritable), and array-record (ArrayRecordReadable/ArrayRecordWritable) surfaces.
  • Tier — FULL = exposes the structural token cursor (TokenReadable/TokenWritable), so it also gets the record cursor for free; PARTIAL = record cursor only (calling structural token methods is a compile error).
  • Cursor streaming — O(1) = a true streaming cursor (constant memory per record/token); buffered = the whole value is materialized (RecordAdapter-backed). "Supported" (✓) and "streams in O(1)" are independent: a format can support a surface yet still buffer it.

Notes

  1. jsonl is naturally multi-record — read/write it with the record cursor (its token cursor doubles as the record cursor), one value per line. Its array-record roles (readArrayRecords / writeArrayRecords) are overridden at the session level to alias that line record stream — each top-level line is one element, emitted/consumed without a surrounding [...] array, streaming O(1). (This corrects the inherited JSON behavior, which targets a single [...]-wrapped document and is invalid for line-delimited JSONL.)
  2. msgpack streams its record/token cursors in O(1), but the array-record cursor is buffered (isArrayRecordStreaming() == false). The writer additionally offers a streaming writeArrayRecords(output, expectedCount) count overload that writes the length-prefix header up front.
  3. jsonschema and jcs are writer-only and token-writable, but their serialize(...) path buffers (isRecordStreaming() == false) because the schema / canonical form is computed over the whole value. The array-record writer is inherited from JSON, and the token writer emits ordinary JSON (the object(Object) databind bridge is disabled — use serialize(...) for schema / canonical output).
  4. xml and html expose the array-record reader (readArrayRecords, buffered, with a caller-specified-root overload) but have no array-record writer.
  5. csv is record-capable but currently buffered (isRecordStreaming() == false); O(1) row streaming is a deferred enhancement.

When to use streaming vs databind​

The POJO databind path (Serializer.write(...) / Parser.read(...)) remains the primary, recommended API. Reach for the streaming surfaces when:

  • You're processing very large or unbounded inputs and don't want to materialize the whole graph.
  • You need to skip parts of a document without binding them.
  • You're composing output from heterogeneous data sources without an intermediate POJO model.
  • You need wire-byte exactness that the POJO path can't guarantee.

The streaming surface is purely structural — object swaps and @Schema annotations apply only on the POJO databind path, never at the token layer.

See also​

In This Section​