Skip to main content

Array-Record Streaming

The readArrayRecords(...) / writeArrayRecords(...) factories — declared by the ArrayRecordReadable / ArrayRecordWritable roles — yield a multi-record cursor over the elements of a top-level wire array, without materializing the whole list. This applies to single-rooted formats whose root value happens to be a list — JSON's [...], CBOR's array major-type, etc.

For naturally-multi-record formats (csv, sse), use readRecords / writeRecords directly — the wire format IS already a sequence of records. JSONL is also natively line-delimited; its readArrayRecords / writeArrayRecords simply alias that line record stream (rather than wrapping the elements in a bracketed [...] array).

See the per-format capability matrix for which formats support array-record reading/writing and whether each streams O(1) or buffers.

Reading large arrays element-by-element​

// Wire payload is one big JSON array; we want O(1) memory per element.
try (RecordReader r = JsonParser.DEFAULT.readArrayRecords(input)) {
    while (r.canRead()) {
        Bean b = r.read(Bean.class);
        process(b);
    }
}

Writing large arrays element-by-element​

try (RecordWriter w = JsonSerializer.DEFAULT.writeArrayRecords(output)) {
    for (Bean b : sourceOfBeans()) {
        w.write(b);
    }
}
// output is now: [{...},{...},{...}]

Format applicability​

FormatReaderWriterNotes
JSON / JSON5streamingstreamingO(1) memory per element.
CBORstreamingstreamingReader uses indefinite-length array peek (BREAK). Writer always emits indefinite-length so no element count needed.
MsgPackstreamingbuffered; streaming with count overloadReader streams O(1) per element. Writer's no-count writeArrayRecords(output) buffers; writeArrayRecords(output, expectedCount) writes the length-prefix header up front and serializes each element directly.
BSONbufferedbufferedBSON requires byte-length prefix at every nesting level; streaming without buffering isn't possible.
Parquet / YAMLbufferedbuffered
XML / HTMLbuffered with caller-specified rootnot supported (reader only)Reader has the readArrayRecords(input, "rootElementName") overload; there is no array-record writer.
JSONLstreamingstreamingarrayRecords aliases the JSONL line record stream — each top-level line is one element, written/read without a surrounding [...]. O(1) memory per element.
CSV / SSEn/an/aUse readRecords/writeRecords — format is already row-streamed.
TOML / HOCON / Hjson / INI / PlainText / Markdown / JsonSchema / JCS / Prototextnot supportednot supported

Length-prefixed writers (MsgPack count overload)​

For binary formats whose array wire encoding starts with a length prefix, the count overload streams without buffering — the caller pre-declares the element count:

ByteArrayOutputStream baos = new ByteArrayOutputStream();
try (RecordWriter w = MsgPackSerializer.DEFAULT.writeArrayRecords(baos, 3)) {
    w.write(bean1);
    w.write(bean2);
    w.write(bean3);
}
// Bytes equal to: MsgPackSerializer.DEFAULT.write(List.of(bean1, bean2, bean3))

Mismatch between the declared count and the actual write count throws on close(). Writing a 4th element when 3 were declared throws IllegalStateException immediately.

Capability check​

boolean supported = MsgPackSerializer.DEFAULT instanceof ArrayRecordWritable;   // true
if (MsgPackSerializer.DEFAULT instanceof ArrayRecordWritable w) {
    boolean streaming = w.isArrayRecordStreaming();   // false — buffered without the count overload
}

See also​