Package org.apache.juneau.marshall.json

Class JsonTokenReader

java.lang.Object

org.apache.juneau.marshall.json.JsonTokenReader

All Implemented Interfaces:

Closeable, AutoCloseable, RecordReader, TokenReader

Direct Known Subclasses:

Json5TokenReader, JsonlTokenReader

public class JsonTokenReader
extends Object
implements TokenReader

Reference implementation of the public TokenReader surface for the JSON format.

Backed by the existing internal ParserReader plumbing. Operates as a hand-rolled state machine over the JSON grammar — structural tokens ({, }, [, ], ,, :) and the four scalar shapes (string, number, boolean, null).

Pipe-level settings (charset, auto-close, unbuffered) are configured on the ParserPipe the caller hands in. Cursor-level behavior is driven by a small JsonTokenReader.Settings bundle — currently just trimStrings.

Notes:

• This class is not thread safe.
• Whitespace and JSON comments are tolerated — comments are silently skipped to mirror Juneau's existing JSON parser behavior.
• The cursor is purely structural; object swaps and @Schema annotations have no effect. Use the POJO databind path on JsonParser for those.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final record	JsonTokenReader.Settings	Cursor-level settings honored by JsonTokenReader.

Field Summary

Fields

Modifier and Type	Field	Description
protected boolean	currentBoolean	Boolean value buffer for the current TokenType.VALUE_BOOLEAN token.
protected String	currentFieldName	Field name buffer for the current TokenType.FIELD_NAME token.
protected String	currentNumberLexeme	Numeric lexeme buffer for the current TokenType.VALUE_NUMBER token (preserved verbatim from the input).
protected String	currentString	String value buffer for the current TokenType.VALUE_STRING token.
protected TokenType	currentToken	The token most recently emitted by next().
protected int	depth	Current container nesting depth (0 = root).
protected final ParserReader	r	Underlying character-level reader; subclasses use this for direct read/unread/mark/getMarked calls during dialect-specific tokenization.
protected static final int	S00_expectValue	State-machine state: at root, after [, after :, or after , inside an array.
protected static final int	S01_expectFieldName	State-machine state: after { or after , inside an object.
protected static final int	S02_expectColon	State-machine state: just emitted a TokenType.FIELD_NAME.
protected static final int	S03_expectCommaOrEndObject	State-machine state: just emitted a value inside an object.
protected static final int	S04_expectCommaOrEndArray	State-machine state: just emitted a value inside an array.
protected static final int	S05_end	State-machine state: the root document has been fully consumed.
protected final JsonTokenReader.Settings	settings	Cursor-level settings (currently just trimStrings).
protected int	state	Current state-machine state; one of the STATE_* constants.

Fields inherited from interface org.apache.juneau.marshall.stream.TokenReader

MSG_READ_NON_VALUE_STATE

Constructor Summary

Constructors

Constructor	Description
JsonTokenReader(ParserPipe pipe)	Constructor with default JsonTokenReader.Settings.
JsonTokenReader(ParserPipe pipe, JsonTokenReader.Settings settings)	Constructor with explicit JsonTokenReader.Settings.
JsonTokenReader(ParserPipe pipe, JsonTokenReader.Settings settings, JsonParserSession session)	Constructor used by JsonParser.readTokens(Object) to plumb the calling parser's session in so that read(Class) can delegate to the existing databind path.

Method Summary

Modifier and Type	Method	Description
protected void	afterValue()	Sets the post-value state based on the enclosing container.
protected boolean	allowsTrailingComma()	Subclass extension point: returns whether trailing commas are accepted before ] / }.
boolean	canRead()	Returns true if TokenReader.read(Class) is currently callable on this cursor — i.e. the cursor is positioned at a value boundary.
void	close()	Closes the underlying stream.
byte[]	getBinary()	Returns the current binary value.
boolean	getBool()	Returns the current boolean value.
TokenType	getCurrentToken()	Returns the current token type without advancing.
int	getDepth()	Returns the current depth of the cursor.
String	getFieldName()	Returns the field name when the current token is TokenType.FIELD_NAME.
Number	getNumber()	Returns the current numeric value as a parsed Number.
String	getNumberLexeme()	Returns the original numeric lexeme for the current numeric token.
String	getString()	Returns the current scalar value as a string.
protected static boolean	isDigit(int c)	Returns whether the given character is an ASCII decimal digit.
protected boolean	isInArray()	Returns whether the cursor's current container (the one most recently pushed) is an array.
protected static boolean	isNumberStart(int c)	Returns whether the given character could start a JSON number lexeme (digit or minus sign).
TokenType	next()	Advances the cursor to the next token.
protected ParseException	parseException(String message, Object... args)	Builds a ParseException with the given message + String.format-style (%s) arguments.
protected void	popContainer()	Pops the top container off the cursor's depth stack.
protected void	pushContainer(boolean isArray)	Pushes a new container onto the cursor's depth stack.
<T> T	read(Class<T> type)	Consumes the upcoming value (object, array, or scalar) and binds it to a POJO via the format's existing parser.
<T> T	read(Type type, Type... args)	Same as TokenReader.read(Class) but accepts a parameterized Type and optional type arguments (the same shape as Parser.read(Object, Type, Type...)).
<T> T	read(ClassMeta<T> type)	Same as TokenReader.read(Class) but accepts a pre-resolved ClassMeta.
protected void	readCommaOrEndArray(int c)	State-machine dispatch: handles the next character at S04_expectCommaOrEndArray.
protected void	readCommaOrEndObject(int c)	State-machine dispatch: handles the next character at S03_expectCommaOrEndObject.
protected void	readFieldNameOrEndObject(int c)	State-machine dispatch: handles the next character at S01_expectFieldName.
protected void	readKeywordValue(int c)	Reads one of the JSON keyword literals (true, false, null) starting with the given lookahead character.
protected void	readNumberValue()	Reads a JSON number from the current input position.
protected int	readSkipWsAndComments()	Reads the next non-whitespace, non-comment character or -1 at end of stream.
protected String	readString()	Reads a double-quoted JSON string from the current input position.
protected void	readValueStart(int c)	State-machine dispatch: handles the next character at S00_expectValue.
void	skipChildren()	Skips over the children of the current structural token.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.juneau.marshall.stream.RecordReader

iterator, stream

Methods inherited from interface org.apache.juneau.marshall.stream.TokenReader

getExtType, getNativeKind, getSimpleValue, getTag, getTagCount, isStreaming

Field Details

S00_expectValue

protected static final int S00_expectValue

State-machine state: at root, after [, after :, or after , inside an array. The next read should be a JSON value (object, array, or scalar).

See Also:

• Constant Field Values

S01_expectFieldName

protected static final int S01_expectFieldName

State-machine state: after { or after , inside an object. The next read should be a field name (string).

See Also:

• Constant Field Values

S02_expectColon

protected static final int S02_expectColon

State-machine state: just emitted a TokenType.FIELD_NAME. The next char should be :.

See Also:

• Constant Field Values

S03_expectCommaOrEndObject

protected static final int S03_expectCommaOrEndObject

State-machine state: just emitted a value inside an object. The next char should be , (followed by another field name) or }.

See Also:

• Constant Field Values

S04_expectCommaOrEndArray

protected static final int S04_expectCommaOrEndArray

State-machine state: just emitted a value inside an array. The next char should be , (followed by another value) or ].

See Also:

• Constant Field Values

S05_end

protected static final int S05_end

State-machine state: the root document has been fully consumed. The next call to next() returns TokenType.END_OF_STREAM.

See Also:

• Constant Field Values

r

protected final ParserReader r

Underlying character-level reader; subclasses use this for direct read/unread/mark/getMarked calls during dialect-specific tokenization.

settings

protected final JsonTokenReader.Settings settings

Cursor-level settings (currently just trimStrings).

currentToken

protected TokenType currentToken

The token most recently emitted by next(). Subclasses set this when overriding state-machine dispatch methods.

state

protected int state

Current state-machine state; one of the STATE_* constants.

depth

protected int depth

Current container nesting depth (0 = root).

currentFieldName

protected String currentFieldName

Field name buffer for the current TokenType.FIELD_NAME token.

currentString

protected String currentString

String value buffer for the current TokenType.VALUE_STRING token.

currentNumberLexeme

protected String currentNumberLexeme

Numeric lexeme buffer for the current TokenType.VALUE_NUMBER token (preserved verbatim from the input).

currentBoolean

protected boolean currentBoolean

Boolean value buffer for the current TokenType.VALUE_BOOLEAN token.

Constructor Details

JsonTokenReader

public JsonTokenReader(ParserPipe pipe)
                throws IOException

Constructor with default JsonTokenReader.Settings.

Parameters:

pipe - The parser input pipe to read from. Must not be null.

Throws:

IOException - If a problem occurred opening the underlying reader.

JsonTokenReader

public JsonTokenReader(ParserPipe pipe,
 JsonTokenReader.Settings settings)
                throws IOException

Constructor with explicit JsonTokenReader.Settings.

Parameters:

pipe - The parser input pipe to read from. Must not be null.

settings - The cursor-level settings. Must not be null.

Throws:

IOException - If a problem occurred opening the underlying reader.

JsonTokenReader

public JsonTokenReader(ParserPipe pipe,
 JsonTokenReader.Settings settings,
 JsonParserSession session)
                throws IOException

Constructor used by JsonParser.readTokens(Object) to plumb the calling parser's session in so that read(Class) can delegate to the existing databind path.

Parameters:

pipe - The parser input pipe to read from. Must not be null.

settings - The cursor-level settings. Must not be null.

session - The JsonParserSession to use for read(Class) delegation, or null to disable read on this reader (it will throw UnsupportedOperationException).

Throws:

IOException - If a problem occurred opening the underlying reader.

Method Details

next

public TokenType next()
               throws IOException,
ParseException

Description copied from interface: TokenReader

Advances the cursor to the next token.

Specified by:

next in interface TokenReader

Returns:

The token type of the new current token, or TokenType.END_OF_STREAM when the underlying input is exhausted. Never returns TokenType.NOT_AVAILABLE — that value is reserved for the initial pre-next() state.

Throws:

IOException - If a problem occurred reading from the underlying stream.

ParseException - If the input is malformed for the format.

getCurrentToken

public TokenType getCurrentToken()

Description copied from interface: TokenReader

Returns the current token type without advancing.

Specified by:

getCurrentToken in interface TokenReader

Returns:

The current token type. Returns TokenType.NOT_AVAILABLE before the first call to TokenReader.next(), and TokenType.END_OF_STREAM once the stream is exhausted.

getDepth

public int getDepth()

Description copied from interface: TokenReader

Returns the current depth of the cursor.

Depth starts at 0 at the document root and increases by 1 for each TokenType.START_OBJECT / TokenType.START_ARRAY encountered (and decreases on the matching end tokens).

Specified by:

getDepth in interface TokenReader

Returns:

The current depth.

getFieldName

public String getFieldName()

Description copied from interface: TokenReader

Returns the field name when the current token is TokenType.FIELD_NAME.

Specified by:

getFieldName in interface TokenReader

Returns:

The field name.

getString

public String getString()

Description copied from interface: TokenReader

Returns the current scalar value as a string.

Defined for TokenType.VALUE_STRING (the natural value), TokenType.FIELD_NAME (returns the field name), TokenType.VALUE_NUMBER (returns the lexeme), and TokenType.VALUE_BOOLEAN (returns "true"/"false"). For TokenType.VALUE_NULL returns null.

Specified by:

getString in interface TokenReader

Returns:

The string view of the current value.

getNumber

public Number getNumber()
                 throws ParseException

Description copied from interface: TokenReader

Returns the current numeric value as a parsed Number.

Implementations may parse lazily; the returned Number subtype is implementation- and format-defined (e.g. Long / Double / BigDecimal). Use TokenReader.getNumberLexeme() to recover the original textual form.

Specified by:

getNumber in interface TokenReader

Returns:

The parsed numeric value.

Throws:

ParseException - If the lexeme cannot be parsed.

getNumberLexeme

public String getNumberLexeme()

Description copied from interface: TokenReader

Returns the original numeric lexeme for the current numeric token.

For text formats this is the exact substring that produced the number. For binary formats the implementation returns a canonical text rendering of the native numeric token.

Specified by:

getNumberLexeme in interface TokenReader

Returns:

The numeric lexeme.

getBool

public boolean getBool()

Description copied from interface: TokenReader

Returns the current boolean value.

Specified by:

getBool in interface TokenReader

Returns:

The boolean value.

getBinary

public byte[] getBinary()

Description copied from interface: TokenReader

Returns the current binary value.

Binary scalars come from binary formats only (cbor, msgpack, bson, ...). Text formats deliver embedded binary content as base64-encoded TokenType.VALUE_STRING.

Specified by:

getBinary in interface TokenReader

Returns:

The binary value.

canRead

public boolean canRead()
                throws IOException,
ParseException

Description copied from interface: TokenReader

Returns true if TokenReader.read(Class) is currently callable on this cursor — i.e. the cursor is positioned at a value boundary.

This is the natural loop predicate when streaming records:

try (TokenReader r = Json.DEFAULT.readTokens(input)) { r.next(); // consume the opening [ while (r.canRead()) { Item item = r.read(Item.class); process(item); } r.next(); // consume the closing ] }

Specified by:

canRead in interface RecordReader

Specified by:

canRead in interface TokenReader

Returns:

true if the next operation can be read; false if the cursor is positioned at a structural separator, field name, end-container, or end-of-stream.

Throws:

IOException - If a problem occurred peeking at the underlying stream.

ParseException - If the input is malformed for the format.

allowsTrailingComma

protected boolean allowsTrailingComma()

Subclass extension point: returns whether trailing commas are accepted before ] / }.

Default returns false (strict RFC-8259 JSON). Json5TokenReader overrides this to return true for the JSON5 dialect.

Returns:

true if a trailing comma should be silently accepted; false if it should produce a parse error.

read

public <T> T read(Class<T> type)
           throws IOException,
ParseException

Description copied from interface: TokenReader

Consumes the upcoming value (object, array, or scalar) and binds it to a POJO via the format's existing parser.

Cursor positioning: the cursor must be in a state where the next call to TokenReader.next() would emit a value-shaped token (i.e. TokenType.START_OBJECT, TokenType.START_ARRAY, or one of the VALUE_* tokens). Calling read consumes that upcoming value wholesale and advances the cursor's depth back to where it was before the value would have started — structurally equivalent to calling TokenReader.next() on a scalar, or TokenReader.next() + TokenReader.skipChildren() on a container.

Do NOT call TokenReader.next() before read. Once the structural opener (e.g. { or [) has been consumed by TokenReader.next(), the underlying parser cannot re-bind the value. The contract is "instead-of-next, not after-next."

Default behavior: the default implementation throws UnsupportedOperationException. Format-specific token readers override this by delegating to their existing *ParserSession at the cursor's current input position and resyncing the cursor's state machine afterward.

Specified by:

read in interface RecordReader

Specified by:

read in interface TokenReader

Type Parameters:

T - The expected return type.

Parameters:

type - The target Java type.

Returns:

The bound POJO.

Throws:

IOException - If a problem occurred reading from the underlying stream.

ParseException - If the input is malformed for the format or cannot be bound to the type.

read

public <T> T read(ClassMeta<T> type)
           throws IOException,
ParseException

Description copied from interface: TokenReader

Same as TokenReader.read(Class) but accepts a pre-resolved ClassMeta.

Specified by:

read in interface RecordReader

Specified by:

read in interface TokenReader

Type Parameters:

T - The expected return type.

Parameters:

type - The target type.

Returns:

The bound POJO.

Throws:

IOException - If a problem occurred reading from the underlying stream.

ParseException - If the input is malformed or cannot be bound to the type.

read

public <T> T read(Type type,
 Type... args)
           throws IOException,
ParseException

Description copied from interface: TokenReader

Same as TokenReader.read(Class) but accepts a parameterized Type and optional type arguments (the same shape as Parser.read(Object, Type, Type...)).

Specified by:

read in interface RecordReader

Specified by:

read in interface TokenReader

Type Parameters:

T - The expected return type.

Parameters:

type - The target type.

args - Optional type arguments for parameterized types.

Returns:

The bound POJO.

Throws:

IOException - If a problem occurred reading from the underlying stream.

ParseException - If the input is malformed or cannot be bound to the type.

skipChildren

public void skipChildren()
                  throws IOException,
ParseException

Description copied from interface: TokenReader

Skips over the children of the current structural token.

If the current token is TokenType.START_OBJECT or TokenType.START_ARRAY, the cursor is advanced to the matching TokenType.END_OBJECT or TokenType.END_ARRAY (i.e. the entire sub-tree is consumed). For any other token this is a no-op.

Specified by:

skipChildren in interface TokenReader

Throws:

IOException - If a problem occurred reading from the underlying stream.

ParseException - If the input is malformed for the format.

close

public void close()
           throws IOException

Description copied from interface: TokenReader

Closes the underlying stream.

Implementations should be idempotent — closing a closed reader is a no-op.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Specified by:

close in interface RecordReader

Specified by:

close in interface TokenReader

Throws:

IOException

readValueStart

protected void readValueStart(int c)
                       throws IOException,
ParseException

State-machine dispatch: handles the next character at S00_expectValue.

Subclasses (e.g. Json5TokenReader) override this to add dialect-specific value shapes (single-quoted strings, bare identifiers, missing-value handling).

Parameters:

c - The next non-whitespace, non-comment character read from the input.

Throws:

IOException - If a problem occurred reading from the underlying stream.

ParseException - If the input is malformed at this position.

readFieldNameOrEndObject

protected void readFieldNameOrEndObject(int c)
                                 throws IOException,
ParseException

State-machine dispatch: handles the next character at S01_expectFieldName.

Subclasses (e.g. Json5TokenReader) override this to accept bare-identifier field names and single-quoted field names.

Parameters:

c - The next non-whitespace character read from the input.

Throws:

IOException - If a problem occurred reading from the underlying stream.

ParseException - If the input is malformed at this position.

readCommaOrEndObject

protected void readCommaOrEndObject(int c)
                             throws IOException,
ParseException

State-machine dispatch: handles the next character at S03_expectCommaOrEndObject.

Subclasses (e.g. Json5TokenReader) override this to accept trailing commas before }.

Parameters:

c - The next non-whitespace character read from the input.

Throws:

IOException - If a problem occurred reading from the underlying stream.

ParseException - If the input is malformed at this position.

readCommaOrEndArray

protected void readCommaOrEndArray(int c)
                            throws IOException,
ParseException

State-machine dispatch: handles the next character at S04_expectCommaOrEndArray.

Subclasses (e.g. Json5TokenReader) override this to accept trailing commas before ].

Parameters:

c - The next non-whitespace character read from the input.

Throws:

IOException - If a problem occurred reading from the underlying stream.

ParseException - If the input is malformed at this position.

afterValue

protected void afterValue()

Sets the post-value state based on the enclosing container.

Subclasses (e.g. JsonlTokenReader) override this when the format treats consecutive top-level values differently — JSONL transitions back to S00_expectValue at depth 0 instead of S05_end so the next line is read.

readString

protected String readString()
                     throws IOException,
ParseException

Reads a double-quoted JSON string from the current input position.

Handles escape sequences (\\n, \\t, \\uNNNN, etc.) inline via ParserReader.replace(char). Subclasses (e.g. Json5TokenReader) override this to also accept single-quoted strings.

Returns:

The parsed string (without surrounding quotes).

Throws:

IOException - If a problem occurred reading from the underlying stream.

ParseException - If the input is malformed at this position.

readKeywordValue

protected void readKeywordValue(int c)
                         throws IOException,
ParseException

Reads one of the JSON keyword literals (true, false, null) starting with the given lookahead character. Sets currentToken and currentBoolean accordingly.

Parameters:

c - The first character of the keyword (one of 't', 'f', 'n').

Throws:

IOException - If a problem occurred reading from the underlying stream.

ParseException - If the keyword is malformed.

readNumberValue

protected void readNumberValue()
                        throws IOException,
ParseException

Reads a JSON number from the current input position. Captures the lexeme via ParserReader.readNumberString() and validates it; defers actual numeric parsing to getNumber() (lazy). Sets currentNumberLexeme and currentToken.

Throws:

IOException - If a problem occurred reading from the underlying stream.

ParseException - If the input is not a valid JSON number.

readSkipWsAndComments

protected int readSkipWsAndComments()
                             throws IOException,
ParseException

Reads the next non-whitespace, non-comment character or -1 at end of stream.

Returns:

The next significant character, or -1 at end of input.

Throws:

IOException - If a problem occurred reading from the underlying stream.

ParseException - If a JSON comment is malformed.

isNumberStart

protected static boolean isNumberStart(int c)

Returns whether the given character could start a JSON number lexeme (digit or minus sign).

Parameters:

c - The character.

Returns:

true if the character could start a number.

isDigit

protected static boolean isDigit(int c)

Returns whether the given character is an ASCII decimal digit.

Parameters:

c - The character.

Returns:

true if the character is between '0' and '9' inclusive.

pushContainer

protected void pushContainer(boolean isArray)

Pushes a new container onto the cursor's depth stack.

Parameters:

isArray - true if the new container is an array; false for an object.

popContainer

protected void popContainer()
                     throws ParseException

Pops the top container off the cursor's depth stack.

Throws:

ParseException - If the stack is already empty (defensive only; well-formed JSON cannot trigger this).

isInArray

protected boolean isInArray()

Returns whether the cursor's current container (the one most recently pushed) is an array.

Returns:

true if currently inside an array; false if inside an object.

parseException

protected ParseException parseException(String message,
 Object... args)

Builds a ParseException with the given message + String.format-style (%s) arguments. Used by all dispatch methods to surface input-malformed errors with consistent formatting.

Parameters:

message - The exception message format string (%s, %d, ...).

args - The format arguments.

Returns:

A new ParseException ready to throw.