Token Profile: Schema and Body

This page covers the fields that token adds on top of the shared descriptor. These are the fields that actually explain how the discrete sequence should be interpreted.

token_unit

Field	Data shape	Typical values	Description
token_unit	Enum or short string	bpe, sentencepiece	Defines the segmentation contract for the sequence

sequence_start

Field	Data shape	Typical values	Description
sequence_start	Integer	128	Start position covered by the current chunk

sequence_end

Field	Data shape	Typical values	Description
sequence_end	Integer	136	End position covered by the current chunk

terminal

Field	Data shape	Typical values	Description
terminal	Boolean	true, false	Indicates whether the current operation or profile output has terminated

stop_reason

Field	Data shape	Typical values	Description
stop_reason	Enum or string	none, eos, tool_call	Explains why output stopped when the schema decides to expose it

sequence_start / sequence_end position the chunk in the logical stream. terminal / stop_reason tell the consumer whether more chunks are still expected.