Token

sealed interface Token

A semantic token produced by the lexer (Stage 2).

Block tokens are produced by the block lexer (one per logical line or construct). Inline tokens are produced by the inline lexer when processing paragraph/heading content.

Every token carries a range tracing back to the original source characters via the constituent lexemes' SourceRanges.

Inheritors

Types

AtxHeadingContent

data class AtxHeadingContent(val lexemes: List<Lexeme>, val range: SourceRange) : Token

The inline content of an ATX heading, as raw lexemes for later inline parsing.

AtxHeadingMarker

data class AtxHeadingMarker(val level: Int, val range: SourceRange) : Token

An ATX heading marker (# through ######).

BlankLine

data class BlankLine(val range: SourceRange) : Token

A line containing only whitespace (or empty).

BlockQuoteMarker

data class BlockQuoteMarker(val range: SourceRange) : Token

A block quote marker (>).

BulletMarker

data class BulletMarker(val marker: Char, val range: SourceRange) : Token

A bullet list marker (-, +, or *).

CodeContent

data class CodeContent(val literal: String, val range: SourceRange) : Token

A line of code content (inside a fenced or indented code block).

CodeFenceClose

data class CodeFenceClose(val fenceChar: Char, val fenceLength: Int, val range: SourceRange) : Token

A closing code fence.

CodeFenceInfo

data class CodeFenceInfo(val info: String, val range: SourceRange) : Token

The info string after an opening code fence.

CodeFenceOpen

data class CodeFenceOpen(val fenceChar: Char, val fenceLength: Int, val indent: Int, val range: SourceRange) : Token

An opening code fence (` ``` ` or ~~~). @property fenceChar the fence character ('`' or '~') @property fenceLength the number of fence characters (≥ 3) @property indent the number of leading spaces (0–3)

HtmlBlockLine

data class HtmlBlockLine(val literal: String, val range: SourceRange) : Token

An HTML block line.

IndentedCodeLine

data class IndentedCodeLine(val literal: String, val range: SourceRange) : Token

A line of indented code (4+ spaces or 1+ tab of leading indent).

OrderedMarker

data class OrderedMarker(val number: Int, val delimiter: Char, val range: SourceRange) : Token

An ordered list marker (e.g. 1., 2), etc.).

ParagraphLine

data class ParagraphLine(val lexemes: List<Lexeme>, val range: SourceRange) : Token

A paragraph line — inline content deferred for the inline pass.

SetextUnderline

data class SetextUnderline(val level: Int, val text: String, val range: SourceRange) : Token

A setext heading underline (=== or ---).

ThematicBreakLine

data class ThematicBreakLine(val marker: Char, val range: SourceRange) : Token

A thematic break line (---, ***, ___, etc.).

Properties

range

abstract val range: SourceRange