Add support PSLR(1) parser generation#774
Open
ydah wants to merge 15 commits into
Open
Conversation
ydah
force-pushed
the
pslr_parser
branch
2 times, most recently
from
9846ee2 to
f149937
Compare
ydah
force-pushed
the
pslr_parser
branch
2 times, most recently
from
9105455 to
030c19d
Compare
Introduce the specification language for PSLR(1) pseudo-scanners: * %token-pattern declares a regex pattern for a terminal (with optional tag and alias), registering the terminal on the fly * %lex-prec declares lexical precedence with the seven PSLR operators (<~ <- -~ << -< <s -s), stored as raw declarations for delayed operand expansion; self-pair identity rules are rejected * %symbol-set names a token set usable as operand of other directives * %lex-tie / %lex-no-tie declare lexical ties with reflexive, symmetric, transitive closure and specificity-based no-tie override * %token-action attaches user code to matched token patterns * %lexer-context classifies states for the bridge-mode lexer (Lrama extension kept separate from the PSLR core) Grammar synthesizes implicit literal token patterns for character and string literals used in rules, escaping regex metacharacters, and expands yyall / symbol-set operands lazily via finalize_lexical_declarations! after the synthesis.
Build a single byte-oriented DFA from all %token-pattern regexes via Thompson NFA construction and subset construction (Def 3.2.12/3.2.13). Accepting states keep their full accepting-token sets; identity and length conflicts are intentionally left unresolved inside the DFA so that the parser-state-aware resolver can decide them later (paper section 3.2.6). The regex dialect is a small ASCII subset: literals, escapes, alternation, grouping, quantifiers, character classes with ranges and negation, and dot. Patterns are validated at construction time: nullable patterns, empty patterns/groups/alternatives, dangling escapes, malformed classes and unsupported escapes are rejected (lexemes must be non-empty per Def 2.2.2). ScannerFSA#pairwise_conflict_pairs precomputes the token pairs that can actually be in an identity or length scanner conflict; the set is used to filter %lex-tie expansion and later pair-based analyses.
LengthPrecedences captures the length component of %lex-prec rules as a pairwise resolution table (Def 3.2.15): longest-match operators (<~ -~) prefer the longer match in both directions, shortest-match operators (<s -s) prefer the shorter one, and right-token operators (<< -<) prefer the declared winner regardless of direction. Same-token autolength conflicts default to longest match (Def 3.2.2) unless an explicit -s self-pair is declared. Unrelated pairs stay unresolved on the normal face so that the profile resolver can report them; the fallback face completes them with traditional longest match for the syntax-error row. Contradictory length declarations for the same pair are rejected at construction time with the operator, line and direction of both declarations.
Add States#compute_pslr, an IELR(1) pipeline extended with scanner awareness (paper section 3.4): * Phase 2 builds the scanner FSA and length precedence tables, and the lexical tie closure is finalized against the FSA conflict pairs * Phase 3 state splitting additionally requires PSLR compatibility: states merge only when their acceptable-token sets produce the same pseudo-scanner decisions, checked with unfiltered kernel lookaheads so lookahead filtering cannot mask scanner-visible differences * after conflict resolution, the scanner_accepts table is built with profile-based complete-conflict resolution (Def 3.2.17-3.2.20): each parser state row is computed by walking the DFA with (Ts, ts, Tl) conflict profiles; unresolved complete conflicts are recorded instead of falling back to declaration order * a fallback row over the whole token universe is computed for syntax-error handling, completing residual conflicts with traditional longest-match/declaration-order semantics * acc(sp) uses true reduce lookaheads (not default reductions), expands through lexical ties, and always includes layout tokens States#validate! now fails generation on unresolved scanner conflicts and on pslr.max-states / pslr.max-state-ratio guard violations, and warns about useless %lex-prec declarations. %lexer-context classification and context-based splitting are kept isolated as a Lrama/Ruby extension executed after the PSLR core.
Generate the PSLR(1) runtime as part of the yacc.c template output: * yy_scanner_transition / yy_state_to_accepting (Def 3.2.13), yy_scanner_accepts rows per parser state plus the fallback row (Def 3.2.14, section 3.5.1), length precedence tables for the normal and fallback faces, token-pattern-to-token-id mapping and layout flags * yy_pseudo_scan / yy_pseudo_scan_full implement Def 3.2.16: walk the DFA, prefer matches per the length precedence tables, fall back to the syntax-error row when no normal-row match exists, and flag layout tokens for skipping * %token-action bodies are emitted with yytext/yyleng and accumulated layout text (yypslr_layout_text/leng) available * yy_state_accepts_token and yy_state_eventually_accepts_token allow a hand-written lexer to query acceptability of a token in a state; the eventual variant follows default-reduction chains (Lrama extension) * a LAC (lookahead correction) checker performs exploratory parses so scanner decisions taken from merged states cannot commit erroneous semantic actions (section 3.5.2) * bridge-mode macros (YYSETSTATE_CONTEXT, YYPSLR_PSEUDO_SCAN*) let an existing yylex consult the pseudo-scanner through %define api.pslr.state-member Integration fixtures cover template argument lists, keyword contexts, layout comments, implicit literals and fallback precedence.
* --report=pslr renders scanner metrics: state growth before/after splitting, token pattern and FSA state counts, per-state scanner_accepts rows and unresolved-conflict details * warn about lexical tie candidates (Def 3.3.3): token pairs that have a scanner conflict but appear one-sided in some parser state, unless an explicit %lex-tie/%lex-no-tie already covers the pair * Command runs compute_pslr for %define lr.type pslr grammars and validates states before writing any parser output, so unresolved scanner conflicts and state-growth guard violations abort generation * document the PSLR(1) directives in NEWS
Ruby sources may contain NUL bytes, so a NUL-terminated scan API cannot represent every valid input. yy_pseudo_scan_result and yy_pseudo_scan now take (input, input_len) and treat input_len == 0 as end of input; the YYPSLR_PSEUDO_SCAN* bridge macros and yypslr_scan_with_layout gain the corresponding length argument, and the layout loop keeps the remaining length in sync as it consumes layout tokens. The scan result reports from_fallback when the token came from the fallback row or from character-token handling, letting callers detect a guaranteed upcoming syntax error before handing the token to the parser. The YYEOF token action now runs before returning YYEOF so a grammar can recover trailing layout (paper section 3.6).
Unresolved-scanner-conflict reports now carry a witness: an example input that drives the DFA to the conflicting profile. The witness is collected during the profile DFS and included in the error message so users can reproduce the conflict without reverse-engineering scanner state ids. Useless-%lex-prec detection previously relied on usage marks that nothing recorded, so it could not distinguish referenced rules from dead ones. The profile resolver now marks the identity and length rules that actually decide a normal-row resolution (winner selection in both the shorter-token-wins and longer-token-wins cases). The fallback row intentionally does not mark: it spans the whole token universe and would make every declaration look used. Compatibility probes during state splitting do not mark either.
LAC is orthogonal to PSLR (paper section 3.5.2), so expose it as %define parse.lac with the Bison-compatible values full and none: * PSLR parsers default to full; exploratory parses are what keep pseudo-scanning on merged states equivalent to canonical LR(1) * LALR/IELR parsers can now opt in with parse.lac full to get immediate error detection and accurate expected-token lists * parse.lac none on a PSLR parser is honored but warned about, because token selections from merged states may then run semantic actions before the syntax error is reported * invalid values are rejected at grammar validation time All LAC template blocks (checker function, lookahead verification, eager lookahead fetch at default reductions, expected-token enumeration) are now gated on the LAC setting instead of lr.type.
Phase 3 previously re-ran the dual-profile resolver walk over the whole scanner FSA for every candidate merge, which is the cost the paper's section 3.4.3 warns about. Precompute the FSA's conflict pairs once and reduce the compatibility test to a presence check per pair: two accept sets are compatible when every conflict pair is present identically in both, or absent from one side entirely (unresolved/unresolved pairs merge because their presence patterns match). Only pairs touching the symmetric difference of the two accept sets are examined, so merges that differ in conflict-free tokens are accepted in O(diff). Layout tokens live in every accept set and thus never distinguish states, which realizes the split-stable layout optimization of section 3.6 without extra bookkeeping.
%define api.pslr.lexer generated emits a complete lexer: yylex is generated, drives yypslr_scan_with_layout with the parser state kept current by yysetstate, skips layout tokens, runs %token-action bodies and advances an internal input cursor. Callers hand the input to the parser with yypslr_set_input(input, len) before yyparse; no hand-written lexer is involved, matching the paper's model. %token-action bodies now receive the semantic value slot: yylval inside an action refers to the YYSTYPE the caller passed, both in pure mode and through the bridge's YYPSLR_SCAN_WITH_LAYOUT macro (whose signature gains the value pointer). The dispatch is also emitted before the scan loop that calls it, so token actions compile. Pure mode requires full coverage: generation aborts when a terminal has no %token-pattern. Bridge mode instead warns with the list of uncovered terminals the user lexer must keep producing.
* %token-pattern bodies can reference earlier patterns with {NAME},
the idiom the paper uses for shortest-match comment scanning
(Fig 3.2c/3.2d). Undefined, forward, self and malformed references
are rejected; a literal brace is written \{.
* The scanner alphabet is now bytes (0-255) rather than 7-bit ASCII:
negated character classes and dot pass multi-byte UTF-8 sequences
through unmodified, and Ruby-side scanning walks bytes to match the
generated C runtime.
* Layout tokens are rejected in grammar rules at validation time; the
parser never sees them, so such rules could never match.
* %define pslr.tables canonical-lr switches state splitting to
canonical LR(1) compatibility (kernel lookahead-set equality) for
debugging conflict reports whose contexts IELR merging would blend.
* --report=pslr now renders per-state acceptable-token sets,
scanner_accepts rows including the fallback row, unresolved-conflict
details with witnesses, useless %lex-prec rules and lexical tie
candidates in addition to the split metrics summary.
Annotate %lex-prec operator constants and attributes as ::Symbol so they do not resolve to Lrama::Grammar::Symbol inside the Grammar namespace, and regenerate the rbs-inline signatures for the new pairwise resolution, LAC, pure-mode and reporting code. steep check is clean.
Document token actions, pure and bridge operating modes, the
length-delimited scan API, {NAME} pattern references, byte-oriented
classes, configurable LAC, canonical-lr table debugging, conflict
witnesses, useless-rule reporting and the extended --report=pslr
output.
Array#filter_map is Ruby 2.7+, so the PSLR report renderer now uses map + compact; the test suite must keep passing on Ruby 2.5 because lrama is executed by BASERUBY when building CRuby. The US-ASCII source check scanned vendor/bundle, where rbs 4.0.3 ships a C file containing a UTF-8 comment, failing the job for every pull request. Exclude vendored gems: they are not sources of this repository.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
refs: https://open.clemson.edu/cgi/viewcontent.cgi?article=1519&context=all_dissertations