Expose the YAML frontmatter body in the grammar so a --- inside a value cannot truncate the metadata - #675
Merged
Conversation
…alue cannot truncate the metadata (bd-mjo6ao32, GH #671) The reader located the frontmatter body by `split("---")`-ing the opaque `metadata` token's text and keeping the second piece, so the first `---` inside a value ended the YAML there: a plain scalar silently dropped every later key, a quoted scalar failed with Q-0-99. Since PR #290 the qmd writer canonicalizes every em dash to `---`, which made both shapes reachable from any document with an em dash in a frontmatter string. The tree-sitter scanner was never at fault (it closes the block only on a column-0 `---` line), but it emitted the whole block as one external leaf token, which is why the Rust side had to rediscover the body. The grammar now parses the block as four external tokens and exposes the body as a `yaml` child under the `body` field: (metadata body: (yaml)) pampa takes the YAML text and range straight from that node; `extract_between_delimiters` and the `block.text.find(content)` offset hazard are deleted. Nothing in Rust knows about `---` any more, and the `yaml` node is also what a YAML injection query would target. Scanner: `_minus_metadata_start` (the opening `---`, sharing the thematic break's `mark_end` position so the confirming look-ahead can run speculatively), `_minus_metadata_open_newline`, `_minus_metadata_body`, `_minus_metadata_end`, then the ordinary `choice(_newline, _eof)` tail, so a closing `---` at EOF without a newline now also closes the block. A new `STATE_IN_MINUS_METADATA` bit gates the interior tokens against tree-sitter's mark-everything-valid error recovery. Consequences of the carrier change (`RawBlock quarto_minus_metadata` now holds the body text and the body range): the body is no longer trimmed, the metadata span ends after the final line break, and the source-info chain is one layer shorter. Resolved key/value offsets are unchanged. That renumbers every `s` index in the 20 `ts-packages/annotated-qmd/examples/*.json` fixtures (regenerated with the documented loop; verified that every non-`astContext` difference is an `s` renumbering) and updates 4 insta snapshots under `crates/pampa/snapshots/json/` (002, and three others with frontmatter) the same way. `Block::BlockMetadata` for a lexical block now spans the YAML body rather than the delimited block; see the plan for the follow-up that would carry both ranges. Regenerating `parser.c` renumbers LR states, so the state-keyed `resources/error-corpus/_autogen-table.json` was rebuilt with `crates/pampa/scripts/build_error_table.ts`; only the table changed. Tests: 12 reader/round-trip/tree-shape regressions in `test_frontmatter_delimiters.rs`; 5 GH #671 corpus cases plus the `body: (yaml)` child on 9 existing expectations (one, `div.txt: 2`, also gains the `block_continuation` every block in a div carries). The three file-based tests in `test_meta.rs` now go through the real reader instead of handing whole documents to `rawblock_to_config_value`. Plan: claude-notes/plans/2026-09-10-frontmatter-delimiter-split-gh671.md Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0132TvF41q7CA19fE29UX77X
Contributor
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #671.
Problem
The reader located the frontmatter body by
split("---")-ing the opaquemetadatatoken's text and keeping the second piece, so the first---inside a value ended the YAML there: a plain scalar silently dropped every later key, a quoted scalar failed with Q-0-99. Since #290 the qmd writer canonicalizes every em dash to---, which made both shapes reachable from any document with an em dash in a frontmatter string (in the wild: quarto-web's2026-04-14-chrome-headless-shell/index.qmd).The tree-sitter scanner was never at fault — it closes the block only on a column-0
---line — but it emitted the whole block as one external leaf token, which is why the Rust side had to rediscover the body by hand.Fix
The grammar now parses the block as four external tokens and exposes the body as a
yamlchild under thebodyfield:pampa takes the YAML text and range straight from that node.
extract_between_delimitersand theblock.text.find(content)offset hazard are deleted; nothing in Rust knows about---any more. Theyamlnode is also what a YAML injection query would target in external tooling.Scanner details:
_minus_metadata_startis the opening---(it shares the thematic break'smark_endposition so the confirming look-ahead can run speculatively), then_minus_metadata_open_newline,_minus_metadata_body,_minus_metadata_end, then the ordinarychoice(_newline, _eof)tail — so a closing---at EOF without a newline now also closes the block. A newSTATE_IN_MINUS_METADATAbit gates the interior tokens against tree-sitter's mark-everything-valid error recovery.Consequences worth reviewing
RawBlock quarto_minus_metadatacarrier now holds the body text and the body range, so the body is no longer trimmed, the metadata span ends after the final line break (4..30instead of4..29), and the source-info chain is one layer shorter. Resolved key/value offsets are unchanged. That renumbers everysindex in the 20ts-packages/annotated-qmd/examples/*.jsonfixtures (regenerated with the documented loop; I verified every non-astContextdifference is ansrenumbering) and 4 insta snapshots undercrates/pampa/snapshots/json/.Block::BlockMetadatafor a lexical block now spans the YAML body rather than the delimited block. Carrying both ranges would mean buildingBlockMetadataat tree-conversion time instead of via the RawBlock carrier — flagged as a follow-up in the plan, not done here.parser.cregeneration renumbers LR states, so the state-keyedresources/error-corpus/_autogen-table.jsonwas rebuilt (crates/pampa/scripts/build_error_table.ts); only the table changed.div.txt: 2) gains theblock_continuationchild every block in a div already carries.Tests
crates/pampa/tests/integration/test_frontmatter_delimiters.rs: the three shapes from the issue, an indented---inside a block scalar, trailing whitespace on the closing line, CRLF, metadata nested in a div, exact key offsets after a----containing value, qmd-writer round trips for single- and multi-line em-dash values, and the tree shape/range of theyamlnode (including closing---at EOF).---, which truncates the YAML on re-read #671 cases intest/corpus/metadata.txt; 9 existing(metadata)expectations gained the child.test_meta.rsnow go through the real reader instead of handing whole documents torawblock_to_config_value.cargo xtask verifygreen locally.End-to-end (output inspected):
Plan:
claude-notes/plans/2026-09-10-frontmatter-delimiter-split-gh671.md(bd-mjo6ao32). Found along the way and filed separately: bd-wl58atds (pre-existing provenance warning on block scalars with multi-byte characters).🤖 Generated with Claude Code
https://claude.ai/code/session_0132TvF41q7CA19fE29UX77X