markdown-codec

Hand-written CommonMark+GFM ⇄ ContentDocument codec, built on document-schema.js.

The same "hand-write the format instead of wrapping a third-party library" bet as pdf-codec, aimed at CommonMark and GFM. No micromark/remark/marked/markdown-it/commonmark/mdast/unified/turndown/showdown dependency (enforced by eslint no-restricted-imports). Runtime dependencies: document-schema.js (the shared pivot) and zod. readMarkdown/writeMarkdown read and write that pivot's ContentDocument directly — the same model documents.js builds docx/pptx/odt/odp conversions around.

graph TD
    schema("document-schema.js")
    ooxml("ooxml.js")
    odf("odf.js")
    pdfcodec("pdf-codec")
    bytecodec("byte-codec")
    mdcodec("markdown-codec")
    documents("documents.js")
    mcp("document-mcp")
    cli("document-cli")

    schema --> ooxml
    schema --> odf
    schema --> pdfcodec
    schema --> bytecodec
    schema --> mdcodec
    schema --> documents
    bytecodec --> pdfcodec
    ooxml --> documents
    odf --> documents
    pdfcodec --> documents
    bytecodec --> documents
    mdcodec --> documents
    documents --> mcp
    pdfcodec --> mcp
    documents --> cli
    odf --> cli
    pdfcodec --> cli

    click schema "https://github.com/ExaDev/document-schema.js" "document-schema.js"
    click ooxml "https://github.com/ExaDev/ooxml.js" "ooxml.js"
    click odf "https://github.com/ExaDev/odf.js" "odf.js"
    click pdfcodec "https://github.com/ExaDev/pdf-codec" "pdf-codec"
    click bytecodec "https://github.com/ExaDev/byte-codec" "byte-codec"
    click mdcodec "https://github.com/ExaDev/markdown-codec" "markdown-codec"
    click documents "https://github.com/ExaDev/documents.js" "documents.js"
    click mcp "https://github.com/ExaDev/document-mcp" "document-mcp"
    click cli "https://github.com/ExaDev/document-cli" "document-cli"

    style mdcodec fill:#f9a825,stroke:#333,stroke-width:3px

Loading

Status

The scanner, block parser, and inline parser are complete hand-written implementations of CommonMark 0.31.2's two-phase algorithm plus GFM's table/strikethrough/autolink/task-list-item extensions. readMarkdown/writeMarkdown/markdownCodec are wired and real. Conformance suites measure the full public surface (readMarkdown → writeMarkdown → reparse → render to HTML) against the vendored CommonMark/GFM corpora — see Fidelity for why the rate is below 100% (dominated by what ContentDocument can represent, not parsing gaps).

Getting started

Requires Node.js >=20 and pnpm 11.6.0 (pinned via packageManager in package.json).

pnpm install

Install as a dependency in another project:

pnpm add markdown-codec
# or
npm install markdown-codec

Published to npmjs.org via OIDC trusted publishing. dist/ is committed rather than gitignored — a holdover from a pre-publish period when a git-tarball install needed a working build; now a known cleanup.

Usage

Reading and writing markdown text:

import { readMarkdown, writeMarkdown } from 'markdown-codec';

const { document, diagnostics } = readMarkdown('# Title\n\nSome **bold** text with a [link](https://example.com).', {
  frontMatter: true, // parse a leading YAML front matter block into ContentDocument.metadata
  images: (destination) => undefined, // a synchronous MarkdownImageResolver port for non-data: URI images
});

const markdown = writeMarkdown(document, {
  bulletListMarker: '-',
  emphasisMarker: '_',
  frontMatter: true, // emit ContentDocument.metadata back out as a leading front matter block
});

Both accept an optional signal (AbortSignal) and sink (MarkdownDiagnosticSink, called once per recoverable issue or construct-mapping gap — see Gotchas). writeMarkdown throws MarkdownUnsupportedDocumentKindError for a non-'wordprocessing' ContentDocument.

The same round trip as a schema-validated z.codec() pair, mirroring pdf-codec's pdfCodec:

import { z } from 'zod';
import { markdownCodec, MarkdownBytesSchema } from 'markdown-codec';

const document = z.decode(markdownCodec, bytes); // throws if bytes are not well-formed UTF-8
const bytes2 = z.encode(markdownCodec, document);

MarkdownBytesSchema checks for well-formed UTF-8. The no-options form only; readMarkdown/writeMarkdown remain the entry points for an AbortSignal or diagnostic sink. Every construct-mapping gap reports through the sink as a stable code (e.g. md/nested-emphasis-flattened) — see MarkdownDiagnosticCodes and Gotchas.

Architecture

Modelled on pdf-codec's own layering, aimed at CommonMark+GFM instead of PDF:

• src/diagnostics/ — three-tier diagnostic policy (throw/recover/degrade); MarkdownDiagnosticCodes names every code.
• src/ast/ — markdown AST node types (document/block/inline union), Zod-first.
• src/options/ / src/defaults/ — read/write options (GFM toggles, sink, AbortSignal, write-side style) and defaults.
• src/scan/ — CommonMark line/character scanner, plus entity-table.ts (generated from assets/html-entities/entities.json).
• src/block/ — CommonMark block-structure algorithm (open-block stack, continuation matching): paragraphs, headings, code blocks, block quotes, lists (incl. GFM task-list-item), thematic breaks, link references, GFM tables.
• src/inline/ — emphasis, code spans, links, autolinks, raw HTML, GFM strikethrough, line breaks.
• src/html/ — raw HTML recognition (bounded rules, not a general parser) plus render.ts (conformance oracle; internal only).
• src/image/ — PNG/JPEG dimension reader and base64 codec, shared by src/lower/ and src/emit/.
• src/shared/ — string-shape conventions src/lower/src/emit agree on (style-constants.ts, list-id.ts's opaque numId). Re-exported so documents.js's MarkdownEditor reuses the identical grammar.
• src/lower/ — AST → ContentDocument lowering (thin adapter, not a second parser); top-of-file table maps each construct to its diagnostic gap.
• src/emit/ — ContentDocument → markdown text emission, the structural inverse of src/lower.
• src/read.ts / src/write.ts / src/codec.ts — public readMarkdown/writeMarkdown entry points and markdownCodec (z.codec() pair).

Vendored assets

assets/ holds real, unmodified conformance corpora (each with a NOTICE.md recording source, version, licence). None is read at runtime: assets/html-entities/entities.json is compiled into src/scan/entity-table.ts, and the spec corpora are test-only. So package.json's "files": ["dist"] is correct.

• assets/commonmark/ — CommonMark spec + corpus (652 examples), tag 0.31.2 (CC-BY-SA 4.0).
• assets/gfm/ — GitHub Flavored Markdown Spec (CC-BY-SA 4.0).
• assets/html-entities/ — WHATWG HTML5 named character reference table (BSD 3-Clause).

Build, test, and lint

pnpm build         # turbo run _build (tsdown -> dist/, ESM + CJS + .d.ts)
pnpm typecheck     # turbo run _typecheck _typecheck:node (dual tsconfig)
pnpm lint          # turbo run _lint (eslint . --fix --cache --max-warnings 0)
pnpm test          # turbo run _test (vitest run --project unit, incl. CommonMark/GFM conformance)
pnpm test:workers  # turbo run _test:workers (unit suite under the real Cloudflare Workers/workerd runtime)
pnpm test:watch    # vitest --project unit
pnpm test:coverage # turbo run _test:coverage (vitest run --project unit --coverage)
pnpm test:smoke    # turbo run _test:smoke (rebuilds dist/, verifies ESM/CJS parity + a real round trip per bundle)
pnpm test:corpus   # turbo run _test:corpus (optional, gitignored real-world sanity check -- see Fidelity)

To run a single test file: pnpm vitest run src/path/to/file.test.ts.

Conventions

• Zod-first schema/type/guard, matching pdf-codec/documents.js: every model type inferred from its Zod schema.
• No type assertions. Every loosely-typed value narrowed through a type guard or Zod parse at the boundary.
• No markdown-parsing library dependency, enforced by eslint no-restricted-imports.
• z.codec() for the round trip (markdownCodec), matching pdf-codec's pdfCodec: wraps the independently-tested readMarkdown/writeMarkdown with automatic two-way schema validation (no-options form only).
• Shrink-only conformance exclusion list. Every spec example the read → write → reparse → render pipeline does not reproduce byte for byte is named in src/test-support/conformance-exclusions.ts, with a test asserting it genuinely still fails — the list shrinks as gaps close, never quietly grows.
• Conventional commits, enforced via commitlint + husky.

Gotchas and quirks

Every construct src/lower/src/emit cannot represent losslessly is a documented MarkdownDiagnosticCodes entry:

• md/invented-page-geometry — no page concept in markdown; one ContentSection with A4 + 1in defaults (overridable). Fires once.
• md/nested-emphasis-flattened — same-kind nested emphasis flattens to one run.
• md/link-title-dropped — link/image title has no ContentRun/ContentImageBlock field.
• md/code-block-info-string-dropped — fenced code info string has no ContentParagraph field.
• md/blockquote-nested-depth — nesting beyond one level is indent depth only; same-depth blockquotes are indistinguishable.
• md/list-item-block-unlisted — a table/image in a list item cannot carry ContentListMembership (paragraphs only).
• md/list-item-multi-block-flattened — multi-block list items lose item-boundary identity.
• md/image-unresolved — no resolver, undefined return, or non-PNG/JPEG bytes degrades to alt-text run.
• md/raw-html-preserved-as-text / md/raw-html-dropped — raw HTML kept as literal text (default) or dropped; never interpreted.
• md/front-matter-key-unmapped — no YAML/TOML engine; only five known LayoutMetadata keys recognised.
• md/heading-level-clamped — styleId beyond Heading6 (from another format) clamps to level 6.
• md/adjacent-links-merged / md/code-span-as-monospace-run — same-destination adjacent links merge; monospace runs emit as code spans.
• md/paragraph-indent-dropped — indentLeftPt without a recognised styleId; indent dropped, paragraph renders.
• md/list-numid-fallback — a foreign numId falls back to a plain bullet list.
• md/table-cell-formatting-dropped / md/table-cell-multi-paragraph-joined — GFM cells have no rich-formatting or multi-paragraph representation.

Fidelity

Markdown → ContentDocument is dominated by target-schema limits, not parsing gaps. The parser recognises every construct CommonMark and GFM define; the limiting factor is what ContentDocument can hold — a cross-format pivot shaped around docx/pptx/odt/odp/ods/odg, not markdown's richer model. Each gap is a permanent structural mismatch.

Round-trip conformance rate (read → write → reparse → render to HTML, compared byte for byte against expected HTML):

Corpus	Examples	Passing round trip	Rate
CommonMark 0.31.2 (assets/commonmark/spec.json)	652	461	70.7%
GFM tagged extensions (table/strikethrough/autolink/task-list, assets/gfm/spec.txt)	23	22	95.7%
Combined	675	483	71.6%

Every non-passing example is named individually in src/test-support/conformance-exclusions.ts, attributed to a closed set of causes (shrink-only — see Conventions): most commonly a soft line break collapsing to a space, a dropped title/info string, a flattened list item/blockquote, or touching emphasis spans.

Optional real-world corpus. test/corpus/ (gitignored) holds a pnpm test:corpus project for a manual sanity check against sibling READMEs on disk — asserts no throw and real content on reparse, not byte fidelity. Not part of pnpm test; run locally before significant parser/lower/emit changes.

Release and publishing

.github/workflows/ci.yml runs commitlint, lint, typecheck, unit suite (incl. conformance), and smoke test on every push/PR. On a push to main where those pass, release.config.ts drives semantic-release: commit history decides the version bump, CHANGELOG.md and package.json are committed back to main, a GitHub Release is cut, and the package publishes to npmjs.org via OIDC trusted publishing (no NPM_TOKEN).

Release detection diffs package.json's version before/after the release step. Four further jobs gate on that: a sibling-released repository_dispatch to documents.js; a republish under @exadev/markdown-codec to GitHub Packages (GITHUB_TOKEN); a republish under mrkdwn.js to npmjs.org (same OIDC exchange); and an SPDX SBOM + build-provenance attestation signed against the packed tarball.

Contributing

Conventional Commits enforced by commitlint (commitlint.config.ts) via a husky commit-msg hook and CI job — semantic-release's version bump depends on well-formed messages. pre-commit runs lint-staged (eslint --fix on staged *.ts); pre-push runs the test suite. Single main branch, no open PR workflow.

References

• document-schema.js — owns the shared ContentDocument pivot.
• pdf-codec — the sibling whose scaffold, tooling, and "hand-write the format" philosophy this project mirrors.
• documents.js — bridges markdown to docx/odt/PDF via this package's ContentDocument. Markdown has no presentation/spreadsheet/drawing variant, so pptx/odp/ods/odg are structurally out of reach.
• CommonMark Spec — the base specification targeted.
• GitHub Flavored Markdown Spec — GFM extensions layered on top.
• WHATWG HTML § named character references — the entity table assets/html-entities/ vendors.

npm aliases

This package also publishes under the alternate name — identical build, same version, republished by CI:

• mrkdwn.js

License

MIT