fix(pdf-codec): decode a glyph through the font's own built-in encoding - #852
Merged
Conversation
A simple font's /Encoding may name any of the three base encodings ISO 32000-1 Annex D defines, but only WinAnsiEncoding had a table here: the other two were reported as "approximated as WinAnsiEncoding" and then decoded through it, which is wrong across most of the upper half. Code 0xBD alone is "perthousand" in StandardEncoding, "Omega" in MacRomanEncoding and "onehalf" in WinAnsi -- three unrelated characters at one code, substituted silently. Both tables are generated from fontTools' own StandardEncoding and MacRoman tables rather than transcribed. The approximation diagnostic now fires only for a name genuinely without a table here, MacExpertEncoding being the realistic one. glyphNameToUnicode also resolves the Adobe Glyph List's constructed name forms -- uniXXXX, uXXXX through uXXXXXX, and a base name carrying a .variant suffix -- which a subsetting tool re-emits in place of a font's real glyph names. That folds in the uniXXXX fallback font-read.ts kept privately, so one function answers what a glyph name means for every caller.
parsePost read the italic angle and underline geometry and stopped there, so the names a version 1.0 or 2.0 'post' table gives each glyph -- the most direct statement a TrueType font makes about what its glyphs are -- were unavailable. parsePostGlyphNames reads both versions, resolving an index below 258 through the standard Macintosh glyph ordering and anything above it through the table's own Pascal-string array, and returns undefined for the versions that name nothing rather than treating a nameless font as an error.
buildCmapLookup collapsed a font's whole 'cmap' to one Unicode lookup, which discards the distinction that matters for a symbol font: a (3, 1) subtable is keyed by code points, but a (3, 0) or (1, 0) subtable is keyed by the font's own character codes, and reading one as the other identifies glyphs as the wrong characters. readCmapSubtables returns each readable subtable alongside the platform and encoding IDs that say what its codes are, plus an enumeration of its own mappings, which is what inverting a subtable needs -- no format stores glyph ID to code directly. Format 0 is now read as well, the shape a symbol font's own single-byte subtable often takes. buildCmapLookup is unchanged in behaviour and now built on top of that, still selecting only the Unicode-keyed formats.
A CFF charset names each glyph by SID, and a SID means one of the 391 standard strings or an entry of the font's own String INDEX -- the half of the string space that is not in the font had no representation here at all, so a charset could be read but not resolved. CFF_STANDARD_STRINGS is generated from fontTools' cffStandardStrings rather than transcribed, and cffStringForSid resolves either half through one call.
…ogram A symbol-encoded font subset states what its character codes draw inside the embedded font program and nowhere else. Until now this codec could not open that program, so such a font had no path to its own encoding at all. readFontProgramEncoding reads all three program shapes a PDF can embed, choosing by what the bytes are rather than which /FontFile key they arrived under, since producers do file a program under the wrong one: - TrueType: the 'cmap' table's (3, 0) symbol or (1, 0) Macintosh subtable maps a code to a glyph, and the glyph is identified by its 'post' name or by reading the font's Unicode subtable backwards. - CFF: the Top DICT's Encoding operator maps codes to glyphs and the charset names them. A CID-keyed program is refused outright -- its charset holds CIDs rather than name SIDs, so reading it would name every glyph wrongly. - Type 1: the /Encoding array in the program's own cleartext header, which sits before eexec and so needs no decryption. A code point recovered by inverting a Unicode subtable is discarded when it falls in a private-use plane: that identifies a glyph within one font and says nothing about the character it draws, so it is no answer rather than a wrong one.
A font subset embedded to draw a symbol -- an ohm sign, a micro sign, a less-than-or-equal -- puts it at whatever code its producer picked, and says so only inside the embedded program. Decoding that code through WinAnsi instead yields a plausible different character: an ohm sign drawn at 0x57 extracted as "W", indistinguishable to every consumer from a document that genuinely says W. The base encoding a code falls back to now follows ISO 32000-1 9.6.6 rather than always being WinAnsi. A symbolic font -- Symbolic in /FontDescriptor, or /BaseFont Symbol or ZapfDingbats -- is encoded by its own program, so the embedded program is read first and the two fixed standard-14 symbol tables next; 9.6.6.4 has such a font's /Encoding ignored outright. An ordinary text font takes an explicitly named base encoding first, its own program when it names none, and WinAnsi last. /ToUnicode and /Differences still win over all of it, and a code no source covers is still the replacement character plus a diagnostic rather than a guess. A composite font gains the same fallback where it is sound: Identity-H with the default /CIDToGIDMap makes a CID the program's own glyph ID, so a font carrying no /ToUnicode is identified through the program instead of decoding to replacement characters. Any other /Encoding or a /CIDToGIDMap stream breaks that identity and the program is not consulted. Fixes #834
Mearman
marked this pull request as ready for review
September 3, 2026 06:17
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
The program is now read on the first code that actually falls through to it, and remembered from then on. A font whose /ToUnicode covers every code it shows never opens its program at all, which is where the cost is largest: an embedded face runs to hundreds of kilobytes, and opening one means inflating the stream and parsing its tables. That matters most on a CPU-metered runtime, where the whole parse holds the isolate with no opportunity to yield.
Contributor
|
🎉 This PR is included in version 3.6.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A font subset embedded to draw a symbol — an ohm sign, a micro sign, a less-than-or-equal — puts it at whatever character code its producer picked, and says so only inside the embedded font program.
readPdfhad no way to open that program, so such a code fell through to WinAnsi and decoded as a plausible different character: an ohm sign drawn at 0x57 came out asW, indistinguishable to every consumer from a document that genuinely says W.#827 narrowed this to the two standard-14 symbol faces (Symbol, ZapfDingbats) and made any other Symbolic-flagged font honestly unmapped, but the real offending fonts — arbitrary subset names like
CIDFont+F3, often with the Symbolic bit unset — were left either unmapped or exactly as wrong as before. That gap is what this closes.builtin-encoding.tsreads the encoding out of the embedded program itself, choosing by what the bytes are rather than which/FontFilekey they arrived under (producers do file a program under the wrong one):cmap's (3, 0) symbol or (1, 0) Macintosh subtable maps a code to a glyph, and the glyph is identified by itspostname, or by reading the font's Unicode subtable backwards where a subsetter stripped the names./Encodingarray in the program's own cleartext header, which sits beforeeexecand needs no decryption.The fallback order in
font-read.tsnow follows ISO 32000-1 9.6.6 rather than always ending at WinAnsi. A symbolic font is encoded by its own program, so the program is read first and the two fixed symbol tables next (9.6.6.4 has such a font's/Encodingignored outright); an ordinary text font takes an explicitly named base encoding first, its own program when it names none, then WinAnsi./ToUnicodeand/Differencesstill win over all of it.Two adjacent things the same clause required:
/MacRomanEncodingand/StandardEncodingnow have real tables instead of being "approximated as WinAnsiEncoding" (code 0xBD alone isperthousand,Omegaandonehalfacross the three), andglyphNameToUnicoderesolves the Adobe Glyph List's constructeduniXXXX/uXXXXXXname forms.A composite font gains the same fallback where it is sound: Identity-H with the default
/CIDToGIDMapmakes a CID the program's own glyph ID, so a font with no/ToUnicodeis identified through the program rather than decoding to replacement characters. Any other/Encoding, or a/CIDToGIDMapstream, breaks that identity and the program is not consulted.Nothing here guesses. A code point recovered by inverting a Unicode subtable is discarded when it lands in a private-use plane — that identifies a glyph inside one font and says nothing about the character it draws — and a code no source covers is still the replacement character plus a
text/unmapped-encodingdiagnostic.Tests
Written first, failing for the right reason, against synthetic programs built by literal byte layout in
test-support/sfnt.tsandtest-support/cff.ts(nothing imported from this package's own readers). The end-to-end one is a whole PDF whose page draws code 0x57 from a subset with no/ToUnicode, no/Encodingand a/BaseFontnaming no standard face —readPdfreturnsΩ, where before it returnedW.Four assertions are cross-checks against the real vendored fonts rather than fixtures written to satisfy the parsers: Caladea's version 2.0
postnames glyphs 5 and 35 asAande; Carlito's version 3.0postnames nothing, so glyph 2007 is identified by inverting its Unicode subtable; STIX Two Math's CFF charset names glyphs 5 and 35 asCanduni1EA8. Every expected value was read out of the font files with fontTools, independently of this package.pnpm lint,typecheck,test(1231),test:workersandtest:smokeall pass, as does the full dependent set (turbo run _test _typecheck _lint --filter=...pdf-codec, 27 tasks).Fixes #834