Problem
isDDL matches a QueryEvent with a regex, and invalidate() drops every cached table's column metadata. So:
- A gh-ost/pt-osc run on a table we do not capture, or routine partition maintenance, evicts every active stream. Each reloads from
information_schema on its next row event.
- A reload only ever returns the latest schema. In a backlog with two or more DDLs on one captured table, the reload after the first gives the schema after the last, so the rows in between decode wrong.
A regex cannot fix either — it never learns which table the statement touched. RENAME and DROP take table lists, and the gh-ost cutover swap (RENAME TABLE users TO _users_del, _users_gho TO users) must not be attributed to its first name alone.
Proposed fix
Replace invalidate() with an applyDDL that parses the statement and advances each cached tableMeta in stream order:
ast.AlterTableStmt on a cached table → mutate tableMeta.Columns in place, honouring FIRST/AFTER, then set AnchoredAt to the DDL's position.
ast.RenameTableStmt / ast.DropTableStmt → drop only the named entries, walking every pair so the cutover swap is handled.
ast.CreateTableStmt → drop the named entry, for MariaDB's CREATE OR REPLACE TABLE.
- Index, constraint, partition, table option, DML → ignored. No eviction.
- Parse error, or an alteration naming a column we do not have → today's drop-all, or fail the run naming the table and statement. Never guess.
No new dependency
github.com/pingcap/tidb/pkg/parser is already direct, already imported by both files that would use it (pkg/binlog/filter.go, pkg/binlog/schema.go) for charset.GetCollationByName. ColumnDef.Tp is a resolved types.FieldType giving signedness (mysql.HasUnsignedFlag(ft.GetFlag())) and ENUM/SET members (ft.GetElems()), so a columnMeta rebuilds from DDL without touching information_schema.
Probed against the vendored version: ADD/CHANGE/MODIFY/RENAME/DROP COLUMN all parse with the right spec type and FIRST/AFTER position, the unsigned flag and ENUM members come through, the gh-ost rename swap yields both pairs in order, comment-prefixed DDL parses, and ADD COLUMN d BLOB sets collate="binary" on its own. The empty-string ENUM member survives at index 0 of GetElems(), so the index-0-is-invalid convention carries over unchanged. MariaDB-only syntax (CREATE OR REPLACE TABLE, WITH SYSTEM VERSIONING) and garbage return an error rather than a partial result, so they land cleanly on the drop-all fallback.
Two implementation notes
Driver import is mandatory. parser.New() panics with no parser driver (forgotten import?) unless _ "github.com/pingcap/tidb/pkg/parser/test_driver" is blank-imported. Despite the name it is the standalone driver for parser-only users, same module, builds by default.
Do not source collation from DDL. Measured: an explicit COLLATE latin1_swedish_ci comes back with GetCharset() and GetCollate() both empty, and charset.GetDefaultCollation("latin1") returns latin1_bin (47) rather than MySQL's real default latin1_swedish_ci (8). Apply DDL for names, ordinals, ENUM/SET members and signedness only; leave CollationID to the binlog (which logs charsets under MINIMAL) and information_schema. A DDL changing only a charset still drops the entry.
Any better solution is welcome
Problem
isDDLmatches aQueryEventwith a regex, andinvalidate()drops every cached table's column metadata. So:information_schemaon its next row event.A regex cannot fix either — it never learns which table the statement touched.
RENAMEandDROPtake table lists, and the gh-ost cutover swap (RENAME TABLE users TO _users_del, _users_gho TO users) must not be attributed to its first name alone.Proposed fix
Replace
invalidate()with anapplyDDLthat parses the statement and advances each cachedtableMetain stream order:ast.AlterTableStmton a cached table → mutatetableMeta.Columnsin place, honouringFIRST/AFTER, then setAnchoredAtto the DDL's position.ast.RenameTableStmt/ast.DropTableStmt→ drop only the named entries, walking every pair so the cutover swap is handled.ast.CreateTableStmt→ drop the named entry, for MariaDB'sCREATE OR REPLACE TABLE.No new dependency
github.com/pingcap/tidb/pkg/parseris already direct, already imported by both files that would use it (pkg/binlog/filter.go,pkg/binlog/schema.go) forcharset.GetCollationByName.ColumnDef.Tpis a resolvedtypes.FieldTypegiving signedness (mysql.HasUnsignedFlag(ft.GetFlag())) and ENUM/SET members (ft.GetElems()), so acolumnMetarebuilds from DDL without touchinginformation_schema.Probed against the vendored version:
ADD/CHANGE/MODIFY/RENAME/DROP COLUMNall parse with the right spec type andFIRST/AFTERposition, the unsigned flag and ENUM members come through, the gh-ost rename swap yields both pairs in order, comment-prefixed DDL parses, andADD COLUMN d BLOBsetscollate="binary"on its own. The empty-string ENUM member survives at index 0 ofGetElems(), so the index-0-is-invalid convention carries over unchanged. MariaDB-only syntax (CREATE OR REPLACE TABLE,WITH SYSTEM VERSIONING) and garbage return an error rather than a partial result, so they land cleanly on the drop-all fallback.Two implementation notes
Driver import is mandatory.
parser.New()panics withno parser driver (forgotten import?)unless_ "github.com/pingcap/tidb/pkg/parser/test_driver"is blank-imported. Despite the name it is the standalone driver for parser-only users, same module, builds by default.Do not source collation from DDL. Measured: an explicit
COLLATE latin1_swedish_cicomes back withGetCharset()andGetCollate()both empty, andcharset.GetDefaultCollation("latin1")returnslatin1_bin(47) rather than MySQL's real defaultlatin1_swedish_ci(8). Apply DDL for names, ordinals, ENUM/SET members and signedness only; leaveCollationIDto the binlog (which logs charsets underMINIMAL) andinformation_schema. A DDL changing only a charset still drops the entry.Any better solution is welcome