Skip to content

feat(MySQL): parse DDL and apply it in stream order, instead of dropping every cached table #1200

Description

@github-actions

Problem

isDDL matches a QueryEvent with a regex, and invalidate() drops every cached table's column metadata. So:

  • A gh-ost/pt-osc run on a table we do not capture, or routine partition maintenance, evicts every active stream. Each reloads from information_schema on its next row event.
  • A reload only ever returns the latest schema. In a backlog with two or more DDLs on one captured table, the reload after the first gives the schema after the last, so the rows in between decode wrong.

A regex cannot fix either — it never learns which table the statement touched. RENAME and DROP take table lists, and the gh-ost cutover swap (RENAME TABLE users TO _users_del, _users_gho TO users) must not be attributed to its first name alone.

Proposed fix

Replace invalidate() with an applyDDL that parses the statement and advances each cached tableMeta in stream order:

  • ast.AlterTableStmt on a cached table → mutate tableMeta.Columns in place, honouring FIRST/AFTER, then set AnchoredAt to the DDL's position.
  • ast.RenameTableStmt / ast.DropTableStmt → drop only the named entries, walking every pair so the cutover swap is handled.
  • ast.CreateTableStmt → drop the named entry, for MariaDB's CREATE OR REPLACE TABLE.
  • Index, constraint, partition, table option, DML → ignored. No eviction.
  • Parse error, or an alteration naming a column we do not have → today's drop-all, or fail the run naming the table and statement. Never guess.

No new dependency

github.com/pingcap/tidb/pkg/parser is already direct, already imported by both files that would use it (pkg/binlog/filter.go, pkg/binlog/schema.go) for charset.GetCollationByName. ColumnDef.Tp is a resolved types.FieldType giving signedness (mysql.HasUnsignedFlag(ft.GetFlag())) and ENUM/SET members (ft.GetElems()), so a columnMeta rebuilds from DDL without touching information_schema.

Probed against the vendored version: ADD/CHANGE/MODIFY/RENAME/DROP COLUMN all parse with the right spec type and FIRST/AFTER position, the unsigned flag and ENUM members come through, the gh-ost rename swap yields both pairs in order, comment-prefixed DDL parses, and ADD COLUMN d BLOB sets collate="binary" on its own. The empty-string ENUM member survives at index 0 of GetElems(), so the index-0-is-invalid convention carries over unchanged. MariaDB-only syntax (CREATE OR REPLACE TABLE, WITH SYSTEM VERSIONING) and garbage return an error rather than a partial result, so they land cleanly on the drop-all fallback.

Two implementation notes

Driver import is mandatory. parser.New() panics with no parser driver (forgotten import?) unless _ "github.com/pingcap/tidb/pkg/parser/test_driver" is blank-imported. Despite the name it is the standalone driver for parser-only users, same module, builds by default.

Do not source collation from DDL. Measured: an explicit COLLATE latin1_swedish_ci comes back with GetCharset() and GetCollate() both empty, and charset.GetDefaultCollation("latin1") returns latin1_bin (47) rather than MySQL's real default latin1_swedish_ci (8). Apply DDL for names, ordinals, ENUM/SET members and signedness only; leave CollationID to the binlog (which logs charsets under MINIMAL) and information_schema. A DDL changing only a charset still drops the entry.

Any better solution is welcome

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions