Skip to content

feat(metrics): create the metrics kafka ingest chain via migration #89631

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jzhu13 wants to merge 2 commits into
base: master
Choose a base branch
from
Open

feat(metrics): create the metrics kafka ingest chain via migration #89631

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7e2756f
feat(metrics): create the metrics kafka ingest chain via migration
jzhu13 Aug 26, 2026
f7d3558
feat(metrics): replace the hand-run clickhouse init script with the m…
jzhu13 Aug 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions bin/clickhouse-metrics-init
View file
Open in desktop

This file was deleted.

384 changes: 0 additions & 384 deletions bin/clickhouse-metrics.sql
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion bin/verify-metrics-pipe
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ Check, in order:
curl -s localhost:6743/_metrics | head # logs-ingestion
curl -s localhost:6744/_metrics | head # metrics-ingestion
4. Is capture-logs receiving? docker logs capture-logs | tail
5. Were the CH tables initialized? bin/clickhouse-metrics-init
5. Were the CH tables created? bin/migrate (migration 0305_metrics_kafka_ingest)
6. Give it ~30s after start (15s scrape + ingest lag), then re-run.
MSG
exit 1
Expand Down
3 changes: 3 additions & 0 deletions docker/clickhouse/config.d/default.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,9 @@
<warpstream_traces>
<kafka_broker_list from_env="KAFKA_HOSTS"/>
</warpstream_traces>
<warpstream_metrics>
<kafka_broker_list from_env="KAFKA_HOSTS"/>
</warpstream_metrics>
</named_collections>

<macros>
Expand Down
3 changes: 3 additions & 0 deletions docker/clickhouse/config.d/multinode/ai_events_node.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,9 @@
<warpstream_traces>
<kafka_broker_list from_env="KAFKA_HOSTS"/>
</warpstream_traces>
<warpstream_metrics>
<kafka_broker_list from_env="KAFKA_HOSTS"/>
</warpstream_metrics>
</named_collections>

<macros>
Expand Down
3 changes: 3 additions & 0 deletions docker/clickhouse/config.d/multinode/aux_node.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,9 @@
<warpstream_traces>
<kafka_broker_list from_env="KAFKA_HOSTS"/>
</warpstream_traces>
<warpstream_metrics>
<kafka_broker_list from_env="KAFKA_HOSTS"/>
</warpstream_metrics>
</named_collections>

<macros>
Expand Down
3 changes: 3 additions & 0 deletions docker/clickhouse/config.d/multinode/data_node.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,9 @@
<warpstream_traces>
<kafka_broker_list from_env="KAFKA_HOSTS"/>
</warpstream_traces>
<warpstream_metrics>
<kafka_broker_list from_env="KAFKA_HOSTS"/>
</warpstream_metrics>
</named_collections>

<macros>
Expand Down
3 changes: 3 additions & 0 deletions docker/clickhouse/config.d/multinode/ingestion_events_node.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,9 @@
<warpstream_traces>
<kafka_broker_list from_env="KAFKA_HOSTS"/>
</warpstream_traces>
<warpstream_metrics>
<kafka_broker_list from_env="KAFKA_HOSTS"/>
</warpstream_metrics>
</named_collections>

<macros>
Expand Down
3 changes: 3 additions & 0 deletions docker/clickhouse/config.d/multinode/ingestion_medium_node.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,9 @@
<warpstream_traces>
<kafka_broker_list from_env="KAFKA_HOSTS"/>
</warpstream_traces>
<warpstream_metrics>
<kafka_broker_list from_env="KAFKA_HOSTS"/>
</warpstream_metrics>
</named_collections>

<macros>
Expand Down
3 changes: 3 additions & 0 deletions docker/clickhouse/config.d/multinode/ingestion_small_node.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,9 @@
<warpstream_traces>
<kafka_broker_list from_env="KAFKA_HOSTS"/>
</warpstream_traces>
<warpstream_metrics>
<kafka_broker_list from_env="KAFKA_HOSTS"/>
</warpstream_metrics>
</named_collections>

<macros>
Expand Down
3 changes: 3 additions & 0 deletions docker/clickhouse/config.d/multinode/logs_node.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,9 @@
<warpstream_traces>
<kafka_broker_list from_env="KAFKA_HOSTS"/>
</warpstream_traces>
<warpstream_metrics>
<kafka_broker_list from_env="KAFKA_HOSTS"/>
</warpstream_metrics>
</named_collections>

<macros>
Expand Down
3 changes: 3 additions & 0 deletions docker/clickhouse/config.d/multinode/ops_node.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,9 @@
<warpstream_traces>
<kafka_broker_list from_env="KAFKA_HOSTS"/>
</warpstream_traces>
<warpstream_metrics>
<kafka_broker_list from_env="KAFKA_HOSTS"/>
</warpstream_metrics>
</named_collections>

<macros>
Expand Down
3 changes: 3 additions & 0 deletions docker/clickhouse/config.d/multinode/sessions_node.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,9 @@
<warpstream_traces>
<kafka_broker_list from_env="KAFKA_HOSTS"/>
</warpstream_traces>
<warpstream_metrics>
<kafka_broker_list from_env="KAFKA_HOSTS"/>
</warpstream_metrics>
</named_collections>

<macros>
Expand Down
4 changes: 0 additions & 4 deletions hogli.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -966,10 +966,6 @@ tools:
bin_script: clickhouse-logs-init
description: Initialize ClickHouse logs schema (temporary, pre-migration)
hidden: true
clickhouse:metrics:init:
bin_script: clickhouse-metrics-init
description: Initialize ClickHouse metrics schema (temporary, pre-migration)
hidden: true
clickhouse:multinode:start:
cmd: tools/infra-scripts/clickhouse-multinode/start-multinode-clickhouse
description: Boot the multinode ClickHouse smoke-test stack (one server per logical cluster) for local migration verification
Expand Down
62 changes: 62 additions & 0 deletions posthog/clickhouse/hcl/exclude-local-multi.hcl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Per-env override of exclude.hcl for the local-multi convergence dump (dump-live.sh
# uses exclude-<env>.hcl when present, else exclude.hcl — it REPLACES, not merges, so
# this file must carry everything exclude.hcl does plus the local-multi-only entries).
#
# The only addition over exclude.hcl is the metrics-overlay ingest block: migration
# 0305 creates those objects on the LOGS node, but they are not modeled in the
# local-multi golden. On cloud they live in the roles/logs/shared + cloud layers and
# ARE checked (dev/prod keep using exclude.hcl); metrics1 itself diverges per env
# (value/count codecs on local + prod, none on dev), so a single all-env HCL
# declaration cannot serve both. Until that is reconciled, skip them here so the
# gate does not read migration-created objects as drift on local-multi.

exclude {
# Secret Kafka broker/credential config; never modeled in the goldens.
object_types = ["named_collection"]

patterns = [
# --- transient (ClickHouse atomic CREATE-OR-REPLACE / EXCHANGE) ---
"_tmp_replace_*",

# --- migration / ORM / backfill scratch ---
"tmp_*",
"*_tmp",
"infi_clickhouse_orm_migrations*",

# --- backups / staging / backfills ---
"*_backup",
"*_backup_*",
"*_staging",
"*_backfill",

# --- cross-cluster proxies carried by the node but owned elsewhere ---
# Distributed proxies into the main event cluster; owned by the data role.
"events_main",
"events_recent",

# --- out-of-band managed: real on prod, not created by the local
# migrate_clickhouse path, so the gate ignores them on BOTH sides until
# a proper OPS migration reproduces them locally. Remove each entry once
# its migration lands. ---
# custom_metrics* views are created on NodeRole.DATA (migration 0117), never
# on the OPS node, yet prod OPS carries them (created out-of-band).
"custom_metrics*",
# Orphan: present on prod OPS but no migration or code creates it anywhere.
"events_team_daily_stats",

# --- metrics-overlay ingest: created by migration 0305 on LOGS, modeled in

@orian orian Aug 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's better option to extend the goldens / HCL and not exclude them here, excluding them, basically removes any protection against wrong migrations in the future

It's fine to merge like this but this will have to get sorted out

# roles/logs/shared + cloud for dev/prod but not in the local golden.
# metrics1 diverges per env (codecs), so it is not yet in an all-env layer. ---
"metrics1",
"metrics",
"metric_attributes",
"metrics1_to_metric_attributes",
"metrics1_to_resource_attributes",
"metrics_kafka_metrics",
"kafka_metrics_avro",
"kafka_metrics_avro_mv",
"kafka_metrics_avro_to_metric_samples",
"kafka_metrics_avro_to_metric_series",
"kafka_metrics_avro_kafka_metrics_mv",
]
}
15 changes: 15 additions & 0 deletions posthog/clickhouse/hcl/exclude-local-single.hcl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,5 +42,20 @@ exclude {
# Orphan: no migration or code creates it; roles/coshared/custom_metrics models it
# for the ops nodes, which do carry it.
"events_team_daily_stats",

# --- metrics-overlay ingest: created by migration 0305 on LOGS, modeled in
# roles/logs/shared + cloud for dev/prod but not in the local golden.
# metrics1 diverges per env (codecs), so it is not yet in an all-env layer. ---
"metrics1",
"metrics",
"metric_attributes",
"metrics1_to_metric_attributes",
"metrics1_to_resource_attributes",
"metrics_kafka_metrics",
"kafka_metrics_avro",
"kafka_metrics_avro_mv",
"kafka_metrics_avro_to_metric_samples",
"kafka_metrics_avro_to_metric_series",
"kafka_metrics_avro_kafka_metrics_mv",
]
}
14 changes: 14 additions & 0 deletions posthog/clickhouse/metrics/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,11 @@
from .kafka_metrics import (
KAFKA_METRICS_AVRO_MV,
KAFKA_METRICS_AVRO_TABLE_SQL,
KAFKA_METRICS_AVRO_TO_KAFKA_METRICS_MV,
KAFKA_METRICS_AVRO_TO_METRIC_SAMPLES_MV,
KAFKA_METRICS_AVRO_TO_METRIC_SERIES_MV,
METRICS_KAFKA_METRICS_TABLE_SQL,
)
from .metric_attributes import METRIC_ATTRIBUTES_TABLE_SQL
from .metric_events import (
METRIC_SAMPLES_DISTRIBUTED_TABLE_SQL,
Expand All @@ -22,4 +30,10 @@
"METRIC_SERIES_DISTRIBUTED_TABLE_SQL",
"METRIC_SAMPLES_TABLE_SQL",
"METRIC_SAMPLES_DISTRIBUTED_TABLE_SQL",
"KAFKA_METRICS_AVRO_TABLE_SQL",
"KAFKA_METRICS_AVRO_MV",
"KAFKA_METRICS_AVRO_TO_METRIC_SAMPLES_MV",
"KAFKA_METRICS_AVRO_TO_METRIC_SERIES_MV",
"KAFKA_METRICS_AVRO_TO_KAFKA_METRICS_MV",
"METRICS_KAFKA_METRICS_TABLE_SQL",
]
Loading
Loading