[HEAVILY WIP] Migrate to ClickHouse by skyfallwastaken · Pull Request #1470 · hackclub/hackatime

skyfallwastaken · 2026-07-08T19:08:07Z

Warning

This PR is incomplete.

Summary of the problem

Postgres slow :(
~500GB of storage used for heartbeats alone. This is woefully inefficient.

Describe your changes

Removes dashboard rollups. They caused a ton of DB queries and ClickHouse is fast enough to not need them anyway.
Migrates all queries to ClickHouse.

Not in this PR but there's a PeerDB mirror from Postgres -> CH atm. It'll get turned off once we move

Screenshots / Media

N/A

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

socket-security · 2026-07-08T19:09:06Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	gem/clickhouse-activerecord@1.6.7

View full report

greptile-apps · 2026-07-08T19:14:35Z

Greptile Summary

This PR moves heartbeat storage and analytics from Postgres rollups to ClickHouse. The main changes are:

Native ClickHouse heartbeat model, writer, schema, and migrations.
Heartbeat ingest, import, export, and Wakatime-compatible API paths updated to use ClickHouse.
Dashboard, profile, stats, badge, cache, and admin heartbeat queries moved to ClickHouse-backed helpers.
Dashboard rollups removed in favor of live ClickHouse aggregation.

Confidence Score: 3/5

This PR has a core account-merge bug that should be fixed before relying on the migrated path.

The migration touches heartbeat ingestion, exports, dashboards, API stats, admin tools, and ClickHouse schema. The account-merge path can leave account merges partially applied across Postgres and ClickHouse. The PR is also marked heavily WIP, so changed heartbeat paths need careful validation before merge.

app/controllers/admin/account_merger_controller.rb; app/models/clickhouse/heartbeat.rb

T-Rex Logs

What T-Rex did

T-Rex reproduced the merge-atomicity verification by forcing a ClickHouse failure after the transaction and observed that transaction.commit occurred before the failure.
T-Rex confirmed the runtime environment, finding Ruby 3.1.2 and Bundler 2.3.15 in the sandbox while the app requires Ruby 4.0.5.
T-Rex encountered a Rails boot blocker when Bundler could not parse the Gemfile on Windows, so the narrow executable Ruby harness path requested by the task was used.
T-Rex captured the harness run showing the partial Postgres commit after the ClickHouse failure and the resulting deletion of the newer user with a reassigned Postgres-owned goal.
T-Rex ran the verification but noted that local artifact references were not uploaded for this run.

_{Ran code and verified through T-Rex}

Important Files Changed

Filename	Overview
app/controllers/admin/account_merger_controller.rb	Moves heartbeat reassignment to ClickHouse, but the operation is outside the Postgres transaction and can leave merges partially applied.
app/models/clickhouse/heartbeat.rb	Adds the ClickHouse heartbeat model and query helpers for durations, aggregation, deletion scopes, and user/project stats.
app/models/clickhouse/heartbeat_writer.rb	Introduces native ClickHouse insert, soft-delete, and account-merge write helpers using `ReplacingMergeTree` versions.
app/services/heartbeat_ingest.rb	Routes direct and imported heartbeat ingestion through the ClickHouse writer with normalization and in-batch deduplication.
db/clickhouse/20260709000001_rebuild_heartbeats_for_native_writes.rb	Rebuilds the ClickHouse heartbeat table for native writes with `ReplacingMergeTree` and PeerDB compatibility columns.
app/controllers/api/hackatime/v1/hackatime_controller.rb	Updates heartbeat push and Wakatime-compatible stats endpoints to read and write ClickHouse rows.
lib/wakatime_service.rb	Switches summary generation to ClickHouse-backed scopes and duration helpers.
app/jobs/heartbeat_export_job.rb	Exports user heartbeat data from ClickHouse while preserving the previous export JSON shape.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
  participant Client
  participant API as Hackatime API
  participant Ingest as HeartbeatIngest
  participant Writer as Clickhouse::HeartbeatWriter
  participant CH as ClickHouse heartbeats
  participant Stats as Stats/Dashboard services

  Client->>API: POST heartbeats
  API->>Ingest: normalize direct/import heartbeat payloads
  Ingest->>Writer: insert_rows(rows)
  Writer->>CH: INSERT native heartbeat rows
  Stats->>CH: read deduped FINAL heartbeat scopes
  CH-->>Stats: durations, aggregates, spans, exports
  Stats-->>Client: API/dashboard/export responses

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
  participant Client
  participant API as Hackatime API
  participant Ingest as HeartbeatIngest
  participant Writer as Clickhouse::HeartbeatWriter
  participant CH as ClickHouse heartbeats
  participant Stats as Stats/Dashboard services

  Client->>API: POST heartbeats
  API->>Ingest: normalize direct/import heartbeat payloads
  Ingest->>Writer: insert_rows(rows)
  Writer->>CH: INSERT native heartbeat rows
  Stats->>CH: read deduped FINAL heartbeat scopes
  CH-->>Stats: durations, aggregates, spans, exports
  Stats-->>Client: API/dashboard/export responses

Prompt To Fix All With AI

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
app/controllers/admin/account_merger_controller.rb:109
**Keep merge atomic**
`merge_user_heartbeats!` now runs after the Postgres transaction has committed and after `newer_user.destroy!`. If the ClickHouse insert or tombstone fails, the newer user and related Postgres rows are already deleted while their heartbeats remain under the deleted `user_id`, so the account merge is only partially applied despite the rescue claiming rollback.

_{Reviews (3): Last reviewed commit: "Numerous fixes" | Re-trigger Greptile}

skyfallwastaken · 2026-07-08T21:25:24Z

Note to self: do we need FINAL here?

skyfallwastaken · 2026-07-08T21:25:34Z

@greptileai review

…house

skyfallwastaken · 2026-07-09T07:28:00Z

@greptile-apps review

+          end
+        end
+        connection.execute(<<~SQL.squish)
+          INSERT INTO #{table} (#{column_list(connection)})


+          column == "user_id" ? older_user_id.to_i.to_s : connection.quote_column_name(column)
+        end
+        connection.execute(<<~SQL.squish)
+          INSERT INTO #{table} (#{column_list(connection)})


+          end
+        end
+        connection.execute(<<~SQL.squish)
+          INSERT INTO #{table} (#{column_list(connection)})


greptile-apps · 2026-07-09T07:31:34Z

      results << "user ##{newer_user.id} deleted"
    end

+    Clickhouse::HeartbeatWriter.merge_user_heartbeats!(older_user_id: older_user.id, newer_user_id: newer_user.id)


Keep merge atomic
merge_user_heartbeats! now runs after the Postgres transaction has committed and after newer_user.destroy!. If the ClickHouse insert or tombstone fails, the newer user and related Postgres rows are already deleted while their heartbeats remain under the deleted user_id, so the account merge is only partially applied despite the rescue claiming rollback.

Artifacts

Repro: runtime version check output

Keeps the command output available without making the summary code-heavy.

Repro: Rails boot blocker from bundle check

Keeps the command output available without making the summary code-heavy.

Repro: executable account merge atomicity harness

Contains supporting evidence from the run (text/x-ruby; charset=utf-8).

Repro: harness output showing partial Postgres commit after ClickHouse failure

Keeps the command output available without making the summary code-heavy.

_{Ran code and verified through T-Rex}

Prompt To Fix With AI

This is a comment left during a code review. Path: app/controllers/admin/account_merger_controller.rb Line: 109 Comment: **Keep merge atomic** `merge_user_heartbeats!` now runs after the Postgres transaction has committed and after `newer_user.destroy!`. If the ClickHouse insert or tombstone fails, the newer user and related Postgres rows are already deleted while their heartbeats remain under the deleted `user_id`, so the account merge is only partially applied despite the rescue claiming rollback. How can I resolve this? If you propose a fix, please make it concise.

skyfallwastaken added 3 commits July 6, 2026 13:58

Migrate to ClickHouse

8352e38

Fix N+1 in TrustLevelAuditLogsController

395824e

????? ClickHouse

645ca75

Copilot AI review requested due to automatic review settings July 8, 2026 19:08

Copilot AI reviewed Jul 8, 2026

github-advanced-security AI found potential problems Jul 8, 2026

View reviewed changes

Comment thread app/controllers/admin/account_merger_controller.rb Fixed

Comment thread app/controllers/admin/account_merger_controller.rb Fixed

unfuck the test

551e691

greptile-apps Bot reviewed Jul 8, 2026

View reviewed changes

Comment thread app/models/clickhouse/heartbeat.rb Outdated

Comment thread app/models/clickhouse/heartbeat.rb Outdated

thingamibobs

14c5ee0

github-advanced-security AI found potential problems Jul 8, 2026

View reviewed changes

Comment thread app/models/clickhouse/heartbeat_writer.rb Fixed

Comment thread app/models/clickhouse/heartbeat_writer.rb Fixed

Comment thread app/models/clickhouse/heartbeat_writer.rb Fixed

skyfallwastaken and others added 2 commits July 8, 2026 22:23

greptile

b8f8c7c

Merge branch 'main' into clickhouse

92b7493

skyfallwastaken added 5 commits July 8, 2026 23:53

what the fuck is fable doing

361c425

Merge branch 'clickhouse' of github.com:hackclub/hackatime into click…

18e6db2

…house

clean up the source_type stuff

b16a3bc

well, that ain't meant to be there

ba81be2

Numerous fixes

521973b

github-advanced-security AI found potential problems Jul 9, 2026

View reviewed changes

greptile-apps Bot reviewed Jul 9, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[HEAVILY WIP] Migrate to ClickHouse#1470

[HEAVILY WIP] Migrate to ClickHouse#1470
skyfallwastaken wants to merge 12 commits into
mainfrom
clickhouse

skyfallwastaken commented Jul 8, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

socket-security Bot commented Jul 8, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jul 8, 2026 •

edited

Loading

T-Rex Logs

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

skyfallwastaken commented Jul 8, 2026

Uh oh!

skyfallwastaken commented Jul 8, 2026

Uh oh!

skyfallwastaken commented Jul 9, 2026

Uh oh!

greptile-apps Bot Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

skyfallwastaken commented Jul 8, 2026

Summary of the problem

Describe your changes

Screenshots / Media

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

socket-security Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 3/5

T-Rex Logs

Important Files Changed

Sequence Diagram

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

skyfallwastaken commented Jul 8, 2026

Uh oh!

skyfallwastaken commented Jul 8, 2026

Uh oh!

skyfallwastaken commented Jul 9, 2026

Uh oh!

greptile-apps Bot Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

socket-security Bot commented Jul 8, 2026 •

edited

Loading

greptile-apps Bot commented Jul 8, 2026 •

edited

Loading