Tracks: Studio Code chat & instructions events (STU-2120)

[wojtekn]

Related issues

• Fixes STU-2120

How AI was used in this PR

Claude Code explored the chat architecture, drafted the plan, and wrote the implementation and tests. The event set was decided interactively rather than taken from the issue as written — see the scope note below. I reviewed the emission points, the privacy boundaries, and the edit-session logic, and iterated to fix multiple issues.

Proposed Changes

Studio Code has had no Tracks coverage, so we could not fully answer how much the assistant is used, with which model, or whether turns actually succeed. Although we have server-side tracking on the AI proxy endpoint, having Tracks in place will allow us to see the entire process.

This adds four events:

Event	Answers
studio_code_message_sent	Usage volume, by provider / model / predefined skill
studio_code_turn_completed	Success vs error vs interrupted, and how long turns take
studio_code_session_created	Messages-per-session and sessions-per-user
studio_setting_instructions_change	Whether people customize the agent

Two props from the mandated AI vocabulary are not sent: is_test (dev/E2E/CI runs are suppressed at the source, so it could only ever be false) and agent_version (pi is pinned per Studio release, so app_version already determines it).

No prompts, replies, raw error text, site names, paths, or instructions content are ever sent. ability_name is resolved against the skill catalog, so arbitrary slash text a user typed cannot reach an event prop.

Testing Instructions

Events log instead of sending in dev (the shared core no-ops the network call), so:

npm start

1. Open Studio Code, send a message. The Main-process terminal — the one running npm start logs Would have recorded… studio_code_message_sent with props, then studio_code_turn_completed when it finishes. (The agent fork normally discards the CLI child's stdout, since its events travel over IPC; a dev run now inherits it so this logging is visible at all.)
2. Confirm channel: studio-ui and ui_version are present — this is the fork-env fix. Then run studio code standalone and confirm channel: studio-cli.
3. Invoke a predefined skill (/rank-me-up) and confirm ability_name: rank-me-up. Try it in both UIs — desktop expands the slash command before forking, the browser UI does not, and both shapes must resolve.
4. Start a new chat → studio_code_session_created. Click "new chat" again without sending anything: the empty draft is reused, so no second event.
5. Settings → Studio Code: edit the instructions and navigate away → one studio_setting_instructions_change. Type, pause past the autosave, then leave → still exactly one. Open the tab and leave without editing → none.

Pre-merge Checklist

• Have you checked for TypeScript, React or other console errors?

[wpmobilebot]

📊 Performance Test Results

Comparing 7b2f127 vs trunk

app-size

Metric	trunk	7b2f127	Diff	Change
App Size (Mac)	1406.64 MB	1406.65 MB	+0.02 MB	⚪ 0.0%

site-editor

Metric	trunk	7b2f127	Diff	Change
load	1073 ms	1026 ms	47 ms	⚪ 0.0%

site-startup

Metric	trunk	7b2f127	Diff	Change
siteCreation	7525 ms	7482 ms	43 ms	⚪ 0.0%
siteStartup	2881 ms	2871 ms	10 ms	⚪ 0.0%

---

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

[bcotrim]

LGTM and works as described 👍

Found one issue that I think we should fix before merging, unless I misunderstood the purpose of the is_resumed flag

[bcotrim]

It seems this is always true in the ui since we always create the session.
Should we look into existing messages instead of just session id?

Example:

Would have recorded Tracks event: studio_code_session_created {
  platform: 'darwin',
  arch: 'arm64',
  app_version: '1.18.0-beta1',
  is_a11n: true,
  channel: 'studio-ui',
  ui_version: 'v2',
  ai_session_id: 'a989d09b-8ef5-4059-aceb-0f835203c16f',
  agent_name: 'pi',
  client: 'studio-code',
  has_site: true
}
Would have bumped stat: studio-code-ui-send=darwin
Would have recorded Tracks event: studio_code_message_sent {
  platform: 'darwin',
  arch: 'arm64',
  app_version: '1.18.0-beta1',
  is_a11n: true,
  channel: 'studio-ui',
  ui_version: 'v2',
  ai_session_id: 'a989d09b-8ef5-4059-aceb-0f835203c16f',
  agent_name: 'pi',
  client: 'studio-code',
  provider: 'wpcom',
  model: 'claude-sonnet-5',
  model_family: 'anthropic',
  has_images: false,
  has_files: false,
  is_resumed: true
}

[wojtekn]

Good point, this prop would be meaningful only in CLI.

I decided to remove it in 7b2f127, as we can use studio_code_session_created and studio_code_message_sent to identify the first turn in a conversation, calculate the number of messages per session, and distinguish one-shot conversations from continued ones.