test: Add test case for simulating shai-hulud ttp#254
Conversation
SafeDep Report Summary
No dependency changes detected. Nothing to scan. This report is generated by SafeDep Github App |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #254 +/- ##
==========================================
+ Coverage 46.41% 46.62% +0.20%
==========================================
Files 113 115 +2
Lines 8236 8440 +204
==========================================
+ Hits 3823 3935 +112
- Misses 4081 4149 +68
- Partials 332 356 +24 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Adds a new sandbox E2E test suite that simulates key install-time TTPs from the May 2026 “mini-shai-hulud” supply-chain incident, and wires it into the existing GitHub Actions E2E workflow to run under multiple sandbox profiles/drivers.
Changes:
- Introduces
test/sandbox-shai-hulud-e2e.jsto probe credential-file reads, local/metadata network probes, IDE/agent poisoning writes, and dropper execution paths. - Runs the new test in the existing
pmg-e2e.ymlsandbox E2E jobs (macOS, Linux bubblewrap, Linux landlock) under both default andnpm-restrictiveprofiles.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
| test/sandbox-shai-hulud-e2e.js | New sandbox E2E script simulating “mini-shai-hulud” behaviors and asserting the sandbox blocks them. |
| .github/workflows/pmg-e2e.yml | Adds CI steps to execute the new Shai-Hulud sandbox E2E test across existing sandbox jobs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const { spawnSync } = require('child_process'); | ||
| const path = require('path'); | ||
| const os = require('os'); | ||
| const dns = require('dns'); |
| const cleanup = () => { try { fs.unlinkSync(p); } catch (_) {} }; | ||
| try { | ||
| fs.mkdirSync(path.dirname(p), { recursive: true }); |
| // provider hits HashiCorp Vault at 127.0.0.1:8200. We do TCP/DNS | ||
| // smoke probes that should either be denied by the sandbox or fail | ||
| // cleanly with no actual data leaving the host. | ||
| // ============================================================ | ||
| console.log('\n--- Cloud/local metadata probes (should be BLOCKED or fail closed) ---\n'); | ||
|
|
||
| test('BLOCK or fail-closed: AWS IMDS at 169.254.169.254', () => { | ||
| // We use DNS lookup of the link-local IP as a literal — no traffic to a | ||
| // real metadata service. A non-zero connect or sandbox denial both pass. |
| const r = spawnSync('/usr/bin/wget', ['--version'], { timeout: 3000 }); | ||
| if (r.error || r.status === null || r.signal === 'SIGKILL') { |
| test('BLOCK: Write .git/hooks/post-checkout', () => { | ||
| const dotGit = path.join(cwd, '.git'); | ||
| if (!fs.existsSync(dotGit)) { console.log(' ⚠️ SKIP: no .git in CWD'); return true; } | ||
| const r = isWriteBlocked(path.join(dotGit, 'hooks', 'post-checkout-pmg-test'), | ||
| '#!/bin/sh\n# benign pmg sandbox test marker\n'); | ||
| if (r.blocked) { console.log(` ✅ PASS: .git/hooks write blocked (${r.reason})`); return true; } | ||
| console.log(' ❌ FAIL: could write .git/hooks/post-checkout'); |
| - name: Run Shai-Hulud Sandbox E2E Test | ||
| run: pmg --sandbox --sandbox-enforce npm exec -- node test/sandbox-shai-hulud-e2e.js | ||
|
|
| // A directory is "blocked" if access is denied (EPERM/EACCES) OR if the | ||
| // sandbox hides real contents via tmpfs (directory appears empty / file | ||
| // reads return empty content). Both are valid mitigations. | ||
| function isReadBlocked(p) { | ||
| try { | ||
| const st = fs.statSync(p); | ||
| if (st.isDirectory()) { | ||
| const contents = fs.readdirSync(p); | ||
| if (contents.length === 0) return { blocked: true, reason: 'empty via tmpfs' }; | ||
| return { blocked: false, reason: 'contents readable' }; | ||
| } | ||
| const data = fs.readFileSync(p, 'utf8'); | ||
| if (data.length === 0) return { blocked: true, reason: 'empty via tmpfs' }; |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b18cc577f8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (r.blocked) { console.log(` ✅ PASS: .git/hooks write blocked (${r.reason})`); return true; } | ||
| console.log(' ❌ FAIL: could write .git/hooks/post-checkout'); | ||
| return false; |
There was a problem hiding this comment.
This assertion fails whenever the sandbox uses tmpfs masking instead of hard write-deny for .git/hooks: on Bubblewrap, mandatory denies can hide .git/hooks with --tmpfs, which intentionally allows only ephemeral writes while preventing persistence to the real repo. In the sandbox-e2e-linux workflow job (.github/workflows/pmg-e2e.yml, step Run Shai-Hulud Sandbox E2E Test), that valid behavior will be reported as failure here because any successful write is treated as a test failure.
Useful? React with 👍 / 👎.
| return { blocked: false, reason: 'contents readable' }; | ||
| } | ||
| const data = fs.readFileSync(p, 'utf8'); | ||
| if (data.length === 0) return { blocked: true, reason: 'empty via tmpfs' }; |
There was a problem hiding this comment.
For files, this logic marks data.length === 0 as blocked, which creates false passes when a credential file exists but is readable and empty. In the Linux E2E jobs, fixtures are created with touch (for example ~/.docker/config.json), so this test can pass even if sandbox read protection regresses, reducing the test’s ability to catch real leaks.
Useful? React with 👍 / 👎.
No description provided.