-
Notifications
You must be signed in to change notification settings - Fork 1k
Add Devin skills for testing, setup, codegen, CI, releases, templates, and triage #1755
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
devin-ai-integration
wants to merge
8
commits into
main
Choose a base branch
from
devin/1787338449-devin-skills
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
483416c
Add Devin skills for testing SDKs and CLI visual output
devin-ai-integration[bot] c62f5fe
Fold in tips from prior testing-cli-tables skill
devin-ai-integration[bot] 9fdbb67
Add CI flakiness and mock-suite insights from prior sessions
devin-ai-integration[bot] b056917
Scope skills to the testing process only
devin-ai-integration[bot] 2ca2d87
Remove DEV.md
devin-ai-integration[bot] 97e8957
Restore unit-test rendering section in visual-output skill
devin-ai-integration[bot] c7d7428
Add skills for env setup, codegen, CI debugging, releasing, templates…
devin-ai-integration[bot] 3daee27
Note streaming exception to sync/async test parity
devin-ai-integration[bot] File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| --- | ||
| name: testing-cli | ||
| description: "Build, run, and test the E2B CLI (packages/cli) locally. Use when changing CLI commands, running its vitest suites, or executing the CLI against real sandboxes." | ||
| --- | ||
|
|
||
| # Testing the CLI | ||
|
|
||
| All commands run in `packages/cli`. Use Node 24 (`nvm use 24`) and pnpm. | ||
|
|
||
| ## Build and run locally | ||
|
|
||
| ```bash | ||
| pnpm build # tsc typecheck + tsdown bundle -> dist/index.js | ||
| node dist/index.js --help | ||
| node dist/index.js sandbox list | ||
| ``` | ||
|
|
||
| Auth: the CLI reads `E2B_API_KEY` from the environment first, then `~/.e2b/config.json` (or `.env.local` at the repo root). Never run `e2b auth login` in headless environments — export the key instead. | ||
|
|
||
| Useful non-interactive patterns: | ||
|
|
||
| ```bash | ||
| node dist/index.js sandbox create base --detach # returns sandbox ID, no attached terminal | ||
| node dist/index.js sandbox exec <id> -- bash -lc 'pwd' # `--` stops CLI flag parsing | ||
| node dist/index.js sandbox kill <id> | ||
| ``` | ||
|
|
||
| ## Automated tests | ||
|
|
||
| ```bash | ||
| pnpm run test # vitest; globalSetup runs `pnpm build` first | ||
| npx vitest run tests/utils/table.test.ts # single file | ||
| ``` | ||
|
|
||
| - Tests spawn the **built** CLI (`dist/index.js`) via helpers in `tests/setup.ts` (`runCli`, `runCliWithPipedStdin`) — rebuild happens automatically through globalSetup, but if you bypass vitest, run `pnpm build` yourself after editing `src/`. | ||
| - `tests/commands/**` cover command behavior; some hit the real API and need `E2B_API_KEY`. | ||
| - Unit tests import from `src/` directly (vitest aliases `e2b` to `../js-sdk/src`), so keep command logic in exported, testable functions (see `buildTableRows`/`sortSandboxes` in `src/commands/sandbox/list.ts`). | ||
|
|
||
| For checking rendered output (tables, colors, alignment), use the `verifying-cli-visual-output` skill. | ||
|
|
||
| ## Before committing | ||
|
|
||
| ```bash | ||
| pnpm run lint && pnpm run typecheck | ||
| ``` | ||
|
|
||
| Public-surface changes need a changeset (`pnpm changeset` at repo root). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| --- | ||
| name: testing-js-sdk | ||
| description: "Run and write tests for the E2B JavaScript SDK (packages/js-sdk). Use when changing js-sdk code, debugging its vitest suites, or verifying SDK behavior against real sandboxes." | ||
| --- | ||
|
|
||
| # Testing the JS SDK | ||
|
|
||
| All commands run in `packages/js-sdk`. Use Node 24 (`nvm use 24`) and pnpm. | ||
|
|
||
| ## Test projects | ||
|
|
||
| Tests are organized into vitest projects in `vitest.config.mts`: | ||
|
|
||
| - `unit` — everything in `tests/**/*.test.ts` except runtimes, template, and connectionConfig. Many of these are **integration tests that create real sandboxes** and require `E2B_API_KEY`. | ||
| - `template` — template builder tests (`tests/template/**`), 180s timeout, require `E2B_API_KEY`. | ||
| - `connectionConfig` — offline config tests. | ||
| - `browser` — Playwright/chromium tests (`pnpm run playwright:install` first). | ||
|
|
||
| ## Running | ||
|
|
||
| ```bash | ||
| # everything (needs E2B_API_KEY) | ||
| pnpm run test | ||
|
|
||
| # one file — fastest loop, preferred while iterating | ||
| npx vitest run tests/api/inflight.test.ts | ||
|
|
||
| # one project | ||
| npx vitest run --project connectionConfig | ||
|
|
||
| # alternate runtimes | ||
| pnpm run test:bun # bun: unit + connectionConfig + template | ||
| pnpm run test:deno # deno: same projects | ||
| ``` | ||
|
|
||
| `E2B_API_KEY` is read from the environment or from `.env` via dotenv (the repo also keeps defaults in `.env.local` at the root or `~/.e2b/config.json`). Purely offline unit tests (e.g. `tests/api/inflight.test.ts`, `tests/utils.test.ts`) run without a key. | ||
|
|
||
| ## Writing tests | ||
|
|
||
| - Use the helpers in `tests/setup.ts`: `sandboxTest` / `templateTest` fixtures create and clean up sandboxes; `isDebug` gates behavior when `E2B_DEBUG` is set (local envd at debug port). | ||
| - Offline tests that need HTTP mock the API with `msw`; see `tests/api/` for patterns. Test isolation is required (`isolate: true`) because suites patch global fetch. | ||
| - SDK changes must be mirrored in the Python SDK (sync + async) with equivalent tests — see `testing-python-sdk`. | ||
|
|
||
| ## Before committing | ||
|
|
||
| ```bash | ||
| pnpm run lint && pnpm run typecheck | ||
| ``` | ||
|
|
||
| Public-surface changes need a changeset (`pnpm changeset` at repo root). | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| --- | ||
| name: testing-python-sdk | ||
| description: "Run and write tests for the E2B Python SDK (packages/python-sdk). Use when changing python-sdk code (sync or async), running pytest suites, or mirroring JS SDK changes in Python." | ||
| --- | ||
|
|
||
| # Testing the Python SDK | ||
|
|
||
| All commands run in `packages/python-sdk`. Use uv for everything (`uv run ...`); never pip. | ||
|
|
||
| ## Layout | ||
|
|
||
| - `tests/sync/` and `tests/async/` — integration tests against real sandboxes (need `E2B_API_KEY`). Sync and async variants must stay equivalent. | ||
|
devin-ai-integration[bot] marked this conversation as resolved.
Outdated
|
||
| - `tests/test_*.py` (top level) — offline unit tests (transports, codecs, config parsing, etc.). | ||
| - `tests/conftest.py` — fixtures that create/clean up sandboxes; `pytest.ini` sets `asyncio_mode=auto`, a 30s per-test timeout, and `pythonpath = tests` for shared helpers like `envd_frame_server`. | ||
|
|
||
| ## Running | ||
|
|
||
| ```bash | ||
| # full suite, 4 workers (needs E2B_API_KEY in env) | ||
| pnpm run test # == uv run pytest -n 4 --verbose -x | ||
|
|
||
| # single file / test — preferred while iterating | ||
| uv run pytest tests/test_paginator.py -v | ||
| uv run pytest tests/sync/sandbox_sync/test_create.py -v -k "metadata" | ||
|
|
||
| # offline-only quick check (skip integration dirs) | ||
| uv run pytest tests --ignore=tests/sync --ignore=tests/async -q | ||
| ``` | ||
|
|
||
| The `skip_debug` marker skips a test when `E2B_DEBUG` is set (local envd). | ||
|
|
||
| ## Writing tests | ||
|
|
||
| - Every behavior change must land in **both** sync and async implementations with matching tests in `tests/sync/` and `tests/async/` (and mirror the JS SDK — see `testing-js-sdk`). | ||
|
devin-ai-integration[bot] marked this conversation as resolved.
Outdated
|
||
| - Async tests need no decorator (`asyncio_mode=auto`). | ||
| - Reuse conftest fixtures rather than creating sandboxes by hand. | ||
|
|
||
| ## Before committing | ||
|
|
||
| ```bash | ||
| pnpm run lint && pnpm run typecheck # ruff check/format + ty check | ||
| ``` | ||
|
|
||
| Public-surface changes need a changeset (`pnpm changeset` at repo root). | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| --- | ||
| name: verifying-cli-visual-output | ||
| description: "Verify the visual/terminal output of the E2B CLI: tables, colors, alignment, spinners, TTY vs piped behavior. Use when changing anything the CLI prints or when reviewing output formatting." | ||
| --- | ||
|
|
||
| # Verifying CLI Visual Output | ||
|
|
||
| The CLI's output style follows kubectl: borderless tables, uppercase headers, left-aligned columns with 3-space padding, no trailing whitespace (`packages/cli/src/utils/table.ts` → `renderTable`). Colors/bold come from `chalk` (`src/utils/format.ts`). | ||
|
|
||
| ## 1. Unit-test the rendering (preferred) | ||
|
|
||
| Capture `console.log` lines and assert exact strings — see `tests/utils/table.test.ts`: | ||
|
|
||
| ```ts | ||
| const lines: string[] = [] | ||
| vi.spyOn(console, 'log').mockImplementation((l: string) => lines.push(l)) | ||
| renderTable(rows, columns) | ||
| expect(lines).toEqual(['SANDBOX ID NAME', 'sbx-1 alpha']) | ||
| ``` | ||
|
|
||
| Keep row-building logic in exported pure functions (e.g. `buildTableRows` in `src/commands/sandbox/list.ts`) so formatting is testable without the API. | ||
|
|
||
| ## 2. Eyeball the real output | ||
|
|
||
| ```bash | ||
| cd packages/cli && pnpm build | ||
| node dist/index.js sandbox list | ||
| ``` | ||
|
|
||
| Checks to make by eye: | ||
| - Headers uppercase; columns aligned even with wide cells (widths use `wcswidth`, so CJK/emoji count as 2 cells). | ||
| - No borders, no trailing whitespace (`node dist/index.js sandbox list | cat -A` — no spaces before `$`). | ||
| - Long values (metadata JSON) don't break alignment of preceding columns. | ||
| - Empty result sets print a sensible message, not a lone header or a crash. | ||
|
|
||
| ## 3. TTY vs piped behavior | ||
|
|
||
| chalk auto-strips colors when stdout is not a TTY, so piped output must stay clean and parseable: | ||
|
|
||
| ```bash | ||
| node dist/index.js sandbox list | head # no ANSI escape codes expected | ||
| node dist/index.js sandbox list | grep -c $'\e' # should be 0 | ||
| FORCE_COLOR=1 node dist/index.js sandbox list # force colors while piping, to inspect them | ||
| script -qec "node dist/index.js sandbox list" /dev/null # run under a real PTY | ||
| ``` | ||
|
|
||
| Interactive commands (spinners, prompts via `inquirer`) need a PTY — use the `script` trick above or a tty-enabled shell; never leave them attached in CI-style runs (prefer `--detach` variants). | ||
|
|
||
| ## 4. Screenshot for PRs | ||
|
|
||
| For user-facing output changes, run the command in a real terminal, take a screenshot, and embed it in the PR description — reviewers care about how it looks, not just the strings. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.