Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .claude/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"permissions": {
"allow": [
"Bash(.venv/bin/portable-python:*)",
"Bash(.venv/bin/pytest:*)",
"Bash(.venv/bin/python scripts/check_okf.py:*)",
"Bash(tox:*)"
]
}
}
8 changes: 5 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,9 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

## What This Project Does

portable-python is a CLI tool and Python library for compiling portable CPython binaries from source. Binaries are statically linked so they can be extracted to any folder and used without installation. Supports Linux and macOS (not Windows).
portable-python is a CLI tool and Python library for compiling portable CPython binaries from source.
Binaries are statically linked so they can be extracted to any folder and used without installation.
Supports Linux and macOS (not Windows).

## Common Commands

Expand All @@ -16,14 +18,14 @@ tox
tox -e py313

# Run a single test
pytest tests/test_build.py::test_build_rc -vv
.venv/bin/pytest tests/test_build.py::test_build_rc -vv

# Lint check / auto-fix
tox -e style
tox -e reformat

# Run tests directly (from venv)
pytest tests/
.venv/bin/pytest tests/

# CI uses: uvx --with tox-uv tox -e py
```
Expand Down
51 changes: 51 additions & 0 deletions docs/architecture/build-setup.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
---
type: Class

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is a page about a class better than a docstring in the code? How do we decide what to put where?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! Rephrased everything to avoid repeating anything from code

title: BuildSetup
description: Coordinates the overall compilation — resolves the python spec, prepares folders, builds external modules then CPython, validates, and compresses the result.
resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py
tags: [class, build, coordinator]
timestamp: 2026-06-23T00:00:00Z
---

# BuildSetup

`BuildSetup` drives a build from end to end. The CLI's [`build`](/cli/build.md) command (and library users) construct one and call `compile()`. Defined in `__init__.py`.

```python
from portable_python import BuildSetup

setup = BuildSetup("cpython:3.13.2")
setup.compile()
```

# Schema

| Member | Kind | Purpose |
|--------|------|---------|
| `__init__(python_spec, modules, prefix)` | constructor | Resolve the [`PythonSpec`](/cli/build.md), set up [`Folders`](/architecture/config.md), pick the extension, and instantiate the family's `python_builder`. |
| `python_spec` | attribute | The resolved family + version (full `X.Y.Z` required). |
| `folders` | attribute (`Folders`) | Resolved build/dist/sources/logs/destdir paths. |
| `prefix` | attribute | Optional `--prefix`; when set the build is **not** [portable](/concepts/portability.md). |
| `tarball_name` | attribute | Composed output name, e.g. `cpython-3.13.2-macos-arm64.tar.gz`. |
| `python_builder` | attribute (`PythonBuilder`) | The concrete builder, e.g. [`Cpython`](/architecture/cpython.md). |
| `validate_module_selection(fatal)` | method | Check every selected/candidate module's `linker_outcome`; abort on a `failed` outcome. |
| `compile()` | method | The full pipeline (see below). |

## The `compile()` pipeline

1. **Clean** the build folder (and logs); set up file logging to `logs/00-portable-python.log`.
2. `python_builder.validate_setup()`.
3. Enter a [`BuildContext`](/concepts/folder-masking.md) — applies macOS `/usr/local` masking and other isolation.
4. Log the platform, config files in use, and the build report.
5. `validate_module_selection()` (fatal unless dry-run / debug).
6. Clean `components/` and `deps/`.
7. `build_context.compile()` then `python_builder.compile()` — external modules first, then CPython.
8. If a `dist/` folder is configured, **compress** the install into `dist/{tarball_name}`.

## Spec validation

`BuildSetup` insists on a **full** version: a spec like `3.13` is rejected — you must give `3.13.2`. `"latest"` (or empty) resolves to `PPG.cpython.latest`. Invalid specs abort early with a red error.

# Citations

[1] [src/portable_python/__init__.py — BuildSetup](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this meant to be referred to from somewhere?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed all citations now 👍 they were indeed all superfluous

55 changes: 55 additions & 0 deletions docs/architecture/config.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
---
type: Class
title: Config & Folders
description: Config loads and merges YAML configuration with platform-specific overrides; Folders resolves the templated build/dist/sources paths.
resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/config.py
tags: [class, configuration, folders, yaml]
timestamp: 2026-06-23T00:00:00Z
---

# Config & Folders

`Config` (in `config.py`) loads, merges, and queries the YAML [configuration](/configuration/portable-python-yml.md). `Folders` (in `versions.py`) turns templated path settings into concrete filesystem paths. Both are reached through [`PPG`](/architecture/ppg.md).

# Schema

## Config

| Member | Purpose |
|--------|---------|
| `__init__(paths, target)` | Layer the built-in `DEFAULT_CONFIG` under any user config files, for the given target platform. |
| `get_value(*key, by_platform=True)` | Fetch a config value, honoring platform-specific overrides (most specific wins). |
| `get_entry(*key, by_platform=True)` | Like `get_value` but returns the raw entry. |
| `resolved_path(*key)` | Fetch a value and resolve it to a path. |
| `config_files_report()` / `represented()` | Human-readable summary of which config files are in effect (used by [`diagnostics`](/cli/diagnostics.md)). |
| `cleanup_globs(...)` / `symlink_duplicates(...)` / `ensure_main_file_symlinks(...)` | Finalize-time file housekeeping in the install. |

## Folders

| Member | Purpose |
|--------|---------|
| `build_folder`, `components`, `deps`, `sources`, `logs`, `dist`, `destdir` | Resolved component paths — see [build layout](/concepts/build-layout.md). |
| `ppp-marker` | Template for the in-progress install folder name. |
| `formatted(text)` | Expand `{build}`, `{version}`, `{abi_suffix}`, … placeholders. |
| `resolved_destdir(relative_path)` | Compose a path under the marker'd destdir. |

## Precedence & overrides

Configuration merges multiple sources. The built-in `DEFAULT_CONFIG` provides sane defaults; user files (default `portable-python.yml`, plus any `include:`d files) layer on top. Within a file, **platform-specific** sections override generic ones:

```yaml
ext: gz # generic default
windows:
ext: zip # used only when targeting windows
macos:
env:
MACOSX_DEPLOYMENT_TARGET: 13
```

`get_value(..., by_platform=True)` returns the most specific match for the current `PPG.target`.

# Citations

[1] [src/portable_python/config.py — Config, DEFAULT_CONFIG](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/config.py)
[2] [src/portable_python/versions.py — Folders](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/versions.py)
[3] [CONFIGURATION.md](https://github.com/codrsquad/portable-python/blob/main/CONFIGURATION.md)
44 changes: 44 additions & 0 deletions docs/architecture/cpython.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
---
type: Class
title: Cpython
description: The concrete PythonBuilder that compiles CPython — configure/make/install, optimization flags, and the finalize step that makes the install relocatable.
resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cpython.py
tags: [class, cpython, build, finalize]
timestamp: 2026-06-23T00:00:00Z
---

# Cpython

`Cpython` is the concrete [`PythonBuilder`](/architecture/python-builder.md) that actually compiles CPython. It is returned by `CPythonFamily.get_builder()` (see [`PPG`](/architecture/ppg.md)) and lives in `cpython.py`.

# Schema

| Member | Kind | Purpose |
|--------|------|---------|
| `candidate_modules()` | classmethod | The external modules CPython can statically link (the [external modules](/modules/index.md) set). |
| `url` | property | CPython source URL (`python.org/ftp/...Python-X.Y.Z.tar.xz`), overridable via config. |
| `c_configure_args()` | method | Assemble `./configure` args (from `cpython-configure` config + computed flags). |
| `has_configure_opt(name, *variants)` | method | Test whether a configure option is in effect. |
| `xenv_LDFLAGS_NODIST`, `xenv_LIBZSTD_*`, `xenv_LIBMPDEC_*` | methods | Environment for linking specific static deps (zstd, mpdec). |
| `_do_linux_compile()` | method | configure → make → make install (DESTDIR). |
| `_finalize()` | method | Clean, byte-compile, relativize — see below. |

## Configure arguments

CPython is configured with, by default, `--enable-optimizations`, `--with-lto`, and `--with-ensurepip=upgrade` (from `cpython-configure` in [config](/configuration/portable-python-yml.md)), plus computed flags pointing at the static deps. Following the [no-patches principle](/overview.md), **all** behavior comes from configure flags — never source edits.

## Finalize: making it relocatable

After `make install` the `_finalize()` step is what turns a normal install into a [portable](/concepts/portability.md) one:

- **Clean passes** — remove tests/idle/2to3 before byte-compiling (`cpython-clean-1st-pass`, ~94 MB) and prune seldom-used pycaches after (`cpython-clean-2nd-pass`, ~1.8 MB).
- **Byte-compile** the standard library (`cpython-compile-all`).
- `_relativize_shebangs()` — rewrite `bin/` script shebangs to be relative.
- `_relativize_sysconfig()` (`RelSysConf`) — rewrite absolute paths in the `sysconfig` data so the install resolves relative to its own location.
- `_apply_pep668()` — mark the environment per PEP 668.
- `_validate_venv_creation()` / `_validate_venv_module()` — sanity-check that `venv` works in the finished build.

# Citations

[1] [src/portable_python/cpython.py](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cpython.py)
[2] [src/portable_python/config.py — cpython-* defaults](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/config.py)
31 changes: 31 additions & 0 deletions docs/architecture/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Architecture

The core classes and how they collaborate during a build. See [ARCHITECTURE.md](https://github.com/codrsquad/portable-python/blob/main/ARCHITECTURE.md) for the original hierarchy diagram.

## Global state & configuration

* [PPG](/architecture/ppg.md) - Global singleton holding config, target platform, and version families.
* [Config](/architecture/config.md) - Loads and merges YAML configuration; the `Folders` helper resolves build paths.

## Build coordination

* [BuildSetup](/architecture/build-setup.md) - Drives the overall compilation: resolve spec, select modules, build, validate, compress.
* [ModuleBuilder](/architecture/module-builder.md) - Abstract base for anything that gets compiled (external C libs and Python itself).
* [PythonBuilder](/architecture/python-builder.md) - `ModuleBuilder` specialization for python implementations.
* [Cpython](/architecture/cpython.md) - Concrete builder: CPython's configure/make/install, optimization, and finalization.

## Validation

* [PythonInspector](/architecture/python-inspector.md) - Validates the portability of a built (or any) python by checking shared-library dependencies and paths.

## Collaboration at a glance

```
PPG (config, target, families)
└─ BuildSetup(python_spec)
├─ Folders (resolved build/dist/... paths)
├─ BuildContext (macOS folder masking, isolation)
└─ python_builder : Cpython (a PythonBuilder, a ModuleBuilder)
└─ modules : ModuleCollection
└─ candidates/selected : ModuleBuilder (Openssl, Zlib, ...)
```
59 changes: 59 additions & 0 deletions docs/architecture/module-builder.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
---
type: Class
title: ModuleBuilder
description: Abstract base for everything that gets compiled — external C libraries and CPython itself — providing the common download/configure/make/install flow and environment injection.
resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py
tags: [class, abstract, build, module]
timestamp: 2026-06-23T00:00:00Z
---

# ModuleBuilder

`ModuleBuilder` is the abstract base for anything compiled by the tool. Both [external C modules](/modules/index.md) and [`PythonBuilder`](/architecture/python-builder.md) (hence [`Cpython`](/architecture/cpython.md)) extend it, so every component shares the same [build layout](/concepts/build-layout.md) and flow.

# Schema

## Declarative class attributes

| Attribute | Purpose |
|-----------|---------|
| `m_name` | Module name (derived from the class name). |
| `m_telltale` | Marker file(s) indicating the lib is present on the system. See [telltale detection](/concepts/telltale-detection.md). |
| `m_debian` | Debian dev package, with `!` / `+` / `-` sigils encoding build constraints. |
| `m_include` | Optional subfolder to add to `CPATH` when active (e.g. `openssl`). |
| `m_build_cwd` | Optional subfolder (relative to unpacked source) to run configure/make from. |

## Source resolution (overridable)

| Member | Purpose |
|--------|---------|
| `url` | Download URL of the source tarball. |
| `version` | Version to build (default per module, overridable via config). |
| `headers` / `src_suffix` | HTTP headers and archive suffix when the URL lacks an extension. |
| `cfg_version`, `cfg_url`, `cfg_configure`, `cfg_patches`, `cfg_http_headers` | Read per-module overrides from [config](/configuration/portable-python-yml.md) (keys like `openssl-version`, `openssl-url`, …). |

## Environment injection (`xenv_*`)

Each `xenv_*` method supplies one environment variable for the compile, pointing tools at the shared `deps/` prefix — the mechanism behind [static linking](/concepts/static-linking.md):

`xenv_CPATH`, `xenv_LDFLAGS`, `xenv_PATH`, `xenv_LD_LIBRARY_PATH`, `xenv_PKG_CONFIG_PATH`. `_find_all_env_vars()` gathers every `xenv_*` on the instance just before running.

## Build flow

| Method | Role |
|--------|------|
| `compile()` | Download → unpack → patch → `_prepare()` → platform compile → `_finalize()`. |
| `run_configure(program, *args, prefix)` | Run `./configure` with the deps prefix. |
| `run_make(*args, cpu_count)` | Run `make` (parallelized by CPU count). |
| `_do_linux_compile()` / `_do_macos_compile()` | Platform-specific compile, dispatched by `PPG.target`. |
| `linker_outcome(is_selected)` | Decide `static` / `shared` / `absent` / `failed` for this module. |
| `captured_logs()` | Context manager routing this module's output to its own numbered log file. |

## Adding a module

To add a new external module you subclass `ModuleBuilder`, set the `m_*` attributes, implement `url`/`version`, and implement `_do_linux_compile()` (macOS reuses it unless overridden). See the [guide](/guides/add-an-external-module.md).

# Citations

[1] [src/portable_python/__init__.py — ModuleBuilder](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py)
[2] [src/portable_python/external/xcpython.py — concrete modules](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/external/xcpython.py)
43 changes: 43 additions & 0 deletions docs/architecture/ppg.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
---
type: Class
title: PPG
description: Global singleton holding shared configuration, the target platform, and the registry of supported python version families.
resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/versions.py
tags: [class, global-state, singleton, versions]
timestamp: 2026-06-23T00:00:00Z
---

# PPG

`PPG` is the global state holder for portable-python. Every module reaches shared state — configuration, the target platform, and version families — through `PPG`'s class attributes and classmethods. It is defined in `versions.py`.

# Schema

| Member | Kind | Purpose |
|--------|------|---------|
| `config` | attribute (`Config`) | The active merged [configuration](/architecture/config.md). |
| `target` | attribute (`PlatformId`) | Target OS/arch; drives platform dispatch and telltale expansion. |
| `families` | attribute (`dict`) | Registry mapping family name → `VersionFamily` (default: `{"cpython": ...}`). |
| `cpython` | attribute (`CPythonFamily`) | The built-in CPython family implementation. |
| `grab_config(paths, target)` | classmethod | (Re)load config from the given paths and set `target`. Called by the CLI's `main`. |
| `get_folders(base, family, version, abi_suffix)` | classmethod | Build a [`Folders`](/architecture/config.md) for a given family/version. |
| `family(name, fatal=True)` | classmethod | Look up a `VersionFamily` by name. |
| `find_python(spec)` | classmethod | Resolve a python on `PATH` via a cached `runez` `PythonDepot`. |
| `find_telltale(*telltales)` | classmethod | Expand `{include}` placeholders against `target.sys_include` and return the first existing path. See [telltale detection](/concepts/telltale-detection.md). |

## Version families

A `VersionFamily` (e.g. `CPythonFamily`) knows how to **list available versions** and **provide a builder**:

- `available_versions` / `latest` — lazily fetched and cached (CPython fetches from `python.org/ftp`, or GitHub tags when `cpython-use-github` is set). This lazy fetch is why [`list`](/cli/list.md) hits the network on first use.
- `get_builder()` — returns the concrete builder class ([`Cpython`](/architecture/cpython.md) for the cpython family).
- `min_version` — earliest non-EOL version known to compile well (currently `3.9`).

## Why a singleton

The build touches global, process-wide facts: which config file is active, what platform we are targeting, and which python versions exist. Centralizing them in `PPG` avoids threading that state through every constructor, and gives the CLI a single place ([`main`](/cli/index.md)) to initialize it.

# Citations

[1] [src/portable_python/versions.py](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/versions.py)
[2] [CLAUDE.md — Key classes](https://github.com/codrsquad/portable-python/blob/main/CLAUDE.md)
38 changes: 38 additions & 0 deletions docs/architecture/python-builder.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
---
type: Class
title: PythonBuilder
description: A ModuleBuilder specialization for python implementations — adds module selection, the python install layout, and helpers to run the freshly built interpreter.
resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py
tags: [class, abstract, python, build]
timestamp: 2026-06-23T00:00:00Z
---

# PythonBuilder

`PythonBuilder` extends [`ModuleBuilder`](/architecture/module-builder.md) with behavior specific to building a python interpreter. It is still abstract — the concrete implementation is [`Cpython`](/architecture/cpython.md). Defined in `__init__.py`.

# Schema

| Member | Kind | Purpose |
|--------|------|---------|
| `modules` | attribute (`ModuleCollection`) | The external modules selected/available for this build. |
| `selected_modules()` | method | Builds the `ModuleCollection` from candidates + the desired list. |
| `bin_python` | property | Path to the built interpreter (`.../bin/python`). |
| `version` | property | The python version being built. |
| `validate_setup()` | method | Pre-flight checks before compilation begins. |
| `run_python(*args)` | method | Invoke the freshly built interpreter (used during finalize/validation). |
| `xenv_LDFLAGS()` | method | Python-specific linker flags layered on the base `ModuleBuilder` behavior. |
| `_prepare()` | method | Hook run before the platform compile. |

## ModuleCollection

A `PythonBuilder` owns a `ModuleCollection`, which models the set of candidate external modules and resolves which are actually built:

- `candidates` — every possible module for this builder (from `candidate_modules()`).
- `selected` — only the modules chosen for *this* build (config + `--modules` + auto-selection). This is the list that actually gets compiled.
- `auto_selected` — modules force-selected because a build can't succeed without them (each module's `auto_select_reason()`).
- `report()` / `report_rows()` — the human-readable table shown by [`build-report`](/cli/build-report.md).

# Citations

[1] [src/portable_python/__init__.py — PythonBuilder, ModuleCollection](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py)
Loading
Loading