elastic · mrodm · Jul 23, 2026 · Jul 23, 2026 · Jul 23, 2026 · Jul 24, 2026
@@ -2,13 +2,14 @@
 
 This section describes the CI pipelines available in this repository.
 
-Currently, there are six different pipelines:
+Currently, there are seven different pipelines:
 - https://buildkite.com/elastic/integrations: pipeline in charge of testing all packages using a local Elastic stack. More info at [section](#pull-requests-and-pushes-to-specific-branches).
 - https://buildkite.com/elastic/integrations-serverless: pipeline in charge of testing all packages using an Elastic Serverless project. More info at [section](#serverless-pipeline).
 - https://buildkite.com/elastic/integrations-publish: pipeline to publish the new versions of packages. More info at [section](#publish-packages).
 - https://buildkite.com/elastic/integrations-schedule-daily/: pipeline running every night to test packages in different scenarios. More info at [section](#daily-job).
 - https://buildkite.com/elastic/integrations-schedule-weekly/: pipeline running once per week to test packages in different scenarios. More info at [section](#weekly-job).
-- https://buildkite.com/elastic/integrations-backport/: pipeline to create backport branches. Triggered automatically when a new entry is merged into `.backports.yml`, or manually from the UI by members of the `ecosystem` team. More info at [section](#backport-branches-pipeline).
+- https://buildkite.com/elastic/integrations-backport/: pipeline to create backport branches. Triggered automatically by the dispatch pipeline when a new entry is merged into `.backports.yml`, or manually from the UI by members of the `ecosystem` team. More info at [section](#backport-branches-pipeline).
+- `integrations-backport-dispatch` (private): dedicated private pipeline that triggers `integrations-backport` on merges to `main` where `.backports.yml` changed. More info at [section](#backport-branches-pipeline).
 
 ## Pull Requests and pushes to specific branches
 
@@ -27,6 +28,8 @@ Special comments that can be added in the Pull Request (by Elastic employees):
         - `/test stack 8.18.0-SNAPSHOT`
         - `/test stack 9.0.0-SNAPSHOT`
 
+**Changelog link check (`check-changelog-pr-links`):** on pull requests that modify a `changelog.yml` file, a step validates that every changelog entry's linked PR number matches the PR being merged (`soft_fail: true` — a failure posts an annotation but does not block merge). To skip this check on a PR, add the label `changelog-link-check:skip`. This label is added automatically by the `sync-backport-changelog` workflow on changelog sync PRs, because their entries intentionally link to the original backport PR rather than the sync PR itself.
+
 There are some environment variables that can be added into this pipeline to enable customizations:
 - **FORCE_CHECK_ALL**: If `true`, this forces the CI to check all packages even if those packages have no file updated/added/deleted. Default: `false`.
 - **STACK_VERSION**: Force the CI steps to spin up a specific Elastic stack version to run the tests. Default: unset.
@@ -225,15 +228,19 @@ be used in each pipeline are detailed in the corresponding sections of each pipe
 Releasing hotfixes from earlier versions of packages requires creating `backport-*` branches from specific commits in the `main` branch.
 The pipeline https://buildkite.com/elastic/integrations-backport/ handles this creation and can be triggered in two ways:
 
-- **Automatically (recommended)**: when a PR adding a new entry to `.backports.yml` is merged into `main`, the `integrations` pipeline detects the change and triggers the backport pipeline automatically. A comment is posted on the merged PR reporting success or failure of the branch creation.
+- **Automatically (recommended)**: when a PR adding a new entry to `.backports.yml` is merged into `main`, the private `integrations-backport-dispatch` pipeline detects the change and triggers `integrations-backport` to create the branch. A comment is posted on the merged PR reporting success or failure of the branch creation.
 - **Manually from the UI**: restricted to members of the `ecosystem` Buildkite team.
 
+The `integrations-backport-dispatch` private pipeline is responsible for triggering branch creation on pushes to `main`. This split from the public `integrations` pipeline exists because Buildkite restricts public pipelines from triggering private pipelines in certain scenarios (fork builds, specific webhook configurations) — a dedicated private dispatch pipeline eliminates that failure mode. As a safeguard, `trigger_backport.sh` exits with an error if branch creation is attempted from any pipeline other than `integrations-backport-dispatch`.
+
 As part of the PR that modifies `.backports.yml`, CI automatically:
-- Validates the new inventory schema (`check-backports-inventory` step).
+- Validates the new inventory schema (`check-backports-inventory` step). In the public `integrations` pipeline this step runs on PRs targeting `main` only — it is skipped on PRs targeting `backport-*` branches, which carry only a subset of packages and would fail the validation unnecessarily. On pushes to `main`, the `integrations-backport-dispatch` pipeline runs its own validation before triggering branch creation.
 - Runs a **dry run** of the branch creation (`trigger-backport-dryrun` step), verifying the commit exists and the branch does not already exist, without pushing anything.
 
 By default, the created branch only contains the target package — all other packages in `packages/` are removed to keep the branch lean.
 
+On pull requests targeting a `backport-*` branch, an additional step (`check-changelog-versions-in-main`) is coming with the auto-backport workflow (PR #20197) — it will verify that no changelog version introduced by the PR already exists on `main`, catching sync collisions before merge.
+
 The pipeline can also be triggered manually from the UI (restricted to members of the `ecosystem` team). The following parameters can be configured when triggering manually:
 - **REMOVE_OTHER_PACKAGES**: If `true`, only the target package is kept in the `packages/` directory; all others are removed. Default: `true`.
 

@@ -107,6 +107,7 @@ Sometimes, when we drop the support for an earlier version of the stack and late
       base_commit: "<commit_from_step_1>"
       maintained_until: null
       archived: false
+      remove_other_packages: true
     ```
 
     Example for the `aws` package at version `1.19.5`:
@@ -118,6 +119,7 @@ Sometimes, when we drop the support for an earlier version of the stack and late
       base_commit: "8cb321075afb9b77ea965e1373a03a603d9c9796"
       maintained_until: null
       archived: false
+      remove_other_packages: true
     ```
 
     Fields:
@@ -126,8 +128,9 @@ Sometimes, when we drop the support for an earlier version of the stack and late
     * **`branch`** — name of the backport branch to create, following the format `backport-<package_name>-<major>.<minor>`
     * **`base_version`** — the package version to branch from (e.g. `1.19.5`, `1.0.0-beta1`)
     * **`base_commit`** — the commit SHA found in the previous step
-    * **`maintained_until`** — set to a `YYYY-MM-DD` date if the branch has a known end-of-life, otherwise `null`
-    * **`archived`** — set to `false` for a new active branch
+    * **`maintained_until`** — `null` for a new active branch. Set to a `YYYY-MM-DD` date when the branch has a known end-of-life: the branch is automatically excluded from the checklist and branch creation once that date passes (strictly before today in UTC). Prefer this over `archived: true` when the end-of-life date is known in advance.
+    * **`archived`** — `false` for a new active branch. Set to `true` to immediately exclude the branch from the checklist and branch creation, with no fixed end-of-life date. Archiving does **not** delete the branch — packages can still be published from it; archiving only removes it from automated tooling.
+    * **`remove_other_packages`** — required. `true`: when the branch is created, all packages other than the target are removed from `packages/`. `false`: all packages are kept. Set to `true` for the standard case — it keeps the branch lean and avoids running tests for unrelated packages on every PR.
 
     Once the PR is opened, CI automatically:
 
@@ -140,19 +143,124 @@ Sometimes, when we drop the support for an earlier version of the stack and late
 
 3. **Create a PR for the bug fix**
 
-    Create a new branch in your own remote (it is advised **not to use** a branch name starting with `backport-`), and apply bugfixes there. Remember to update the version in the package manifest (update patch version like `1.19.<x+1>`) and add a new changelog entry for this patch version.
+    **Recommended: use `backport_apply.sh`**
 
-    Once ready, open a PR selecting as a base branch the one created above: `backport-<package_name>-<major>.<minor>` (e.g. `backport-aws-1.19`).
+    `backport_apply.sh` handles the entire process: cherry-picking the commit, bumping the patch version, writing the changelog entry, syncing package owners, and opening a PR.
 
-    Once this PR is merged, this new version of the package is going to be published automatically following the usual CI/CD jobs. Wait for the package to appear in the [Elastic Package Registry](https://epr.elastic.co/) before proceeding to the next step.
+    ```bash
+    dev/scripts/backport_apply.sh \
+      --sha <merge_commit_sha> \
+      --package <package_name> \
+      --target <branch_or_version> \
+      --open-pr
+    ```
+
+    Required arguments:
+
+    | Argument | Description |
+    |----------|-------------|
+    | `--sha` | Merge commit SHA on `main` to cherry-pick (minimum 8 characters; from step 1). |
+    | `--package` | Package name as it appears in `manifest.yml`. |
+    | `--target` | Version series (e.g. `6.14`) or full branch name (e.g. `backport-aws-6.14`); the branch name is derived automatically from the version series. |
+
+    Common optional flags:
+
+    | Flag | Description |
+    |------|-------------|
+    | `--open-pr` | Create a GitHub PR after pushing the working branch. |
+    | `--dry-run` | Commit locally and skip push and PR creation; use to review the result before opening a PR. |
+
+    What the script does, in order:
+
+    1. Fetches the backport branch and creates a local working branch (`auto-backport/<pkg>-<version>-<sha8>`).
+    2. Cherry-picks `<sha>`, auto-resolving version-only conflicts in `manifest.yml`; restores `changelog.yml` to HEAD (it is regenerated in the next step).
+    3. Bumps the patch version in `manifest.yml` and inserts a new `changelog.yml` entry (with a placeholder link that is fixed after the PR is opened).
+    4. Syncs package owners from `main` as a separate commit — see [Package owner synchronization](#package-owner-synchronization).
+    5. Pushes the working branch and opens a PR against the backport branch (with `--open-pr`).
+    6. Replaces the placeholder link in `changelog.yml` with the real backport PR URL and pushes a second `Fix changelog link to backport PR` commit.
+
+    If the cherry-pick conflicts on files beyond a version-line difference in `manifest.yml`, the script reports the conflicting files and cleans up. In this case, apply the fix manually (see the alternative path below).
+
+    > **Coming soon:** once the auto-backport workflow lands, checking a branch checkbox in the backport checklist comment will trigger the same process automatically on merge — no manual invocation of `backport_apply.sh` needed. The script remains useful for ad-hoc backports and retries.
 
-    If it is needed to release a new fix for that version, there is no need to create a new branch. Just create a new PR to merge a new branch onto the same backport branch created previously.
+    **Alternative: manual cherry-pick**
+
+    Create a new branch in your own remote (do **not** use a name starting with `backport-`), apply the bug fix, bump the patch version in `manifest.yml`, and add a `changelog.yml` entry. Open a PR targeting the backport branch.
+
+    Once this PR is merged, the new version of the package is published automatically. Wait for it to appear in the [Elastic Package Registry](https://epr.elastic.co/) before proceeding to the next step.
+
+    For subsequent fixes to the same version, no new branch is needed — open a new PR against the same backport branch.
 
 4. **Update changelog in main**
 
-    Once PR has been merged in the corresponding backport branch (e.g. `backport-aws-1.19`) and the package has been published, a new Pull Request should be created manually to update the changelog in the main branch to include the new version published in the backport branch. Make sure to add the changelog entry following the version order.
+    This step is handled automatically. When a backport PR is merged, the `sync-backport-changelog.yml` workflow fires and opens a PR against `main` that adds the new changelog entry for the backport version. The sync PR is created with two labels:
+
+    - `backport:sync-changelog` — identifies it as an automated sync PR.
+    - `changelog-link-check:skip` — skips the changelog link validation (the entry's link points to the backport PR, not the sync PR itself).
+
+    After the workflow runs, a comment is posted on the merged backport PR linking to the sync PR or reporting a failure. No manual action is needed.
+
+## Package owner synchronization
+
+Backport branches are created from historical commits, so their `manifest.yml` owner field and `.github/CODEOWNERS` entries may be stale from the start and can drift further as packages change hands on `main`. Because GitHub resolves PR reviewers from the CODEOWNERS on the PR's **base branch**, a stale backport branch notifies the wrong team.
+
+Two mechanisms keep owners in sync.
+
+### Automatic sync during apply
+
+When `backport_apply.sh` (or `mage applyBackport`) creates a backport PR, it automatically syncs the package's owners from `main` as a separate commit on top of the cherry-pick:
+
+- **What is synced:** the `owner.github` field in `manifest.yml`, the package's own `.github/CODEOWNERS` line, and any sub-path entries nested under the package (data streams, `kibana/` directory, and other subdirectory overrides).
+- **Commit message:** `Sync <package> package owners from main`
+- **No-op:** if the owners already match `main`, the commit is skipped silently.
+- **Warn-and-continue:** if `main` cannot be fetched, or the package no longer exists on `main`, a warning is printed and the apply continues without syncing. The backport PR is still opened; the CI check below surfaces any remaining mismatch.
+
+### CI check: `check-backport-owners`
+
+A Buildkite step runs on every pull request targeting a `backport-*` branch (triggered when `packages/**` or `.github/CODEOWNERS` changes) and posts a comment on the PR with one of three outcomes:
+
+- **✅ In sync** — `Package owners are in sync with main.` No action needed.
+- **Mismatch** — `Package owners are out of sync with main:` followed by a list of packages and the team(s) they should now be owned by. Update `manifest.yml` (`owner.github`) and `.github/CODEOWNERS` for each listed package to match the teams shown.
+- **Check failed** — `The backport owner check failed to run` with a link to the build log. This is usually a transient network error fetching `main`; re-run the build.
+
+The step is currently `soft_fail: true` — a mismatch posts a warning comment but does not block merge.
+
+## Backport checklist comment
+
+When you open or update a pull request targeting `main`, the `post-backport-checklist.yml` workflow automatically posts a comment listing the active backport branches for every package touched by that PR. The comment is updated on every push — any manual edits are overwritten. It only appears when at least one package in the PR's diff has active backport branches in `.backports.yml`.
+
+Example comment:
+
+```
+## Backport branches
+
+> [!IMPORTANT]
+> Only branches for packages touched by this PR's current diff are shown.
+> This comment is updated automatically on each push — manual edits will be overwritten.
+
+Active backport branches for the packages touched by this PR:
+
+**aws**
+- `backport-aws-1.19` (maintained until 2027-06-30)
+- `backport-aws-6.x`
+
+---
+
+> [!TIP]
+> If a branch above is no longer required, set `archived: true` in its entry in `.backports.yml` to stop it appearing here.
+> If the branch has a known end-of-life date, prefer `maintained_until: "YYYY-MM-DD"` — it will be excluded automatically once that date passes.
+```
+
+The comment is currently **informational only** — no checkboxes are rendered and no automation is triggered by it. If you do not intend to backport, you can safely ignore it.
+
+**Suppressing a branch from the checklist:**
+
+To stop a branch appearing in the checklist, update its entry in `.backports.yml`:
+
+- **`archived: true`** — excludes the branch immediately, with no fixed end-of-life date.
+- **`maintained_until: "YYYY-MM-DD"`** — excludes the branch automatically once that date passes; preferred when the end-of-life date is known.
 
-    In order to keep track, this new PR should have a reference (relates) to the backport PR too in its description.
+Archiving a branch does not delete it. Packages can still be published from an archived branch; archiving only removes the branch from the checklist and branch creation.
 
 ## Known issues