rancher-upgrade

Plan, sequence, and de-risk a community-edition Rancher upgrade across a real fleet: a management ("hosting") cluster and the downstream RKE2/K3s clusters it provisions. The hard part of a Rancher upgrade is almost never the helm upgrade itself — it's the cross-cluster coordination: a management-cluster Rancher bump silently changes which downstream Kubernetes versions the fleet can run, and getting the order wrong strands sub-clusters or forces a rebuild.

Community editions only. SUSE Rancher Prime backports, Prime-only patches, and Prime support matrices are out of scope — flag them when they appear, never build a plan on them. House Rule #1.

Companion to k8s-components-checker — who owns what

This skill is the upgrade-methodology companion to k8s-components-checker (same repo), mirroring the argo-cd-apps ↔ compat/argo-cd.md split. The boundary is load-bearing — respect it to avoid drift:

The mental model — two coupled axes

A Rancher fleet upgrade moves on two axes that are decoupled but ordered:

1. Management-cluster axis — the k8s minor the Rancher server itself runs on, and the Rancher

version. (k8s window → cite compat/rancher.md.)

1. Downstream axis — which RKE2/K3s k8s minors Rancher can provision and manage, governed by

Kontainer Driver Metadata (KDM), NOT by the management cluster's own k8s version.

The rule that ties them: *introducing a downstream k8s minor requires upgrading Rancher first; downstream patch upgrades do not.* So the management Rancher always moves before any downstream minor bump — and a trailing downstream cluster can fall out of support when the host Rancher rolls forward. This is the single most common way a fleet upgrade goes wrong. See references/kdm-downstream-matrix.md.

Workflow

1. Establish the change set and the fleet shape

• Install type first — this skill assumes the Helm-on-Kubernetes install. The same command that

gets the version answers this: an empty result means there is no Helm-managed rancher release, and the likeliest reason is a single-node Docker install (docker run rancher/rancher, still a documented method). That upgrades by stopping the container, backing up its data volume, and starting a new container from the newer image against the same volume — no `helm upgrade`, no cert-manager chart, no BRO, no etcd snapshot, no KDM coordination. Effectively nothing below applies. Say so and stop rather than emitting a Helm plan; point at the Rancher docs' single-node-with-Docker upgrade page.

• Versions: current Rancher minor+patch, target Rancher minor. Get this from the cluster

(helm list -A -n cattle-system | grep rancher), not from memory.

• Fleet shape: does this Rancher actually provision downstream clusters?

kubectl get clusters.cluster.x-k8s.io -A (populated = real fleet; empty = standalone mgmt cluster — the CAPI/Turtles migration is then a near-non-event). kubectl get clusters.provisioning.cattle.io -A — any row whose name ≠ local is a downstream cluster.

• Air-gap? If the registry is internal-mirror-only, the air-gap procedure (what to mirror)

applies — references/air-gap-procedure.md. (Direct-pull clusters skip mirroring entirely.)

• Kubeconfig path: if the operator's kubeconfig is Rancher-proxied (server contains /k8s/clusters/),

switch to a Rancher-independent admin kubeconfig BEFORE upgrading — helm upgrade of the rancher release restarts the proxy pods and severs the API mid-apply. references/prereqs-and-ordering.md § Pre-flight.

2. Compute the upgrade path (no minor skipping)

The only supported path between minors is latest-COMMUNITY-patch-of-current-minor → latest- COMMUNITY-patch-of-next-minor, one minor at a time (2.11→2.12→2.13→2.14→2.15; never 2.11→2.14). Intra-minor patch jumps are fine.

Operator insists on a skip ("one window", "I accept the risk"): never write the skipping helm upgrade, even when told not to lecture. A skip fails mechanically: the management-cluster k8s windows of minors three apart share no k8s minor (2.12 runs on 1.31–1.33, 2.15 on 1.34–1.36), and downstreams can only be lifted to minors the current Rancher's KDM offers, so they strand. Give the fewer-windows version instead: every hop chained in one window with a gate between hops (rollout complete, clusters Active, fresh etcd snapshot), and each required mgmt/downstream k8s bump placed before the hop that needs it.

⛔ The rung is the community ceiling, not the newest tag. Rancher ships both editions to one GitHub feed, so for every minor except the current one the newest tag is Prime-only and uninstallable here — sort -V | tail -1 silently yields a target the operator cannot use. Derive each rung edition-aware (House Rule #1 + #3): references/lifecycle.md § Latest patch per minor.

3. For each minor step, run the pre-flight → upgrade → post-flight runbook

The per-minor breaking changes and the ordered runbook live in references/per-minor-runbook.md. Prerequisites common to every step (cert-manager window, Helm floor, mandatory backup, RKE1 sweep, API aggregation layer) and the cross-cluster ordering + rollback floor are in references/prereqs-and-ordering.md.

4. Coordinate the downstream clusters

Before bumping the host Rancher, check every downstream cluster's k8s minor against the target Rancher's KDM window (references/kdm-downstream-matrix.md). Lift any trailing downstream into the new window first, or it loses manageability after the host upgrade. After the host upgrade (which ships a new KDM branch), the newly-unlocked downstream minors become selectable — bump downstreams then.

5. Produce the plan

Emit an ordered plan: per-minor-step pre-flight gates, the upgrade command (air-gap variant if applicable), post-flight verification, and the downstream-coordination steps interleaved at the right points. Cite the reference + grounded version for every specific claim. Recognizable shape:

<cluster/fleet> — Rancher <current> → <target>   (path: <minor steps, no skips>)

prerequisites  - cert-manager <window> · Helm <floor> · RKE1 sweep · BRO backup + etcd snapshot
downstream     - <cluster>: k8s <ver> — <in target's KDM window | lift to ≥X BEFORE host upgrade>
                 source: references/kdm-downstream-matrix.md
per step (×N)  1. pre-flight gates  2. upgrade mgmt Rancher  3. post-flight  4. bump downstream
blockers       - <one-way rollback / known regression / Prime-gated path>  source: references/<file>.md

Every cited version is cluster-reported or freshly gh-grounded (House Rule #3), never from memory; target versions follow look-ahead (House Rule #4).

House rules (encode into every plan)

1. Community editions only. Flag Prime-gated versions/patches/features; never plan on them.

The reliable community-vs-Prime signal is the GitHub release-notes first line — see references/lifecycle.md.

1. Cite, don't restate, the mgmt-cluster k8s window. That lives in

k8s-components-checker/references/compat/rancher.md. Pointing at it keeps the two skills from drifting.

1. Never invent versions; ground or abstain (House Rule #8 in k8s-components-checker). k8s windows and KDM

mechanics are methodology the skill states. Specific release/patch numbers — "latest 2.13 patch", "Turtles version on 2.14", "BRO chart for 2.12" — are volatile and the #1 fabrication risk. State a specific version only if it is cluster-reported, freshly grounded via gh, or explicitly marked UNVERIFIED. Grounding method (anti-confirmation): anchor on gh api repos/rancher/rancher/releases/latest, enumerate-and-derive the real latest patch, never ask "does vX.Y.Z exist?" (existence/list queries get rubber-stamped). gh must be run with valid auth from the operator's workstation — anonymous is 60 requests/hour and exhausts almost immediately on an enumeration sweep (gh auth status first). Repo map + protocol: references/lifecycle.md § Grounding.

1. Look-ahead version targeting. When recommending any target version, pick the one that covers

the operator's next planned hop too, not the bare immediate minimum — a version sitting at its own support ceiling forces an avoidable second upgrade. (Same rule as k8s-components-checker House Rule #9.)

And ground the patch floor on advisories, not just on "latest". Rancher ships security advisories in batches, and they set a minimum patch per minor that the latest-patch lookup does not tell you. Enumerate them the same way you enumerate releases:

   gh api repos/rancher/rancher/security-advisories --paginate \
     --jq '.[] | "\(.severity) \(.published_at[0:10]) \(.ghsa_id) \(.summary)"'
   # then, for one advisory, the per-minor patched floors:
   gh api repos/rancher/rancher/security-advisories --paginate \
     --jq '.[] | select(.ghsa_id=="GHSA-...") | .vulnerabilities[]
           | "\(.vulnerable_version_range) -> \(.first_patched_version)"'

One advisory sets several floors at once: CVE-2026-44945 (CRITICAL, cross-cluster impersonation) lists < 2.14.2, < 2.13.8, < 2.12.12, < 2.11.16. A ladder targeting "a recent 2.11" lands inside it. `first_patched_version` is almost always null on this feed — derive the floor from vulnerable_version_range; "no patched version listed" does not mean "no fix".

On community edition most of those floors are unreachable, and that decides the plan. Rancher publishes community patches for a minor only while it is the newest minor; the day minor N+1 GAs, minor N's next patch becomes a Prime-docs redirect. Verified across every 2.11–2.15 release (2026-09-15): v2.11.4 turned Prime the day 2.12.0 shipped, v2.12.4 the day before 2.13.0, v2.13.4 and v2.14.4 on their successors' GA dates. So the community ceiling is frozen at the last patch before the next GA, while advisory floors keep climbing:

Consequence: every community minor except the newest is inside an unpatched CRITICAL. 2.13, 2.12 and 2.11 all sit below CVE-2026-44939's floors (command injection via unsanitized YAML, < 2.13.6 / < 2.12.10 / < 2.11.14), and 2.13/2.12/2.11 below CVE-2026-44945's as well. There is no patch-in-place remediation for any of them. The current top floor on every minor is CVE-2026-88804 (CRITICAL, unauthenticated UI-settings write → stored XSS); its ranges carry a v prefix (>=v2.14.0, <v2.14.6), so a < 2. regex misses them.

Fleet is a second feed with the same shape. rancher/fleet advisories (CVE-2026-88808, CRITICAL, agent writes with cluster-admin) floor at Fleet 0.16.2 / 0.15.7 / 0.14.11; Rancher pins the Fleet it installs (fleetVersion in the tag's build.yaml), so community 2.14.3 runs Fleet 0.15.4 and only 2.15.2 (Fleet 0.16.2) clears it. Sweep both feeds.

So do not answer a community security question with a patch number. The remediation is the minor hop — which makes House Rule #5's ladder the security path, not merely the supported one, and makes the look-ahead rule above mandatory rather than advisory: land on the newest minor or you land somewhere unpatchable. Re-derive both columns at plan time; the ceilings move only on a GA, but the floors move with every batch.

1. Management Rancher upgrades BEFORE downstream k8s minor bumps, always. And **no minor

skipping** on the Rancher axis. Violating either is the classic fleet-stranding / rebuild path.

1. Back up before every step. An RKE2 etcd snapshot of the management cluster is the real

rollback floor — the 2.14 CAPI-v1beta2 boundary is one-way without it. Add a backup-restore-operator backup if BRO is installed; many community installs have none, so don't block on it (substitute targeted resource exports — Rancher/Fleet objects + the auth AuthConfig & secrets). See references/prereqs-and-ordering.md § Backup & rollback.

Decision guide

Each reference carries a header stating what was grounded and when — and what was NOT re-derived on the latest pass. Read that header before citing anything from the file: a claim under a "not re-derived" disclaimer must be re-grounded at use time regardless of the file's headline date. Otherwise re-ground per House Rule #3 only if the stamp is stale (≳1–2 weeks) — same-day re-grounding is wasted churn the operator will (rightly) push back on; don't present it as an unconditional step.