Karpenter 1.15 node repair: replace-first, until the pool is full

Karpenter got node repair in v1.1.0 on 30 November 2024. For almost two years the feature was simple: a node reported an unhealthy condition for long enough and Karpenter deleted it. Disruption budgets did not apply, PodDisruptionBudgets were skipped, nothing was launched in advance, and the drain grace period was clamped to zero. Issue #2042, opened in February 2025, asked for repair to honour PDBs and was finally closed on 25 September 2026.

Karpenter v1.15.0 shipped on 10 October 2026 and rewrites repair as a voluntary disruption, in the same category as consolidation and drift. For most fleets that is an improvement. For fleets that cannot grow, it forces a choice the old controller never offered, and the upgrade also changes the meaning of budgets you already have.

What actually changed

The design is set out in two RFCs in the Karpenter repository, making-node-repair-voluntary.md and terminate-first-disruption.md. What follows comes from the v1.15.0 code and the AWS provider’s v1.15.0 defaults, because in two places the RFCs and the code disagree.

Everything is still behind alpha feature gates that are off by default. In the AWS chart this is how you turn it on:

# values.yaml, karpenter chart v1.15.0
settings:
  featureGates:
    nodeRepair: true             # ALPHA, default false
    terminateFirstRepair: false  # ALPHA, requires nodeRepair
    terminateFirstDrift: false   # ALPHA, same idea for drift
  awsFeatureGates:
    rebootForRepair: false       # ALPHA, needs ec2:RebootInstances
  legacyNodeRepair: false        # true = the old controller; incompatible with terminateFirstRepair

The AWS provider’s default repair policies set the timing. Ready=False and Ready=Unknown wait 30 minutes and drain for at most 10. The same goes for the Node Monitoring Agent conditions (StorageReady, NetworkingReady, KernelReady, ContainerRuntimeReady). AcceleratedHardwareReady is split by XID code: known fatal and reboot-clearable XIDs wait 10 minutes and drain for 5, while unrecognised GPU faults fall back to 30 and 10. The source comment says why a bound exists at all: without one, a nil NodeClaim terminationGracePeriod would let a drain hang for roughly 19 days.

The upgrade trap: budgets without reasons

A NodePool budget with no reasons applies to every disruption method. In v1.15 that list includes Unhealthy. The check in GetAllowedDisruptionsByReason is a single line: a budget counts if budget.Reasons == nil or the list contains the reason being evaluated.

Plenty of NodePools carry a freeze like the one below, written to stop consolidation from shuffling pods during business hours. Under the legacy controller it had no effect on repair. Under v1.15, a node that goes NotReady at 10:00 waits for the window to close at 18:00 before anything replaces it.

# Before: freezes consolidation, drift AND repair in v1.15
disruption:
  consolidationPolicy: WhenEmptyOrUnderutilized
  budgets:
    - nodes: "0"
      schedule: "0 8 * * mon-fri"
      duration: 10h
    - nodes: "10%"

# After: the freeze names its reasons, repair gets its own pace
disruption:
  consolidationPolicy: WhenEmptyOrUnderutilized
  budgets:
    - nodes: "0"
      reasons: [Underutilized, Empty, Drifted]
      schedule: "0 8 * * mon-fri"
      duration: 10h
    - nodes: "10%"
      reasons: [Underutilized, Empty, Drifted]
    - nodes: "1"
      reasons: [Unhealthy]

Percentages round up, the same way maxUnavailable does on a PDB, so 10% on a five-node pool still allows one. An explicit "1" for Unhealthy is easier to reason about during an incident than a percentage of a pool whose size is changing under you.

The trade-off: a pool with no room to replace into

Voluntary disruption replaces first: launch the new node, wait for it, drain the old one. That needs headroom, and some fleets have none by design. One is a NodePool pinned to an EC2 capacity reservation sized to exactly N GPU instances with no on-demand fallback. The other is a static NodePool (spec.replicas) at its limits.nodes. In both cases the replacement would be instance N+1, which does not exist.

With terminateFirstRepair off, which is the default, the scheduling simulation cannot place the pods, repair publishes a DisruptionBlocked event and moves on. The broken node keeps its reserved slot. On a GPU reservation you are paying for an accelerator that runs nothing, and the node stays that way until a person replaces it.

With it on, Karpenter issues a delete-only command. It cordons and drains the node, bounded by the policy’s grace period and honouring PDBs until then, terminates it, and lets normal provisioning launch the replacement into the freed slot. The RFC is honest about the cost. With no replacement serving yet, drain time is downtime. On a shared reservation, another consumer can claim the slot first, so the relaunch hits insufficient capacity and the pool stays a node short until a slot frees up.

Two details decide how much this matters to you. First, terminate-first fires only when capacity is structurally full: no-reserved-capacity or static-at-limit. A Spot or on-demand launch that fails with insufficient capacity does not switch a pool to terminate-first. It keeps replacing first under the normal launch backoff, so a pool narrowed to one Spot family is not covered by this flag. Second, the gate is controller-wide. The RFC planned it as on by default in beta, but in v1.15.0 it is alpha and off, and there is no per-NodePool switch. Per pool, the levers you have are the Unhealthy budget, which paces how many swaps run at once, and the veto annotation.

# Hold repair on one node while you investigate (blocks all repair on it)
kubectl annotate node "$NODE" karpenter.sh/do-not-repair=true

# karpenter.sh/do-not-disrupt does NOT stop repair. Repair admits nodes with
# blocking pods on the promise of its drain bound; the coupling is still open
# upstream (kubernetes-sigs/karpenter#2424).

# See why a repair did not happen
kubectl get events -A --field-selector reason=DisruptionBlocked
kubectl get events -A --field-selector reason=NodeRepairBlocked

The breaker that is still there

The voluntary-repair RFC says the budget “replaces the binary 20% breaker outright”. The v1.15.0 code keeps it. repairUnhealthyThreshold = "20%" is still a constant in repair.go, now evaluated per NodePool, and while it is tripped no repair runs in that pool. The event it emits is NodeRepairBlocked, not DisruptionBlocked.

The threshold rounds up and trips when the unhealthy count exceeds it. In a four-node GPU pool, 20% rounds up to one node, so a single broken node repairs and a second one stops repair for the whole pool. That is the right call for a correlated failure such as a bad AMI. It also means small pools hit the breaker after one extra fault, and the budget you tuned has no say in it.

What we would alert on

Repair that does nothing is silent by default. These are the series we would add next to the existing Karpenter dashboards. Reason labels are lower case in the metrics.

# Repair is waiting on budget: unhealthy nodes are eligible, budget is zero
min by (nodepool) (karpenter_nodepools_allowed_disruptions{reason="unhealthy"}) == 0
  and on() sum(karpenter_voluntary_disruption_eligible_nodes{reason="unhealthy"}) > 0

# Terminate-first is happening, by pool and by why the replacement could not go first
sum by (nodepool, terminate_first_reason) (
  increase(karpenter_voluntary_disruption_terminate_first_decisions_total{reason="unhealthy"}[1h])
) > 0

# What repair actually did, by condition and drain mode (graceful | eventual | forceful)
sum by (nodepool, condition, termination_mode) (
  increase(karpenter_nodeclaims_unhealthy_disrupted_total[1h])
)

# The capacity gap terminate-first leaves behind
# (the RFC calls this karpenter_pods_unschedulable_count; v1.15.0 exports it under this name)
sum(karpenter_scheduler_unschedulable_pods_count) > 0

Pair the first alert with the business-hours freeze above. If it fires every weekday at 08:00 and clears at 18:00, a reasonless budget is holding repair back.

Reboot before replace, for GPUs

The release also adds reboot as a node action. In the AWS provider it applies to one policy only: AcceleratedHardwareReady=False with a reboot-clearable XID (46, 48, 54, 62, 63, 95, 109, 110, 136, 140, 143, 155, 156, 158). With awsFeatureGates.rebootForRepair on and ec2:RebootInstances granted to the controller role, Karpenter reboots the instance in place instead of replacing it. On a reservation that is the better outcome, because the instance and its slot never leave. Fatal XIDs (64, 74, 79, 119, 120) are always replaced.

Honest trade-offs

• All of it is alpha. NodeRepair, TerminateFirstRepair and RebootForRepair are alpha gates, so enable them on a non-production cluster first and read the events for a week.
• If you depend on the old controller’s speed, you can keep it. settings.legacyNodeRepair: true brings back delete-first repair, without terminate-first, reboot or do-not-repair. The flag is marked for deprecation, so treat it as a dated exception with an owner.
• Leave terminate-first off where one node is the service. If a reserved pool runs a single-replica inference endpoint, terminate-first means an outage for the length of the drain plus the relaunch. Keeping the broken node and paging a human is the more honest choice there.
• Budgets cut both ways. nodes: "1" for Unhealthy limits the damage when a detector misfires, and it also serialises repair during a real zonal event.

How we would roll it out

Karpenter is the node layer on the EKS estates we build, defined in Terraform and Terragrunt with the chart values in the same repository. Our order for this release: add explicit reasons to every existing budget and a separate Unhealthy budget per NodePool, ship the alerts, then enable nodeRepair on a non-production cluster. Terminate-first stays off until each capacity-constrained pool has a written decision about which outcome it prefers. That last step is a conversation more than a config change. On a reserved GPU pool, the person who owns the reservation and the person who owns the service are rarely the same, and the flag affects both.

Naviteq. DevOps, FinOps and AI-driven cloud automation, delivered at scale.