Nodes can finally say they are leaving on purpose

A node disappearing from an autoscaled fleet looks identical to a node that died. Kubernetes has never had a place for the node to say which one it was, and every tool that needed to know invented its own way of guessing. Version 1.37 finally gives that fact a name.

Four dialects for one fact

On a cluster where the node layer is managed by anything more active than a fixed node group, nodes leave constantly and on purpose. Consolidation replaces one. An upgrade drains another. A spot instance gets reclaimed with two minutes of warning. The cloud provider schedules host maintenance. All four are deliberate, and none of them looks any different from a kubelet that stopped reporting.

The signals exist. They are just not the same signal, and none of them lives where a workload controller would look.

The KEP behind this change says the quiet part out loud: lifecycle state is inferred today from “a mix of Node readiness, taints, Pod state, controller status, labels, annotations, and provider-specific APIs”. Each works locally, none produces reusable data, and so every consumer guesses separately. The DaemonSet controller reschedules a pod that was never missing, and your alerting pages someone because the autoscaler did its job.

What 1.37 adds

KEP-5683, owned by SIG Node with SIG Apps participating, adds five well-known node condition types. They follow the same model Kubernetes already uses for Ready, MemoryPressure and DiskPressure: publish the signal first, teach core controllers to react to it later.

// k8s.io/api/core/v1: new NodeConditionType constants in 1.37

// GracefulNodeShutdownInProgress reports whether Graceful Node Shutdown
// is determined to be in progress on this Node.
GracefulNodeShutdownInProgress NodeConditionType = "GracefulNodeShutdownInProgress"

// DrainInProgress reports that this Node is actively being drained.
DrainInProgress NodeConditionType = "DrainInProgress"

// Drained reports that this Node has reached the drain criteria
// selected by the actor managing the lifecycle.
Drained NodeConditionType = "Drained"

// MaintenancePlanned reports that this Node is expected to undergo maintenance.
MaintenancePlanned NodeConditionType = "MaintenancePlanned"

// MaintenanceInProgress reports that this Node is actively undergoing maintenance.
MaintenanceInProgress NodeConditionType = "MaintenanceInProgress"

Each carries the usual True / False / Unknown status, plus a reason that is a machine-readable cause category. The KEP names four to start with: AdminRequested, DrainCompleted, MaintenanceWindow and NodeShutdown. On the node it reads the way every other condition does.

# kubectl get node ip-10-0-3-91 -o jsonpath='{.status.conditions}' | jq
{
  "type": "MaintenancePlanned",
  "status": "True",
  "reason": "MaintenanceWindow",
  "message": "host maintenance window opens 02:00 UTC",
  "lastTransitionTime": "2026-09-01T21:40:11Z"
}

Nothing sets that for you. In this first release the conditions are admin managed: an administrator, or a maintenance controller the administrator has authorised, writes them through the status subresource. Clearing the state means setting the status to False or removing the condition entirely.

# A maintenance controller declaring intent before it touches anything
kubectl patch node ip-10-0-3-91 --subresource=status --type=merge -p '{
  "status": {
    "conditions": [{
      "type": "MaintenancePlanned",
      "status": "True",
      "reason": "MaintenanceWindow",
      "message": "kernel patch, batch 3 of 7",
      "lastTransitionTime": "2026-09-01T21:40:11Z"
    }]
  }
}'

Clearing it is the half people forget, and the writer owns it. A condition left at True after the node returns to service is worse than none.

kubectl patch node ip-10-0-3-91 --subresource=status --type=merge -p '{
  "status": {
    "conditions": [{
      "type": "MaintenancePlanned",
      "status": "False",
      "reason": "AdminRequested",
      "lastTransitionTime": "2026-09-02T04:05:00Z"
    }]
  }
}'

The feature is alpha in 1.37, behind a gate that has to be enabled on two control plane components.

# kube-apiserver and kube-controller-manager both need it
--feature-gates=NodeLifecycleConditions=true

Delegating the write to a controller means granting the node status subresource, which is a permission worth being deliberate about: whoever holds it can also make a healthy node look drained.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-lifecycle-writer
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["nodes/status"]
    verbs: ["patch", "update"]

What you can build on it today

Because no core controller consumes these conditions yet, what you get in this release is observability and glue code. That sounds like a consolation prize and mostly is not: fleets already automate the disruption fine, they just cannot tell planned churn from failure once it reaches a graph or a pager.

kube-state-metrics already exports arbitrary node conditions, so the conditions become alertable the moment something writes them. Two rules earn their place immediately: one that catches a drain which has stopped making progress, and one that stops the generic node-down alert from firing on deliberate disruption.

groups:
- name: node-lifecycle
  rules:
  # A drain that has not finished inside its budget is a real problem.
  - alert: NodeDrainStalled
    expr: |
      kube_node_status_condition{condition="DrainInProgress",status="true"} == 1
    for: 20m
    labels:
      severity: warning
    annotations:
      summary: "Node {{ $labels.node }} has been draining for 20 minutes"

  # A node that is gone on purpose should not page anyone.
  - alert: NodeNotReady
    expr: |
      kube_node_status_condition{condition="Ready",status="true"} == 0
      unless on (node)
      kube_node_status_condition{condition=~"DrainInProgress|MaintenanceInProgress|GracefulNodeShutdownInProgress",status="true"} == 1
    for: 5m
    labels:
      severity: critical

The second rule is the one that changes an on-call rotation, and you can write it today against a vendor signal instead. Most teams already have, each in their own dialect. What the standard buys is that the same rule survives the next cluster without being rewritten for whatever manages its nodes.

The part that costs more than alert noise

Noisy pages are the visible cost. The expensive one is that PodDisruptionBudgets are blind in exactly the same way. A budget does not know why an eviction is being attempted, only that it is. A minAvailable: 2 on three replicas blocks the third eviction whether the node underneath is being consolidated on a quiet Tuesday or is halfway through a hardware failure.

Karpenter honours that budget, which is correct and also means consolidation stalls on a half-drained node nobody is watching. The documented way out is terminationGracePeriod on the NodePool, and it is blunter than most teams realise: with it set, a node can be disrupted on drift even when pods have blocking PDBs or the karpenter.sh/do-not-disrupt annotation.

apiVersion: karpenter.sh/v1
kind: NodePool
spec:
  # Past this deadline, blocking PDBs and do-not-disrupt stop being respected.
  terminationGracePeriod: 6h
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 15m
    budgets:
      - nodes: "20%"
      - nodes: "0"           # no voluntary churn during business hours
        schedule: "0 8 * * mon-fri"
        duration: 10h
        reasons: ["Underutilized", "Drifted"]

So the real choice on a churning fleet today is consolidation that never finishes, or disruption budgets that are advisory past a deadline. Neither is wrong. Both exist because the eviction path has no way to ask whether this particular departure was planned, and a budget that could tell the difference would not need the deadline in the first place.

Where we hit this

We run this shape for one of our client, a SaaS marketing platform we moved off Heroku onto AWS: EKS with Karpenter scaling the nodes, a second tenant in another region, and PagerDuty on the far end of the alerting stack we built with it. Karpenter earns its keep by treating nodes as disposable, so the fleet churns by design.

The instinct on a fleet like that is to call the nodes flaky. The actual problem is that intent has to be reconstructed from vendor-specific side effects. We made it explicit instead: the disruption taint gates the alerting, NodePool budgets confine voluntary churn to a schedule, and do-not-disrupt goes only on pods that genuinely cannot move. The conditions in 1.37 are the standard version of the same idea.

Three limits worth knowing first

You probably cannot turn it on. The gate has to be set on kube-apiserver and kube-controller-manager, and a managed control plane does not take flags, so on EKS, GKE and AKS this does not exist in 1.37. Beta is targeted for 1.38 and stable for 1.39. For most readers that makes it a design to plan against rather than a switch to flip; self-managed and on-prem clusters can enable it today.

No core controller reads them. The DaemonSet controller still reschedules, the Job controller still waits, the autoscaler still picks its own scale-in target. Consuming the conditions is explicitly scheduled as separate follow-up work, one controller at a time. Anyone expecting MaintenancePlanned=True to protect a workload today will be disappointed.

Any authorised actor can write them, and the KEP knows it. There is deliberately no ownership, no locking and no handoff protocol. Drained means the drain criteria chosen by whoever manages the lifecycle, not a fixed definition, and the KEP does not define what happens to the node afterwards. Three actors touching node lifecycle means three possible meanings for one condition. The KEP lists this under risks and accepts it for the first release. Adopt early and you should write down what each condition means in your organisation before anything depends on it.

Four open bugs this does not close

The KEP is honest about being a foundation rather than a fix, and it names the bugs it is a foundation for. Each is a controller guessing at lifecycle state and getting it wrong. Publishing a condition resolves none of them yet.

• kubernetes#122912 (open since January 2024): the DaemonSet controller and the Graceful Node Shutdown manager disagree about workload placement, so a DaemonSet pod the kubelet stopped on purpose is counted as unavailable.
• kubernetes#139226 (open): DaemonSet status cannot say why rollout pods are unavailable, so a rollout verifier cannot separate a bad image from a node under maintenance.
• kubernetes#138719 (open): the ReplicaSet controller ignores node lifecycle state when picking pods to delete during scale-down, so it can remove a pod on a healthy node while a pod on a draining node goes too. You lose two instead of one.
• kubernetes#134038 (open): pods stuck terminating have no timeout, which is how a Job with podReplacementPolicy: Failed stalls behind a drain.

The oldest thread here is older than all of them. Issue 25625 asked Kubernetes to own drain server-side rather than leaving it as a client-side loop in kubectl. It was opened in May 2016, collected seventy-three comments, and was closed carrying a lifecycle/rotten label. KEP-5683 cites it. Ten years is a long time for a fleet to guess.

The practical read

Treat 1.37 as the moment to standardise the glue code you already have. If your alerting already separates planned disruption from failure through a taint or an annotation, that logic is correct and now has a target to migrate onto. If it does not separate them at all, the fix needs neither 1.37 nor alpha gates. It needs you to decide which node departures are deliberate, and to say so somewhere your alerting can read.