Most engineers interact with the Kubernetes scheduler only when pods get stuck in Pending state. They open kubectl describe pod, see SchedulingFailed, adjust CPU or memory requests, and move on. This surface-level engagement with the scheduler leaves significant operational capability unexplored — capabilities that can make the difference between a cluster that is resilient to zone failures and one that is not, between a multi-tenant cluster where teams do not interfere with each other and one where a runaway workload degrades every service, and between a cluster that runs at 65% efficiency and one that runs at 85% efficiency.
The Kubernetes scheduler in 2026 is a sophisticated system with plugins for predicates (can this pod fit on this node?), priorities (which of the fitting nodes is best?), and post-bind hooks (what happens after scheduling?). Beyond the scheduler itself, complementary tools like the Descheduler, the Cluster Autoscaler’s balance-similar-node-groups mode, and Karpenter’s consolidation extend scheduling intelligence to running workloads. This guide covers seven techniques that experienced platform engineers use but rarely document.
HOW THE SCHEDULER WORKS
The Kubernetes scheduler runs in two phases for each unscheduled pod: (1) Filter — eliminate nodes that cannot fit the pod (insufficient resources, taints the pod does not tolerate, node affinity not satisfied, PVC not accessible). (2) Score — rank surviving nodes by priority plugins (spread, image locality, resource balance). The highest-scoring node wins. Most scheduler customisation happens in these two phases — either by adding filter constraints (affinity, topology spread) or by influencing scoring (resource LeastAllocated vs MostAllocated).
#1 Topology Spread Constraints — Zone-Aware Pod Distribution — Guarantee availability across failure domains without manual node selection
What It Does
topologySpreadConstraints distributes pod replicas across topology domains (availability zones, nodes, racks) according to a maxSkew parameter. maxSkew: 1 means no zone can have more than one pod more than any other zone. This guarantees that a zone failure takes down at most 1/N of your replicas — without requiring you to know node names, use pod anti-affinity, or manually distribute replicas.
vs Pod Anti-Affinity
Pod anti-affinity (requiredDuringSchedulingIgnoredDuringExecution with topologyKey=kubernetes.io/hostname) prevents two pods from sharing a node. It does not balance across zones — all pods could end up in zone A on separate nodes. Topology spread constraints provide zone-level and node-level balancing simultaneously.
whenUnsatisfiable Options
DoNotSchedule: the pod remains Pending if no topology-valid node exists (strict, safer for stateful workloads). ScheduleAnyway: the pod schedules even if it violates maxSkew (best-effort, better for availability). For production stateless services, use DoNotSchedule with zone spread and ScheduleAnyway with node spread — this guarantees zone balance while allowing node imbalance to proceed rather than blocking scheduling.
labelSelector
The constraint applies only to pods matching the labelSelector. A Deployment’s topologySpreadConstraints should use matchLabels identical to the Deployment’s pod selector — otherwise existing pods in other Deployments count toward the skew and the constraint behaves unexpectedly.
# Topology spread: zone balance + node spread for HA
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-api
spec:
replicas: 6
template:
spec:
topologySpreadConstraints:
# Hard constraint: spread evenly across zones (max 1 skew)
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels: {app: payments-api}
# minDomains: 3 # K8s 1.28+: require at least 3 zones
# Soft constraint: also spread across nodes within each zone
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway # Best-effort node spread
labelSelector:
matchLabels: {app: payments-api}
# Result: 6 replicas distributed as 2-2-2 across 3 zones
# Zone failure takes down at most 2 replicas (1/3 of capacity)
# Zero manual node selection required
#2 Priority Classes — Control Which Pods Survive Resource Pressure — Define eviction ordering before resource pressure decides for you
What They Are
PriorityClass assigns an integer priority value to pods. When a node runs out of resources, the scheduler evicts lower-priority pods to make room for higher-priority pods. Without explicit PriorityClasses, Kubernetes evicts pods in a less predictable order that may sacrifice production services to keep batch jobs running.
Recommended Hierarchy
system-cluster-critical (built-in, priority 2000001000): kube-system components. system-node-critical (built-in, priority 2000000000): node-level components. platform-critical (1000000): monitoring, service mesh, certificate management. production (500000): production application pods. staging (100000): staging and QA pods. batch (10000): batch jobs, ML training. development (1000): development environment pods.
preemptionPolicy
preemptionPolicy: PreemptLowerPriority (default): high-priority pods can evict lower-priority pods. preemptionPolicy: Never: the pod gets priority in scheduling queue but cannot preempt running pods. Use Never for production pods that should get priority over new scheduling but should not disrupt running workloads by evicting pods mid-flight.
PodDisruptionBudget Interaction
Priority-based eviction respects PodDisruptionBudgets. If evicting a pod would violate its PDB, the scheduler evicts the next-lowest-priority pod instead. This means PDBs + PriorityClasses together correctly protect production services even during node pressure events.
# PriorityClass: production hierarchy
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: production
value: 500000
globalDefault: false
preemptionPolicy: Never # Priority in queue, no mid-flight eviction
description: 'Production workloads — prioritised over staging and batch'
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: batch-jobs
value: 10000
globalDefault: false
preemptionPolicy: Never
description: 'Batch processing — first evicted under node pressure'
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: platform-critical
value: 1000000
preemptionPolicy: PreemptLowerPriority
description: 'Monitoring, mesh, cert-manager — evicts prod if needed'
---
# Apply to a Deployment
spec:
template:
spec:
priorityClassName: production
# Pods with priority 500000 survive node pressure before
# staging (100k) and batch (10k) pods are evicted
#3 The Descheduler — Rebalance Running Pods — Fix placement that was correct at schedule time but is wrong now
What It Is
The Kubernetes Descheduler (SIG scheduling, kubernetes-sigs/descheduler) is a controller that periodically evicts pods that are suboptimally placed based on configurable strategies. It solves a fundamental scheduling limitation: the scheduler places pods optimally at schedule time, but cluster topology changes (new nodes added, nodes removed, pod resource usage changing) make the original placement suboptimal over time.
RemovePodsViolatingTopologySpreadConstraint
The most important strategy: evicts pods that violate topologySpreadConstraints that could not be satisfied at the time the pod was scheduled but can be satisfied now (because new nodes were added). If your 6-replica Deployment landed 4 pods in zone A and 2 in zone B during a zone-A-node-addition event, the descheduler evicts 2 zone-A pods — they reschedule to zone B, achieving the 2–2–2 balance.
LowNodeUtilization
Evicts pods from over-utilised nodes and reschedules them to under-utilised nodes. This works alongside Karpenter consolidation but focuses on redistribution across existing nodes rather than node removal. Useful for long-lived clusters where resource usage has drifted from initial placement.
RemovePodsHavingTooManyRestarts
Evicts pods that have restarted more than a threshold number of times and are in CrashLoopBackOff — their current node may have a resource constraint or configuration issue that is causing the restart. Rescheduling to a different node often resolves the underlying cause without manual intervention.
Deployment
Descheduler runs as a Deployment (periodic) or CronJob. Periodic mode runs the descheduler’s strategies every configuredInterval. For topology rebalancing, run every 5–10 minutes. For resource rebalancing, run every 30–60 minutes to avoid excessive pod churn.
# Descheduler: deploy and configure rebalancing strategies
helm install descheduler \
kubernetes-sigs/descheduler \
--namespace kube-system \
--set schedule='*/10 * * * *'
# Runs every 10 minutes
# descheduler-config.yaml
apiVersion: descheduler/v1alpha2
kind: DeschedulerPolicy
profiles:
- name: default
plugins:
balance:
enabled:
# Core strategy: fix topology spread violations
- name: RemovePodsViolatingTopologySpreadConstraint
# Evict pods from over-utilised nodes
- name: LowNodeUtilization
pluginConfig:
- name: LowNodeUtilization
args:
thresholds:
cpu: 20 # Node is under-utilised if CPU < 20%
memory: 20
pods: 20
targetThresholds:
cpu: 50 # Evict from nodes above 50% CPU
memory: 50
pods: 50
numberOfNodes: 0 # 0 = apply to all nodes
- name: RemovePodsViolatingTopologySpreadConstraint
args:
constraints:
- DoNotSchedule # Only fix hard constraints
# ScheduleAnyway violations are best-effort, skip them
#4 Node Affinity vs Node Selector — The Right Tool for Each Job — Two mechanisms for node selection with different trade-offs
nodeSelector
The simplest node selection mechanism: a key-value map of labels that the node must have. nodeSelector: {node-type: gpu} schedules the pod only on nodes labelled node-type=gpu. Simple, readable, but binary — a node either matches or it does not. No preference mechanism, no soft constraints.
Node Affinity
Node affinity extends nodeSelector with: requiredDuringSchedulingIgnoredDuringExecution (hard constraint, pod does not schedule if unsatisfied), preferredDuringSchedulingIgnoredDuringExecution (soft preference with a weight, pod still schedules if preference cannot be met), and more expressive operators (In, NotIn, Exists, DoesNotExist, Gt, Lt vs just equality).
The IgnoredDuringExecution Caveat
Both hard and soft node affinity are only evaluated at schedule time — if a node’s labels change after a pod is scheduled there, the pod continues running. requiredDuringSchedulingRequiredDuringExecution (planned but not yet implemented in stable) would evict pods when node labels change. Account for this by using node labels that are stable and not changed post-provisioning.
Practical Pattern
Use nodeSelector for simple, stable constraints (GPU nodes, ARM64 nodes). Use node affinity preferredDuringScheduling for soft preferences (prefer nodes with local SSDs, prefer the same zone as a dependency service). Use required affinity only when the pod genuinely cannot run elsewhere.
#5 Pod Anti-Affinity for Stateful Services — Spread StatefulSet replicas without topology spread constraints
When to Use
StatefulSets for databases (Cassandra, Kafka, Elasticsearch) that manage their own replication should use pod anti-affinity rather than topology spread constraints. Each replica of these databases should be on a separate node and ideally a separate zone — anti-affinity enforces this with the correct granularity.
requiredDuringScheduling vs preferredDuringScheduling
requiredDuringScheduling: the pod will not schedule if anti-affinity cannot be satisfied (all nodes already have a matching pod). Use for databases where co-location genuinely degrades reliability. preferredDuringScheduling: the scheduler tries to satisfy anti-affinity but will co-locate pods if necessary. Use when anti-affinity is a preference, not an absolute requirement.
Performance Consideration
Pod anti-affinity has O(N*P) complexity where N is node count and P is pod count. In clusters with thousands of pods, widespread use of pod anti-affinity significantly increases scheduler latency. Reserve hard anti-affinity for stateful services with genuine co-location constraints; use topology spread constraints for stateless service spreading.
#6 Taints and Tolerations — Reserve Nodes for Specific Workloads — Dedicate nodes to specific workload types without label-only node selection
Taints
Taints are applied to nodes: kubectl taint nodes gpu-node-1 workload=gpu:NoSchedule. NoSchedule: pods without the matching toleration will not be scheduled on this node. PreferNoSchedule: the scheduler prefers not to schedule here but will if no other options exist. NoExecute: existing pods without the toleration are evicted.
Tolerations
Tolerations are added to pod specs to allow scheduling on tainted nodes: tolerations: [{key: workload, value: gpu, effect: NoSchedule}]. A toleration allows a pod to schedule on a tainted node but does not require it — combine with node affinity to both tolerate and prefer a specific node type.
Practical Pattern
GPU nodes: taint with workload=gpu:NoSchedule. Only GPU workloads (with the toleration) can schedule there; CPU-only pods never consume GPU capacity. System nodes: taint with CriticalAddonsOnly=true:NoSchedule. Only platform components (kube-proxy, CoreDNS, CNI) with the built-in toleration run on these nodes.
Karpenter NodePools
Karpenter NodePools apply taints automatically when provisioning nodes: the NodePool spec includes taints that are applied to all nodes in that pool. Combined with workload-specific tolerations, this creates dedicated pools without manual taint management.
#7 Bin Packing vs Spread — The Resource Efficiency Trade-off — Control whether the scheduler optimises for density or resilience
The Default Scheduler
By default, the Kubernetes scheduler uses LeastAllocated scoring: it prefers nodes with the most available resources, spreading pods across many nodes. This maximises resilience (no node is heavily loaded) at the cost of efficiency (many lightly loaded nodes cannot be consolidated by Karpenter).
MostAllocated Scoring
MostAllocated: prefer nodes that are already more utilised, filling them up before using new nodes. This is the bin-packing strategy — more compute-efficient but less resilient to node failures (a failed node takes down more pods). Enable via the KubeSchedulerConfiguration API.
Karpenter Consolidation as the Right Abstraction
Rather than configuring the scheduler for bin packing (which affects resilience), use Karpenter consolidation instead. Karpenter consolidates after the fact: LeastAllocated scheduler places pods for resilience, Karpenter then evicts pods from underutilised nodes and reschedules them on the remaining nodes, then terminates the empty node. This achieves the efficiency of bin packing without compromising the initial scheduling resilience.
When to Use MostAllocated
GPU clusters where each GPU node is expensive and must be fully utilised. Batch processing environments where node failure is acceptable and cost efficiency is paramount. Not recommended for production microservice clusters.
START WITH TOPOLOGY SPREAD AND PRIORITY CLASSES
Of the seven techniques in this guide, topology spread constraints and priority classes deliver the most operational value with the lowest risk. Add topologySpreadConstraints to every production Deployment today — it takes 10 lines of YAML and immediately improves your cluster’s resilience to zone failures. Add PriorityClasses to every workload type to prevent batch jobs from evicting production services during node pressure events. The descheduler and advanced affinity patterns are powerful but require more careful tuning before production deployment.
