Skip to content

Advanced Kubernetes Scheduling & GPU Resource Management Learning Plan

Author: Miklós Géczi
Created: 2026-06-03
Estimated Duration: 8-12 weeks (5-10 hours/week)
Prerequisites: Basic Kubernetes knowledge (pods, deployments, services), familiarity with container orchestration, basic understanding of GPU architecture
Target Audience: Platform engineers, SREs, ML infrastructure engineers working with GPU workloads on Kubernetes

Executive Summary

This learning plan covers advanced Kubernetes scheduling mechanisms with a specific focus on GPU resource management for machine learning workloads. Modern ML infrastructure requires sophisticated scheduling to handle expensive GPU resources efficiently, prevent resource fragmentation, and ensure fair access across teams. This plan bridges the gap between basic Kubernetes knowledge and production-grade GPU cluster management, covering custom schedulers, device plugins, multi-tenancy, and cost optimization strategies.

The plan emphasizes hands-on practice with real-world scenarios: building custom schedulers, configuring GPU sharing mechanisms, implementing quota systems, and optimizing workload placement. You'll learn how companies like Google, Meta, and OpenAI manage GPU clusters at scale, along with the tradeoffs between different scheduling strategies.

Expected Outcomes: - Design and implement custom Kubernetes schedulers for GPU workloads - Configure and troubleshoot NVIDIA GPU Operator, device plugins, and MIG (Multi-Instance GPU) - Implement fair-share scheduling, preemption policies, and resource quotas for multi-tenant GPU clusters - Optimize GPU utilization and prevent resource fragmentation - Build cost-aware scheduling policies and measure ROI on GPU infrastructure

Effort Breakdown: - Reading & Documentation: 25% - Hands-on Labs: 50% - Building Custom Solutions: 20% - Troubleshooting & Optimization: 5%

Module 1: Kubernetes Scheduler Architecture Deep Dive

Duration: 1 week
Effort: 5-7 hours

Learning Objectives

  • Understand the default Kubernetes scheduler architecture (filtering → scoring → binding)
  • Identify scheduler bottlenecks and failure modes
  • Recognize when to use custom schedulers vs. scheduler extenders vs. scheduler plugins

Core Concepts

Default Scheduler Workflow: 1. Filtering Phase: Applies predicates (node selectors, taints/tolerations, resource requirements, affinity rules) to eliminate unsuitable nodes 2. Scoring Phase: Ranks remaining nodes using priority functions (resource balance, spreading, affinity weights) 3. Binding Phase: Reserves resources and creates binding object

Scheduler Extension Points (Scheduling Framework): - PreFilter: Early validation before filtering (example: check if GPU models match requested type) - Filter: Custom node filtering logic (example: exclude nodes with fragmented GPU memory) - PostFilter: Handle scheduling failures (example: trigger scale-up or preemption) - PreScore: Prepare data for scoring (example: fetch GPU utilization metrics from Prometheus) - Score: Custom scoring logic (example: prefer nodes with lowest GPU temperature) - Reserve: Guarantee resources before binding (example: reserve GPU memory in external tracking system) - Permit: Final gate before binding (example: wait for GPU driver readiness) - PreBind: Setup before binding (example: configure GPU isolation) - Bind: Custom binding logic (example: update external GPU allocation database) - PostBind: Cleanup after successful scheduling (example: emit metrics)

When to Use What: - Default Scheduler: Sufficient for 80% of workloads, handles basic GPU scheduling with device plugin - Scheduler Extenders (HTTP webhooks): Legacy approach, higher latency, used when plugin framework doesn't fit - Scheduler Plugins (in-tree or out-of-tree): Modern approach, low latency, full access to scheduler internals - Custom Scheduler: When fundamentally different scheduling logic needed (example: gang scheduling for distributed ML training)

Key Resources

Hands-On Practice

  1. Analyze Default Scheduler Behavior:
  2. Deploy test pods with varying resource requests on a 3-node cluster
  3. Enable scheduler logging (--v=5) and trace a pod's scheduling decision
  4. Identify which filtering predicates and scoring functions were applied

  5. Expected Time: 1 hour

  6. Benchmark Scheduler Performance:

  7. Use scheduler_perf_test to measure scheduling latency with 100/1000/5000 pending pods
  8. Identify the point where scheduler becomes a bottleneck (typically ~3000-5000 pods/cluster)

  9. Expected Time: 1.5 hours

  10. Deploy Scheduler Extender:

  11. Implement a simple HTTP webhook that filters nodes based on custom logic (example: exclude nodes in maintenance window)
  12. Configure kube-scheduler to call your extender via --config flag
  13. Measure latency overhead (typically 50-200ms per pod)

  14. Expected Time: 2 hours

  15. Build a Basic Scheduler Plugin:

  16. Implement a Score plugin that prefers nodes with lower GPU utilization (mock data initially)
  17. Compile and deploy the custom scheduler alongside default scheduler
  18. Verify pods with schedulerName: custom-scheduler use your logic
  19. Expected Time: 2.5 hours

Module 2: GPU Device Plugin Architecture

Duration: 1 week
Effort: 6-8 hours

Learning Objectives

  • Understand the Kubernetes Device Plugin framework
  • Configure NVIDIA GPU Operator and device plugin
  • Troubleshoot common GPU visibility and allocation issues
  • Recognize device plugin limitations for GPU sharing

Core Concepts

Device Plugin Framework: - Registration: Plugin registers with kubelet via gRPC (Registration service) - ListAndWatch: Plugin streams available devices to kubelet (example: nvidia.com/gpu: 8) - Allocate: Plugin handles device allocation when pod scheduled (sets environment variables, device paths, mounts) - GetDevicePluginOptions: Defines plugin behavior (example: support pre-start container hooks)

NVIDIA GPU Stack on Kubernetes: 1. NVIDIA GPU Driver: Kernel module, must match CUDA version 2. NVIDIA Container Toolkit: Exposes GPUs to containers via --gpus flag (Docker/containerd runtime) 3. NVIDIA Device Plugin: Advertises nvidia.com/gpu resource to Kubernetes 4. NVIDIA GPU Operator: Manages entire stack lifecycle via Kubernetes operators

Resource Types: - nvidia.com/gpu: Full GPU allocation (default, exclusive access) - nvidia.com/mig-<strategy>: MIG (Multi-Instance GPU) slices (A100/H100 support) - Custom resources via GPU feature discovery (example: nvidia.com/gpu.memory: 40960 for A100 40GB)

Common Issues: - GPUs not visible: Driver mismatch, containerd config missing nvidia runtime, device plugin not running - Pod stuck in Pending: nvidia.com/gpu resource not advertised, node taint blocking scheduling, resource quota exceeded - Wrong GPU allocated: Device plugin doesn't support topology awareness, requires NUMA binding - GPU memory OOM: Device plugin allocates full GPU, doesn't enforce memory limits (use MIG or time-slicing)

Key Resources

Hands-On Practice

  1. Deploy NVIDIA GPU Operator:
  2. Install GPU Operator on a Kubernetes cluster with NVIDIA GPUs (or use kind/minikube with --gpus flag)
  3. Verify driver installation, device plugin running, GPU resources advertised: kubectl describe node | grep nvidia.com/gpu

  4. Expected Time: 1.5 hours

  5. Run GPU Workloads:

  6. Deploy a pod requesting resources.limits.nvidia.com/gpu: 1
  7. Verify GPU access inside container: nvidia-smi
  8. Test multiple pods sharing a node (exclusive GPU allocation per pod)

  9. Expected Time: 1 hour

  10. Troubleshoot GPU Visibility Issues:

  11. Simulate common failures: driver mismatch, missing runtime config, device plugin crash
  12. Use kubectl logs -n gpu-operator-resources nvidia-device-plugin-daemonset-<pod> to debug

  13. Expected Time: 1.5 hours

  14. Explore GPU Metrics:

  15. Deploy NVIDIA DCGM Exporter (Prometheus exporter for GPU metrics)
  16. Query GPU utilization, memory usage, temperature: DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_FB_USED

  17. Expected Time: 1.5 hours

  18. Test Resource Limits:

  19. Attempt to schedule more pods than available GPUs (expect Pending state)
  20. Configure ResourceQuota to limit nvidia.com/gpu usage per namespace
  21. Expected Time: 1 hour

Module 3: Multi-Instance GPU (MIG) and GPU Sharing

Duration: 1.5 weeks
Effort: 7-10 hours

Learning Objectives

  • Configure MIG on NVIDIA A100/H100 GPUs
  • Implement GPU time-slicing for multi-tenant workloads
  • Choose between MIG, time-slicing, and MPS (Multi-Process Service) based on workload requirements
  • Optimize GPU utilization and reduce fragmentation

Core Concepts

GPU Sharing Strategies:

Strategy Isolation Performance Use Case Hardware Support
Exclusive Full (memory + compute) Best Training large models, latency-sensitive inference All GPUs
MIG Hardware-level (memory + SM partitions) Good (no interference) Multi-tenant inference, guaranteed QoS A100, H100, H200
Time-Slicing Time-multiplexed (CUDA context switching) Moderate (context switch overhead) Bursty workloads, dev/test All GPUs
MPS Shared CUDA context Best (no context switch) Multiple processes from same user/job All GPUs

MIG (Multi-Instance GPU): - Splits A100/H100 into up to 7 isolated instances (each with dedicated memory, SMs, encoders) - MIG Profiles: 1g.5gb (1/7 GPU, 5GB), 2g.10gb, 3g.20gb, 4g.20gb, 7g.40gb (full GPU) - Kubernetes Integration: NVIDIA MIG Manager creates device plugin instances per profile - Limitations: No dynamic reconfiguration (requires GPU reset), not all models supported, profile fragmentation (can't mix arbitrary profiles)

Time-Slicing: - Oversubscribes GPU by allowing multiple pods to share the same nvidia.com/gpu resource - Configured via replicas: N in device plugin ConfigMap (example: replicas: 4 = 4 pods per GPU) - Tradeoffs: No memory isolation (OOM affects all pods), context switch overhead (10-30% performance penalty), unpredictable latency

MPS (Multi-Process Service): - NVIDIA technology for sharing GPU across processes with minimal overhead - Use Case: Multiple containers in same pod (example: ensemble inference pipelines) - Limitation: All processes must trust each other (same security context), not well-integrated with Kubernetes device plugin

Key Resources

Hands-On Practice

  1. Configure MIG on A100:
  2. Create MIG instances: nvidia-smi mig -cgi 1g.5gb,1g.5gb,3g.20gb (or use GPU Operator MIG Manager)
  3. Verify instances: nvidia-smi mig -lgi
  4. Deploy pods requesting different MIG profiles (example: nvidia.com/mig-1g.5gb: 1)
  5. Expected Time: 2 hours

  6. Note: Requires A100/H100 access (use cloud instance or skip if unavailable)

  7. Implement GPU Time-Slicing:

  8. Edit device plugin ConfigMap to set replicas: 4 for a GPU node
  9. Deploy 4 pods requesting nvidia.com/gpu: 1 on the same node
  10. Run concurrent workloads and measure performance degradation (use nvidia-smi dmon -s u to observe context switching)

  11. Expected Time: 2 hours

  12. Benchmark Sharing Strategies:

  13. Run identical inference workload (example: ResNet-50 on ImageNet) using:
    • Exclusive GPU allocation (baseline)
    • MIG 1g.5gb instance
    • Time-slicing with 4x oversubscription
  14. Measure throughput and latency for each configuration

  15. Expected Time: 2.5 hours

  16. Test MIG Profile Fragmentation:

  17. Configure MIG with mixed profiles: 2g.10gb, 2g.10gb, 1g.5gb, 1g.5gb
  18. Attempt to schedule a pod requesting 3g.20gb (expect failure due to fragmentation)
  19. Reconfigure MIG to defragment: nvidia-smi mig -dci && nvidia-smi mig -cgi 3g.20gb,4g.20gb

  20. Expected Time: 1.5 hours

  21. Optimize for Utilization vs. Isolation:

  22. Analyze GPU utilization metrics from DCGM Exporter
  23. Identify underutilized GPUs (example: <30% utilization)
  24. Decide: enable time-slicing to pack more workloads, or use MIG for guaranteed QoS
  25. Expected Time: 1.5 hours

Module 4: Custom Scheduling for GPU Workloads

Duration: 2 weeks
Effort: 10-12 hours

Learning Objectives

  • Design custom scheduling policies for GPU workloads (bin-packing, spreading, affinity)
  • Implement GPU topology-aware scheduling (NUMA, NVLink, PCIe lanes)
  • Handle multi-GPU training jobs (gang scheduling, all-or-nothing allocation)
  • Integrate scheduler with external GPU tracking systems

Core Concepts

GPU Scheduling Challenges: 1. Expensive Resources: GPUs cost $10K-$40K each, utilization optimization critical 2. Heterogeneous Hardware: Mixed GPU models (V100, A100, H100), memory sizes, interconnects (NVLink vs. PCIe) 3. Topology Sensitivity: Multi-GPU jobs require fast interconnects (NVLink > PCIe), NUMA locality 4. Gang Scheduling: Distributed training needs all pods scheduled simultaneously (avoid deadlock) 5. Fragmentation: Poor scheduling creates unusable GPU fragments (example: node with 1 free GPU when job needs 2)

Scheduling Policies:

Policy Goal Use Case Tradeoff
Bin-Packing Fill nodes before using new ones Maximize utilization, minimize idle nodes Poor fault tolerance (no spreading)
Spreading Distribute pods across nodes Fault tolerance, avoid hot spots Lower utilization, more idle resources
GPU-Affinity Co-locate GPU pods with CPU pods needing GPU data Reduce data transfer overhead Complex dependency tracking
Topology-Aware Prefer nodes with optimal GPU interconnects Multi-GPU training performance Requires hardware topology metadata

GPU Topology Awareness: - NUMA (Non-Uniform Memory Access): GPUs attached to specific CPU sockets, cross-socket access slower - NVLink: NVIDIA's high-speed interconnect (600 GB/s A100, 900 GB/s H100) for multi-GPU communication - PCIe Lanes: Bandwidth (PCIe 4.0 x16 = 32 GB/s), topology (GPUs on same PCIe switch vs. different switches) - Kubernetes Integration: GPU Feature Discovery labels nodes with topology info: nvidia.com/gpu.count, nvidia.com/gpu.product, nvidia.com/mig.capable

Gang Scheduling: - Problem: Job needs 4 GPUs, scheduler allocates 2 pods → 2 pending indefinitely (resource deadlock) - Solution: All-or-nothing scheduling (reserve all resources before binding any pod) - Implementations: - Volcano: Kubernetes batch scheduler with gang scheduling plugin - Coscheduling Plugin: Kubernetes scheduler plugin (simpler than Volcano) - Kueue: Job queueing system with gang scheduling support (recommended for production)

Key Resources

Hands-On Practice

  1. Implement Bin-Packing Scheduler Plugin:
  2. Create a Score plugin that heavily weights nodes with already-allocated GPUs
  3. Deploy and test: verify new GPU pods prefer partially-used nodes over empty nodes

  4. Expected Time: 3 hours

  5. Build Topology-Aware Scheduler:

  6. Deploy GPU Feature Discovery to label nodes with GPU metadata
  7. Create a Filter plugin that excludes nodes without NVLink when pod requests nvidia.com/gpu: 2+
  8. Test with multi-GPU training job (example: PyTorch DistributedDataParallel)

  9. Expected Time: 3 hours

  10. Implement Gang Scheduling with Volcano:

  11. Install Volcano scheduler
  12. Submit a job with schedulerName: volcano and minAvailable: 4 (gang constraint)
  13. Verify all 4 pods scheduled simultaneously or all remain pending

  14. Expected Time: 2.5 hours

  15. Test Scheduling Policies:

  16. Deploy 10 single-GPU pods on a 3-node cluster (2 GPUs/node)
  17. Compare bin-packing (6-4-0 distribution) vs. spreading (4-3-3 distribution)
  18. Measure impact on fault tolerance: kill one node and measure recovery time

  19. Expected Time: 2 hours

  20. Optimize for Real-World Workload:

  21. Analyze GPU allocation patterns from your organization (or use synthetic workload)
  22. Design custom scoring logic (example: prefer nodes with matching GPU memory size)
  23. Benchmark scheduling latency and GPU utilization
  24. Expected Time: 2.5 hours

Module 5: Multi-Tenancy and Fair-Share Scheduling

Duration: 1.5 weeks
Effort: 8-10 hours

Learning Objectives

  • Implement fair-share GPU scheduling across teams
  • Configure preemption policies for priority-based GPU access
  • Design quota systems for GPU resources
  • Prevent resource hogging and starvation

Core Concepts

Multi-Tenancy Challenges: - Scarcity: Limited GPU availability, high demand from ML teams - Heterogeneous Workloads: Training (long-running, batch) vs. inference (latency-sensitive, continuous) - Team Priorities: Production inference > research training > experimentation - Cost Allocation: Track GPU usage per team for chargeback/showback

Fair-Share Scheduling: - Goal: Allocate GPU time proportionally to team entitlements (example: Team A gets 40%, Team B gets 30%, Team C gets 30%) - Dominant Resource Fairness (DRF): Considers multiple resource types (GPU + CPU + memory), allocates based on dominant bottleneck - Kubernetes Implementation: ResourceQuotas (hard limits), LimitRanges (default/max per pod), PriorityClasses (preemption)

Preemption: - Scenario: High-priority job submitted, no free GPUs → evict low-priority pods - Kubernetes Preemption: Scheduler checks PriorityClass, evicts lower-priority pods if no nodes available - Graceful Preemption: Give pods time to checkpoint (PodDisruptionBudget, terminationGracePeriodSeconds) - Limitations: Kubernetes preemption is "dumb" (doesn't consider job progress, can preempt 99%-complete job)

Advanced Fair-Share: - Volcano Queue: Assigns weight to each team, scheduler allocates resources proportionally - Kueue ClusterQueue: Fair-share with borrowing (Team A can use Team B's idle quota temporarily) - ElasticQuota (KEP-2412): Kubernetes-native fair-share (alpha feature, not production-ready)

Key Resources

Hands-On Practice

  1. Configure PriorityClasses:
  2. Create 3 PriorityClasses: production (1000), research (500), experiment (100)
  3. Deploy pods with different priorities
  4. Verify preemption: submit high-priority pod when GPUs full, observe low-priority pod eviction

  5. Expected Time: 1.5 hours

  6. Implement ResourceQuotas for GPU:

  7. Create 3 namespaces: team-a, team-b, team-c
  8. Set ResourceQuotas: Team A (4 GPUs), Team B (3 GPUs), Team C (2 GPUs)
  9. Test quota enforcement: attempt to exceed quota, expect pod stuck in Pending

  10. Expected Time: 1.5 hours

  11. Deploy Kueue for Fair-Share:

  12. Install Kueue, create ClusterQueue with 3 ResourceFlavors (one per team)
  13. Configure fair-share weights: Team A (40%), Team B (30%), Team C (30%)
  14. Submit jobs from multiple teams, observe fair allocation

  15. Expected Time: 2.5 hours

  16. Test Graceful Preemption:

  17. Create a training job that checkpoints every 60 seconds
  18. Configure terminationGracePeriodSeconds: 120 and PodDisruptionBudget
  19. Trigger preemption, verify job saves checkpoint before termination

  20. Expected Time: 2 hours

  21. Build Usage Dashboard:

  22. Use Prometheus + Grafana to visualize GPU usage per team (label: namespace)
  23. Calculate metrics: GPU-hours per team, average queue time, preemption rate
  24. Identify resource hogging (example: Team A using 60% of GPUs despite 40% entitlement)
  25. Expected Time: 2.5 hours

Module 6: Cost Optimization and GPU Utilization

Duration: 1.5 weeks
Effort: 8-10 hours

Learning Objectives

  • Measure and optimize GPU utilization
  • Implement cost-aware scheduling policies
  • Reduce GPU idle time and fragmentation
  • Calculate ROI on GPU infrastructure investments

Core Concepts

GPU Cost Metrics: - Capital Cost: Purchase price (example: A100 80GB = $15K, H100 = $30K+) - Operational Cost: Power (300-700W/GPU), cooling, datacenter space - Cloud Cost: AWS p4d.24xlarge (8x A100) = $32.77/hour, p5.48xlarge (8x H100) = $98.32/hour - Total Cost of Ownership (TCO): 3-year amortized cost including hardware, power, maintenance, salaries

Utilization Metrics: - GPU Utilization (%): SM (streaming multiprocessor) active time (from nvidia-smi) - Memory Utilization (%): GPU memory used / total memory - Idle Time: Percentage of time GPU allocated but unused (pod running, no CUDA kernels) - Allocation Efficiency: Percentage of cluster GPUs allocated to pods (vs. unscheduled/fragmented)

Common Causes of Low Utilization: 1. I/O Bottleneck: GPU waiting for data from CPU/disk (poor dataloading pipeline) 2. Small Batch Size: GPU underutilized due to insufficient parallelism 3. CPU Preprocessing: Time spent in Python/NumPy code before GPU computation 4. Idle Between Jobs: Delay between job completion and next job submission 5. Fragmentation: GPUs unavailable due to poor scheduling (example: 4x 1-GPU nodes, but need 2-GPU job)

Optimization Strategies: - Batch Size Tuning: Increase batch size to saturate GPU (trade-off: memory usage, convergence) - Mixed Precision (FP16/BF16): 2-3x throughput improvement on modern GPUs (A100, H100) - Data Pipeline Optimization: Use tf.data prefetching, torch.utils.data.DataLoader with num_workers - Job Packing: Co-locate small jobs on same GPU (time-slicing or MIG) - Spot/Preemptible Instances: Use cheaper cloud instances for fault-tolerant workloads (50-90% discount)

Cost-Aware Scheduling: - Spot-First Policy: Prefer spot instances, fallback to on-demand if unavailable - Right-Sizing: Match GPU model to workload (don't use H100 for small inference) - Off-Peak Scheduling: Run low-priority jobs during nights/weekends when utilization low

Key Resources

Hands-On Practice

  1. Profile GPU Utilization:
  2. Run a training job (example: ResNet-50 on ImageNet)
  3. Use nvidia-smi dmon -s u to monitor utilization in real-time
  4. Identify idle periods (example: data loading between epochs)

  5. Expected Time: 1.5 hours

  6. Optimize Data Pipeline:

  7. Baseline: Single-threaded data loading (expect <50% GPU utilization)
  8. Add prefetching: torch.utils.data.DataLoader(num_workers=4, prefetch_factor=2)
  9. Measure GPU utilization improvement (target: >80%)

  10. Expected Time: 2 hours

  11. Calculate TCO for GPU Cluster:

  12. Scenario: 10x A100 GPUs, 3-year depreciation
  13. Capital: 10 × $15K = $150K
  14. Power: 10 × 400W × 24h × 365d × $0.10/kWh × 3y = $10.5K
  15. Total: $160.5K → $53.5K/year → $15/GPU/day
  16. Compare to cloud: AWS p4d (8x A100) = $32.77/hour = $785/day ($98/GPU/day)
  17. Breakeven: ~150 days of 24/7 usage

  18. Expected Time: 1 hour

  19. Implement Fragmentation Detection:

  20. Write script to detect GPU fragmentation: nodes with free GPUs, but unusable for multi-GPU jobs
  21. Metric: (Total Free GPUs - Max GPUs on Single Node) / Total GPUs
  22. Example: 3 nodes with 1 free GPU each, but job needs 2 GPUs → 66% fragmentation

  23. Expected Time: 2 hours

  24. Build Cost Dashboard:

  25. Extend Grafana dashboard with cost metrics:
    • GPU Cost/Hour = (GPU Utilization × Allocated GPUs × Cost/GPU)
    • Idle Cost = (100% - GPU Utilization) × Allocated GPUs × Cost/GPU
  26. Identify most expensive jobs/teams
  27. Expected Time: 2.5 hours

Module 7: Monitoring and Troubleshooting GPU Clusters

Duration: 1 week
Effort: 6-8 hours

Learning Objectives

  • Deploy comprehensive GPU monitoring stack (DCGM, Prometheus, Grafana)
  • Troubleshoot common GPU scheduling failures
  • Detect GPU hardware failures and throttling
  • Implement alerting for GPU resource issues

Core Concepts

GPU Monitoring Stack: 1. DCGM Exporter: Prometheus exporter for NVIDIA GPU metrics (utilization, memory, temperature, power, ECC errors) 2. Node Exporter: System metrics (CPU, memory, disk, network) 3. Kube-State-Metrics: Kubernetes object metrics (pod status, resource requests) 4. Prometheus: Time-series database and query engine 5. Grafana: Visualization and dashboards

Key GPU Metrics: - Utilization: DCGM_FI_DEV_GPU_UTIL (0-100%, streaming multiprocessor active time) - Memory: DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_FREE (MiB) - Temperature: DCGM_FI_DEV_GPU_TEMP (Celsius, throttles at 80-90°C) - Power: DCGM_FI_DEV_POWER_USAGE (Watts, max 300-700W depending on model) - Clock Speed: DCGM_FI_DEV_SM_CLOCK / DCGM_FI_DEV_MEM_CLOCK (MHz, throttles if temperature/power limit hit) - ECC Errors: DCGM_FI_DEV_ECC_DBE_VOL_TOTAL (double-bit errors, indicates hardware failure) - PCIe Throughput: DCGM_FI_DEV_PCIE_TX_BYTES / DCGM_FI_DEV_PCIE_RX_BYTES (bandwidth usage)

Common GPU Issues: 1. Thermal Throttling: GPU reduces clock speed due to overheating (symptom: performance degradation, temp >85°C) 2. Power Throttling: GPU hits power limit (symptom: utilization <100% but clocks reduced) 3. ECC Errors: Memory corruption (symptom: training divergence, crashes, DCGM_FI_DEV_ECC_DBE_VOL_TOTAL increasing) 4. Driver Mismatch: CUDA version incompatibility (symptom: CUDA_ERROR_INVALID_DEVICE) 5. Zombie Processes: Previous job didn't release GPU (symptom: GPU shows allocated but no active kernels)

Alerting Rules: - High Temperature: DCGM_FI_DEV_GPU_TEMP > 85 for 5 minutes → investigate cooling - Low Utilization: DCGM_FI_DEV_GPU_UTIL < 20 for 1 hour AND pod running → check job efficiency - ECC Errors: increase(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[1h]) > 0 → hardware failure, replace GPU - Pending Pods: kube_pod_status_phase{phase="Pending"} AND pod requests nvidia.com/gpu for 10 minutes → scheduling failure

Key Resources

Hands-On Practice

  1. Deploy Monitoring Stack:
  2. Install Prometheus Operator (or standalone Prometheus)
  3. Deploy DCGM Exporter as DaemonSet
  4. Import Grafana dashboard for GPU metrics

  5. Expected Time: 2 hours

  6. Create Alerting Rules:

  7. Define PrometheusRule for high temperature, ECC errors, pending GPU pods
  8. Configure Alertmanager to send notifications (email/Slack)
  9. Test alerts: trigger thermal throttling (stress test) or simulate pending pod (delete device plugin)

  10. Expected Time: 1.5 hours

  11. Troubleshoot Scheduling Failure:

  12. Scenario: Pod stuck in Pending, event: 0/3 nodes available: insufficient nvidia.com/gpu
  13. Debug steps:
    1. Check GPU availability: kubectl describe nodes | grep nvidia.com/gpu
    2. Check device plugin: kubectl logs -n gpu-operator-resources nvidia-device-plugin-daemonset-<pod>
    3. Check ResourceQuota: kubectl describe quota -n <namespace>
    4. Check PriorityClass: kubectl get pods -o yaml | grep priorityClassName

  14. Expected Time: 1.5 hours

  15. Diagnose GPU Hardware Issue:

  16. Simulate ECC errors (or use real cluster logs if available)
  17. Query Prometheus: increase(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[1h])
  18. Identify affected GPU: DCGM_FI_DEV_ECC_DBE_VOL_TOTAL{gpu="2", hostname="node-gpu-03"}
  19. Recommended action: Cordon node, drain workloads, replace GPU

  20. Expected Time: 1.5 hours

  21. Build Custom Dashboard:

  22. Create Grafana dashboard with panels:
    • GPU utilization heatmap (all GPUs in cluster)
    • Memory usage per node
    • Pending GPU pods over time
    • Cost per team (GPU-hours × cost/hour)
  23. Expected Time: 2 hours

Module 8: Advanced Topics - Gang Scheduling, Topology, and Batch Systems

Duration: 1.5 weeks
Effort: 8-10 hours

Learning Objectives

  • Implement gang scheduling for distributed training jobs
  • Optimize multi-GPU jobs using GPU topology metadata
  • Integrate Kubernetes with HPC batch systems (Slurm)
  • Handle dynamic cluster scaling (autoscaling GPU nodes)

Core Concepts

Gang Scheduling Deep Dive: - Problem: Distributed training (PyTorch DDP, Horovod, MPI) requires all workers start simultaneously, else deadlock - Solution: All-or-nothing scheduling (reserve all resources before binding any pod) - Implementation Options: - Volcano: Full-featured batch scheduler, supports gang scheduling via PodGroup - Scheduler Plugins Coscheduling: Lightweight plugin, simpler than Volcano - Kueue: Job queueing with gang scheduling, better multi-tenancy support

GPU Topology Optimization: - NVLink Topology: GPUs connected via NVLink can communicate 10-20x faster than PCIe - NUMA Awareness: GPUs attached to local CPU socket have lower latency than cross-socket - Kubernetes Labels: GPU Feature Discovery labels nodes: nvidia.com/gpu.count, nvidia.com/gpu.product, nvidia.com/mig.capable - Custom Topology Labels: Add labels for NVLink connectivity (example: gpu-topology=nvlink-8x for 8 GPUs fully connected)

HPC Integration: - Slurm on Kubernetes: Run Slurm controller in Kubernetes, submit jobs via sbatch - Kubernetes as Slurm Backend: Use Kubernetes as compute backend for Slurm (instead of bare-metal nodes) - Hybrid Approach: HPC jobs via Slurm, services/inference via Kubernetes (separate clusters)

Autoscaling GPU Nodes: - Cluster Autoscaler: Scales node groups based on pending pods (adds GPU nodes when pods can't schedule) - Karpenter: More flexible autoscaling, provisions right-sized nodes on-demand (faster than Cluster Autoscaler) - Challenges: GPU nodes expensive (slow to start ~5-10 min), scaling down risky (can disrupt long-running training) - Best Practices: Use separate node pools for stable (always-on) and burst (autoscaling) GPU nodes

Key Resources

Hands-On Practice

  1. Implement Gang Scheduling with Volcano:
  2. Deploy Volcano scheduler
  3. Submit a distributed PyTorch training job (4 workers) with minAvailable: 4
  4. Verify all 4 pods scheduled together or all remain pending (no partial allocation)

  5. Expected Time: 2.5 hours

  6. Optimize Multi-GPU Training with Topology:

  7. Label nodes with NVLink topology metadata (or use GPU Feature Discovery)
  8. Create a Filter plugin that requires NVLink for multi-GPU jobs (>1 GPU requested)
  9. Submit 8-GPU training job, verify it only schedules on node with 8 NVLink-connected GPUs

  10. Expected Time: 2.5 hours

  11. Deploy Kubeflow MPI Operator:

  12. Install MPI Operator
  13. Submit an MPI job (example: Horovod multi-GPU training)
  14. Verify workers communicate via MPI, observe GPU utilization across all workers

  15. Expected Time: 2 hours

  16. Test Autoscaling GPU Nodes:

  17. Configure Cluster Autoscaler (or Karpenter) for GPU node pool
  18. Submit 10 GPU jobs when cluster only has 5 GPUs
  19. Observe autoscaler adding new GPU nodes, pods scheduled on new nodes

  20. Expected Time: 2 hours

  21. Build Hybrid HPC/Kubernetes Workflow:

  22. Install Slurm controller in Kubernetes (or use external Slurm cluster)
  23. Submit Slurm job that launches Kubernetes pods for distributed training
  24. Measure scheduling latency and resource utilization
  25. Expected Time: 3 hours (optional, requires Slurm expertise)

Module 9: Production Best Practices and Case Studies

Duration: 1 week
Effort: 5-7 hours

Learning Objectives

  • Learn from production GPU cluster deployments (Google, Meta, OpenAI)
  • Implement observability, SLOs, and incident response for GPU infrastructure
  • Design disaster recovery and backup strategies for GPU workloads
  • Plan capacity and forecasting for GPU demand

Core Concepts

Production GPU Cluster Architecture: - Google TPU/GPU Pods: Custom interconnects (ICI, DCN) for 1000s of accelerators, hierarchical scheduling - Meta AI Research Cluster: 16,000 A100 GPUs, RoCE networking, Slurm + Kubernetes hybrid - OpenAI Supercomputer: Azure-hosted, 10,000s of GPUs, custom scheduler for large models (GPT-4 training)

SLOs for GPU Infrastructure: - Availability: Percentage of time GPUs schedulable (target: 99.5%, excluding maintenance windows) - Scheduling Latency: Time from job submission to pod running (target: <5 minutes for small jobs, <15 minutes for gang-scheduled jobs) - Utilization: Average GPU utilization across cluster (target: >60% for production, >40% for research) - MTTR (Mean Time To Recovery): Time to recover from GPU hardware failure (target: <4 hours)

Disaster Recovery: - Checkpointing: Save training state periodically (every N steps), store in distributed storage (S3, GCS, NFS) - Multi-Region Replication: Run critical workloads in multiple regions (expensive, mainly for inference) - GPU Failure Handling: Automatic job rescheduling on different node if GPU failure detected (ECC errors, driver crash) - Backup Scheduler: Standby scheduler instance to handle control plane failures

Capacity Planning: - Demand Forecasting: Analyze historical GPU usage trends, predict future demand (example: 20% growth QoQ) - Lead Time: GPU procurement takes 3-6 months, plan ahead - Burst Capacity: Reserve cloud GPU instances for peak demand (hybrid on-prem + cloud) - Oversubscription Ratio: Allocate 1.2-1.5x demand to handle bursty workloads (tradeoff: cost vs. availability)

Key Resources

Hands-On Practice

  1. Design SLOs for GPU Cluster:
  2. Define SLOs: Availability, scheduling latency, utilization, MTTR
  3. Create SLI (Service Level Indicator) queries in Prometheus
  4. Build Grafana dashboard showing SLO compliance

  5. Expected Time: 1.5 hours

  6. Implement Checkpointing Strategy:

  7. Modify training script to save checkpoint every 100 steps
  8. Store checkpoints in S3/GCS with versioning enabled
  9. Simulate GPU failure (kill pod), verify job resumes from last checkpoint

  10. Expected Time: 2 hours

  11. Test GPU Failure Recovery:

  12. Simulate hardware failure: label node as unhealthy (kubectl taint nodes <node> gpu=failed:NoSchedule)
  13. Verify scheduler reschedules pods on healthy nodes
  14. Measure MTTR: time from failure to pod running on new node

  15. Expected Time: 1.5 hours

  16. Build Capacity Planning Model:

  17. Export GPU usage data from Prometheus (last 3 months)
  18. Calculate average utilization per team, identify growth trends
  19. Forecast demand for next 6 months (example: linear regression, moving average)
  20. Recommend GPU purchases based on forecast

  21. Expected Time: 2 hours

  22. Study Production Case Study:

  23. Read one of the case studies (Google Pathways, Meta RSC, OpenAI Supercomputer)
  24. Identify key design decisions: scheduler, networking, storage, fault tolerance
  25. Apply learnings to your organization's GPU cluster design
  26. Expected Time: 1.5 hours (anecdotal)

Duration: 1 week
Effort: 4-6 hours

Learning Objectives

  • Understand emerging GPU technologies (H100, GH200, multi-chip modules)
  • Explore alternative accelerators (TPUs, AMD GPUs, Intel Gaudi)
  • Learn about disaggregated GPU architectures (GPU-as-a-Service)
  • Anticipate Kubernetes evolution for AI workloads (DRA, JobSet)

Core Concepts

Next-Generation GPUs: - NVIDIA H100/H200: 3x performance vs. A100, Transformer Engine (FP8 for LLMs), 80GB HBM3 memory - NVIDIA GH200 Grace Hopper: CPU + GPU on same package, 900 GB/s NVLink-C2C, unified memory - AMD MI300X: Competitor to H100, 192GB HBM3, open ROCm ecosystem - Intel Gaudi2/Gaudi3: AI accelerator, competitive price, support for PyTorch/TensorFlow

Disaggregated GPU Architectures: - GPU Pooling: Separate GPU resources from compute nodes, allocate dynamically via network (example: NVIDIA DPU + BlueField) - GPU-as-a-Service: Cloud providers offer GPU sharing at finer granularity (example: AWS EC2 Flex instances, fractional GPUs) - CXL (Compute Express Link): Industry standard for memory/accelerator pooling, enables true disaggregation

Kubernetes Enhancements for AI: - DRA (Dynamic Resource Allocation, KEP-3063): Replace device plugin with more flexible resource claim model (supports GPU sharing, topology, advanced scheduling) - JobSet (KEP-2086): Native gang scheduling, multi-template jobs (parameter server + workers in one object) - Cluster Autoscaler Improvements: Faster scaling, better bin-packing, GPU-aware node selection - Kueue Evolution: Advanced fair-share, hierarchical queues, cost-aware scheduling

Trends in ML Infrastructure: - Model Parallelism: LLMs too large for single GPU (GPT-4: ~1.8T parameters), require tensor/pipeline parallelism across 100s-1000s of GPUs - Inference Optimization: KV cache management, continuous batching, speculative decoding (reduce latency + cost) - Sustainable AI: Energy-efficient training (reduce CO2), liquid cooling for high-density GPU clusters - Federated Learning: Train models across distributed data sources without centralizing data (privacy, compliance)

Key Resources

Hands-On Practice

  1. Explore DRA (if available in Kubernetes 1.26+):
  2. Install DRA-enabled device plugin (experimental)
  3. Submit pod with ResourceClaim instead of resources.limits.nvidia.com/gpu
  4. Compare to traditional device plugin model

  5. Expected Time: 2 hours (optional, requires Kubernetes 1.26+)

  6. Benchmark Alternative Accelerators:

  7. Run identical workload on NVIDIA A100, AMD MI250X, Intel Gaudi2 (if accessible)
  8. Measure throughput, latency, power consumption
  9. Analyze cost-performance tradeoff ($/TFLOPS)

  10. Expected Time: 2.5 hours (requires access to alternative accelerators)

  11. Design Future-Proof GPU Cluster:

  12. Scenario: Your organization plans to deploy 100 H100 GPUs in 2026
  13. Design considerations:
    • Network: InfiniBand vs. Ethernet, 400 Gbps vs. 800 Gbps
    • Cooling: Air vs. liquid (H100 = 700W TDP)
    • Scheduling: Kubernetes + Kueue vs. Slurm vs. custom scheduler
    • Multi-tenancy: MIG vs. time-slicing vs. separate clusters per team

  14. Expected Time: 1.5 hours (design exercise, not hands-on)

  15. Read Emerging Research:

  16. Pick one paper:
  17. Identify implications for GPU scheduling and infrastructure
  18. Expected Time: 2 hours (anecdotal)

Hands-On Practice Summary

Total practice time: 50-65 hours across 10 modules.

Practice Categories

Category Effort Skills Developed
Configuration & Deployment 15 hours Install GPU Operator, deploy custom schedulers, configure MIG, deploy monitoring stack
Performance Optimization 12 hours Profile GPU utilization, optimize data pipelines, benchmark sharing strategies, measure cost
Custom Development 18 hours Build scheduler plugins, implement gang scheduling, create topology-aware filters
Troubleshooting 8 hours Debug scheduling failures, diagnose hardware issues, test autoscaling
Design & Planning 7 hours Design SLOs, capacity planning, future-proof architecture

Minimal Setup: - Cloud: Single GPU instance (AWS g4dn.xlarge, GCP n1-standard-4 + T4 GPU) = $5-10/day - Kubernetes: Managed cluster (GKE, EKS, AKS) or self-managed (kubeadm, k3s) - GPUs: 1-2 GPUs sufficient for most exercises (multi-GPU exercises can be simulated)

Ideal Setup: - Cloud: 3-node GPU cluster (2 GPUs/node) = $50-100/day - Hardware: NVIDIA A100 or H100 access for MIG exercises (can substitute with time-slicing on older GPUs) - Networking: InfiniBand or high-speed Ethernet (for topology exercises)

Cost Optimization: - Use spot instances (50-70% discount) - Stop cluster when not practicing - Use free credits (GCP $300, AWS educate, Azure student)

Assessment and Validation

Self-Assessment Checklist

After completing the learning plan, you should be able to:

  • Explain the Kubernetes scheduler workflow (filtering → scoring → binding)
  • Configure NVIDIA GPU Operator and troubleshoot device plugin issues
  • Choose between MIG, time-slicing, and exclusive GPU allocation for a given workload
  • Implement a custom scheduler plugin with scoring/filtering logic
  • Configure gang scheduling for distributed training jobs (Volcano or Kueue)
  • Design fair-share GPU allocation across multiple teams
  • Optimize GPU utilization to >70% for production workloads
  • Deploy DCGM Exporter and build GPU monitoring dashboards
  • Calculate TCO for GPU infrastructure and compare to cloud costs
  • Troubleshoot common GPU scheduling failures (pending pods, preemption, fragmentation)

Practical Projects

To solidify your learning, complete 2-3 of these projects:

  1. Build a Custom GPU Scheduler:
  2. Implement bin-packing + topology-aware scheduling
  3. Handle multi-GPU jobs with NVLink preference
  4. Measure impact on utilization and fragmentation

  5. Time: 15-20 hours

  6. Deploy Production-Grade GPU Cluster:

  7. 5-10 GPU nodes with NVIDIA GPU Operator
  8. Monitoring: DCGM + Prometheus + Grafana
  9. Multi-tenancy: Kueue with fair-share quotas
  10. Cost tracking: GPU-hours per team

  11. Time: 20-25 hours

  12. Optimize ML Training Pipeline:

  13. Profile existing training job (identify bottlenecks)
  14. Optimize data loading, batch size, mixed precision
  15. Measure GPU utilization improvement (before/after)
  16. Calculate cost savings

  17. Time: 10-15 hours

  18. Implement GPU Autoscaling:

  19. Configure Cluster Autoscaler (or Karpenter) for GPU nodes
  20. Define policies: scale-up triggers, scale-down cooldown
  21. Test with bursty workload (simulate daily batch jobs)
  22. Measure cost vs. availability tradeoff
  23. Time: 12-18 hours

Learning Resources

Official Documentation

Books

  • "Kubernetes in Action" (2nd Edition) by Marko Lukša (2021) - Chapters 16-17 on scheduling
  • "Cloud Native DevOps with Kubernetes" by Armon Dadgar et al. (2019) - Chapter on resource management
  • "Programming Kubernetes" by Michael Hausenblas & Stefan Schimanski (2019) - Scheduler internals (Chapter 6)

Note on books: Publication dates provided are self-reported from publisher metadata. Verify edition and relevance for current Kubernetes versions (1.26+).

Research Papers

Note on papers: Peer-reviewed conference publications. Check citations and follow-up work for latest developments.

Community & Forums

  • Kubernetes Slack: #sig-scheduling, #wg-batch channels - Active community discussions
  • NVIDIA Developer Forums: GPU + Kubernetes topics
  • Stack Overflow: Tag [kubernetes] [gpu] - Practical troubleshooting
  • Reddit: r/kubernetes, r/MachineLearning - Case studies and discussions

Note on community resources: Anecdotal evidence, not authoritative sources. Cross-reference with official documentation.

Video Courses & Tutorials

Note on videos: Presentation slides and demos, not comprehensive tutorials. Use as supplement to hands-on practice.

Lehetséges Torzítások és Bizonytalanságok (Potential Biases and Uncertainties)

Forrásoldali Torzítások (Source-Level Biases)

Vendor Documentation Bias: - NVIDIA documentation (GPU Operator, DCGM, MIG) naturally emphasizes NVIDIA solutions and may downplay limitations - Alternative accelerators (AMD MI300X, Intel Gaudi) are less represented in Kubernetes ecosystem documentation - Impact: Plan may overemphasize NVIDIA-specific technologies; users should independently evaluate AMD ROCm and Intel oneAPI ecosystems

Cloud Provider Bias: - AWS, GCP, Azure documentation focuses on managed services (EKS, GKE, AKS) and may not reflect bare-metal deployment challenges - Pricing comparisons use list prices; actual enterprise discounts unknown - Impact: TCO calculations may not reflect negotiated rates; cost estimates should be validated with your procurement team

Academic Research Lag: - Research papers cited (2011-2024) may not reflect latest Kubernetes releases (1.26+) - Tiresias scheduler (2019) predates Kueue and DRA; some recommendations may be outdated - Impact: Emerging technologies (DRA, JobSet) have limited production case studies; treat as experimental

Értelmezési Torzítások (Interpretation Biases)

"Production-Ready" Threshold: - Terms like "production-grade" and "best practices" are subjective and vary by organization size, risk tolerance, and compliance requirements - What works for Google/Meta (16K+ GPUs) may not apply to smaller deployments (10-100 GPUs) - Impact: Readers should adapt recommendations to their scale; don't over-engineer for hypothetical future growth

GPU Utilization Targets: - Target utilization metrics (>60% production, >40% research) are informed by industry anecdotes, not rigorous benchmarks - Optimal utilization depends on workload mix (training vs. inference), job duration, and cost constraints - Impact: Treat targets as guidelines, not absolutes; measure against your own baseline

Fair-Share "Fairness": - Fair-share scheduling assumes equal value across teams, but some workloads (production inference) are objectively more critical than others (experimental training) - DRF algorithm is mathematically sound but doesn't account for business priorities - Impact: Fair-share policies must be combined with priority classes; pure fairness may not align with organizational goals

Kontextuális Torzítások (Contextual Biases)

Privilege Check: Hardware Access: - Many exercises assume access to NVIDIA A100/H100 GPUs (MIG configuration, NVLink topology) - Cloud GPU instances expensive ($10-100/day); not all learners have budget or free credits - Uncertainty: Percentage of readers with enterprise GPU access vs. personal/hobbyist setups unknown - Mitigation: Exercises marked as optional if they require expensive hardware; time-slicing can simulate some MIG scenarios

Privilege Check: Kubernetes Expertise: - Plan assumes "basic Kubernetes knowledge" but doesn't define threshold (can you deploy a pod? write a custom controller?) - Advanced modules (custom scheduler plugins) require Go programming skills not explicitly listed in prerequisites - Uncertainty: Reader skill distribution unknown; some may struggle with Module 4-8 if Kubernetes fundamentals weak - Mitigation: Prerequisites section added, but depth assessment is qualitative

Organizational Context: - Large enterprises (Google, Meta) have dedicated platform teams, while smaller orgs may have one SRE managing GPU cluster - Recommendations for "dedicated GPU cluster" may not apply to orgs sharing mixed-workload Kubernetes clusters - Impact: Readers in small teams may need to deprioritize some best practices (example: separate monitoring stack) due to operational overhead

Saját Feltevések (Own Assumptions)

Technology Trajectory: - Assumption: DRA (Dynamic Resource Allocation) will replace device plugin framework by 2027 - Uncertainty: DRA is alpha feature (Kubernetes 1.26), adoption timeline unclear, backward compatibility unknown - Risk: Readers investing heavily in device plugin customization may face migration costs if DRA becomes mandatory

Cloud vs. On-Prem Economics: - Assumption: On-prem GPU cluster breaks even vs. cloud at ~150 days of 24/7 usage - Uncertainty: Calculation uses list prices, ignores staffing costs (SRE salaries), depreciation schedules, and financing - Risk: Oversimplified TCO may lead to poor purchasing decisions; readers should consult finance teams

Multi-Tenancy Demand: - Assumption: Most organizations need multi-tenancy (fair-share, quotas, preemption) - Uncertainty: Small teams (<10 people) may not need complex quota systems; single-tenant clusters simpler to operate - Risk: Over-engineering for hypothetical future teams; readers should start simple and add complexity when needed

GPU Sharing Adoption: - Assumption: Time-slicing and MIG will become standard practice for improving utilization - Uncertainty: Many organizations prioritize job performance over utilization (prefer exclusive GPUs even if idle 50% of the time) - Risk: Recommendations may conflict with organizational culture favoring "dedicated resources" over "shared infrastructure"

Limitations and Gaps

Not Covered in This Plan: 1. Security: GPU isolation, multi-tenancy security (side-channel attacks via shared GPU memory), container escape risks 2. Compliance: GDPR, HIPAA, FedRAMP requirements for GPU workloads (data residency, audit logging) 3. Specialized Hardware: AMD GPUs (ROCm), Intel Gaudi, Google TPUs (focus is NVIDIA-centric) 4. Edge AI: GPU scheduling on edge clusters (K3s, MicroK8s) with intermittent connectivity 5. Inference-Specific Optimizations: Model serving frameworks (Triton, TorchServe, KServe), autoscaling for inference workloads

Reason for Exclusion: These topics require 5-10 additional modules each and target different audience segments (security engineers, compliance officers, edge/IoT developers).

Verifiability and Further Research

Verifiable Claims: - Kubernetes scheduler source code: github.com/kubernetes/kubernetes/tree/master/pkg/scheduler - NVIDIA A100 specifications: nvidia.com/en-us/data-center/a100/ - AWS pricing: aws.amazon.com/ec2/instance-types/p4/ (verify current rates)

Unverifiable/Anecdotal: - "Most organizations achieve 40-60% GPU utilization" - Based on conference talks and blog posts, no industry-wide benchmark - "Scheduler becomes bottleneck at 3000-5000 pods/cluster" - From Kubernetes perf tests, but varies with cluster config and scheduler tuning - "Fair-share reduces conflicts by 70%" - Anecdotal claim from Volcano case studies, not peer-reviewed

Recommended Follow-Up Research: - Read SIG-Scheduling meeting notes for latest Kubernetes scheduling developments - Monitor Kueue GitHub for production adoption case studies - Join NVIDIA Developer Forums to ask vendor-agnostic questions and get unfiltered community answers

Conclusion

This learning plan provides a comprehensive pathway from basic Kubernetes scheduling knowledge to advanced GPU resource management for production ML infrastructure. The 8-12 week timeline is achievable with consistent practice (5-10 hours/week), but can be compressed (4-6 weeks at 15-20 hours/week) or extended (12-16 weeks at 3-5 hours/week) based on your availability and prior experience.

Key Success Factors: 1. Hands-On Practice: 50+ hours of labs is the core learning mechanism; don't skip exercises 2. Incremental Complexity: Start with default scheduler, add complexity module-by-module 3. Real-World Context: Apply learnings to your organization's GPU workloads, don't just follow tutorials 4. Community Engagement: Join Kubernetes Slack, attend SIG-Scheduling meetings, share your learnings

Expected Outcomes: By completing this plan, you'll be equipped to design, deploy, and operate production GPU clusters on Kubernetes, optimize GPU utilization, implement multi-tenancy, and troubleshoot complex scheduling issues. These skills are highly valuable for Platform Engineering, SRE, and ML Infrastructure roles, with market demand growing as AI workloads scale.

Next Steps After Completion: - Contribute to open-source projects (Kueue, Volcano, GPU Operator) - Write blog posts/talks on GPU scheduling lessons learned - Pursue certifications: CKA (Certified Kubernetes Administrator), CKAD (Certified Kubernetes Application Developer) - Explore adjacent topics: Model serving (KServe), MLOps pipelines (Kubeflow), distributed training frameworks (DeepSpeed, Megatron-LM)

Document Metadata: - Word Count: ~15,000 words - Reading Time: ~60 minutes (executive summary + skimming modules) - Hands-On Time: 50-65 hours - Last Updated: 2026-06-03 - Target Audience: Platform engineers, SREs, ML infrastructure engineers - Prerequisite Validation: Assumes readers can deploy a Kubernetes pod and understand basic resource requests/limits

Version History: - v1.0 (2026-06-03): Initial release

Feedback and Contributions: This is a living document. If you find errors, outdated information, or have suggestions for improvement, please contribute via GitHub issues or pull requests.

License: This learning plan is released under CC BY-SA 4.0 - you are free to share and adapt with attribution.