Season's Savings | ₹5/mo for 3 months

₹199/month after first 3 months. Renews automatically. Terms apply.
Kubernetes Bytes cover art

Kubernetes Bytes

Kubernetes Bytes

Written by: Ryan Wallner & Bhavin Shah
Listen for free

About this listen
Kubernetes Bytes is a podcast bringing you the latest from the world of cloud native data management. Hosts Ryan Wallner and Bhavin Shah come to you from Boston, Massachusetts with experienced backgrounds in cloud-native tech. They'll be sharing their thoughts on recent cloud native news and talking to industry experts about their experiences and challenges managing the wealth of data in today's cloud-native ecosystem.Ryan Wallner & Bhavin Shah
Episodes
  • Running LLMs is easy. Scaling Inference on Kubernetes is Hard!
    Nov 9 2026

    Running an open-source LLM locally is increasingly easy.

    Running that same model efficiently for hundreds or thousands of users is an entirely different engineering problem.

    In this episode of Kubernetes Bytes, Bhavin sits down with Cedric Clyburn from Red Hat to unpack what actually happens when organizations move from experimenting with local models to operating AI inference at scale on Kubernetes.

    They discuss why traditional application metrics don’t fully describe LLM performance and introduce the three important inference metrics: time to first token, inter-token latency, and throughput.

    Cedric also explains what inference servers such as vLLM actually do, how KV cache consumes GPU memory, why agentic workloads dramatically increase inference demand, and how optimizations such as continuous batching and paged attention help make expensive GPUs more efficient.

    The conversation goes deeper into GPU architecture and memory movement, including HBM, SRAM, prefill and decode, before exploring model compression and quantization as another way to reduce infrastructure requirements.

    Finally, they discuss llm-d and the emerging Kubernetes-native distributed inference stack, including KV-cache-aware routing, Day 2 model operations, model upgrades, and why AI sovereignty and compliance may drive more organizations toward self-hosted inference.

    Topics include:

    • Hosted AI APIs vs. running your own models
    • AI inference economics
    • Time to first token (TTFT)
    • Inter-token latency (ITL)
    • Token throughput
    • Why inference servers exist
    • vLLM and SGLang
    • KV cache
    • GPU memory and HBM
    • Prefill vs. decode
    • Continuous batching
    • Paged attention
    • Distributed inference
    • Model compression and quantization
    • LLM evaluation and benchmarking
    • vLLM vs. llm-d
    • Kubernetes inference-aware routing
    • Day 2 operations for AI models
    • AI sovereignty and private inference
    • Running smaller models locally on CPUs and edge devices

    If you’re a Kubernetes, platform engineering, infrastructure, or AI engineering practitioner trying to understand how production LLM serving actually works, this episode provides a practical introduction to the infrastructure underneath modern AI applications.

    Show Notes:

    • Cedric Clyburn - https://www.linkedin.com/in/cedricclyburn/
    • IBM Technology Youtube - https://www.youtube.com/@IBMTechnology
    • https://github.com/vllm-project/vllm
    • https://llm-d.ai/
    • https://github.com/kserve/kserve
    • https://gateway-api.sigs.k8s.io/
    • https://github.com/vllm-project/llm-compressor
    • https://github.com/vllm-project/guidellm
    Show More Show Less
    55 mins
  • We Built an AI Developer Platform on Kubernetes
    Oct 26 2026

    What does it actually look like when a platform team starts using AI agents in production engineering workflows?

    In this episode of Kubernetes Bytes, we sit down with Berkay from Futureplay to talk about how a small infrastructure and server engineering team is using Kubernetes to support mobile games, internal developer platforms, and increasingly AI-assisted software development.

    The conversation starts with Futureplay’s Kubernetes architecture and its philosophy of keeping infrastructure lean and avoiding unnecessary complexity. Berkay explains how the team uses a decision journal to evaluate technology choices, how they decide between managed services and self-hosting, and how they provide self-service environments for developers.

    Then we get into how AI is changing their engineering workflow.

    Futureplay has built an internal system where users can submit requests through Slack, an orchestrator interprets and scopes the request, and Kubernetes Jobs spin up specialized AI workers for building, evaluating, and verifying the change.

    The system can ask clarifying questions, create code, open a pull request, deploy a sandbox preview, collect feedback, and iterate — while still keeping human approval and security boundaries in place.

    Topics include:

    • Why Futureplay chose Kubernetes
    • Keeping Kubernetes infrastructure lean
    • Buy vs. build decisions for databases and infrastructure
    • Developer self-service and ephemeral environments
    • Using AI for Terraform, Helm, CI/CD, and infrastructure changes
    • Why AI-generated production changes still require validation
    • Agentic software development with Codex and Claude Code
    • From Slack requests to pull requests and live previews
    • Running AI workers as Kubernetes Jobs
    • Building an in-house AI orchestrator
    • Agent isolation, network policies, identity, and permissions
    • Why Futureplay uses a dedicated cluster for AI workloads
    • Why they currently choose hosted models over self-hosted GPUs
    • Measuring platform engineering success
    • Running a small team with near-five-nines availability

    If you’re trying to understand what practical AI adoption looks like inside a real engineering organization — beyond demos and vendor promises — this episode offers a detailed look at one team’s evolving approach.

    Subscribe to Kubernetes Bytes for conversations about Kubernetes, cloud native infrastructure, platform engineering, AI, security, and the CNCF ecosystem.

    Show More Show Less
    49 mins
  • What if Kubernetes Could Resume Instead of Restart? | Checkpoint / Restore Explained
    Oct 2 2026

    Kubernetes usually handles failures and rescheduling by restarting workloads. But what if Kubernetes could save the exact running state of a workload, move it somewhere else, and resume from where it stopped?

    In this episode of Kubernetes Bytes, Bhavin sits down with Radostin Stoyanov, maintainer of CRIU (Checkpoint/Restore In Userspace) and a contributor helping lead the Kubernetes Checkpoint/Restore Working Group.

    They explore how checkpoint/restore works, why it is becoming increasingly relevant for Kubernetes and AI workloads, and how the community is working toward pod-level checkpoint and restore.

    The conversation covers how checkpointing can help reduce expensive AI inference cold starts, improve fault tolerance for large GPU training jobs, enable faster model swapping, and potentially influence future Kubernetes scheduling and preemption workflows.

    They also discuss the challenges of checkpointing distributed applications, GPU memory, compression, security, encryption, and integrating checkpoint/restore directly into Kubernetes APIs.

    Topics include:

    • What checkpoint/restore actually saves
    • Application-level vs. infrastructure-level checkpointing
    • CRIU and Kubernetes
    • Why AI inference workloads can take minutes to initialize
    • Saving CPU and GPU execution state
    • Pod-level vs. container-level checkpointing
    • Kubernetes Checkpoint/Restore Working Group
    • Distributed checkpoint/restore
    • Fault tolerance for large GPU training jobs
    • Accelerating model startup and model swapping
    • Memory compression for large checkpoints
    • Kubernetes APIs and controllers for checkpoint/restore
    • Security risks of checkpoint files
    • Encryption and validation
    • Scheduler preemption and workload migration
    • Checkpoint/restore for AI-agent sandboxes


    If you work with Kubernetes, GPUs, AI infrastructure, inference, distributed training, or scheduling, this episode provides a look at a Kubernetes capability that could become increasingly important as workloads become more expensive to restart.

    Subscribe to Kubernetes Bytes for conversations about Kubernetes, cloud native infrastructure, AI, platform engineering, security, and the CNCF ecosystem.

    Show More Show Less
    49 mins
adbl_web_anon_alc_button_suppression_t1
No reviews yet