Running GenAI in containers: GPU

From the creator

In this episode, Ayan Gupta is joined by Brian Benz, who demonstrates why GPUs are essential for running generative AI at scale. Just like using an electric frother versus whisking milk by hand, GPUs don't just speed things up, they make complex AI workloads practical in production. And Brian shows you exactly how to harness that power! This session tackles a real-world challenge: generating images locally using AI without relying on expensive external image services. Brian has built a Spring Boot application that uses Stable Diffusion 1.5 with ONNX Runtime and Nvidia CUDA to create images directly on your machine—or in containers on Azure. The demo generates beautiful watercolor images in about 90 seconds with GPU acceleration versus over 5 minutes on CPU alone. You'll learn the architecture behind running GenAI in containers with GPU support. Brian explains the three key components: ONNX (Open Neural Network Exchange) for model interoperability, Stable Diffusion models from Hugging Face's ONNX Community, and SD4J (Stable Diffusion for Java), an Oracle open-source library that provides Java bindings for Stable Diffusion operations. The hardest part? Making sure all the versions of ONNX Runtime, SD4J, and CUDA work together seamlessly. Brian also reveals his secret weapon: he used GitHub Copilot in agent mode with Claude Sonnet 4.5 to generate much of the integration code, turning what could have been a month-long project into just 2-3 days of work. This is a masterclass i

Published 2025-11-05 · Watch on YouTube ↗
Availability and playback restrictions are controlled by the original channel.