IBM at PyTorch 2025

Description:

Developers:

• Andre Fritolli: Open Source Developer Advocate from IBM, (specializing in CI/CD)
• Thanh Ha: Engineer from Linux Foundation (CI/CD Infrastructure)
• Jordan Conway: Engineer from Linux Foundation (CI/CD Infrastructure)
• Zhe Zhang: Distinguished Engineer from NVIDIA (CI/CD Infrastructure)
• Eli Uregas: Leads CI/CD infrastructure efforts in collaboration with PyTorch Foundation
• Andrey Talman: Engineer (PyTorch releases)
• Nikita Shulga: Core PyTorch OSS Developer and domain expert.
• Anita Katahoire: Technical Program Manager leading PyTorch release activities
• Yang Wang: Engineer (benchmarking, monitoring)
• Armen Donigian: Engineering manager

Join informal discussion, provide feedback, and uncover opportunities to collaborate.

Description:

Mert Toslali & Yu Chin Fabian Lim, IBM Research 

Training LLMs with online RL methods like GRPO presents a unique challenge: inference is required at every training step. In the standard Hugging Face TRL setup, inference is handled by vLLM running as a separate server on dedicated GPUs, communicating via HTTP. This creates a “ping-pong” inefficiency—training GPUs wait during generation, and inference GPUs wait during training—leading to poor GPU utilization and high cost.

Our talk introduces co-located vLLM, a key optimization that enables training and inference to run on the same GPUs. Built on vLLM’s external_launcher, it allows in-process, torch-compatible execution. We contributed a now-merged PR to TRL that eliminates the need for HTTP calls or separate servers. Our setup supports torchrun, TP/DP, and scales to training large models (like 72B). This setup improves training throughput by up to 1.7×, reduces # of GPUs needed, and is now part of the official TRL repo.

Description:

Andrea Frittoli, IBM

Session: Poster Presentations -  Responsible AI & Community

Description:

Cong Liu, Google; Carlos Costa, IBM

Session: Poster Presentations - Generative & Large Models

Description:

Maroon Ayoub, IBM & Tyler Michael Smith, Red Hat

Session: Poster Presentations - Generative & Large Models

Description:

Martin Hickey, IBM & Junchen Jiang, University of Chicago

Session: Poster Presentations - Generative & Large Models

Description:

Mehant Kammakomati, IBM Research; Amal Joe R S, IIT Bombay

Session: Poster Presentations - Generative & Large Models

Description:

Yidi Wu, Meta & Thomas Ortner, IBM Research Europe

Session: Poster Presentations - PyTorch Core 

Description:

Sahdev Zala, IBM

Session: Poster Presentations - PyTorch Core 

Description:

Maroon Ayoub, IBM Research & Cong Liu, Google

As PyTorch-based LLMs scale in complexity and user concurrency, their inference demands diverge across stages. Prefill is compute-heavy; decode is latency-sensitive. In this talk, we introduce a disaggregated serving pattern for PyTorch LLMs using llm-d—a Kubernetes-native, open-source framework co-developed by IBM Research, Google, and Red Hat. We'll walk through how llm-d separates prefill and decode into orchestrated sidecars, improving GPU utilization and QoS alignment. You'll learn how the Gateway API Inference Extension (GIE) enables routing based on load, cache locality, and session affinity. The talk includes real-world benchmarks and a visual demo of llm-d serving PyTorch models with vLLM across heterogeneous hardware on Kubernetes.

Description:

Burkhard Ringlein, IBM

Today, vLLM (part of pytorch) is the de-facto industry standard for serving Large Language Models. vLLM is increasingly being adopted in production and can be executed on NVIDIA GPUS, AMD GPUs, as well as custom accelerators like AWS Inferentia.

However, for most of the past, vLLM’s state-of-the-art performance war largely depending on a number of hand-written CUDA or HIP kernels. These kernels have typically been carefully optimized for a specific GPU platform and may pose a serious obstacle to the portability of vLLM across different hardware.

Leveraging Open AI Triton, we were able to introduce a “triton backend” to vllm that produces state-of-the-art performance across GPU platforms with a single code base, without involving hand written CUDA or HIP kernels.

In this talk, we will present our recent advances that lead to state-of-the-art performance on both NVIDIA and AMD GPUs with a single Triton-only code-base. We will present the engineering and science behind this triton-only backend, including autotuning for different platforms, system aspects like the launch overhead of Tritons Just-in-time compiler, and different kernel optimizations.