Post by DeepInfra

When we built DeepInfra, we made a deliberate bet on NVIDIA’s inference ecosystem. Not as a hedge as a conviction. Here’s what that looks like in practice: We run TensorRT-LLM and NVIDIA Dynamo for distributed serving, and use ModelOpt to quantize models to NVFP4 on Blackwell GPUs including B300s. When DeepSeek V4 dropped, we served it in production on day 0. After measuring a 4x performance increase on B300 over H200, we migrated quickly. A workload that previously needed 4×H200 now runs on a single B300 at higher tokens per second. The reason this works isn’t just the hardware. The components of the NVIDIA inference software stack work together...NVFP4 reduces memory pressure. Dynamo handles routing and disaggregated serving. TensorRT-LLM optimizes the kernels for the hardware underneath. When they work together, software improvements keep flowing through to production without developers having to do anything. NVIDIA published a blog today on exactly this: how their full inference software stack turns hardware potential into delivered token economics. We’re featured in it. https://lnkd.in/gpj4CyDN We also wrote our own post on why we made this bet and what it’s delivered. https://lnkd.in/gCKgPaee