Post by DeepInfra

3,661 followers

šŸ†• šŸš€ We just launched the Batch API on DeepInfra Run large inference jobs asynchronously at 20% below real-time pricing. A lot of inference work isn't latency-sensitive at all: ā†’ Embedding an entire knowledge base or product catalog for RAG ā†’ Classifying, summarizing, or labeling millions of rows in a dataset ā†’ Running a model over an evaluation set or benchmark For these workloads, paying real-time prices and managing rate limits doesn't make sense. The Batch API is built for exactly this: submit a JSONL file of requests, let the job run, get the results back within 24 hours at 20% off. A few things worth knowing: ā†’ Fully OpenAI-compatible If you've used OpenAI's Batch API, your existing code just works. Point the SDK at DeepInfra, swap in your API key, done. ā†’ Works with /v1/chat/completions, /v1/completions, and /v1/embeddings ā†’ Every model available for real-time inference can also be used in batch ā†’ Up to 50,000 requests per file, 200MB max file size ā†’ Batch usage doesn't affect your real-time rate limits The discount is applied automatically, nothing extra to configure. Read more here šŸ‘‡ ā†’ https://lnkd.in/gWAndEFc

Post content