Senior Software Engineer - Model Performance
Inference.net provides fully managed AI infrastructure for deploying, observing, tracing, training, and evaluating open-source, custom, and fine-tuned AI models at scale. It serves AI-native product teams looking to switch from providers like OpenAI, Anthropic, and Gemini to optimized infrastructure for lower cost and faster performance.
Funding
Projects
About Inference.net
Inference.net offers a suite of infrastructure products for AI-native teams, including Deploy (turn-key, globally distributed model hosting with high uptime), Observe (LLM observability with monitoring, tracing, and debugging tools), Trace (capturing agent LLM calls, tool calls, and framework steps), Train (custom model fine-tuning in days), Evaluate (rigorous benchmarking before production deployment), and HALO (open-source agent optimization). The platform hosts models such as GLM-5.2, Kimi K2.6, MiniMax-M2.5, and GPT-OSS 120B, and offers proprietary open-source models like ClipTagger and Schematron. Clients include engineering teams such as Cal AI and Gravity Ads, who use the platform to cut latency and costs while maintaining frontier-level model quality. The company is SOC 2 Type II compliant, indicating a focus on production-grade, enterprise-ready AI infrastructure.
Skills
About the Role
You will be responsible for making our inference stack as fast and efficient as possible. Your work spans from implementing known optimization techniques to experimenting with novel approaches, always with the goal of serving models faster and cheaper at scale. Your north star is inference performance: latency, throughput, cost efficiency, and how quickly new model architectures can be brought into production. You'll work across the full inference stack—from CUDA kernels to serving frameworks—to find and eliminate bottlenecks. This role reports directly to the founding team. You'll have the autonomy, a large compute budget, and technical support to push the limits of what's possible in model serving.
Requirements
- 2+ years of experience in ML systems, inference optimization, or GPU programming
- Strong proficiency in Python and familiarity with C++
- Hands-on experience with LLM inference frameworks (vLLM, SGLang, TensorRT-LLM, or similar)
- Deep understanding of GPU architecture and experience profiling GPU workloads
- Familiarity with LLM optimization techniques (quantization, speculative decoding, continuous batching, KV cache management)
- Experience with PyTorch and understanding of how models execute on hardware
- Track record of measurably improving system performance
Responsibilities
- Implement and productionize optimization techniques including quantization, speculative decoding, KV cache optimization, continuous batching, and LoRA serving
- Deep dive into inference frameworks (vLLM, SGLang, TensorRT-LLM) and underlying libraries to debug and improve performance
- Profile and optimize CUDA kernels and GPU utilization across serving infrastructure
- Add support for new model architectures, ensuring they meet performance standards before going to production
- Experiment with novel inference techniques and bring successful approaches into production
- Build tooling and benchmarks to measure and track inference performance across the fleet
- Collaborate with applied ML engineers to ensure trained models can be served efficiently
Benefits
- Equity
- Comprehensive benefits