Software Engineer Production Inference Distributed Inference

Skills

About the Role

You will build, scale, and operate distributed production inference systems for Inkling, Inkling-Small, and Tinker. You will improve request routing, batching, multi-node serving, GPU utilization, latency, throughput, reliability, and observability. You will help translate evolving model architectures into production-ready serving systems, troubleshoot issues across the serving stack, and participate in an on-call rotation.

Requirements

• 3+ years of experience building and operating distributed systems in production
• Strong systems programming skills in Python, C++, Rust, or similar languages
• Experience with production infrastructure at scale, including reliability, observability, and performance under real-world load
• Solid understanding of networking, concurrency, and distributed systems fundamentals

Responsibilities

• Design, build, and operate distributed infrastructure for large-scale model serving, including request routing, load balancing, batching, and multi-node coordination
• Optimize production inference latency and throughput through KV cache management, continuous batching, speculative decoding, and quantization
• Build and maintain high-concurrency serving systems with strong uptime, low tail latency, and observability
• Benchmark, tune, and extend inference engines for new model architectures
• Partner with research and infrastructure teams to translate model designs into production-ready serving systems
• Build tooling to trace, debug, and resolve issues across the serving stack, from orchestration to GPU kernels
• Participate in the on-call rotation for production inference systems

Benefits

• Health, dental, and vision benefits
• Unlimited PTO
• Paid parental leave
• Relocation support as needed