Software Engineer, Model Runtime

Skills

About the Role

You will build the model runtime within the inference engine that executes frontier models at scale on OpenAI's custom silicon. You'll design scheduling, batching, memory and KV-cache management, distributed execution, and profiling tools, working with kernel, compiler, and silicon teams.

Requirements

• Strong systems programming experience in C++, Rust, Python, or comparable environments
• Experience building or optimizing runtimes, distributed systems, compilers, kernels, or model-serving infrastructure
• Understanding of modern LLM inference including prefill, decode, batching, KV-cache, and model parallelism
• Ability to reason quantitatively about latency, throughput, and memory bandwidth
• Experience profiling and debugging across hardware-software stack

Responsibilities

• Design and implement the LLM inference runtime for frontier models on custom silicon
• Build scheduling, continuous batching, memory management, KV-cache management, and execution orchestration
• Develop distributed execution strategies across chips, hosts, and racks
• Optimize latency, throughput, memory efficiency, and hardware utilization
• Partner with kernel, compiler, architecture, and silicon teams
• Enable new model features, execution patterns, numerical formats, and hardware capabilities
• Create profiling, observability, benchmarking, and performance-modeling tools
• Debug complex correctness, performance, and reliability issues
• Turn workload insights into requirements for future silicon

Benefits

• Generous equity
• Performance-related bonus
• Medical, dental, and vision insurance with HSA employer contributions
• Pre-tax Health FSA, Dependent Care FSA, and commuter accounts
• 401(k) retirement plan with employer match
• Paid parental leave (up to 24 weeks for birth parents and 20 weeks for non-birthing parents)
• Paid medical and caregiver leave (up to 8 weeks)
• Flexible PTO for exempt employees
• 13+ paid company holidays and coordinated company office closures
• Paid sick or safe time
• Mental health and wellness support
• Employer-paid basic life and disability coverage
• Annual learning and development stipend
• Daily meals in offices and meal delivery credits
• Relocation support for eligible employees
• Charitable donation matching and wellness stipends