Staff Senior Staff AI Infrastructure Engineer Inference Infrastructure

Skills

About the Role

You will own production reliability for a Kubernetes-native control plane that provisions and runs GPU inference infrastructure. You will operate host-lifecycle workflows, investigate networking and serving failures, extend automated remediation, manage SLOs and incident response, and turn production lessons into improvements in controllers, automation, observability, and platform design.

Requirements

• Experience operating production provisioning or host-lifecycle pipelines at fleet scale.
• Deep experience owning reliability for large-scale distributed systems.
• Hands-on InfiniBand and RDMA troubleshooting experience.
• Familiarity with multi-tenant model serving, LoRA or adapter serving, and differentiated SLOs.
• Experience eliminating operational toil through self-built automation and tooling.
• Hands-on experience with Kubernetes controllers, CRDs, reconciliation loops, and node-level remediation systems.
• Knowledge of event-driven architectures, backpressure, circuit breaking, metrics, logs, and traces.
• Ability to write and ship production code for GPU infrastructure.

Responsibilities

• Operate the production provisioning state machine and host-lifecycle pipeline.
• Diagnose failed state transitions, GPU driver and CUDA validation issues, health-check failures, and decommissioning issues.
• Investigate InfiniBand, RDMA, GPU, hero-model, and LoRA-serving failures.
• Extend automated self-healing with detection and remediation plugins.
• Own SLOs and error budgets for provisioning and fleet health and lead incident response.
• Feed production topology, interconnect, and scheduling failures into platform requirements.
• Improve inference-stack performance, latency, throughput, and efficiency.
• Build observability tools and implement solutions for bottlenecks and instability.
• Lead cross-functional reliability planning and stakeholder management.

Benefits

• Startup equity.
• Health insurance.