Principal Infrastructure Engineer AI Cluster Performance and Validation
Nscale is a London-based, full-stack AI cloud and infrastructure company that provides GPU compute, managed AI services, orchestration software, data centers, and power infrastructure for AI training, fine-tuning, and inference.
Funding history
About Nscale
Nscale builds and operates vertically integrated AI infrastructure spanning software, GPU compute, networking, storage, purpose-built data centers, and power. Its active cloud platform offers self-service inference endpoints, fine-tuning, managed Kubernetes and Slurm, virtual machines, and GPU clusters.
Skills
Candidate Availability
Required and preferred rules are kept separate and reflect the wording in the original posting.
About the Role
You will define cluster acceptance criteria, run AI workloads at scale, diagnose performance failures, and build automated validation systems. You will optimize GPU, network, storage, scheduler, and host configurations, create Python tooling, and establish reliability and benchmarking standards.
Requirements
- 10+ years of relevant large-scale compute infrastructure experience
- AI workload experience at scale
- Distributed training
- PyTorch, Megatron-LM, DeepSpeed, or equivalent
- Cluster validation
- Performance debugging
- InfiniBand or RoCEv2
- RDMA
- GPUDirect
- TCP/IP
- BGP
- Linux systems
- Python
- C, C++, or Go
- Ansible or Terraform
- SLURM or Kubernetes
Responsibilities
- Define architecture, roadmaps, acceptance criteria, and performance standards for GPU clusters
- Run distributed training and inference workloads to validate cluster behavior
- Diagnose cluster failures and performance regressions across the full stack
- Design and build automated validation and burn-in systems
- Optimize fabric, communication, storage, and host-level configurations
- Translate performance needs into engineering solutions with partner teams
- Build Python systems for automated triage, telemetry correlation, and regression detection
- Establish standards for reliability, observability, benchmarking, and operational excellence
- Mentor engineers and produce runbooks and post-incident write-ups