Senior AI R&D Engineer Performance Pre-training
German enterprise AI company building and deploying specialized large language models for sovereign, mission-critical use cases.
About Aleph Alpha GmbH
Aleph Alpha designs, builds, and deploys specialized large language models with enterprises and public institutions, emphasizing European infrastructure, data sovereignty, compliance, explainability, and domain-specific performance.
Skills
Candidate Availability
Required and preferred rules are kept separate and reflect the wording in the original posting.
About the Role
You will engineer systems for training foundation models at scale. You will profile and optimize training loops, tune distributed parallelism strategies, and work with researchers to develop hardware-efficient model architectures. You will remove bottlenecks from Python layers through GPU kernels while improving utilization and throughput on large GPU clusters.
Requirements
- Proficiency in Python and PyTorch.
- Strong engineering background in parallel and/or distributed systems with a proven record of excellence.
- Hands-on experience with modern machine learning techniques, especially large language models and their lifecycle.
- Deep understanding of the CUDA programming model.
- Experience with distributed programming APIs such as NCCL or MPI.
- Experience analyzing profiling traces with PyTorch Profiler and Nvidia Nsight.
- Regular on-site collaboration in Heidelberg.
- Contributions to distributed training frameworks such as TorchTitan, Megatron-LM, or DeepSpeed.
- Familiarity with low-precision training formats including MXFP4 and MXFP8.
- Understanding of NCCL communication primitives, NVSHMEM, or CUDA IPC.
- Experience implementing and optimizing transformer-based model training.
- Experience with NVIDIA Blackwell architecture.
Responsibilities
- Profile training loops with PyTorch Profiler, Nsight Systems, and Nsight Compute to identify system- and kernel-level bottlenecks.
- Configure and tune composite parallelism strategies for large-scale LLM training.
- Optimize load balancing, critical-path bottlenecks, and communication-to-computation trade-offs.
- Partner with AI researchers to define hardware-efficient model architectures without compromising convergence.
Benefits
- 30 days of paid vacation.
- Fitness and wellness offerings through Wellhub.
- Mental health support through nilo.health.
- Subsidized company pension plan.
- Subsidized Germany-wide transportation ticket.
- Budget for additional technical equipment.
- Flexible working hours.
- Hybrid working model.
- JobRad bike lease.