Research Engineer (ML Infrastructure)
About
A VC-backed robotics startup in London is building a transformer-based foundation model for robotic manipulation. The team is fewer than ten people, technically ambitious and highly hands-on, with most training data generated through simulation.
Their next challenge is scaling training from the current cloud setup to hundreds of GPUs. That introduces a different class of engineering problems: distributed communication across nodes, reliable checkpointing, high-throughput storage and data loading, GPU utilisation, observability and compute cost.
They are looking for an ML Infrastructure Engineer to take ownership of this scaling work, improving both training speed and compute efficiency. You do not need to have solved every problem at this scale already. Strong ML infrastructure foundations, an understanding of how distributed systems begin to fail under load, and the appetite to learn quickly in an early-stage environment matter more than extensive experience.
What you'll do
- Build and operate the infrastructure behind multi-node GPU training, including job orchestration, scheduling, environments and recovery
- Make training reliable, so a failed node costs minutes rather than days
- Profile the stack end to end and remove the constraints limiting throughput
- Solve the less glamorous but critical problems across checkpointing, data loading, storage, networking and image builds
- Build the observability the team currently lacks, making utilisation, performance and cost visible
- Work directly with researchers and take infrastructure problems off their plate
- Grow into decisions about how a large compute budget is allocated and committed
What you'll need
- Experience building ML infrastructure, with some exposure to multi-node GPU training
- Strong Python and confidence working in a PyTorch codebase
- A working understanding of distributed training, including NCCL, FSDP or DeepSpeed, and where these systems tend to break
- Experience working with cloud GPUs, ideally on AWS
- Willingness to debug below the orchestration layer across communications, storage, I/O and hardware
- A genuine appetite for the pace, ownership and ambiguity of an early-stage startup
Bonus
- Experience supporting researchers or research engineers directly
- Kernel or performance work using CUDA or Triton
- Deeper expertise in storage or networking
- Open-source contributions to training or inference infrastructure
- Exposure to vision, multimodal or robotics model training
Shortlisted candidates will be contacted within 48 hours.