Senior AI Infrastructure Engineer (Model Training)

• Kodiak’s AI is only as good as the speed at which we can train it
• Every improvement to our models – from GigaFusionNet to large-scale world models – depends on infrastructure that turns thousands of hours of multimodal driving data into training throughput
• We are looking for engineers who make model training fast: streaming massive camera, LiDAR, and radar datasets without stalling a single GPU, sharding data and models efficiently across nodes, and extracting every FLOP from the latest hardware
• If you measure your impact in tokens per second and GPU utilization, this role is for you
• Design high-throughput data loading and streaming systems for multimodal sensor data (camera, LiDAR, radar), including dataset formats, sharding strategies, and prefetching pipelines that keep GPUs saturated
• Build and optimize distributed training infrastructure across multi-node GPU clusters, applying data, tensor, pipeline, and fully sharded (FSDP/ZeRO) parallelism to models that don’t fit on a single device
• Maximize utilization of modern accelerators such as NVIDIA B200s through mixed-precision training (BF16/FP8), fused kernels, memory optimization, and communication/computation overlap
• Profile end-to-end training pipelines to find and eliminate bottlenecks across storage, network, CPU preprocessing, and GPU compute
• Develop scalable dataset construction pipelines that convert petabytes of raw driving logs into training-ready, streamable formats
• Partner with ML teams to scale new architectures from prototype to full-cluster training runs efficiently and reliably

Benefits

• Remote-friendly work environment
• 401k
• Generous PTO policy
• Life insurance
• Assortment of medical (PPO/HMO/HDHP), dental, vision, and FSA plans
• Family-friendly company events
• Dog-friendly office
• Employee-driven fitness classes
• Beautiful, renovated facilities in Mountain View, CA and Lancaster, TX
• Free EV charging
• Trivia and board game nights

• BS, MS, or PhD in Computer Science or a related field, and at least 2-3 years of industry experience in ML systems or infrastructure
• Experience building high-performance data pipelines for large-scale training, including streaming dataset formats (WebDataset, MosaicML Streaming/MDS, or similar), sharding, and storage/network-aware loading
• Strong Python skills and proficiency in PyTorch internals; systems-level experience (C++/CUDA/Triton) a plus
• Passion for building the infrastructure that lets AI for the physical world train faster, scale further, and improve continuously
• Hands-on experience with distributed training frameworks and techniques (PyTorch DDP/FSDP, DeepSpeed, Megatron, NCCL) and a strong grasp of parallelism trade-offs
• Deep understanding of GPU performance: mixed precision, memory hierarchy, kernel fusion, profiling tools (Nsight, PyTorch Profiler), and interconnects (NVLink, InfiniBand)