Software Engineer, ML Training Optimization

• Design, implement, and operate cutting-edge ML training performance optimization techniques — distributed training (data/tensor/pipeline parallelism), mixed-precision, gradient checkpointing, and GPU kernels — to scale our VLM, VLA, and Foundational models training efficiency.

• Work on RL framework and RL performance optimization for driving foundation model

• Profile and debug training throughput bottlenecks across the compute, compiler, framework, and cluster-networking stack.

• Collaborate closely with ML researchers, data engineers, and hardware engineers to define requirements and align on architectural decisions for large-scale training infrastructure.

• Implement custom kernels and build benchmarks to track and validate training-performance optimizations across the stack.

Note: You do not have to meet all the requirements below to be considered for this position:

• 4+ years of total experience, including 2+ years working on large-scale model training platforms.

• Experience with distributed training frameworks (Megatron-LM, TorchTitan), efficiently utilizing GPU/accelerator clusters at scale.

• Experience with RL framework such as Verl/MILES and RL algorithms(GRPO, PPO)

• Experience with collective communication libraries (e.g., NCCL) and optimizing communication/compute overlap in multi-node training.

• Experience using profiling tools (NVIDIA Nsight, PyTorch Profiler, or XLA/JAX profiling) to identify and resolve training bottlenecks.

• Proficient in Python or C++.

• Experience with ML compilers, performance tooling, or hardware/software co-design (e.g., GPU/TPU/accelerator architectures).