Skip to content

Thinking Machines Lab

NewBe an early applicant

Software Engineer, Training Infrastructure

Posted Updated

Description

You will build the distributed training stack that runs supervised and reinforcement learning jobs reliably across large GPU clusters. You'll work on scheduling, orchestration, checkpointing and recovery, optimize throughput and cost, bring up new model architectures including mixture-of-experts, build reliability and observability, support multi-tenant LoRA fine-tuning, and collaborate with API and product teams.

Responsibilities

  • Design, build, and operate distributed training infrastructure spanning scheduling, orchestration, checkpointing, and recovery
  • Improve training throughput and cost efficiency through profiling, parallelism, memory optimization, and kernel- and communication-level tuning
  • Bring up and optimize training support for new model architectures including mixture-of-experts
  • Build reliability: failure detection, job recovery, multi-tenant isolation, and observability
  • Support multi-tenant LoRA fine-tuning and coupled sampling and training for reinforcement learning
  • Collaborate with Tinker API, frameworks, and product teams

Requirements

  • 4+ years of software engineering including building and operating production distributed systems
  • Strong Python plus experience in C++, Rust, or Go
  • Hands-on experience with large-scale model training or inference infrastructure using frameworks such as PyTorch
  • Track record of debugging complex performance and reliability issues in distributed environments

Benefits

  • Health, dental, and vision benefits
  • Unlimited PTO
  • Paid parental leave
  • Relocation support
  • Visa sponsorship

Skills

Apply

Apply to Software Engineer, Training Infrastructure at Thinking Machines Lab

Create a free account first, or go straight to the employer.

Free RoleWatch account

  • Get an email when new jobs match your search
  • Save jobs and track every application
  • See how well your CV matches each job

Already have one?

No account needed. Opens the employer's own site in a new tab.

See also

Software Engineering jobs by country — openings, pay and top skills →