Senior Staff Inference Performance Engineer
Posted Updated
Description
- The Inference Cloud Engineering team builds the deployment and orchestration technology behind d-Matrix’s inference services: an internal cloud where teams test and tune models, and a customer-facing Cloud with self-service inference endpoints and compute
- As the team’s Inference Performance Engineer, you will own end-to-end performance of the workloads we deploy, from the moment a request arrives to the moment a result is returned
- The standard is that a workload on our orchestrated platform should perform at least as well as it does on bare metal
- Today’s workloads include speculative decoding and attention-FFN disaggregation across d-Matrix Corsair accelerators and GPUs, with more heterogeneous architectures coming
- Because there is no fixed system to master, the job is a method: profile first, form a hypothesis, test it, and then change the system
- You will own the outcome, not every fix
- Many root causes sit with other teams, so you will fix what you understand well and hand off the rest with evidence the owning team can act on
- Diagnose performance problems across the full serving path, from orchestration (scheduling, placement, networking, autoscaling) to inference internals (batching, KV-cache, prefill/decode, serving framework, driver and runtime, interconnect, host OS and NUMA, accelerator)
- Isolate each problem to a specific layer and owner, then decide whether to fix it directly, route it to the owning team, or both, with a workaround now and a tracked handoff for the permanent fix
- Write minimal, reproducible problem statements for cross-stack findings so owning teams can act without repeating the investigation
- Protect throughput, latency, and stability targets for orchestrated workloads by finding and closing gaps against bare metal, such as placement overhead, non-performance power defaults, and extra network hops
- Assess what it takes to scale from single deployments to fleet operation, including routing, placement, and shared-resource contention
- Capture each workload’s node requirements (CPU placement, memory and NUMA locality, network path) in the orchestration layer’s configuration so performant deployment is the default
- Build lightweight telemetry and benchmarks that reach into prototype stacks to isolate or hand off problems
- Turn recurring investigations into playbooks and mentor engineers in the profiling-first method
Benefits
- Equity
- Health care
- Flexible time-off
- Paid paternity leave
- 401k retirement plan
- Remote & hybrid working model
- Work with world-leading engineers and researchers
- On-site fitness center
- Team-building activities
- Free food and snacks
- Peer coaching & development
- Happy hours
- Sound judgment about ownership, with examples of correctly handing off problems that were not yours to fix
- Clear technical writing that hardware, firmware, or platform teams can act on without context from your part of the stack
- 8+ years of hands-on performance or systems engineering, including resolving hard regressions that crossed multiple layers of a stack
- Proficiency in Python and C/C++, sufficient to instrument, debug, and patch code across the stack
- Deep systems fundamentals, including Linux internals, NUMA, CPU pinning and cpusets, PCIe, memory and cache behavior, and network paths
- Experience running performance-sensitive workloads on Kubernetes, including identifying when scheduler defaults, network overlays, or orchestration abstractions cost latency or throughput, and closing the gap
- Track record of finding performance overhead through profiling in unfamiliar, sparsely documented systems, valued more than experience with any specific vendor stack or accelerator
- Fluency with AI and agentic tools for investigation, code search, log triage, and drafting instrumentation, with the discipline to verify output against real data
- Working knowledge of LLM inference serving, including batching, KV-cache, prefill/decode, quantization, and speculative decoding
- Experience with LLM serving frameworks such as vLLM, SGLang, or TensorRT-LLM
- Hands-on experience building or tuning speculative decoding, disaggregated serving, or heterogeneous serving architectures
- Experience tuning KV-cache management, request and batch scheduling, or weight prefetch and residency in an LLM serving system
- GPU programming and profiling experience (such as CUDA, Nsight, NCCL), valued as transferable background
- Experience with pre-production or bring-up hardware, or in organizations where root causes routinely cross team boundaries
- Rust experience
- Startup experience or comfort in environments where priorities shift quickly
Skills
Sign in to read and join the discussion.