Skip to content

d-Matrix

Senior Staff Inference Performance Engineer

Posted Updated

Description

  • The Inference Cloud Engineering team builds the deployment and orchestration technology behind d-Matrix’s inference services: an internal cloud where teams test and tune models, and a customer-facing Cloud with self-service inference endpoints and compute
  • As the team’s Inference Performance Engineer, you will own end-to-end performance of the workloads we deploy, from the moment a request arrives to the moment a result is returned
  • The standard is that a workload on our orchestrated platform should perform at least as well as it does on bare metal
  • Today’s workloads include speculative decoding and attention-FFN disaggregation across d-Matrix Corsair accelerators and GPUs, with more heterogeneous architectures coming
  • Because there is no fixed system to master, the job is a method: profile first, form a hypothesis, test it, and then change the system
  • You will own the outcome, not every fix
  • Many root causes sit with other teams, so you will fix what you understand well and hand off the rest with evidence the owning team can act on
  • Diagnose performance problems across the full serving path, from orchestration (scheduling, placement, networking, autoscaling) to inference internals (batching, KV-cache, prefill/decode, serving framework, driver and runtime, interconnect, host OS and NUMA, accelerator)
  • Isolate each problem to a specific layer and owner, then decide whether to fix it directly, route it to the owning team, or both, with a workaround now and a tracked handoff for the permanent fix
  • Write minimal, reproducible problem statements for cross-stack findings so owning teams can act without repeating the investigation
  • Protect throughput, latency, and stability targets for orchestrated workloads by finding and closing gaps against bare metal, such as placement overhead, non-performance power defaults, and extra network hops
  • Assess what it takes to scale from single deployments to fleet operation, including routing, placement, and shared-resource contention
  • Capture each workload’s node requirements (CPU placement, memory and NUMA locality, network path) in the orchestration layer’s configuration so performant deployment is the default
  • Build lightweight telemetry and benchmarks that reach into prototype stacks to isolate or hand off problems
  • Turn recurring investigations into playbooks and mentor engineers in the profiling-first method

Benefits

  • Equity
  • Health care
  • Flexible time-off
  • Paid paternity leave
  • 401k retirement plan
  • Remote & hybrid working model
  • Work with world-leading engineers and researchers
  • On-site fitness center
  • Team-building activities
  • Free food and snacks
  • Peer coaching & development
  • Happy hours
  • Sound judgment about ownership, with examples of correctly handing off problems that were not yours to fix
  • Clear technical writing that hardware, firmware, or platform teams can act on without context from your part of the stack
  • 8+ years of hands-on performance or systems engineering, including resolving hard regressions that crossed multiple layers of a stack
  • Proficiency in Python and C/C++, sufficient to instrument, debug, and patch code across the stack
  • Deep systems fundamentals, including Linux internals, NUMA, CPU pinning and cpusets, PCIe, memory and cache behavior, and network paths
  • Experience running performance-sensitive workloads on Kubernetes, including identifying when scheduler defaults, network overlays, or orchestration abstractions cost latency or throughput, and closing the gap
  • Track record of finding performance overhead through profiling in unfamiliar, sparsely documented systems, valued more than experience with any specific vendor stack or accelerator
  • Fluency with AI and agentic tools for investigation, code search, log triage, and drafting instrumentation, with the discipline to verify output against real data
  • Working knowledge of LLM inference serving, including batching, KV-cache, prefill/decode, quantization, and speculative decoding
  • Experience with LLM serving frameworks such as vLLM, SGLang, or TensorRT-LLM
  • Hands-on experience building or tuning speculative decoding, disaggregated serving, or heterogeneous serving architectures
  • Experience tuning KV-cache management, request and batch scheduling, or weight prefetch and residency in an LLM serving system
  • GPU programming and profiling experience (such as CUDA, Nsight, NCCL), valued as transferable background
  • Experience with pre-production or bring-up hardware, or in organizations where root causes routinely cross team boundaries
  • Rust experience
  • Startup experience or comfort in environments where priorities shift quickly

Skills

Apply

Apply to Senior Staff Inference Performance Engineer at d-Matrix

Create a free account first, or go straight to the employer.

Free RoleWatch account

  • Get an email when new jobs match your search
  • Save jobs and track every application
  • See how well your CV matches each job

Already have one?

No account needed. Opens the employer's own site in a new tab.

See also