Skip to content

Domino Data Lab

Staff Performance & Reliability Engineer

Posted Updated

Description

  • The Automation Team at Domino acts as a force multiplier for engineering, building the tools and systems that enable teams to ship code confidently and consistently
  • A core part of this mission is Tempest, an in-house platform that orchestrates realistic, long-duration workloads against live Kubernetes clusters and validates the results against real observability data
  • Today, when scale testing surfaces a bottleneck, a resource misconfiguration, or a regression in system behavior, the team can identify and report the issue — but we need someone who can take the next step: profiling services, tracing root causes through Prometheus and New Relic data, and partnering with platform engineers to drive durable fixes
  • Focused on iteration and continuous improvement, the team looks for targeted enhancements that create outsized impact, and this role will close the gap between detection and resolution at the infrastructure level
  • Serve as the technical owner of Tempest, Domino’s scale and reliability platform, ensuring it remains reliable, extensible, and aligned with evolving infrastructure needs
  • Diagnose and drive resolution of performance bottlenecks and resource misconfigurations surfaced by scale testing — working directly with platform and infrastructure teams to ship fixes, not just file tickets
  • Deliver accurate, data-driven sizing recommendations for customer-facing documentation based on rigorous empirical testing across deployment sizes
  • Strengthen observability across scale testing by improving Prometheus and New Relic instrumentation, making it faster to pinpoint root causes during and after multi-day load runs
  • Establish and operationalize scale testing on cloud platforms, ensuring appropriate sizing and configuration guidance for this increasingly divergent product line
  • Partner with platform teams to enable effective scale and reliability testing across additional cloud providers, helping position Domino for future multi-cloud success
  • Increase the efficiency and leverage of a small team by building infrastructure automation that scales operationally as the product and customer base grow

Benefits

  • Health & wellness: Premium medical, dental, and vision insurance plans, including a free option for you and your family
  • Commuter benefits: What’s your transportation of choice? We support your daily commute, whether Uber, Lyft, bus, or train
  • Love for parents: New parents receive up to eight weeks of fully paid-parental leave. P.S. Got baby pics? Share them in our Slack channel dedicated to kids
  • Flexible paid time off: We value your right to disconnect. Take the time you need to recharge, spend time with family, and explore the world
  • Annual education reimbursement: Pursue the professional growth that will help you excel in your role with educational opportunities that work for your needs, and at your speed
  • Own your outcome: Feel supported to get your work done how and when you need to, regardless of your location
  • Strong proficiency in Python and comfort working in a large, modular codebase that spans orchestration, infrastructure automation, and systems integration
  • Familiarity with performance and load testing methodologies (e.g., Locust, k6, or similar) as part of a broader infrastructure or reliability practice
  • Demonstrated ability to go beyond detection to resolution: profiling services, identifying resource bottlenecks, and working with engineering teams to ship durable fixes
  • Experience with observability stacks (Prometheus, Grafana, New Relic, or similar) — writing queries, building dashboards, and using metrics to diagnose performance and reliability issues at the systems level
  • Background in SRE, platform engineering, or infrastructure with hands-on experience operating and troubleshooting distributed systems in production Kubernetes environments
  • Clear ownership mindset — self-directed, accountable, and able to communicate priorities and status effectively in a remote, async environment

Skills

Apply

Apply to Staff Performance & Reliability Engineer at Domino Data Lab

Create a free account first, or go straight to the employer.

Free RoleWatch account

  • Get an email when new jobs match your search
  • Save jobs and track every application
  • See how well your CV matches each job

Already have one?

No account needed. Opens the employer's own site in a new tab.

See also