AI Site Reliability Engineer Sr

Standard Job Description

Your Mission:

The AI Factory team is seeking an experienced Site Reliability Engineer to lead significant reliability improvements across the platforms and services that support AI development and deployment. You will turn recurring operational problems into scalable engineering solutions through software development, automation, observability, and reliability testing.

You will own a defined technical area within the reliability roadmap—from identifying and scoping work through design, implementation, and adoption. Working with platform, application, and operations teams, you will establish reusable approaches that improve reliability across the AI Factory rather than optimizing a single tool or service.

Key Responsibilities:
• Independently lead a significant reliability capability, such as incident-response automation, platform health and observability, release validation, or resilience testing.
• Identify recurring failure modes and operational toil; design and implement software, automation, and process improvements that address their underlying causes.
• Build and maintain operational tooling that helps teams validate system health, investigate issues, and recover more effectively.
• Lead and implement approaches for standardizing and correlating metrics, logs, and traces across services, improving how teams detect and troubleshoot reliability issues.
• Define and standardize approaches for validating upgrades, releases, and platform changes, including automated testing of failure and recovery scenarios.
• Define and standardize CI/CD, GitOps, and progressive-delivery practices across teams to reduce deployment risk.
• Lead technical reviews with partner teams, communicate reliability recommendations, and coordinate implementation across team boundaries.
• Establish reusable engineering patterns, review code and designs, and mentor less-experienced engineers.
• Contribute to reliability platforms such as Cluster Concierge while retaining ownership of broader reliability outcomes, not just a single product.

Responsible for autonomy hardware and software system integration and testing including verification and validation.Translates customer requirements into product and systems specifications while addressing technical, schedule, and cost considerations; Establishes functional and technical specifications and standards for autonomous systems; Determines sensing hardware components; Recommends and selects appropriate control systems; Integrates and optimizes the autonomous compute and sensing hardware and software; Solves hardware/software interface problems; Develops plan(s) to integrate autonomous functionality into product(s) and platform(s) and other system(s); Develops test plans for validation and verification and procedures for test and evaluation requirements in collaboration with designers and developers; Performs integration testing and coordinates subsystem and/or system testing activities for autonomous programs; Documents and conducts analysis of test results and recommends fixes to software, hardware components, subsystems and systems; Interfaces with other teams involved the development lifecycle for perception

Basic Qualifications

• Experience designing, developing, and maintaining production software or automation using Go, Python, or a comparable language.
• Experience operating or engineering Kubernetes-based platforms, including troubleshooting complex service or infrastructure issues.
• Experience building or improving CI/CD, GitOps, infrastructure automation, or deployment workflows.
• Experience using observability data—including metrics, logs, or traces—to diagnose problems and improve system health.
• Demonstrated ability to own a technical initiative through design, implementation, and coordination with other teams.

Desired Skills

• Experience with OpenShift, GitLab CI/CD, Argo CD, Argo Rollouts, or similar GitOps and progressive-delivery tooling.
• Experience with Prometheus, Grafana, OpenTelemetry, or comparable observability tools.
• Experience designing automated reliability tests, upgrade validation, resilience tests, or failure-mode analyses.
• Experience improving incident response, reducing operational toil, or defining actionable service-health measures.
• Experience developing internal platforms or tools used by multiple engineering teams.
• Familiarity with AI/ML platforms, GPU-based infrastructure, or deployments in disconnected environments.
• Experience mentoring engineers and establishing reusable development or operational practices.
• Strong oral and written communication skills, and ability to collaborate with cross-functional partners
• Creative and resourceful when it comes to problem-solving
• Ability to work with internal stakeholders to collect feedback, prioritize tasks, and manage the engineering backlog
• Self-motivated, self-directed, and the ability to thrive in a fast-paced environment in an industry that constantly changes