Skip to content

Graphcore

Site Reliability Engineering Lead

Posted Updated

Description

  • We are seeking an experienced Site Reliability Engineering leader to build and lead a new SRE organization responsible for the production operation of a rapidly scaling AI supercomputing platform. The environment combines highly customized compute, high-performance networking, storage and supporting infrastructure, and will grow through multiple phases of deployment
  • This is a rare opportunity to establish the reliability function for a new platform from the ground up. The platform and its operational model are being developed in parallel and will ultimately support a 24x7x365 production service with stringent availability requirements
  • You will take the SRE organization from initial formation through production launch, stabilization and scale. This includes hiring and developing the team, defining the operating model, establishing production readiness and incident-management practices, and ensuring reliability and operability are engineered into the platform from the outset
  • SRE is responsible for the operational capability required to run the platform reliably in production, while partnering with engineering teams that remain accountable for the reliability and operability of the systems they build
  • This is not a purely managerial position. During the development and early production phases, the SRE Manager will be expected to work directly with engineering teams, develop a deep understanding of the platform, and participate in troubleshooting and incident response
  • Over time, success will increasingly mean building the people, processes, automation, tooling, and operational discipline that allow the organization to operate effectively without depending on you for day-to-day escalation
  • Build the SRE Organization:
  • Build and develop the team from its initial formation through full 24x7x365 production operations, including defining roles, interviewing and hiring team members, establishing career expectations, and developing future technical leaders
  • Mentor engineers and team leads, develop successors, and build an organization capable of operating effectively without depending on any single individual
  • Work with leadership to forecast staffing requirements as the platform grows from initial deployment through full production scale
  • Establish the Production Operating Model:
  • Define the operating model for an SRE organization, including staffing and coverage model, escalation paths, on-call responsibilities, incident management, handoffs, production access, change management, and operational readiness requirements
  • Establish clear operational interfaces with Datacenter Operations, engineering teams, vendors, and other service owners
  • Establish and continuously improve production readiness standards, runbooks, operational procedures, failure-mode documentation, escalation processes, and incident response practices with a strong emphasis on automation and engineering over manual operational work
  • Develop training, cross-training, simulation, and production incident-response exercises to ensure the team can operate independently and confidently
  • Engineer Reliability Into the Platform:
  • Embed with platform engineering teams during development to gain deep technical knowledge of the system and ensure reliability, serviceability, observability, and operational requirements are incorporated into the platform before production
  • Lead the development of SLOs, operational health indicators, alerting standards, incident severity definitions, and reliability reporting appropriate for a large-scale production infrastructure service
  • Build a culture in which recurring operational problems are engineered out through automation, improved observability, better platform design, and elimination of unnecessary toil
  • Ensure the SRE organization can rapidly diagnose and mitigate issues across compute, networking, storage, and supporting infrastructure
  • Develop strong technical depth within the team while maintaining access to specialist expertise in critical areas such as high-performance networking, storage, observability, and platform scheduling
  • Lead Production Operations:
  • Lead or participate in major production incidents as necessary, particularly during platform development, launch, and early production
  • Establish a blameless post-incident review process focused on identifying systemic improvements and ensuring corrective actions are completed
  • Serve as the senior operational authority for the SRE organization and represent production reliability concerns in engineering and leadership discussions
  • What Success Looks Like:
  • The SRE organization is staffed, trained and capable of supporting the platform through production launch and subsequent growth
  • Production readiness, observability, incident management and operational engineering practices are embedded into how the platform is developed and operated
  • The team has the technical depth, automation and operating discipline to resolve most production issues independently, with leadership escalation reserved for genuinely exceptional events

Benefits

  • Work-Life Balance: We believe you should have a good work-life balance. Everyone’s situation is different, so we offer flexible working arrangements and start and finish times, allowing you to choose when to work from home or come into the office
  • Healthcare: We offer Private Medical Insurance (PMI) to our employees to help them take care of their health and that of their loved ones
  • Pension: We believe that our employees deserve the best when it comes to planning for their future
  • Income Protection: Graphcore understands that unforeseen circumstances can happen, and we want to ensure our employees have the necessary financial security in place
  • Progression: We prioritise learning and development as an integral part of our culture at Graphcore. Our Talent Development team is dedicated to equipping all our employees with the necessary tools, resources, and support to enhance their skills and reach their full potential in their careers
  • Wellbeing: We want to ensure that our employees have the resources and support they need to prioritise their health and wellbeing
  • Significant experience leading or building an SRE, Production Engineering, Infrastructure Reliability, or comparable function supporting large-scale, highly available production infrastructure
  • Strong judgment regarding when to solve an immediate operational problem and when to invest in eliminating the underlying failure mode
  • Strong written and verbal communication skills, particularly during incidents and when communicating technical risk to senior leadership
  • A strong systems engineering background with a working understanding of Linux, networking, storage, distributed systems, automation, and production infrastructure
  • Ability to operate effectively at both the leadership and hands-on technical levels, moving comfortably between organizational design, architecture discussions, troubleshooting, and incident response
  • Ability to manage conflicting priorities under a pressured environment
  • Strong incident leadership experience, including managing high-severity, multi-team production incidents under significant time pressure
  • Experience building and operating sustainable 24x7x365 production support or on-call organizations
  • Experience taking a new or rapidly evolving platform through production readiness, launch, stabilization, and ongoing operation
  • Demonstrated ability to lead, hire, mentor, develop, and retain a growing team of strong technical engineers at various levels of capability
  • Strong understanding of modern Site Reliability Engineering principles, including SLOs, incident management, observability, automation, toil reduction, capacity management, and production readiness
  • Candidates are not expected to have experience in all the areas below. Experience in several would be particularly valuable:
  • High-performance networking technologies such as InfiniBand, RDMA, RoCE, or large-scale Ethernet fabrics
  • Large-scale parallel or distributed storage systems
  • Workload schedulers and orchestration platforms such as Kubernetes, Slurm, or comparable systems
  • Production observability and service-level monitoring for complex distributed infrastructure
  • Operational relationships and escalation processes involving datacenter operations and infrastructure/hardware/networking vendors
  • Custom or early-generation hardware, firmware, or environments where hardware and software are developed concurrently
  • Rapid capacity expansion, datacenter migration, or transitions between temporary and permanent production environments

Skills

What Lead SRE jobs ask for — and how much of it you have →
Apply

Apply to Site Reliability Engineering Lead at Graphcore

Create a free account first, or go straight to the employer.

Free RoleWatch account

  • Get an email when new jobs match your search
  • Save jobs and track every application
  • See how well your CV matches each job

Already have one?

No account needed. Opens the employer's own site in a new tab.

See also

SRE jobs by country — openings, pay and top skills →