Data Platform Engineer
Posted Updated
Description
- The Data Platform Engineer will own the reliability and operational performance of large-scale distributed systems and data infrastructure
- This role will strengthen the production operation of streaming and Kubernetes workloads while ensuring infrastructure changes are managed through repeatable, version-controlled practices
- The role will also improve the scalability and operational efficiency of the infrastructure that engineering teams depend on
- Enable critical product initiatives to launch on time. Deliver the streaming and data infrastructure required by CRE, CCIP, Data Feeds and Data Streams, translating product requirements into reliable Redpanda/Kafka and Flink capabilities
- Maintain 99.9% availability for critical data infrastructure as usage grows. Scale streaming and Kubernetes workloads, resolve throughput and recovery bottlenecks, and define SLIs that reflect the experience of consuming services
- Keep operational maintenance below 20% of team capacity. Reduce recurring incidents, manual interventions and routine maintenance through automation, actionable alerts, tested recovery procedures and repeatable vulnerability remediation
- Reduce cost to serve and make consumption accountable. Connect Redpanda topic traffic, Flink processing and BigQuery usage to the teams and services consuming them
- Increase infrastructure self-service and operational safety. Establish reusable infrastructure modules, deployment workflows and documentation that enable engineering teams to provision, test, scale and retire AWS and Kubernetes resources independently
- Use service adoption and consumer needs to guide infrastructure improvements. Establish visibility into infrastructure consumers, service adoption, reliability, cost and maintenance effort
- Deep expertise in one or more of the following technologies: Apache Flink, Redpanda/Kafka, Iceberg, PostgreSQL, Airflow, or Trino, including system internals, complex troubleshooting, or performance optimization
- Experience deploying, operating, and troubleshooting production Kubernetes workloads, including resource management, scaling, and safe rollouts
- Proven experience operating large-scale distributed systems, streaming platforms and data processing infrastructure, with a strong understanding of availability, consistency, failure modes, and performance trade-offs
- Experience building reusable infrastructure modules, self-service capabilities, and CI/CD workflows that enable engineering teams to deploy and operate services independently
- Experience building and operating Apache Flink pipelines in production, including monitoring, diagnosing failures, and managing recovery
- Hands-on experience operating AWS Aurora/RDS in production, including provisioning, backup and recovery, failover, rightsizing, and cost optimization
- Hands-on experience operating Kafka or Kafka-compatible platforms such as Redpanda in production, including capacity planning, performance tuning, and troubleshooting
- Proficiency in reading, debugging, and writing production-quality code in Java and/or Go
- Contributions to open-source infrastructure or data projects through code, tooling, documentation, or actionable bug reports
- Experience using Terraform to provision and manage production infrastructure through version-controlled, reviewed, and repeatable changes
- We carefully review all applications and aim to provide a response to every candidate within two weeks after the job posting closes