Data Engineer
Posted Updated
Description
- We see a Data Engineer as a software engineer who specialises in distributed data systems
- You’ll join the Data Engineering team, whose prime responsibility is the development and operation of the Data Collection Hub, a platform that ingests data from many sources, processes/enriches it, and distributes it to multiple downstream systems
- Engineer distributed ingestion services that reliably pull data from diverse sources, handle messy real-world edge cases, and deliver clean, well-structured outputs to multiple downstream products
- Build high-throughput processing components (batch and/or near-real-time) with a focus on performance, scalability, and predictable cost, using strong profiling and measurement practices
- Design and evolve data contracts (schemas, validation rules, versioning, backward compatibility) so downstream teams can build with confidence
- Own production quality: write maintainable code, strong unit/integration tests, and add the observability you need (metrics/logs/tracing) to diagnose issues quickly
- Improve platform reliability by hardening pipelines against partial failures, retries, rate limits, data drift, and infrastructure issues—then codify those learnings into better tooling and guardrails
- Contribute to CI/CD and developer experience: faster builds, better test signal, safer releases, and automated operational checks
- Participate in design reviews, code reviews, incident retrospectives, and iterative delivery—making pragmatic trade-offs and documenting them clearly
- Technology Stack:
- Languages: Predominantly Python and Node.js
- Distributed/data platforms: HDFS, HBase, Spark, plus increasing use of Kubernetes and cloud services
- Storage/search: MongoDB, OpenSearch
- Orchestration: Airflow, Dagster, NiFi
- Tooling: GitHub, GitHub Actions, Rundeck, Jira, Confluence
- Deployment/config: Ansible (physical), Terraform / Argo CD / Helm (Kubernetes)
- Development environment: MacBook (typical)
Benefits
- Pension: Our company pension scheme is a great way of building a pot of money for your retirement and as your employer we match your contributions up to 4%
- Private Family Health insurance: We provide private medical insurance through Vitality Health. As well as giving staff general advice about eating greens and moving about, Vitality Health gives personal, achievable goals to get healthier with Vitality partners helping along the way with discounts, cashback and reward
- 25 days holidays: We offer candidates 25 days annual leave plus bank holiday per year, increasing to 30 after 5 years’ service
- EMI Share Options: We offer an employee share option scheme to recognise and reward staff members for their support, service and contribution to the business
- Flexible Working: Employees have the option to work from one of our offices located in Cheltenham, Bristol or London or to work fully remote
- Wellbeing Allowance: We provide employees with a £500 allowance, paid quarterly in the sum of £125 per quarter to support a flexible, holistic approach to employees’ well-being
- Free lunch and snacks: There is a well-stocked pantry with food, snacks and drinks. Lunch is also provided and ranges from a selection of hot and cold food
- We’re looking for someone with 2+ years of industry experience building and operating production software who enjoys working across data pipelines, distributed systems, and operational reliability
- Fluency in at least one programming language (Python/Node.js a plus)
- 2+ years building and operating production software systems
- Strong fundamentals: data structures, testing, version control, Linux basics
- Experience debugging moderately complex systems and improving reliability/performance
- Spark/PySpark experience
- Hadoop ecosystem exposure (HDFS/HBase)
- Workflow orchestration (Airflow/Dagster/NiFi)
- Search/indexing (OpenSearch, MongoDB)
- Kubernetes and infrastructure-as-code
- Degree in Computer Science or numerical degree
Skills
- Airflow
- Ansible
- Argo CD
- CI/CD
- Cloud
- Confluence
- Dagster
- Data Engineering
- Data Pipelines
- Developer Experience
- Distributed Systems
- GitHub
- GitHub Actions
- Hadoop
- HBase
- Helm
- Infrastructure as Code
- Jira
- Kubernetes
- Linux
- MongoDB
- NiFi
- Node.js
- Observability
- OpenSearch
- PySpark
- Python
- Spark
- Terraform
- Version Control
- Workflow Orchestration
Sign in to read and join the discussion.