Senior Data Engineer
Posted Updated
Description
- As a Senior Data Engineer, you will be a key player in the end-to-end lifecycle of our large-scale ingestion and transformation engine. You will design, develop, scale, and maintain data pipelines that handle massive data streams, on the order of 100 TB per day. This role offers a dynamic mix of advanced greenfield data engineering and improving the stability and quality of our existing pipelines
- Design, build, and deploy scalable data pipelines that ingest data from various sources like APIs, Webhooks, and Cloud Storage
- Implement complex domain modeling and transformation logic to unify disparate data sources into high-fidelity data products
- Contribute to architectural improvements by transitioning legacy processes into modern, scalable, and stable data architectures
- Own the support and troubleshooting of live integrations, from diagnosing hotel and internal issues to communicating clearly with stakeholders during incidents
- Drive operational excellence by improving the reliability and quality of our data pipelines through advanced observability, automated testing, and automation
- Collaborate closely with product, engineering, support, and business teams, as well as customers and partners
- Investigate and resolve complex technical issues by performing deep-dive analysis into our code and systems to identify root causes
- Leverage AI and automation to increase engineering velocity, streamline complex debugging, and handle recurring tasks
- You will join one of our Integration Engineering teams as a senior data engineer. Our team’s mission is to reliably transform, integrate, and store the vast data sources that power our BI products. We handle a staggering amount of information, processing over 100TB of data daily and managing a dataset with over 3 trillion hotel rates. You’ll be part of a highly talented and collaborative group of about 6-8 data engineers, working together to power Lighthouse’s core intelligence
- Technologies you will work with: Python, dbt, airflow, Google Cloud Platform (GCP), BigQuery, Spanner, Pub/Sub, Kubernetes, Claude.
Benefits
- Vision & dental Insurance
- Work from home opportunities
- Health insurance
- You are a forward-thinking builder who uses AI-assisted tools to ship high-quality code faster
- Knowledge of the complex business of hospitality data domains
- You have strong knowledge of data cloud technologies and hands-on experience with a major cloud platform like GCP, AWS, or Azure
- You have professional proficiency in Python for large-scale data processing and pipeline development
- You have experience building and maintaining production data pipelines, with a growing interest in shaping the architecture around them
- You take pride in the software you build, from initial design to production monitoring and continuous improvement
- You are fluent in English with excellent communication skills to discuss technical details and explain complex data flows to technical stakeholders
- Not ticking every box? No problem! We value diverse backgrounds and unique skill sets, and we encourage individuals from all walks of life to apply. If your experience looks a little different from what we’ve described, but you’re passionate about what we do and are a quick learner, we’d love to hear from you!
Skills
Sign in to read and join the discussion.