Principal Data Engineer
Posted Updated
Description
- Octus is seeking a Principal Data Engineer to design, develop, and lead scalable data ingestion and transformation pipelines.
- This is a hands-on Individual Contributor (IC) role that encompasses some management duties, including mentoring, architecture guidance, and providing technical leadership.
- You’ll play a key role in designing and maintaining robust data infrastructure that powers data platforms, products and automation initiatives across the firm.
- Lead the overarching technical strategy for the data platform, ensuring alignment between data infrastructure and long-term business goals
- Lead the design and development of data ingestion and transformation pipelines, ensuring scalability, efficiency, and reliability across diverse data sources (APIs, web data, internal feeds, etc.)
- Serve as a foundational technical leader and mentor for senior engineers within the data platform team, guiding architecture, design, and implementation decisions
- Architect and manage data pipelines and orchestration workflows using AWS services such as MWAA (Airflow), Lambda, ECS, and SQS
- Implement and maintain infrastructure as code (IaC) using Terraform, ensuring reproducibility and compliance with cloud standards
- Partner with data analysts, scientists, and backend engineers to ensure data consistency, discoverability, and reliability
- Apply best practices in data modeling, schema design, and ETL/ELT processes for high-volume structured and semi-structured data
- Ensure data quality and lineage through automated testing, monitoring, and alerting
- Promote continuous improvement through code reviews, observability practices, and team-wide knowledge sharing
- Collaborate closely with technology leadership to align data platform development with business strategy and product goals
- Stay up to date with industry trends in data engineering, cloud architecture, AI/ML integration, and automation
Benefits
- Medical, dental, vision and 401(k) benefits
- A flexible, hybrid working policy with home office reimbursements
- Reimbursements for career development and continuing education
- Generous paid time off – including vacations, sick leave and parental leave
- Employee referral bonuses
- Gym subsidies and other fun fitness benefits
- Kudos program to recognize and reward colleagues
- Manager training programs
- Familiarity with CI/CD pipelines, automated testing, and modern DevOps practices
- Expert proficiency in Python for data pipeline and automation development
- Familiarity with containerization (Docker) and workflow orchestration best practices
- Strong understanding of data ingestion, transformation, and orchestration tools and patterns, including those used in AI/ML pipelines
- Extensive experience in a technical leadership capacity, including mentoring and leading complex data infrastructure projects end-to-end
- Strong foundation in software engineering principles, including SOLID design, modularity, and scalability
- Excellent communication, collaboration, and problem-solving skills
- Experience with data warehousing or lakehouse technologies (Redshift, Snowflake, Databricks, etc.)
- Advanced SQL skills and experience optimizing complex queries and data models
- Experience implementing and managing Terraform or similar IaC frameworks
- 8+ years of experience in data engineering or backend development, with a focus on scalable data solutions
- Proven experience designing and maintaining cloud-native data pipelines on AWS (e.g., MWAA/Airflow, Lambda, ECS, SQS, Glue, S3, Redshift, etc.)
- Exposure to streaming data technologies (Kafka, Kinesis, Flink)
- Experience integrating data quality and observability tools (Great Expectations, Monte Carlo, etc.)
- Familiarity with Scrapy, BeautifulSoup, or other data extraction frameworks for ingestion pipelines
Skills
- AI
- Airflow
- API
- Automation
- AWS
- CI/CD
- Cloud
- Cloud Native
- Containerization
- Data Engineering
- Data Ingestion
- Data Modeling
- Data Pipelines
- Data Quality
- Data Warehousing
- Databricks
- DevOps
- Docker
- ECS
- ELT
- ETL
- Flink
- Infrastructure as Code
- Kafka
- Kinesis
- Lakehouse
- Lambda
- Machine Learning
- Observability
- Python
- Redshift
- S3
- Snowflake
- SQL
- SQS
- Terraform
- Test Automation
- Workflow Orchestration
Sign in to read and join the discussion.