Databricks Data Engineer and Scientist

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Databricks Data Engineer and Scientist based in United States.

In this role, you will design, develop, and optimize enterprise-scale data pipelines within a Databricks and AWS-based Lakehouse environment. You will transform complex data into reliable, analytics-ready assets that support informed business decisions and advanced analytical initiatives. Working across data engineering, data modeling, and AI-assisted development, you will modernize legacy systems while ensuring data accuracy, performance, and scalability. You will contribute to a Bronze, Silver, and Gold medallion architecture and build robust ETL processes with comprehensive logging and error handling. The role combines hands-on technical delivery with collaboration across cross-functional teams in support of a critical federal data environment. This is a full-time, remote opportunity for an experienced professional who enjoys solving complex data challenges and improving enterprise data infrastructure.

• Design, build, deploy, and maintain end-to-end data pipelines for an enterprise Lakehouse using Databricks and AWS.

• Develop and manage reliable ETL processes with comprehensive logging, error handling, troubleshooting, and issue resolution.

• Design and maintain data models within a Bronze, Silver, and Gold medallion architecture, selecting appropriate dimensional or relational modeling approaches.

• Analyze and decompose complex legacy SQL, rebuild pipelines and data models, and validate results for accuracy and performance.

• Optimize data pipeline performance, scalability, and query execution using partitioning, bucketing, caching, and other relevant techniques.

• Use AI-assisted coding tools to accelerate pipeline analysis, data engineering, debugging, and data modeling tasks.

• Support data security and compliance requirements while collaborating with cross-functional teams to deliver high-quality, analytics-ready data.

Requirements

• Bachelor's degree in Computer Science, Computer Engineering, Engineering, Mathematics, Physics, or Information Systems and Technology.

• At least 8 years of experience designing, developing, deploying, and managing ETL processes and data pipelines.

• At least 3 years of experience building, debugging, and optimizing data pipelines using Databricks notebooks, clusters, and job orchestration tools, including Delta Lake and Unity Catalog.

• At least 3 years of experience developing Python and PySpark code for ETL scripts and custom data transformations.

• Practical experience using AI-assisted coding tools, such as Claude Code or Cursor, to support data engineering and data modeling activities.

• Experience designing data models using dimensional and relational approaches within a Bronze, Silver, and Gold medallion architecture.

• Hands-on experience with AWS services, including AWS Glue, Amazon S3, and Amazon RDS.

• Knowledge of ETL and query performance optimization techniques, including partitioning, bucketing, and caching.

• Demonstrated ability to analyze complex legacy SQL, reconstruct data pipelines and models, and compare outputs to validate accuracy and performance.

• Ability to obtain and maintain a Public Trust or other required Suitability/Fitness determination, depending on client requirements.

• U.S. citizenship is required to support the federal contract.

Benefits

• Work arrangement: Remote position based in the United States.

• Employment type: Full-time.

• Technical environment: Opportunity to work with Databricks, AWS, Lakehouse architecture, Python, PySpark, and AI-assisted development tools.

• Compensation: Salary, bonuses, and other financial benefits were not specified in the provided job description and should be confirmed during the hiring process.

• Additional benefits: Healthcare, paid time off, retirement plans, and other employee perks were not detailed in the provided description and should be confirmed with the hiring team.

We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.

We appreciate your interest and wish you the best!

Why Apply Through Jobgether?

Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.

#LI-CL1