Skip to content

InFynd

New

Senior Data Engineer

Posted Updated

Description


  • Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory and Azure Databricks.
  • Develop data ingestion and transformation workflows using Python, SQL, and PySpark.
  • Work with Apache Spark and PySpark for large-scale distributed data processing.
  • Implement batch processing, incremental loading, full loading, and Change Data Capture (CDC) mechanisms.
  • Integrate data from multiple sources, including relational databases, APIs, and cloud storage.
  • Work with Azure Data Lake Storage Gen2 and Azure Blob Storage.
  • Develop and optimize data models, fact tables, dimension tables, and data warehouse structures.
  • Perform data cleansing, validation, deduplication, and aggregation.
  • Implement and manage workflows using Azure Data Factory and Apache Airflow.
  • Optimize SQL queries, Spark jobs, and data pipelines for better performance.
  • Work with Azure Synapse Analytics and relational databases.
  • Implement error handling, logging, monitoring, and alerting for production pipelines.
  • Manage version control and CI/CD pipelines using Git, GitHub, and Azure DevOps.
  • Troubleshoot production issues, perform root cause analysis, and provide production support.
  • Maintain technical documentation and collaborate with cross-functional teams.


Requirements


Programming Languages

  • Python, SQL, PySpark.
  • Pandas, NumPy, OOP concepts, exception handling, and API integration.

Big Data Technologies

  • Apache Spark and PySpark.
  • Spark DataFrames, Spark SQL, transformations, actions, joins, aggregations, and window functions.
  • Partitioning, caching, and performance optimization.

Azure Cloud Technologies

  • Azure Data Factory (ADF).
  • Azure Databricks.
  • Azure Data Lake Storage Gen2 (ADLS Gen2).
  • Azure Blob Storage.
  • Azure Synapse Analytics.
  • Azure Key Vault and Azure Monitor.

ETL & Data Engineering

  • ETL/ELT pipeline development.
  • Data ingestion, transformation, and migration.
  • Change Data Capture (CDC).
  • Incremental loading and full loading.
  • Batch and distributed processing.

Database Technologies

  • Microsoft SQL Server, PostgreSQL, MySQL, and Oracle.
  • Complex SQL queries, indexing, stored procedures, and query optimization.

Data Modeling & Warehousing

  • Dimensional and relational data modeling.
  • Fact and dimension tables.
  • Data normalization and denormalization.
  • Schema design and data warehouse development.

Workflow Orchestration & DevOps

  • Azure Data Factory and Apache Airflow.
  • Git, GitHub, Azure DevOps, and CI/CD pipelines.

File Formats

  • CSV, JSON, XML, Parquet, and Delta Lake.
  • 4–5 years of relevant experience in Data Engineering.
  • Strong hands-on experience with Azure Data Factory, Azure Databricks, PySpark, Python, and SQL.
  • Experience developing and optimizing scalable data pipelines.
  • Good understanding of cloud-based data engineering and data warehousing concepts.
  • Strong analytical, debugging, and problem-solving skills.
  • Good communication and collaboration skills.


Skills

What Senior Data Engineering jobs ask for — and how much of it you have →
Apply

Apply to Senior Data Engineer at InFynd

Create a free account first, or go straight to the employer.

Free RoleWatch account

  • Get an email when new jobs match your search
  • Save jobs and track every application
  • See how well your CV matches each job

Already have one?

No account needed. Opens the employer's own site in a new tab.

See also

Data Engineering jobs by country — openings, pay and top skills →