Senior Data Engineer
Posted Updated
Description
- Design,
develop, and maintain scalable ETL/ELT data pipelines using Azure Data
Factory and Azure Databricks.
- Develop
data ingestion and transformation workflows using Python, SQL, and
PySpark.
- Work
with Apache Spark and PySpark for large-scale distributed data processing.
- Implement
batch processing, incremental loading, full loading, and Change Data
Capture (CDC) mechanisms.
- Integrate
data from multiple sources, including relational databases, APIs, and
cloud storage.
- Work
with Azure Data Lake Storage Gen2 and Azure Blob Storage.
- Develop
and optimize data models, fact tables, dimension tables, and data
warehouse structures.
- Perform
data cleansing, validation, deduplication, and aggregation.
- Implement
and manage workflows using Azure Data Factory and Apache Airflow.
- Optimize
SQL queries, Spark jobs, and data pipelines for better performance.
- Work
with Azure Synapse Analytics and relational databases.
- Implement
error handling, logging, monitoring, and alerting for production
pipelines.
- Manage
version control and CI/CD pipelines using Git, GitHub, and Azure DevOps.
- Troubleshoot
production issues, perform root cause analysis, and provide production
support.
- Maintain
technical documentation and collaborate with cross-functional teams.
Requirements
Programming Languages
- Python,
SQL, PySpark.
- Pandas,
NumPy, OOP concepts, exception handling, and API integration.
Big Data Technologies
- Apache
Spark and PySpark.
- Spark
DataFrames, Spark SQL, transformations, actions, joins, aggregations, and
window functions.
- Partitioning,
caching, and performance optimization.
Azure Cloud Technologies
- Azure
Data Factory (ADF).
- Azure
Databricks.
- Azure
Data Lake Storage Gen2 (ADLS Gen2).
- Azure
Blob Storage.
- Azure
Synapse Analytics.
- Azure
Key Vault and Azure Monitor.
ETL & Data Engineering
- ETL/ELT
pipeline development.
- Data
ingestion, transformation, and migration.
- Change
Data Capture (CDC).
- Incremental
loading and full loading.
- Batch
and distributed processing.
Database Technologies
- Microsoft
SQL Server, PostgreSQL, MySQL, and Oracle.
- Complex
SQL queries, indexing, stored procedures, and query optimization.
Data Modeling & Warehousing
- Dimensional
and relational data modeling.
- Fact
and dimension tables.
- Data
normalization and denormalization.
- Schema
design and data warehouse development.
Workflow Orchestration & DevOps
- Azure
Data Factory and Apache Airflow.
- Git,
GitHub, Azure DevOps, and CI/CD pipelines.
File Formats
- CSV,
JSON, XML, Parquet, and Delta Lake.
- 4–5
years of relevant experience in Data Engineering.
- Strong
hands-on experience with Azure Data Factory, Azure Databricks, PySpark,
Python, and SQL.
- Experience
developing and optimizing scalable data pipelines.
- Good
understanding of cloud-based data engineering and data warehousing
concepts.
- Strong
analytical, debugging, and problem-solving skills.
- Good
communication and collaboration skills.
Skills
- Airflow
- Analytics
- API
- Azure
- Azure Data Factory
- Azure DevOps
- Azure Monitor
- Azure Synapse
- Big Data
- CI/CD
- Cloud
- Data Engineering
- Data Ingestion
- Data Lake
- Data Modeling
- Data Pipelines
- Data Warehousing
- Databricks
- Delta Lake
- DevOps
- ELT
- ETL
- Git
- GitHub
- JSON
- MySQL
- NumPy
- OOP
- Oracle
- pandas
- Parquet
- PostgreSQL
- PySpark
- Python
- RDBMS
- Root Cause Analysis
- Spark
- SQL
- SQL Server
- Vault
- Version Control
- Workflow Orchestration
- XML
Sign in to read and join the discussion.