Solution Architect
Description
This role requires strong technical knowledge of enterprise data platforms, excellent stakeholder management skills, and proven experience delivering complex projects within the banking and financial services industry.
- 10+ years of experience.
- 5+ years managing Enterprise Data Lake, Data Warehouse or Big Data delivery initiatives.
- Proven experience in Hadoop/Cloudera environments.
- Experience managing multiple concurrent project deliveries and BAU portfolios.
- Banking and Financial Services experience preferred.
Data Engineering & Integration
· Design, develop, and maintain scalable data pipelines within the Enterprise Data Lake platform using Cloudera CDP, Hadoop, Spark, Hive, Java, Python, and SQL.
· Develop high-performance batch and near real-time data ingestion frameworks for onboarding enterprise source systems into the Data Lake.
· Develop complex transformation logic using Spark, HiveQL, SQL, and Java-based processing frameworks.
· Design and implement data ingestion patterns supporting structured, semi-structured, and unstructured datasets.
· Build and maintain batch and near real-time data integration solutions.
· Implement source-to-target mappings and ingestion specifications.
Hadoop & Cloudera Platform Development
· Develop and support solutions within the Cloudera Data Platform ecosystem.
· Build and maintain data pipelines leveraging:
· Hadoop HDFS
· Hive
· Impala
· Spark
· Kafka
· YARN, Oozie
· Perform deployment, troubleshooting, and performance tuning of applications running on the Hadoop ecosystem.
· Partner with platform teams to ensure optimal cluster utilization and workload management.
Data Quality & Governance
· Ensure adherence to Data Lake onboarding standards and governance requirements.
· Investigate and resolve data quality issues and production incidents.
· Ensure compliance with enterprise data governance standards.
DevOps & Platform Automation
· Integrate data engineering solutions into CI/CD pipelines using Git, Jenkins, Azure DevOps, or equivalent tools.
· Automate deployment, testing, monitoring, and operational support activities.
· Develop reusable frameworks and utilities to improve engineering productivity and platform scalability.
Production Support & Incident Management
· Provide L3 support for Data Lake application.
· Perform root cause analysis for production incidents and implement permanent fixes.
· Monitor pipeline execution, cluster health, and workload performance.
· Participate in release implementation, deployment validation, and post-production support activities.
· Strong hands-on experience with Hadoop ecosystem and data pipeline orchestration.
· Hadoop/HDFS
· Hive
· Spark
· Unix/Linux scripting
· Shell scripting
· ETL/ELT development experience.
· Data modelling fundamentals.
· Performance tuning and optimization.
· Experience in Data Warehousing and Data Lake environments.
· Strong Java / Python competency with experiences.
· Strong SQL development skills.
Skills
- Automation
- Azure
- Azure DevOps
- Bash
- Big Data
- CI/CD
- Cloudera
- Data Engineering
- Data Governance
- Data Ingestion
- Data Lake
- Data Modeling
- Data Pipelines
- Data Quality
- Data Warehousing
- DevOps
- ELT
- ETL
- Git
- Hadoop
- Hive
- Incident Management
- Java
- Jenkins
- Kafka
- Linux
- Python
- Root Cause Analysis
- Spark
- SQL
- Stakeholder Management
- Unix
Sign in to read and join the discussion.