Azure Datafactory

Responsibilities

Key Responsibilities:

• Design, develop, and maintain end-to-end data pipelines using Azure Data Factory for batch and scheduled workloads.
• Build and orchestrate data transformations and processing workflows using Databricks aligned to business requirements.
• Implement robust pipeline monitoring, alerting, and failure handling to ensure reliable and repeatable executions.
• Perform data validation and reconciliation checks to ensure accuracy, completeness, and consistency across sources and targets.
• Optimize pipeline performance by tuning ADF activities, improving orchestration logic, and streamlining transformations in Databricks.
• Collaborate with cross-functional teams to gather requirements, estimate effort, and deliver solutions within timelines.
• Maintain technical documentation for pipelines, datasets, schedules, dependencies, and operational runbooks.
• Support deployments and environment promotions by following structured release and change management practices.

Minimum Qualifications:

• Education: BTECH, MTECH, MCA, MSC.
• Overall experience: 3–5 years in data engineering / data integration roles.
• Hands-on experience with Azure Data Factory including pipeline development, scheduling, triggers, and integration patterns.
• Hands-on experience with Databricks for data processing and transformation workflows.
• Ability to troubleshoot pipeline failures, analyze logs, and implement corrective actions to improve stability.

Additional Responsibilities

• Experience designing scalable orchestration patterns in ADF (parameterization, reusable components, dependency handling).
• Strong understanding of data transformation best practices and implementing efficient processing in Databricks.
• Exposure to building data quality checks and operational dashboards for pipeline health and SLA tracking.
• Experience collaborating with stakeholders to translate business requirements into technical pipeline designs and delivery plans.
• Familiarity with performance tuning approaches for cloud data pipelines and distributed processing workloads.

Technical and Professional Requirements

Azure Synapse Analytics, Azure Data Lake Storage (ADLS), PySpark, Delta Lake, CI/CD for data pipelines, AZURE DATAFACTORY

Preferred Skills

• Azure Data Factory (ADF)

Educational Requirements

MCA,MSc,MTech,Bachelor of Engineering