Azure Databricks Consultant

Responsibilities

Design, develop, and maintain scalable Azure Databricks solutions for enterprise data platforms.

Build and optimize end-to-end ETL/ELT pipelines using PySpark, Spark SQL, and Azure Data Factory (ADF).

Develop robust data ingestion frameworks from multiple structured and unstructured data sources.

Implement and manage Delta Lake and Lakehouse Architecture for high-volume data processing.

Create and optimize data models, transformations, and workflows to support analytical and business requirements.

Perform performance tuning and troubleshooting of Databricks workloads and Apache Spark jobs.

Work with Azure services including ADLS Gen2, Azure Synapse Analytics, Azure Key Vault, and Azure Data Factory.

Collaborate with business stakeholders, architects, and cross-functional teams to deliver data solutions.

Ensure adherence to data governance, security, and best practices.

Participate in code reviews, CI/CD implementation, testing, deployment, and production support activities.

Technical and Professional Requirements

6 to 8 years of experience in Data Engineering and Big Data technologies.

Strong hands-on experience with Azure Databricks and PySpark.

Expertise in Apache Spark, Spark SQL, and distributed data processing.

Experience with Azure Data Factory (ADF) and building enterprise-scale data pipelines.

Strong knowledge of Azure Data Lake Storage (ADLS Gen2).

Preferred Skills

• Azure Data Factory (ADF)
• Azure Virtual Machines
• Azure Application Gateway
• Databricks

Educational Requirements

MBA,MCA,ME,MSc,MTech,Bachelor of Engineering,BCA,BSc,BTech