Spark-Scala, Databricks

Responsibilities

Key Responsibilities:

Data Engineering & ETL Development

• Design, develop, and maintain ETL pipelines using Spark-Scala on Databricks for batch and incremental processing.
• Implement data transformations, joins, aggregations, and validations to ensure accurate and consistent outputs.
• Build reusable Spark components and follow best practices for modular, maintainable code.

Performance, Reliability & Operations

• Tune Spark jobs (partitioning, caching, shuffle optimization) to improve performance and cost efficiency.
• Monitor job execution, troubleshoot failures, and provide timely production support with root-cause analysis.
• Implement logging, error handling, and data quality checks to improve pipeline reliability.

Collaboration & Delivery

• Work with cross-functional teams to gather requirements and translate them into technical solutions.
• Participate in code reviews, documentation, and knowledge sharing to uplift team standards.
• Support release cycles by validating outputs, ensuring backward compatibility, and maintaining deployment readiness.

Additional Responsibilities

Minimum Qualifications:

• Bachelor’s/Master’s degree (BE/BTech/MSc/MCA/MTech or equivalent).
• 3–5 years of experience in data engineering or ETL development roles.
• Strong hands-on experience with Spark using Scala and working on Databricks.
• Solid understanding of ETL concepts, data transformations, and pipeline troubleshooting.
• Ability to write clean, testable code and collaborate effectively with technical and non-technical stakeholders.

Preferred Qualifications:

• Experience building end-to-end pipelines on Databricks including notebooks, jobs/workflows, and cluster configuration basics.
• Strong SQL skills and experience integrating Spark pipelines with structured data sources and curated datasets.
• Familiarity with data quality frameworks, reconciliation strategies, and automated validation checks.
• Exposure to CI/CD practices for data engineering (version control, automated testing, release management).
• Proven ability to optimize distributed workloads and deliver measurable improvements in runtime and stability.

Good to have skills:

SQL, Delta Lake, Apache Airflow, Azure Data Lake Storage (ADLS), Git

Technical and Professional Requirements

Primary skills:Technology->Big Data – Data Processing->Spark

Technology->Data Engineering->Databricks

Technology->Functional Programming->Scala

Preferred Skills

• Spark
• Scala
• Databricks

Educational Requirements

MCA,MSc,MTech,Bachelor of Engineering,BTech