Python+ETL+Pyspark

Responsibilities

ETL Development

Design, develop, and maintain ETL pipelines using Python and PySpark

Extract, transform, and load data from multiple structured and unstructured sources

Build reusable and scalable data processing frameworks

Ensure data quality, validation, and consistency

PySpark / Big Data Processing

Develop and optimize PySpark jobs for large-scale data processing

Work with Spark DataFrames and RDDs

Implement transformations, aggregations, and joins in Spark

Optimize jobs for performance and scalability

Python Development

Develop backend logic and data processing scripts using Python

Write modular, reusable, and efficient code

Integrate APIs and automate workflows

Data Management & Integration

Work with data lakes and warehouses (S3, HDFS, Redshift, Hive)

Handle file formats like Parquet, ORC, JSON, CSV

Perform data cleansing, enrichment, and transformation

Collaboration & Support

Work with data engineers, analysts, and business stakeholders

Debug and troubleshoot ETL/data pipeline issues

Participate in Agile/Scrum ceremonies

Maintain documentation and coding standards

Additional Responsibilities

Core Skills

2–5 years of experience in Python and ETL development

Hands-on experience with PySpark (mandatory)

Strong understanding of data processing and pipelines

Solid knowledge of SQL and database concepts

Technical Skills

Experience with Apache Spark ecosystem

Good knowledge of data structures and algorithms (basic to intermediate)

Familiarity with Big Data technologies (Hadoop, Hive)

Experience with version control (Git)

Understanding of REST APIs and integrations

Technical and Professional Requirements

• Primary skills:Domain->Finacle-Core-Functional->Finacle-Core-WMS->Grand Master,Technology->ETL & Data Quality->ETL – Others,Technology->Machine Learning->Python

Preferred Skills

• PYTHON
• ETL – Others
• PySpark

Educational Requirements

MCA,MSc,MTech,Bachelor of Engineering,BCA,BSc,BTech