What are the responsibilities and job description for the Data Engineer position at Talent Groups?
Responsibilities:
1. Work with client business / technical users and offshore teams continuously
2. Design, Develop and deploy ETL processes and data pipelines using Databricks
3. Implement data ingestion from various sources into databricks, ensuring data quality and reliability
4. Build, Optimize and fine-tune Databricks workflows to improve performance and reliability
Must have skills:
1. 8 years of programming experience in PySpark, Python and SQL
2. Experienced in designing and implementing complex and scalable Data pipelines / ETL using Databricks.
3. Experienced with cloud-based data storage and processing technologies, particularly AWS services(S3, Step functions, Lambda, Airflow)
4. Familiarity with CI/CD practices, version control (Git), automated testing and Agile environment
5. Hands-on experience writing complex SQL queries, exporting, and importing large amounts of data using utilities.
