What are the responsibilities and job description for the Spark position at Tekishub Consulting Services?
Spark
Request ID : 106682-1
Work Location : Plano, TX 75024 (ONSITE)
Job Details : Must Have Skills :
Must Have Skills : Spark
Spark
Databricks
AWS Cloud
Proficiency of Python, PySpark for data manipulation and visualization of large datasets.
Experience querying databases and reporting with SQL.
Experience in automating Data Engineering pipelines
Nice to have skills
Datadog monitoring & alerting for Data, Reporting and infra pipelines.
Detailed Job Description :
Key Responsibilities
Collaborate with data scientists and analysts to understand data requirements for supply chain projects.
Design, develop, and maintain scalable data pipelines using Spark, Databricks.
Work with Business Product Owners to refine the requirement and acceptance criteria.
Lead other junior data engineers within the team by mentoring, reviewing the code and providing feedback.
Implement and optimize ETL processes to extract, transform, and load data from various sources into the data lake or warehouse.
Work closely with AWS services, ensuring efficient data storage, processing, and retrieval.
Apply data quality and validation techniques to ensure accuracy and reliability of datasets.
Troubleshoot and resolve data-related issues, ensuring smooth project execution.
Collaborate with cross-functional teams to integrate data engineering solutions into the overall project framework.
Required Qualifications
7 years of experience developing data pipelines handling large volume of data.
Proficiency of Spark for data manipulation and visualization of large datasets.
Experience querying databases and reporting with SQL.
Experience in automating Data Engineering pipelines Datadog monitoring & alerting for Data, Reporting and infra pipelines.
Preferred Qualifications
Experience with Databricks, Airflow and AWS tools like Lambda, Glue, EC2, S3 Applied experience in supply chain, logistics, or related areas.
Top 3 responsibilities you would expect the Subcon to shoulder and execute :
Datadog monitoring alerting for Data, Reporting and infra pipelines.
Ability to clearly communicate technical information.
Oniste and offshore coordination.
Regards
Arun
arunkumar@tekishub.com ( 1 3023302253)
Last updated : 2024-09-25
