What are the responsibilities and job description for the Data Engineer position at ClifyX?
Job Description
Job Title: Data Engineer - Databricks Experience
Location: New Jersey (Remote)
Job Type: Contract/Fulltime
JD :
Experience with PySpark or Python
DataBricks experience is mandatory
Experience with big data tools Spark, Hadoop Kafka would be good to have
Hands on experience in Microsoft Azure Data Factory Azure Logic apps service Data Bricks
Experience integrating different data sources
Should have experience in Azure services e.g., Azure Data factory, Azure Data Lake (Gen 2) , Azure SQL, Blob Storage and Azure DevOps, ARM
Should have experience in Azure Databricks PySpark, Databricks Delta, n ice to have experience in Scala, python
Experience building data pipeline by using Azure Big data stack
Extensive ingesting, cleaning, transformation, and aggregation of massive amounts of data from multiple internal and external sources (Salesforce, Google Analytics, etc.) by using Azure Services
Create data pipeline that enable various systems within ecosystem to stream high volume data from multiple sources into central repository for processing
Reporting at scale enabled by Databrick s Delta for incremental processing
Understanding of spark framework and tuning of spark applications Extensive experience with horizontally scalable and highly available system design and implementation, with focus on performance and resiliency
Coordinate with all stakeholders and understand the requirement and meet the delivery deadlines
I appreciate your time. Have a nice day!
