What are the responsibilities and job description for the Data Engineer position at DATAMAXIS?
Rate: depending on experience
Work location: MI (hybrid)
We are seeking a highly skilled and certified Data Engineer with expertise in Big Data technologies, Kafka, and Google Cloud Platform (GCP). The ideal candidate will have a strong background in designing, building, and maintaining scalable data pipelines, as well as experience with real-time data processing. You will be responsible for ensuring the reliability, efficiency, and scalability of our data infrastructure.
Key Responsibilities:
- Design, develop, and maintain scalable data pipelines and ETL processes using Big Data technologies.
- Implement and manage real-time data streaming solutions using Apache Kafka.
- Architect and manage data storage solutions on Google Cloud Platform (GCP), ensuring high availability and reliability.
- Collaborate with data scientists, analysts, and other engineering teams to integrate data solutions into various applications.
- Optimize data systems and pipelines for performance, scalability, and cost-efficiency.
- Ensure data quality and integrity across all data pipelines.
- Monitor and troubleshoot data pipelines to ensure continuous and reliable data flow.
- Document data engineering processes and maintain comprehensive records of all data infrastructure components.
- Stay up to date with the latest trends and best practices in data engineering, Big Data, Kafka, and GCP.
Qualifications:
Certification: Google Cloud Certified - Professional Data Engineer or equivalent certification.
Experience: 3 years of experience in data engineering with a focus on Big Data technologies.
Technical Skills:
- Proficiency in Apache Kafka for real-time data streaming.
- Hands-on experience with Google Cloud Platform (GCP) services such as BigQuery, Dataflow, Pub/Sub, Cloud Storage, etc.
- Strong knowledge of ETL processes and data pipeline orchestration tools.
- Experience with distributed data processing frameworks like Apache Hadoop, Apache Spark, etc.
- Proficiency in programming languages such as Python, Java, or Scala.
- Experience with SQL and NoSQL databases.
