Data Engineer

Job Title: Data Engineer

Location: Chicago, IL

Contract: W2 Only

Duration: 12 Months

Job Overview: The Data Engineer will be responsible for building and maintaining scalable data pipelines and data integration solutions using various technologies, including Big Data tools, Java, Python, SQL, Google Cloud Platform, AWS, and Snowflake. The ideal candidate will have a strong background in data engineering, experience with cloud platforms, and a solid understanding of data mining and data mapping techniques. This role requires a detail-oriented professional with excellent problem-solving skills who is passionate about data and technology.

Key Responsibilities:

1. Data Pipeline Development:
   • Design, build, and maintain scalable data pipelines for extracting, transforming, and loading (ETL) data from various sources into data warehouses and data lakes.
   • Develop and optimize ETL processes using Java, Python, and SQL to ensure high performance and data integrity.
   • Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver robust solutions.
2. Big Data Engineering:
   • Work with Big Data technologies to handle large datasets and perform data processing tasks.
   • Implement data integration and data processing solutions using tools such as Hadoop, Spark, and Kafka.
   • Develop and maintain data models and data architecture that support analytical and business intelligence needs.
3. Cloud Platform Expertise:
   • Design and implement cloud-based data solutions using platforms such as Google Cloud Platform (Google Cloud Platform) and Amazon Web Services (AWS).
   • Develop and maintain data warehouses and data lakes on cloud platforms, ensuring scalability, reliability, and security.
   • Leverage cloud-native tools and services (e.g., BigQuery, Redshift, Snowflake) for data storage, processing, and analytics.
4. Database Management:
   • Manage and optimize relational databases, including MySQL, Oracle, and SQL Server.
   • Develop complex SQL queries and stored procedures for data extraction, transformation, and analysis.
   • Ensure database performance, reliability, and security through effective monitoring and optimization techniques.
5. Data Visualization and Reporting:
   • Develop and maintain interactive dashboards and reports using Tableau or similar BI tools to provide actionable insights to business users.
   • Collaborate with business stakeholders to define key performance indicators (KPIs) and design visualizations that effectively communicate data trends.
6. Data Mining and Mapping:
   • Perform data mining and analysis to uncover patterns, correlations, and insights that drive business decisions.
   • Develop data mapping specifications to ensure accurate data integration and transformation processes.
   • Work with data governance teams to ensure data quality, consistency, and compliance with regulatory requirements.
7. Collaboration and Communication:
   • Work closely with cross-functional teams, including data scientists, business analysts, and product managers, to deliver high-quality data solutions.
   • Provide technical guidance and mentorship to junior data engineers and team members.
   • Document data processes, architectures, and technical specifications to facilitate knowledge sharing and collaboration.

Qualifications:

• Experience:
• 12 years of experience in data engineering, data integration, and data management.
• Strong experience with Big Data technologies (Hadoop, Spark, Kafka) and cloud platforms (Google Cloud Platform, AWS).
• Proven experience in developing data pipelines and ETL processes using Java, Python, and SQL.
• Technical Skills:
• Proficiency in SQL and database management (MySQL, Oracle, SQL Server).
• Experience with data warehousing solutions like Snowflake, BigQuery, or Redshift.
• Familiarity with data visualization tools such as Tableau.
• Strong understanding of data modeling, data architecture, and data governance principles.
• Soft Skills:
• Excellent problem-solving and analytical skills.
• Strong communication skills, both verbal and written.
• Ability to work independently and as part of a collaborative team.
• Detail-oriented with a focus on data accuracy and quality.

Preferred Qualifications:

• Experience with NoSQL databases (e.g., Cassandra, MongoDB).
• Knowledge of machine learning and data science concepts.
• Familiarity with DevOps practices and CI/CD pipelines.
• Advanced degree in Computer Science, Data Science, or a related field.