Data Engineer

Description

Who We Seek:

• Passion Seekers.  You genuinely care about the work that you do and its impact on society.
• Self-Starters.  You’re a go-getter who isn’t afraid to step up and disrupt the status quo.
• Entrepreneurs.  You bring fresh ideas to the table, work hard, develop business and consistently seek new challenges.
• Collaborators.  You’re a great contributor to a high performing team that accomplishes great feats for our clients

Duties:

• Designing, implementing, and maintaining batch and streaming data pipelines between various SQL sources, including Aurora and SQL Server, and a target data lake in S3, as well as enterprise data stored in Redshift/Snowflake.
• Expertise in AWS cloud services, such as Kinesis, S3, Lake Formation, Glue, and Step Functions, to build scalable, reliable, and high-performance data pipelines that enable seamless data integration and empower data-driven insights.
• Data Pipeline Design: Design end-to-end data pipelines that efficiently extract, transform, and load data from SQL sources (Aurora, SQL Server) to the target data lake in S3 and the enterprise data in Redshift/Snowflake.
• Batch and Streaming Integration: Implement both batch and real-time streaming data integration solutions using AWS Kinesis and other relevant technologies.
• Data Transformation: Develop data transformation processes using AWS Glue or other ETL tools to harmonize, cleanse, and enrich data for analytical use.
• Data Lake Management: Oversee the setup and configuration of the data lake in S3,  applying AWS Lake Formation best practices for data organization, cataloging, and access control.
• Data Governance: Ensure adherence to data governance and security standards across the data pipelines, guaranteeing data privacy and compliance.
• Performance Optimization: Continuously monitor and optimize the performance of the data pipelines, addressing bottlenecks and ensuring efficient data processing and delivery.
• Error Handling and Monitoring: Implement error handling mechanisms and robust data monitoring to identify and resolve data pipeline issues proactively.
• Data Cataloging and Lineage: Establish and maintain data cataloging and lineage information using AWS Glue Data Catalog to enable data discoverability and traceability.
• Documentation: Create comprehensive technical documentation, including design specifications, data flow diagrams, and operational guides.
• Collaboration: Collaborate with data analysts, data scientists, and other stakeholders to understand data requirements and deliver reliable data solutions.
• Data Governance: Ensure data governance principles are implemented throughout the data pipelines to maintain data quality and integrity.

Qualifications

Education Qualification:

• Bachelor's Degree from an accredited college or university with a major in Computer Science, Information Systems, Engineering, Business, or other related scientific or technical discipline.

General Experience:

• At least six (6) years of experience working on AWS cloud-based batch and streaming data pipelines.
• Strong proficiency in AWS cloud services, including Kinesis, S3, Lake Formation, Glue, and Step Functions.
• In-depth knowledge of SQL databases, such as Aurora and SQL Server, and data lakes in S3, as well as enterprise data in Redshift/Snowflake.
• Hands-on experience with ETL tools, data transformation, and data integration techniques.
• Familiarity with data governance, data privacy, and security best practices in AWS environments.
• Strong problem-solving skills and the ability to troubleshoot complex data pipeline issues.
• Excellent communication and teamwork skills to collaborate effectively with cross functional teams.
• AWS certifications, such as AWS Certified Data Analytics - Specialty or AWS Certified Big Data - Specialty, are advantageous.

About Us:

Attain Partners is a place for great ideas and the people who have them. As a management, technology, and strategy consulting firm, our professionals provide innovative solutions to revolutionize government, education, health, and nonprofit organizations and positively impact those they serve. We are business analysts, technologists, digital strategists, managers of change, and forward thinkers, with the entrepreneurial drive to shape the future. Our team is present in 35 states and the District of Columbia.

Visit  https://attainpartners.com/careers  to explore your path forward with Attain Partners. 

Attain Partners is an Equal Opportunity/Affirmative Action employer. All qualified applicants will receive consideration for employment without regard to sex, gender identity, sexual orientation, race, color, religion, national origin, disability, protected Veteran status, age, or any other characteristic protected by law.

Applicants have rights under Federal Employment Laws. For more Information visit EEO, EEO Poster Supplement, Family and Medical Leave Act (FMLA), and Employee Polygraph Protection Act (EPPA).

If you are unable to complete this application due to a disability, contact this employer to ask for an accommodation or an alternative application process.