What are the responsibilities and job description for the Data Engineer position at Mavinsys?
Hello,
We are from Mavinsys Talent Acquisition team based on One World Trade Centre, New York. We are specializing in IT services and staffing majorly in lateral hiring/contract. Below is one of our requirement to fill immediately, if you're interested, please share your candidature to joinus@mavinsys.com
Job Title: Data Engineer
Location: Missouri, MO
Duration: 12months
Job Description;
- Proficient IT professional experience with 8 years of expertise as a Data Engineer, ETL Developer and Software Engineer includes designing,
developing, & implementing data models for enterprise-level applications.
- Worked on ETL Migration services by creating & deploying AWS Lambda functions to provide a serverless data pipeline that can be written
to Glue Catalog & queried from Athena.
- Experience in Analytics & cloud migration from on-premises to AWS Cloud with AWS EMR, S3, & DynamoDB.
- Experience in creating & managing reporting & analytics infrastructure for internal business clients using AWS services including Athena,
Redshift, Spectrum, EMR, & Quick Sight.
- Extensive expertise with Amazon Web Services such as Amazon EC2, S3, RDS, IAM, Auto Scaling, CloudWatch, SNS, Athena, Glue, Kinesis,
Lambda, EMR, Redshift, & DynamoDB.
- Proficiency in setting up the CI/CD pipelines using Jenkins, GitHub, Chef, Terraform & AWS.
- Created an Azure SQL database, monitored it, & restored it. Migrated Microsoft SQL server to Azure SQL database.
- Experience with Azure Cloud, Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, Azure Analytical services, Big Data
Technologies (Apache Spark), & Data Bricks.
- Developed ETL pipelines in & out of the data warehouse using a mix of Python & Snowflakes, SnowSQL Writing SQL queries against Snowflake.
- Extensive experience developing & implementing cloud architecture on Microsoft Azure.
- Excellent understanding of connecting Azure Data Factory V2 with a range of data sources & processing the data utilizing pipelines, pipeline
parameters, activities, activity parameters, & manually/window-based/event-based task scheduling.
- Created a connection from Azure to an on-premises data center using the Azure Express Route for Single & Multi-Subscription.
- Working knowledge in Python programming with a variety of packages such as NumPy, Matplotlib, SciPy, & Pandas.
- Integrated Jenkins with Docker container using Cloud bees Docker pipeline plugin & provisioned the EC2 instance using Amazon EC2 plugin
- Extensive experience creating Web Services with the Python programming language, including implementation of JSON-based RESTful & XML-
based SOAP web services.
- Experienced in writing complex Python scripts with Object-Oriented principles such as class creation, constructors, overloading, & modules.
- Experience establishing & maintaining multi-node development & production Hadoop clusters.
- Worked with Spark to improve the speed & optimization of current Hadoop algorithms utilizing SparkContext, Spark-SQL, Data Frame, Pair RDD,
& Spark YARN.
- Worked with the Map Reduce programming paradigm & the Hadoop Distributed File System.
