What are the responsibilities and job description for the Big Data Engineer position at Coventek Inc?
Responsibilities
· Extract and integrate data from multiple data systems and organize data in a format that can be easily read by human or machine.
· Transition of legacy ETLs with Java and Hive queries to Spark ETLs.
· Design and develop data processing solutions and custom ETL pipelines for varied data formats like parquet and Avro.
· Design, develop, test and release ETL mappings, mapplets, workflows using Streamsets, Java MapReduce, Spark and SQL.
· Performance tuning of end-to-end ETL integration processes.
· Analyze and recommend optimal approach for obtaining data from diverse source systems.
· Work closely with the data architects and interface with business stakeholders to understand requirements and offer solutions.
Preferred Qualifications
· BS in Information Technology, Computer Science, Software Engineering or related field.
· Understanding of distributed computing principles and hands-on experience in Big Data Analytics and development.
· Good knowledge of Hadoop and Spark ecosystems including HDFS, Hive, Spark, Yarn, MapReduce and Sqoop.
· 4 years of Big Data development experience.
· 3 years of working experience in ETL development and functional programming knowledge preferably with Scala, Spark, Java, Python, R.
· 3 years of experience tuning Spark/Java coding, SQL and No SQL.
· AWS development using big data technologies preferred.
· AWS cloud-certified, Databricks, and Snowflake experience a plus.
