What are the responsibilities and job description for the Sr. Data Architect position at Diverse Lynx?
Job Description
- Minimum 12 years of IT experience and at least 5 years in cloud native data engineering solution design/architecture and development.
- Strong hands-on expertise in building Data engineering solutions using Apache Spark and Scala.
- Deep expertise in Performance tuning and optimization of Spark/Scala programs.
- Ability to design and build data engineering tasks using Data bricks framework.
- Hands on experience working with Azure Cloud services like Azure Data Factory, Azure Data Lake Storage, Azure Even hub, Azure SQL, Azure Analytical services, Azure Cosmos DB,
- Should be able to demonstrate expertise in designing and building efficient and scalable data engineering solutions using Massively Parallel Processing (MPP) architecture.
- Micro-batch streaming and continuous streaming process in Databricks for high latency, low latency and ultra-low latency of data accordingly by using inbuilt Apache spark modules
- Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in Azure Databricks.
- Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL
- Experience building and optimizing big data data pipelines, architectures and data sets.
- Experience in writing HQL queries in Hive Data warehouse and performance tuning of HIVE scripts
- Well versed with Relational and Dimensional Modeling techniques like Star, Snowflake Schema, OLTP, OLAP, Normalization, Fact and Dimensional Tables
- Proficient in creating Technical Design Documents (TDD).
- Familiar with DevOps principles and experience in CD/CI implementations using Azure native services.
- Very good customer relationship and stakeholder management skills
- Excellent communication and presentation skills.
Salary : $149,123
Sr .Net Engineer
Hudson Data -
Jersey, NJ
Sr Lead Software Engineer- Solutions Architect - AWS, Spark, Data Lake
JPMorgan Chase -
Jersey, NJ
Sr. Python AI/ML Engineer
Data Wave Technologies Inc -
Morris, NJ
