What are the responsibilities and job description for the Big Data Lead position at Diverse Lynx?
Role: Big Data Tech Lead
Location: Schenectady, NY (100% Remote)
Contract and Full-Time Both
Responsibilities:
ESSENTIAL SKILLS:
Location: Schenectady, NY (100% Remote)
Contract and Full-Time Both
Responsibilities:
- Design & Implement Data ingestion and Data lakes-based solutions
- using Big Data Technologies.
- The Tech Lead should be highly proficient in the use of Big Data / Open-Source Technologies and standard techniques of Data Integration,
- Data Manipulation.
- Should be able to design and develop cost efficient and performance data pipelines in the cloud platform.
- Create data environment to support our data analytics, reporting and data science teams
- Experience with integration of data from multiple data sources
- Knowledge of various Data Pipeline techniques and frameworks
- Performance optimization - need to monitor the complete process and apply necessary infrastructure changes to speed up the query execution.
- Efficient data ingestion - Discovering patterns in data sets with datamining techniques and using different data ingestion APIs and inject data into the data lake as per need.
ESSENTIAL SKILLS:
- Hands on knowledge on SQL as well as SQL/NoSQL databases
- " Proficient in programming languages such as Python, PySpark, Scala and Java
- Experience with Spark
- Working knowledge of
- Experience with integration of data from multiple data sources
- Experience with NoSQL databases, such as HBase, Cassandra, MongoDB
- Knowledge of various ETL techniques and frameworks, such as Flume
- Experience with various messaging systems, such as Kafka or RabbitMQ.
- " Experience with building stream-processing systems, using solutions such as Storm or Spark-Streaming
- " Working knowledge and experience in Big data services in one of the
- Cloud Provider will be good (AWS or Azure or GCP)
Recommended Skills
- Relational Databases
- Extract Transform Load (Etl)
- Teradata Sql
- Microsoft Business Intelligence
- Microstrategy
- Data Architecture
Data Engineer
PlaceMe -
Galway, NY
IT Data Analyst
Glens Falls National Bank & Trust Company -
Glens Falls, NY
Associate for Research and Data Analytics
Iroquois Healthcare Association -
Clifton Park, NY
