Senior Data Engineer

Remote an option with onsite visits for critical meetings as needed.

Phone/Skype Hire.

Location: Brooklyn, NY

Duration: 12 months (35 hrs/week)

For the initial phases, the Senior Data Engineer will assist the Application Engineering in building a robust, secure, and modern data pipelines to ingest, process, transform applications data using Informatica Intelligent Cloud Services or a comparable ETL/ELT tool employing modern data movement strategies and methodologies. The data needs to be store in an Azure Data Lake and eventually brought into a cloud-based data store such as Snowflake or a similar data store. Then, the Analytics reporting solution needs to be built either using cloud-based Google Looker or Microsoft Power BI or another similar tool.

Using the Cloud OTI Data Platform, the Senior Data Engineer will build highly available, robust, concrete data pipelines and reporting solutions following industry best practices, adhering to OTI security guidelines, modifying, and running automated CI/CD pipelines used for releasing code, and modifying and executing terraform modules for deploying infrastructure components, working collaboratively under the direction of OTI data engineering management and leads. The resource will be a person with integrity who is dependable and is fully focused on delivering optimal solutions with little to no maintenance and operations overhead.

Responsibilities

• Be experienced in Data Engineering best practices, technologies, tools and processes.
• Bring sound knowledge of Data Warehouses and LakeHouse concepts and practical implementation experience.
• Build a framework of repeatable solutions and playbooks enabling efficient and predictable data pipelines.
• Have hands-on development experience in the implementation of an agile, cloud centric data warehousing and reporting platform with team members of various experience level.
• Interact with clients, both technical and non-technical stakeholders.
• Handle relationships with end users. Interact regularly to gather feedback, listen to their issues and concerns, and recommend solutions.
• Meet critical deadlines and deliver in short sprints.
• Ensure successful delivery of new reports and dashboards as needed.
• Maintain and curate data documentation including Architectural Decision Records (ADR), how-to guides, data lineage and ownership using Azure DevOps or similar tool.
• Maintain query performance and tuning to ensure cost optimization.
• Participate in joint application development sessions with co-engineers and end users and be willing to brainstorm.
• Complete technical documentation and be willing to transfer knowledge as needed.
  
  

Skills Needed

12 years developing Data Pipelines / Flows using ETL/ELT tools and technologies.

10 years of strong SQL fluency (query optimization, windowing functions, aggregation, etc.).

5 years building complex Analytics and Reporting solutions.

3 years' experience with a cloud data lake/warehouse solution (Snowflake, Redshift, GCP etc.).

Hands on experience working with data integration tools like Informatica Intelligent Cloud Services, Informatica Power Center or SSIS or a similar tool.

Extensive experience developing production grade, large scale data solutions.

Experience performing conceptual, logical, and physical data modeling using data modeling tools in complex, large-scale environments.

Experience working with Microsoft Azure cloud computing platform and services.

Experience managing data orchestration at scale using tools such as Airflow, Prefect and Dagster.

Experience with traditional RDMS platforms (Oracle and SQL Server).

Experience working with version control systems (e.g., Git)

Good understanding of CI/CD principles.

Experience developing dashboards and reports in applications such as Oracle Analytics Server (OAS), Microsoft Power BI and Google Looker.

Experience using Azure services for Security, Blob Storage, Data Lake, Databricks, Data Factory etc.

Programming experience with Python or Java

Experience with Azure Monitoring services

Microsoft Certified Azure Solutions Architect Expert or a Snowpro Certification or a similar one