Senior Data Engineer

COMPANY DESCRIPTION

Pavilion is building the next generation Intelligent Data Platform for the AI/ML world that extends from edge to cloud. Its unique system design that gets its underpinnings from a typical network switch architecture coupled with software layered on top enables unprecedented and breakthrough offerings with industry-leading performance, reliability, and density. HyperOS ( Distributed storage stack that provides blocks, files, and objects) leverages modern technologies such as NVMe over Fabrics, containers, and data pipeline technologies. It is designed and written to take advantage of the made for NVMe appliance, thereby giving itself an unfair advantage to deliver breakthrough acceleration for data pipelines by enabling high-performance storing, retrieving, and data transformation. The current offering on the market is rated as the fastest, densest and easiest to use data platform by order of magnitude, along with boasting marquee customers. You can read what the customers are saying about it here.  Pavilion is a high-growth and well-funded company which boasts investment from Tier 1 Venture Capitalists. 

JOB DESCRIPTION

Senior Data Engineer loves data. We have a culture that welcomes new insights, loves to dig deep, and values better decision-making. This role reports to the Senior Director of Data Engineering.

You will be part of a team of Engineers and Data Scientists that will

• Build scalable, reliable, and reactive data pipelines to deal with of data
• Building the first generation low latency data infrastructure and systems to support our customers’ warehouse operational excellence and our in-house tool development. 
• Define the roadmap for our Data Infrastructure Platform and how clients interact with them. 
• Be the organization’s data SME; best practices, processes, systems, etc 
• Plan for the future and ensure our data platform remains scalable and fast. 
• Identify data sources and work with and build APIs from/for other microservices and outside services to build a holistic business perspective using data. 

About you

• You’ve worked as a data engineer for at least eight years in a production environment. 
• You are fluent coding in a data capacity Python at a minimum with at least five years’ experience in one or more of the languages.
• You have strong experience optimizing, hardening, and configuring Kafka streams at high throughput and data volumes. You should be able to “Talk Kafka,” i.e., understand Kafka very well.
• You’ve got at least eight years of SQL experience. 
• You have strong experience with data pipeline, data modeling, query optimization, and tuning, etc. 
• You have production experience with data streaming technologies - such as Spark, Kinesis, etc. 
• You have built dashboards with tools like Grafana and Kibana

We welcome people of diverse backgrounds, experiences, abilities, and perspectives. We are an equal opportunity employer and a fun place to work. Come join us at Pavilion