Sr. Data Engineer

Overview

Pyramid Systems is looking for a Data Engineer (Senior) who is passionate about bringing creative architect solutions to end customers.

Responsibilities

• You'll work with AI team members to operationalize data pipelines and ML tasks
• You'll provide day-to-day support of deploying Python-native ML pipelines
• You'll perform data engineering tasks to enable AI/ML capabilities
• You'll present results to a diverse audience in presentation or report form
• You'll support architectural leadership, technical support, and advisement services to ensure identity management system technologies are integrated and meeting the appropriate security requirements
• You’ll support leadership who engage with senior level executives at a public facing Federal agency and provide subject matter expertise in security architecture and other key domain areas

Qualifications

• You have the ability to obtain and maintain DHS Suitability
• You are a US Citizen
• Plan, create, and maintain data architectures, ensuring alignment with business requirements.
• Obtain data, formulate dataset processes, and store optimized data.
• Experiance in Databricks, PySpark and SQL
• Identify problems and inefficiencies and apply solutions.
• Determine tasks where manual participation can be eliminated with automation.
• "Identify and optimize data bottlenecks, leveraging automation where possible
• Create and manage data lifecycle policies (retention, backups/restore, etc.)"
• Create, maintain and manage ETL/ELT pipelines & data transformations
• Create, maintain and manage data quality gates (Great Expectations) to ensure high data quality
• Support the AI/ML teams with optimizing feature engineering code
• Lots and lots of Spark updates
• Create, maintain and manage Spark Structured Streaming jobs, including using the newer Delta Live Tables and/or DBT
• Research existing data in the data lake to determine best sources for data
• Create, manage and maintain ksqlDB and Kafka Streams queries/code
• Maintain and update Python-based data processing scripts executed on AW Lambdas
• Unit tests for all our Spark, Python data processing, and Lambda codes
• Maintain our PCIS Reporting Database data lake with optimizations and maintenance (perf tuning, etc.)

Education 

• Bachelor's degree in Computer Science or related discipline