What are the responsibilities and job description for the Staff Data Engineer position at BP?
Key Responsibilities:
- Architects, designs, implements, and maintains reliable and scalable data solutions
- Establishes metadata management, data catalogs, data lineage, data standards, data stewardship, and best practices across the organization
- Leads the team to establish Data Governance model including people, process, and technology to sustain data quality
- Researches and promotes new tools and techniques to shape the future of the data engineering environment
- Works closely with data scientists, micro-service developers, and security authorities to build out data platform incrementally and securely
- Develops data modelling best practices and acts as a subject matter expert for data models across all areas of the business
- Leads the team to write, deploy, and maintain software to build, integrate, manage, maintain, and quality-assure data
- Architects, designs, develops, and delivers large-scale data ingestion, data processing, and data transformation projects on the Azure cloud
- Mentors and shares knowledge with the team to provide design reviews, discussions, and prototypes
- Leads customer discussions from a technical standpoint to deploy, run, and audit best practices for cloud products
- Leads the team to follow software & data engineering best practices (e.g., technical design and review, unit testing, monitoring, alerting, source control, code review & documentation)
- Leads the team to deploy secure and well-tested software that meets privacy and compliance requirements; develops, maintains, and improves CI / CD pipeline
- Leads the team in following site-reliability engineering best practices: on-call rotations for services they maintain, responsible for defining and maintaining SLAs. Design, build, deploy and maintain infrastructure as code. Containerizes server deployments.
- Actively contributes to improve developer velocity
- Part of a cross-disciplinary team working closely with other data engineers, software engineers, data scientists, data managers and business partners in a Scrum/Agile setup
Job Requirements:
Education:
Bachelor or higher degree in computer science, Engineering, Information Systems, or other quantitative fields
Experience:
-
Years of experience: 10 to 15 years with minimum of 10 years relevant experience
-
Deep and hands-on experience (typically 5 years) designing, planning, productionizing, maintaining, and documenting reliable and scalable data infrastructure and data products in complex environments
-
Hands on experience with:
-
Databricks and using Spark for data processing (batch and/or real-time)
-
Configuring Delta Lake on Azure Databricks
-
Languages: Python, Scala, SQL
-
Cloud platforms: Azure (ideally) or AWS
-
Azure Data Factory
-
Azure Data Lake, Azure SQL DB, Synapse, and Cosmos DB
-
Data Management Gateway, Azure Storage Options, Stream Analytics and Event Hubs
-
Designing data solutions in Azure incl. data distributions and partitions, scalability, disaster recovery and high availability
-
Data modeling with relational or data-warehouse systems
-
Advanced hand-on experience with different query languages
-
Azure DevOps (or similar tools) for source control & building CI/CD pipelines
-
Understanding Data Structures & Algorithms & their performance
-
Experience designing and implementing large-scale distributed systems
-
Deep knowledge and hands-on experience in technologies across all data lifecycle stages
-
Stakeholder management and ability to lead large organizations through influence
Desirable Criteria:
- Strong stakeholder management
- Continuous learning and improvement mindset
- Proactive mindset to leave the system better than you found it
Key Behaviors:
- Empathetic: Cares about our people, our community, and our planet
- Curious: Seeks to explore and excel
- Creative: Imagines the extraordinary
- Inclusive: Brings out the best in each other
