Lead Data Engineer

Job Description

An experienced software engineer who will be responsible for the design, development, implementation and on-going support of .NET applications as well as database ETL processes, managing pipelines, data cleaning, MLOps and data modeling. The position will be comprised work closely with data science team productionizing concepts and models developed in Python and other DS technologies. Should have experience with .Net development of web services and using Git version control repositories. Experience with dependency injection, unit testing, and software design patterns.

Should have knowledge of and experience with Machine Learning and AI concepts and be able to articulate differences between them and data requirements that they may have. Particularly the candidate should have knowledge of NLP based services, such as Chat GPT/OpenAI, and experience of prompt engineering, embeddings, fine tuning etc. via the api. Should understand the flow of data and be able to extract/clean data, into appropriate structures, in order to train models. Should understand synchronous and asynchronous data pipelines and training methodologies.

Must be able to work on multiple projects simultaneously, including both enhancements as well as new project development. The software engineer's responsibilities will be to design and develop these applications, and to coordinate with the rest of the team working on different layers of the infrastructure. The candidate must be a self-starter with a sense of urgency and a commitment to quality and professionalism.

Responsibilities:

Ultimate responsibility for the accuracy of ingested data

Work closely with Dev Lead and Architect and other data providers to ensure implementation follows best practices

Technical design process of efficient, scalable, secure and performant data processing pipelines

Ensure modern practices and techniques are brought to the design process

Translate application stories and use cases into functional applications

Design, build, and maintain efficient, reusable, and reliable C# and or Python code

Integrate with AI models, 1st and 3rd party

Integrate with 1st and 3rd party API's for data consumption

Integrate with data storage solutions such as SQL Server/Spark/Vector DB/Search

Ensure the best possible performance, quality, and responsiveness of applications

Identify bottlenecks and bugs, and devise solutions to mitigate and address these issues

Integration of the front-end and back-end aspects of the web application

Collaborate with other team members and stakeholders