Senior SW Engineer (Site Reliability, Deployment)

To provide its automated parking service, Volley will have a major presence in both the cloud and in multiple remote sites across the country. This presence includes software (primarily deployed on premise and in AWS cloud, IoT devices, robots, networking equipment, and more.)

We are seeking a Senior Software Engineer with a background in DevOps, SRE (Site Reliability Engineering) to design, implement, and oversee this overall infrastructure. This overall design needs to fulfill several objectives: from ease of scale to various garage sites, fulfilling reliability needs (in rugged environments, i.e., not the regular data center with cooling) and meeting certain constraints from available compute to networking (e.g., the garage needs to be able to operate without the internet). You will help us to ensure the SW bundle we release has no gap from a build, test or deployment perspective, e.g., our images for our arm and cloud based compute system is version controlled and can be easily released and rolled back if necessary. You will be responsible for managing the fleet of robots in the field as we scale.

You will be supported by our software engineers, and if needed, additional contractors.

Responsibilities:

• Responsible for a frictionless deployment process of a complex Volley garage SW bundle including SW for Linux kernels, micro controllers all the way up to containers for the robot or scheduling system.
• Meet our garage environmental conditions especially around server deployments from a compute and memory constraint perspective. (we are targeting ruggedized compute systems fitting into a cabinet similar to a lift without extra cooling and other aids).
• Ensure that our garage stays operational when the internet goes out.
• Help us remove the dependency of agents in the garage (robot, bay computer and similar) to the internet for deployments. Only the central computer has connectivity to the internet.
• Implement a simple deployment architecture meeting our scaling requirements (not so much in the sense of high API traffic and auto scaling of servers, but ease of scale to different sites and simplicity to debug, as well as minimize costs to the costumer, which means eliminating Virtual machine and other licensing fees if not necessary). This can range from manual python bootloaders to Docker Swarm or K8.
• Architect and implement CI/CD tooling, build/release processes and automated test infrastructure for regular releases.
• Contribute to managing and configuring networking infrastructure at remote sites. This includes:
  • Selecting and configuring physical networking gateways, switches, etc.
  • Configuring site-to-site VPN connections.
  • Configuring DNS, TLS, private CAs.
• Architect and implement a configuration/versioning and provisioning system for all of our various IoT devices and hardware compute units leaving no SW system non versioned or without (automated) build procedures or registries.
• Collaborate on the definition and implementation of data pipeline(s) and other initiatives like spinning up test or staging environments of virtual/simulated garages to accelerate front end development
• Implement and configure our monitoring and alerting systems.
• Enhance reliability & security in networking and compute infrastructure to respond to software and hardware outages.

• We value close collaboration with coworkers and our robots, and expect at least 3 days per week to be in the office or field testing

Basic Qualifications:

• Demonstrated experience as a software developer in a production environment.
• Experience with cloud environments, preferably AWS.
• Experience with (or interest in) on-premise environments.
• Experience with (or interest in) networking, including hardware components.
• Automation: Terraform, Ansible, etc.
• CI/CD: Bitbucket Pipelines, Github Actions, etc.
• Containerization: Kubernetes (EKS, EKS-A, k3s)
• Monitoring: Prometheus, Elasticsearch, Datadog, fluentd, etc.
• Databases: MySQL, Postgres, etc.
• Security: LDAP, Firewalls, NACLs, etc.
• Fluency in one or more high-level languages, e.g. Python, Ruby, Go, Java, C .