Senior Performance Test Engineer

You Lead the Way. We’ve Got Your Back. At American Express, we know that with the right backing, people and businesses have the power to progress in incredible ways. Whether we’re supporting our customers’ financial confidence to move ahead, taking commerce to new heights, or encouraging people to explore the world, our colleagues are constantly redefining what’s possible — and we’re proud to back each other every step of the way. When you join #TeamAmex, you become part of a diverse community of over 60,000 colleagues, all with a common goal to deliver an exceptional customer experience every day.

What we are looking for:

The Performance and Chaos Engineering team is crucial to the success of our distributed microservices. As a senior performance & Chaos test engineer on this team, you will be working on sophisticated systems running on-prem, relational databases and large and complex datasets. You will focus on optimizing overall product performance and reliability. Additionally, you will define and improve an automate-able performance benchmarking/testing framework to understand and optimize application performance as data volume grows.

The key qualities we’re looking for is a true self-started, problem solver, someone who enjoys digging deeper and has a need to understand systems and how they’re wired. Someone who has DevOps/SRE, and Strong Performance testing background would be ideal for this role. This person will have experience with performance & reliability analysis and the ability to develop your own analysis tools (with open-source technology) and performance testing tools.

This position is an individual contributor and reports to the Director of Performance & Chaos Engineering.

Job Responsibilities:

• Develop, implement, and analyze performance tests with complex workload and large simulated test datasets for distributed system running in Kubernetes environment

• Develop tools or use open-source frameworks to automate performance & Chaos testing, application monitoring & analyzing results, and reporting issues

• Define performance & Chaos test strategy, publish metrics and complete performance related requirements

• Setup test environment and test data required to execute performance & Chaos tests

• Identify and analyze performance bottlenecks in the product and work with engineering to resolve the issue

• Own performance, Chaos engineering Planning, design, implementation and execution

• Prepare and present performance benchmarks, standard methodologies, test results, comparisons and analysis

• Provide day to day technical guidance and hands-on help to other engineers on the team

• Analyze performance results to identify performance bottlenecks and suggest optimizations for our systems

As part of our diverse tech team, you can architect, code and ship software that makes us an essential part of our customers’ digital lives. Here, you can work alongside dedicated engineers in an open, supportive, inclusive environment where your voice is valued, and you make your own decisions on what tech to use to solve ambitious problems. Amex offers a range of opportunities to work with the latest technologies and encourages you to back the broader engineering community through open source. And because we understand the importance of keeping your skills fresh and relevant, we give you dedicated time to invest in your professional development. Find your place in technology on #TeamAmex.

Basic Qualifications:

• Passion and drive for Performance, Scalability and Reliability testing of all systems.

• 8 years industry performance testing experience within a large software organization 

• Hands-on experience testing distributed microservices and event driven architecture

• Hands-on performance testing timeseries and SQL datastores

• Ability to implement Industry standard performance testing tools & frameworks (Jmeter, Gatlin etc.) 

• Strong experience in bottleneck analysis, profiling, and distributed tracing tools

• Hands on experience on system monitoring & profiling tools and frameworks like Grafana, InfluxDB, Prometheus, Dynatrace, Splunk etc.)

• Experience performance testing APIs and/or SDKs

• Comfortable using Linux, Docker, Kubernetes (any flavor of Kubernetes) & GIT

• Experience working on multiple projects and features in tandem

• Experience with writing performance test plans and test cases for assigned features

Preferred Qualifications:

• Experience in performance test engineering or SRE roles

• OOP Programming or scripting skills (Java preferred)

• Experience working in globally distributed teams

• Knowledge of how to communicate performance & reliability metrics during a release cycle

• Advanced interpersonal skills: demonstrated ability to explain complex technical issues to both technical and non-technical audiences.

• Bachelor’s in computer science or related field.

Bonus Points:

• Domain knowledge of Cards Payments systems. Understanding of E2E workflows of Authorization Approval and clearing & reconciliation processes.

We back our colleagues with the support they need to thrive, professionally and personally. That’s why we have Amex Flex, our enterprise working model that provides greater flexibility to colleagues while ensuring we preserve the important aspects of our unique in-person culture. Depending on role and business needs, colleagues will either work onsite, in a hybrid model (combination of in-office and virtual days) or fully virtually.

In order to work in or visit any of our offices in the United States, we require colleagues to be fully vaccinated against COVID-19.

American Express is an equal opportunity employer and makes employment decisions without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, protected veteran status, disability status, age, or any other status protected by law.

Employment eligibility to work with American Express in the U.S. is required as the company may pursue visa sponsorship for these positions.