Tailored answers, filled into supported job forms.Your tailored resume and answers, filled into supported job forms for you to review.

Download the Chrome Extension
AI AdoptionJob SimulationCertificationsRoadmapsPricing
Sign In
OneRoadmap

OneRoadmap is a career platform built around ORI, its AI career agent. ORI finds overlooked job opportunities, matches them to your profile and shows the skill gaps to close, with roadmaps, challenges, job simulations and certifications to close them. When you are ready, it prepares a tailored resume, application answers and an application strategy, with a cover letter where the application asks for one. The OneRoadmap Chrome extension fills supported application forms for you to review and submit, and you keep track of every application in one place.

gaurav.ghai@oneroadmap.io
Delhi NCR, India

Platform

  • AI Roadmaps
  • Free Certifications
  • Learning Resources
  • Pricing

Training

  • AI Adoption Workshops
  • Expert Sessions
  • Upcoming Events
  • Workshop Gallery

Company

  • About
  • Blog
  • Contact

Legal

  • Privacy
  • Terms
  • Refunds
  • Delete your data

© 2026 OneRoadmap

Operated by Ghai Technologies, India · International operations through One Roadmap Marketing Management, Dubai, UAE

Built for your next chapter.

Open roles

· Risk Management

Senior Technical Product Manager – Agent Evaluation & Observability

American Express

Senior · 6+ yrsOn-site · New York, NY, United States / Phoenix, AZ, United StatesListed 13d ago
Apply now

Backed by

VC portfolio

HQ

🇺🇸 New York City, NY, United States

Open roles

304

How to stand out for Senior Technical Product Manager – Agent Evaluation & Observability at American Express

Auto Match agent

Let ORI find you the best jobs.

Set up your Auto Match agent once - your target role, level and where you want to work. It searches every day, scores each opening against your profile and resume, and delivers the ones worth applying to, with a prepared application a click away.

Searches every day Scored against your profile Applications prepared for you
Sign in & set up Auto Match agent

Resume & career call

Get your resume reviewed for this role - 30-minute 1:1 call

Line-by-line resume feedback for this application, how to position your Role Readiness, and a clear plan for what to do next - with a OneRoadmap career coach.

Experience Senior · 6+ yrs (6–8 years)

About the role

from listing

Agentic Partner Membership Experiences

The core of our company is not just the products we offer, but the personal connections our customers have with our brand. You can create digital experiences, from payments to rewards to servicing, that keep us connected to our customers and serve them at every touchpoint. From digital servicing and lifestyle features to secure, frictionless payment options, you can make a meaningful difference in our customers’ lives and help set us apart in the industry. Find your place in digital products on #TeamAmex.

The Amex Digital Labs team drives innovation at American Express by originating, incubating, and managing products and platforms that are essential to our customers’ digital lives. We are a fast-paced team that leverages emerging technologies to develop digital payment and financial services products and engaging Membership experiences for Card Members. We often collaborate with technology companies and strategic partners to bring products to the market.

We are looking for a Senior Technical Product Manager to lead the roadmap for tools and workflows that hlep teams test and improve agent-enabled customer experiences. You will help teams measure how well agent's complete customer tasks, understand failures and use the findings to improve their experiences.

You will own the product roadmap, requirements and adoption of shared evaluation tools and workflows. Working with engineering and enterprise platform teams, you will build on existing capabilities to help product teams assess agent performance and decide what to improve. Domain product owners will remain accountable for release decisions.

Primary responsibilities:

Own Evaluation and Quality workflows

Own the roadmap for shared agent evaluation and observability capabilities, from user discovery and requirements through launch, adoption and improvement.

Partner with domain owners to define representative journeys, prompt sets, expected tool behavior and outputs, scoring rubrics and acceptance criteria.

Define requirements for automated evaluation, human review, and tools that help teams diagnose failures.

Connect Evaluations with Operational Evidence

Define product requirements for traces, events, diagnostic views and review tools that connect customer intent, tool or API calls and observed outcomes where data access permits.

Work with engineering, analytics and existing enterprise evaluation and observability teams to extend shared capabilities, reuse instrumentation and standardize evidence.

Identify where limited access to partner data prevents a complete assessment.

Partner with Identity and domain owners to test policy-relevant behavior, including denied actions, revoked access and required approvals.

Measure and Improve Agent Experiences

Define meaningful measures such as task completion, correct tool use, response quality, failure and recovery rates, latency and cost.

Define how testing and monitoring work together across controlled test environments, end-to-end partner integrations and production. Clarify what each can measure and use the findings to prioritize improvements with the responsible teams.

Measure adoption and usefulness of evaluation workflows, including review effort, coverage and time to identify problems. Communicate the limits of findings based on sampled data.

Lead a Reusable Product Capability

Translate measurement gaps into a prioritized backlog, clear requirements, and actionable acceptance criteria. Recommend when to extend existing tools, integrate a solution or build a missing capability.

Build privacy and data governance into evaluation and telemetry requirements, including permitted uses of partner-provided data, appropriate access, minimization and retention. Communicate uncertainty, release risks and tradeoffs clearly.

Minimum Qualifications

6-8+ years of experience in product management, product development, software engineering, technical platforms, analytics, quality tooling, or AI products.

Experience owning evaluation, testing, observability, analytics or related technical products through launch, adoption and iteration, with measurable user or business outcomes.

Practical understanding of AI or agent-enabled systems, including nondeterministic behavior, tool or API interactions, and the need for both automated tests and human judgment.

Ability to define metrics, interpret evidence and identify sampling, coverage, or measurement limitations. Familiarity with traces, logs, APIs and technical troubleshooting.

Strong product discovery, roadmap and prioritization skills, including translating ambiguous quality problems into usable tools and repeatable workflows.

Cross-functional leadership and clear communication, with the ability to balance customer outcomes, engineering constraints and privacy or risk requirements.

Preferred Qualifications

Experience with LLM or agent evaluation, scenario design, human review, model-assisted scoring or production failure analysis.

Familiarity with MCP, function calling, telemetry standards or evaluation frameworks; ability to use code or queries to investigate evidence.

Experience building shared capabilities adopted by multiple teams or delivering products in financial services or another regulated industry.

Bachelor’s degree or equivalent experience.

Employment eligibility to work with American Express in the United States is required as the company will not pursue visa sponsorship for these positions.

Hybrid
Apply on company site

Meet Ori - your career agent on WhatsApp

Find jobs, get your roadmap, check if you're ready for a role and prepare applications - in chat, any language.

Ask Ori about this role
Checking your fit…

Opportunity details

Deadline
Closing in 2d · 18 Nov

As stated by the source. Anything not shown was not stated.

American Express

Financial Services

At American Express, we know that with the right backing, people and businesses have the power to progress in incredible ways. Whether we’re supporting our customers’ financial confidence to move ahead, taking commerce to new heights, or encouraging people to explore the world, our colleagues are constantly striving to uphold our powerful backing promise to our customers and each other every day.

These beliefs have been our North Star for 170 years as our business transformed – from helping evacuate travelers during World Wars, to ensuring the safety of our customers’ funds during the Great De

Company pageWebsite

More at American Express

American ExpressEngineering & ArchitecturePreferredFunded

On-site · Toronto, ON, Canada

1–3 yrs · 1d ago

Campus Undergraduate Full-Time Engineer - 2027 Software Engineer I, Enterprise Technology Services- Toronto, Ontario, Canada
Machine LearningBig Data+11 more
View role
American ExpressData Mgmt and AnalyticsPreferredFunded

On-site · Gurugram, HR, India

1–3 yrs · 1d ago

Business Analyst-Data Analytics
View role
American ExpressEngineering & ArchitecturePreferredFunded

On-site · Phoenix, AZ, United States / Atlanta, GA, United States

Senior · 6+ yrs · 1d ago

Senior Software Engineer II - Enterprise Architecture
View role
American ExpressPreferredFunded

On-site · Phoenix, AZ, United States / Sandy, UT, United States / Sunrise, FL, United States / Sunnyvale, CA, United States / New York, NY, United States / Atlanta, GA, United States / Charlotte, NC, United States

Senior · 1d ago

Director Cybersecurity - Certificate and Public Key Infrastructure Management
View role
Apply