Tailored answers, filled into supported job forms.Your tailored resume and answers, filled into supported job forms for you to review.

Download the Chrome Extension
AI AdoptionJob SimulationCertificationsRoadmapsPricing
Sign In
OneRoadmap

OneRoadmap is a career platform built around ORI, its AI career agent. ORI finds overlooked job opportunities, matches them to your profile and shows the skill gaps to close, with roadmaps, challenges, job simulations and certifications to close them. When you are ready, it prepares a tailored resume, application answers and an application strategy, with a cover letter where the application asks for one. The OneRoadmap Chrome extension fills supported application forms for you to review and submit, and you keep track of every application in one place.

gaurav.ghai@oneroadmap.io
Delhi NCR, India

Platform

  • AI Roadmaps
  • Free Certifications
  • Learning Resources
  • Pricing

Training

  • AI Adoption Workshops
  • Expert Sessions
  • Upcoming Events
  • Workshop Gallery

Company

  • About
  • Blog
  • Contact

Legal

  • Privacy
  • Terms
  • Refunds
  • Delete your data

© 2026 OneRoadmap

Operated by Ghai Technologies, India · International operations through One Roadmap Marketing Management, Dubai, UAE

Built for your next chapter.

This role is no longer listed. It is kept for reference for 30 days. Browse open roles

Open roles

AI / ML

Agentic AI Evaluation Engineer

EY

4–7 yrsOn-site · Kolkata, WB, IN, 700091Listed 9d ago
Apply now

Backed by

VC portfolio

HQ

🇬🇧 London, United Kingdom

Open roles

12303

How to stand out for Agentic AI Evaluation Engineer at EY

Auto Match agent

Let ORI find you the best jobs.

Set up your Auto Match agent once - your target role, level and where you want to work. It searches every day, scores each opening against your profile and resume, and delivers the ones worth applying to, with a prepared application a click away.

Searches every day Scored against your profile Applications prepared for you
Sign in & set up Auto Match agent

Resume & career call

Get your resume reviewed for this role - 30-minute 1:1 call

Line-by-line resume feedback for this application, how to position your Role Readiness, and a clear plan for what to do next - with a OneRoadmap career coach.

Experience 3–6 yrs (4–7 years)

About the role

from listing

At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all.

EY- Assurance – Senior – Digital

Role : GenAI / Agentic AI Evaluation Engineer (Quality, Safety & Reliability)

Position Details

As part of EY GDS Assurance Digital, you will help design, build, and scale a standardized evaluation capability, focused on evaluating GenAI, RAG-based, and Agentic AI solutions before deployment.

This role sits at the intersection of AI evaluation engineering, Responsible AI, and GenAI security/red teaming. The primary objective is to ensure GenAI/agentic systems are safe, reliable, robust, and fit-for-purpose, by designing evaluation strategies, building repeatable test harnesses, and generating auditable evidence that supports go/no-go decisions.

You will work with global stakeholders (product teams, solution architects, risk & compliance, and assurance leadership) to define evaluation requirements, request test datasets from product teams, execute rigorous evaluations (functional + non-functional), and recommend mitigations and controls to reduce risk.

This is a core full-time role that requires a hands-on AI Development mindset, strong evaluation mindset, and the ability to translate risk concerns into practical testing strategies and measurable acceptance criteria.

Responsibilities

Define and operationalize evaluation strategies for GenAI systems across use cases like Q&A assistants, summarization, extraction, drafting, agentic systems, and multi-step workflows.

Translate business use-cases into a structured evaluation plan: scope, assumptions, success criteria, datasets, metrics, red-team scenarios, thresholds, and reporting requirements.

Drive standardization: reusable evaluation templates, test case libraries, scoring rubrics, and reporting formats across product teams.

Design structured dataset requirements for product teams and ensure coverage across:

Core user journeys and primary business intents

Edge cases (rare prompts, ambiguous queries, incomplete context)

Adversarial cases (malicious prompts, jailbreak attempts, prompt injections)

Bias & fairness cases (sensitive demographic proxies, protected attributes, stereotyping patterns)

Define guidance for dataset sufficiency and statistical coverage (e.g., minimum samples, distribution balance, scenario matrices, stratification by intent/risk).

Build reusable evaluation pipelines for:

Answer quality (correctness, relevance, completeness, clarity)

Grounding & faithfulness (RAG-specific: faithfulness, context precision/recall, hallucination rate, citation quality)

Agentic behavior (tool-call accuracy, tool misuse, goal completion, step correctness, unnecessary actions, loop detection, safety of tool outputs)

Operational quality (latency, cost/token budget, throughput, stability, retries, failure recovery)

Combine LLM-as-judge and human evaluation in a calibrated way (rubric design, sampling plans, agreement checks).

Implement automated evaluation harnesses in Python (preferred), enabling:

batch runs on scenario suites

configurable metric definitions

reproducible runs with run IDs and artifacts

storage of traces and outputs for auditability

Execute structured red teaming aligned to OWASP Top 10 for LLM Applications, covering (examples):

Prompt injection (direct + indirect) and tool hijacking

Sensitive data disclosure / PII leakage

Insecure output handling (downstream injection)

Training data leakage / memorization probes

Model denial-of-service / denial-of-wallet patterns

Integrate evals into development lifecycle: pre-release regression gates, CI checks, benchmark comparisons across model versions/prompts/tools/retrievers.

Perform adversarial testing for agentic workflows:

tool misuse / over-permissioned tool access

unauthorized action execution

exfiltration via tools/connectors

prompt injection via retrieved documents (RAG poisoning)

Recommend mitigations: input validation, retrieval filtering, tool sandboxing, least-privilege permissions, guardrails, policy prompting, refusal logic, output encoding, monitoring alerts.

Produce high-quality evaluation reports that are auditable and decision-ready, including:

methodology, datasets, metrics, thresholds

quantitative results

qualitative results

risk assessment summary and recommended control actions

Present findings to stakeholders in a crisp, risk-informed manner; clearly explain residual risk, limitations, and rationale for go/no-go.

Key Requirements/Skills & Qualification:

Excellent academic background, including at a minimum a bachelor’s or a master’s degree in data science, Statistics, Engineering, Operational Research, or other related field with strong focus on modern data architectures, processes, and environments.

4–7+ years of relevant experience in one or more areas:

ML/AI/GenAI/Agentic engineering (NLP/LLMs), evaluation engineering, applied research

security testing / red teaming

building and designing evaluation harness that ensures safety, reliability and robustness.

Strong hands-on Python for building evaluation harnesses (data processing, metric computation, orchestration, reporting pipelines).

Practical understanding of GenAI system architectures: RAG, embeddings/vector search, prompt orchestration, tool calling, multi-agent systems, memory, routing.

Experience designing metrics and evaluation methods (rubrics, automated scoring, sampling strategy, regression design).

Familiarity with LLM risks and mitigations, especially for enterprise contexts (data leakage, hallucinations, prompt injection, unsafe content, bias).

Security / Red Teaming Skills (Strong Preference)

Understanding of OWASP Top 10 for LLM Applications and how to translate it into test cases and controls.

Experience with adversarial testing approaches: jailbreak prompts, injection patterns, tool misuse scenarios, retrieval poisoning patterns.

Familiarity with secure-by-design practices for LLM apps: least privilege, safe tool invocation, output encoding/validation, monitoring.

Evaluation frameworks and tooling: RAGAS, DeepEval, LangSmith, Phoenix/Arize, custom eval harnesses.

Experimentation practices: A/B testing mindset, baseline comparisons, statistical rigor for sample sizes.

Observability/tracing: structured logging, OpenTelemetry, Langfuse-style traces, dashboards.

Basic DevOps practices: Git, CI/CD, containerization (Docker), reproducible environments.

Strong written communication to produce clear evaluation plans and reports for technical + non-technical stakeholders.

Ability to challenge assumptions constructively (“effective challenge”) and influence engineering teams toward remediation.

Comfort operating in ambiguity with fast-evolving GenAI tooling and risk landscape.

Preferred / Nice-to-Have

Experience in Assurance/Finance/Regulatory environments (model validation, risk acceptance workflows, audit evidence mindset).

Familiarity with responsible AI frameworks (NIST AI RMF, ISO/IEC 42001, EU AI Act concepts).

Experience evaluating multilingual systems or domain-heavy enterprise assistants.

Hands-on with Azure ecosystem (Azure OpenAI, AI Search, Function Apps, App Insights, Key Vault).

Additional skills requirements:

Excellent written, oral, presentation and facilitation skills

Ability to coordinate multiple projects and initiatives simultaneously through effective prioritization, organization, flexibility, and self-discipline.

Must have demonstrated project management experience.

Knowledge of firm’s reporting tools and processes.

Proactive, organized, and self-sufficient with ability to priorities and multitask.

Analyses complex or unusual problems and can deliver insightful and pragmatic solutions.

Ability to quickly and easily create/ gather/ analyze data from a variety of sources.

A robust and resilient disposition able to encourage discipline in team behaviors

What we look for

A Team of people with commercial acumen, technical experience, and enthusiasm to learn new things in this fast-moving environment

An opportunity to be a part of market-leading, multi-disciplinary team of 7200 + professionals, in the only integrated global assurance business worldwide.

Opportunities to work with EY GDS Assurance practices globally with leading businesses across a range of industries

What working at EY offers

At EY, we’re dedicated to helping our clients, from startups to Fortune 500 companies — and the work we do with them is as varied as they are.

You get to work with inspiring and meaningful projects. Our focus is education and coaching alongside practical experience to ensure your personal development. We value our employees, and you will be able to control your own development with an individual progression plan. You will quickly grow into a responsible role with challenging and stimulating assignments. Moreover, you will be part of an interdisciplinary environment that emphasizes high quality and knowledge exchange. Plus, we offer:

Support, coaching and feedback from some of the most engaging colleagues around

Opportunities to develop new skills and progress your career

The freedom and flexibility to handle your role in a way that’s right for you

EY | Building a better working world

EY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets.

Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate.

Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.

Apply on company site

Meet Ori - your career agent on WhatsApp

Find jobs, get your roadmap, check if you're ready for a role and prepare applications - in chat, any language.

Ask Ori about this role
Checking your fit…

Opportunity details

Deadline
Closed · 21 Nov

As stated by the source. Anything not shown was not stated.

EY

Professional Services

EY is building a better working world by creating new value for clients, people, society, the planet, while building trust in the capital markets. Enabled by data, AI and advanced technology, EY teams help clients shape the future with confidence and develop answers for the most pressing issues of today and tomorrow. EY teams in more than 150 countries work across a full spectrum of services in assurance, consulting, tax, strategy and transactions, strengthened by sector experience and diverse ecosystem partners. Find out more about the EY global network: http://ey.com/en_gl/legal-statement

Company pageWebsite

More at EY

EYPreferredFunded

On-site · Ha Noi, VN

Senior · 3+ yrs · 10h ago

Senior Associate - Government & Public Sector, Economics & Infrastructure Advisory | Hanoi Office
View role
EYPreferredFunded

On-site · Bengaluru, KA, IN, 560016

Senior · 8+ yrs · 10h ago

Agile-EY - GDS Consulting - AI And DATA - Data Modeller -Senior
View role
EYPreferredFunded

On-site · Chennai, TN, IN, 600032

Senior · 4+ yrs · 10h ago

Agile-DE - Cloud - Senior Data Platform Engineer - GDSN02
View role
EYPreferredFunded

On-site · Bengaluru, KA, IN, 560016

Senior · 5+ yrs · 10h ago

DE-RCE-Risk Java Full stack Sr-GDSN02
View role
Apply