Evaluation and red-team data should expose failures before deployment

The purpose of evaluation and red-teaming data is to find failures before real users do. Clean benchmark prompts are not enough. Production models fail on rare domain inputs, unsafe requests, prompt injection attempts, hallucination traps, malformed tool outputs, bad retrieval context, policy conflicts, and long-horizon workflow drift.

NVIDIA NeMo Data Designer generates and validates red-team and evaluation datasets as part of a broader evaluation workflow. It creates domain-specific stress-test examples from seeds, prompts, schemas, samplers, validators, and judge rubrics. The value is the repeatable workflow: generate targeted eval cases, score them, inspect failures, adjust the scenario design, and regenerate better coverage. For adversarial and refusal coverage, NVIDIA’s open Nemotron safety and content-moderation datasets are a published starting point, and the Nemotron agentic and tool-use datasets help ground agent stress tests. For reward-based evaluation, the NeMo Gym collection provides RL verifiable data across math, code, agentic, instruction-following, and safety tasks.

A strong red-team dataset should include both ordinary failures and adversarial failures. For LLMs, that may mean unsupported claims, unsafe prompts, sensitive-topic handling, refusal boundaries, and domain-specific hallucination traps. For RAG systems, it may mean hard negatives, stale evidence, contradictory sources, missing context, and citation failures. For agents, it should include prompt injection, unsafe tool calls, permission violations, hallucinated tool results, partial tool failures, escalation mistakes, and long-horizon drift.

Evaluation should be tied to clear scoring criteria. Some checks are deterministic: schema validity, code execution, SQL execution, policy rules, citation presence, or tool-call arguments. Others need judge rubrics: answer faithfulness, helpfulness, refusal correctness, escalation quality, recovery behavior, and whether the final answer follows from retrieved or tool-produced evidence.

Key takeaways

• Red-team and eval data should expose failures before production users do.
• NVIDIA NeMo Data Designer generates and validates stress-test datasets through a repeatable workflow.
• The open Nemotron safety and agentic datasets give published starting points for adversarial and agent coverage.
• Strong eval data varies by system type: LLM, RAG, agent, code, SQL, physical AI, or regulated-domain workflow.
• Agent evals should include tool-use failures, prompt injection, permission boundaries, retrieval misses, refusals, escalation, and long-horizon drift.