Use production logs as evidence for missing edge-case coverage

Production logs are useful, but they are biased toward what has already happened often enough to be recorded. Rare failures, dangerous scenarios, unusual users, policy conflicts, and new product states may be missing or too sparse to train on. Simply upsampling the few rare events in the logs can cause overfitting, while generating from scratch can drift into unrealistic examples.

A better workflow starts with the logs as evidence, not as the whole dataset. Teams identify missing scenarios, extract or summarize safe seed patterns, remove or protect sensitive information, define edge-case taxonomies, generate variants, validate the results, and review high-risk examples. NVIDIA NeMo Data Designer supports this process through seed grounding, sampler columns, dependency-aware generation, validators, judges, preview, and iteration.

Privacy matters because production logs often contain customer data, employee data, secrets, PII, PHI, financial information, legal material, or proprietary workflows. Production logs need controls before they are used as seeds. A responsible workflow includes permissioning, redaction, retention review, privacy review, and legal or compliance approval. For sensitive tabular log data, NVIDIA NeMo Safe Synthesizer may be the fit when teams need a private synthetic version of the source data.

For agents, rare log gaps include tool failures, partial API responses, permission denials, stale retrieval, prompt injection, ambiguous user requests, handoffs, escalation, and unsafe actions. The generated examples should test whether the agent handles those states correctly, not merely whether it produces a plausible final response.

Key takeaways

• Production logs underrepresent rare and future edge cases.
• Use logs to identify and seed missing scenarios; do not simply upsample or blindly copy them.
• NVIDIA NeMo Data Designer supports controlled rare-case expansion.
• Production logs require privacy, permissioning, retention, and compliance review.
• Agent rare cases should include tool, retrieval, permission, safety, handoff, and recovery failures.