Client overview
A US-based AI client collecting realistic web-browsing interaction data to evaluate AI agent performance. The project focuses on validating step-by-step reasoning, action logic, screenshot fidelity, and final answer quality during real-world website navigation.
Business Challenges
- Delayed QA feedback loop: QA review initiated 3 weeks post-production, necessitating large-scale retroactive corrections and rapid data reconciliation.
- High-frequency guideline updates: Frequent, real-time updates to complex interaction rules (pop-ups, backspace usage, and copy-paste restrictions).
- High-precision technical constraints: Strict adherence to WebOlmo-specific execution, including letter-by-letter typing and screenshot fidelity under tight deadlines.
Project Detail
Expected output: Web Navigation AI
Solutions
- Manage guideline changes effectively
- Tracked and applied Golden Guide updates in real time
- Conducted fast retraining after each update
- Held daily calibration sessions to keep teams aligned
- Build a clear browsing process
- Created a structured WebOlmo execution checklist
- Controlled letter-by-letter typing and screenshot accuracy
- Used reasoning templates to validate AI logic and final answers
- Maintain quality and timeline
- Formed a dedicated correction team when QA started
- Applied multi-layer review for high-risk batches
- Split production and revision teams to protect delivery speed
