What we can take on
Search relevance
Graded judgements against your own guidelines, in your own markets.
Assistant evaluation
Multi-turn testing for logic, safety, tone and instruction following.
ASR benchmarking
Word error rate against verified ground truth, by accent and condition.
Bias and fairness
Outcome differences measured across demographic slices you define.
Output ranking
Candidate generations ranked by quality with written justification.
Error taxonomy
Failures categorised so engineering knows what to fix first.
The types of model evaluation we run
The full breakdown, not the headline categories. Anything not on this list is a scoping call away.
Human evaluation. Likert-scale rating, Pairwise preference judging, Blind side-by-side comparison, Rubric-based scoring.
Benchmark testing. ASR word error rate, Search and retrieval relevance, Assistant instruction-following, Multi-turn dialogue coherence.
Bias and fairness. Demographic slice comparison, Language and dialect parity, Outcome disparity testing, Stereotype and representation review.
Error analysis. Failure taxonomy tagging, Root-cause categorisation, Severity grading, Release-over-release regression tracking.
The forms the data takes
What you actually receive: file types, structure and the metadata that travels with it.
- Scorecards
- Per-item scores by criterion with rater IDs and rubric version.
- Side-by-side preferences
- Win, tie and loss counts with written rationales.
- Error taxonomies
- Each failure tagged with a type, severity and example.
- Gold sets
- Reviewed reference answers you can reuse in your own pipeline.
- Agreement reports
- Inter-rater statistics per slice.
- Regression diffs
- What changed between two model versions, with the items that moved.
How the data is collected
From the first brief to the batch you receive, step by step.
- 01
Test sets built from your use case
We build prompts, audio or images that reflect the cases your model meets in production, including the hard ones, instead of a generic benchmark.
- 02
Raters qualified on your rubric
Evaluators are trained on your criteria, tested against a gold set and re-certified on a schedule.
- 03
Blind comparisons
Outputs from different models or versions are shown without labels in random order, so a preference reflects the output and not the brand.
- 04
Disagreement is data
Each item is judged by several raters. Where they split, we report the split and the reasons instead of averaging it away.
- 05
Reported at the slice you care about
Results are cut by language, topic, difficulty and user segment, so a regression in one slice is not hidden by a good average.
How a project runs
- 01
Scoping call
We agree volume, languages, demographic quotas, formats and what counts as a rejected item.
- 02
Guidelines and gold set
Written guidelines with worked examples, plus a gold set every contributor is tested against.
- 03
Pilot batch
A small paid batch you accept before anything scales. Most guideline problems surface here.
- 04
Production
Work flows through capture checks, automated QC, peer review and expert review.
- 05
Delivery
Incremental batches in your format with a data card, QA report and agreement scores.
- 06
Iteration
Rejections feed back into the guidelines so the same mistake is not made twice.
How quality is checked
Checks that run during the work, not a sample at the end.
- Raters qualified against a gold set before live work
- Several raters per item, with agreement reported per slice
- Rubric versioned and pinned to the report
- Spot audits of rationales for copying and low effort
- Raw judgements delivered, not just a summary score
Industries that use it
Where teams bring us model evaluation, and what they use it for.
- Technology and softwareRelease gating for assistants and search.
- Banking and financePolicy and compliance checks on model answers.
- HealthcareClinician review of medical outputs.
- TelecomVoice-bot and agent-assist evaluation.
- AutomotiveIn-car assistant and perception evaluation.
- Retail and e-commerceRecommendation and shopping-assistant quality.
Why teams pick us for model evaluation
Agreement first
If raters cannot agree, the metric is noise and we say so before you ship it.
Slice reporting
Results broken out by accent, region, gender and content category.
Repeatable protocol
The same protocol re-run next quarter gives a comparable number.
Independent pools
Evaluators are not the people who produced your training data.
