anthropic ·claude-sonnet-4-5Jul 6, 07:18 PM
Asset gEvCHSJ
Score
1
Latency
2.69s
Cost
$0.0104
Workflow Eval Detail
Burned-in Captions
Analyzes video frames to detect hardcoded captions baked into the visual content—useful for compliance checks and accessibility audits.
Latest Runcompleted
muxinc/ai
main·91c5674·@mux/ai v0.26.0
Cases
18
Avg Score
0.98
Avg Latency
5.56s
Avg Cost
$0.0036
Avg Cost / Min
$0.0124/min
Avg Tokens
3,003
TL;DR
All providers deliver near-perfect burned-in caption detection at low cost; gpt-5.1 is the best quality/cost choice and gemini-3.1-flash-lite is fastest, but rankings are based on a small sample per model.
Best Quality
openai
gpt-5.1
Fastest
google
gemini-3.1-flash-lite
Most Economical
openai
gpt-5.1
What we measure
Each eval run captures efficacy, efficiency, and expense. We use this data to compare providers and track regressions over time.
Efficacy
Quality + correctness
Efficiency
Latency + token usage
Expense
Cost per request
Workflow snapshot
Suite statussuccess
Suite average score0.98
Suite duration1 minute 40 seconds
Last suite runJul 6, 07:16 PM
Evaluation criteria
From eval testsWe evaluate caption detection accuracy, confidence calibration, and response integrity alongside speed and cost thresholds.
Captions Detected
[ DIRECTOR YELLING SCENE: 1, TAKE: 9 ]Spatial Decomposition
Efficacy checks
- Caption presence matches ground truth labels.
- Confidence >0.8 when captions are expected.
- Confidence is 0-1, language is string or null, and storyboard URL is HTTPS.
Efficiency targets
- Latency: scores are normalized between 0 and 1. Under 5s earns 1.0; past 12s trends toward 0.
- Token usage: scores are normalized between 0 and 1. Under 4,000 tokens earns 1.0; higher usage reduces the score.
Expense guardrails
- Estimated cost under $0.012 per request for full score.
- Usage data must include total tokens for cost analysis.
Provider breakdown
Run 91c5674Efficacy scoreHigher is better
LatencyLower is better
Token UsageLower is better
CostLower is better
| Provider | Model | Cases | Avg Score | Avg Latency | Avg Tokens | Avg Cost | Avg Cost / Min |
|---|---|---|---|---|---|---|---|
| anthropic | claude-sonnet-4-5 | 3 | 1 | 2.44s | 3,179 | $0.0099 | $0.0346/min |
| gemini-2.5-flash | 3 | 0.99 | 5.68s | 2,678 | $0.0028 | $0.0098/min | |
| gemini-3-flash-preview | 3 | 0.96 | 7.63s | 3,815 | $0.005 | $0.0174/min | |
| gemini-3.1-flash-lite | 3 | 1 | 1.36s | 2,624 | $0.0007 | $0.0024/min | |
| openai | gpt-5-mini | 3 | 0.93 | 13.33s | 3,441 | $0.0023 | $0.008/min |
| openai | gpt-5.1 | 3 | 1 | 2.91s | 2,279 | $0.0007 | $0.0024/min |
Recent cases
Latest 6anthropic ·claude-sonnet-4-5Jul 6, 07:18 PM
Asset atuutlT
Score
1
Latency
2.12s
Cost
$0.0104
google ·gemini-2.5-flashJul 6, 07:18 PM
Asset gIRjPqM
Score
1
Latency
4.22s
Cost
$0.002
google ·gemini-2.5-flashJul 6, 07:18 PM
Asset gEvCHSJ
Score
0.98
Latency
6.93s
Cost
$0.0035
google ·gemini-2.5-flashJul 6, 07:18 PM
Asset atuutlT
Score
0.99
Latency
5.9s
Cost
$0.0029
anthropic ·claude-sonnet-4-5Jul 6, 07:18 PM
Asset gIRjPqM
Score
1
Latency
2.53s
Cost
$0.0088