anthropic ·claude-sonnet-4-5Apr 3, 07:58 PM
Asset gEvCHSJ
Score
1
Latency
2.21s
Cost
$0.0082
Workflow Eval Detail
Burned-in Captions
Analyzes video frames to detect hardcoded captions baked into the visual content—useful for compliance checks and accessibility audits.
Latest Runcompleted
muxinc/ai
main·b7cce22·@mux/ai v0.13.1
Cases
18
Avg Score
0.98
Avg Latency
5.8s
Avg Cost
$0.0029
Avg Cost / Min
$0.01/min
Avg Tokens
2,327
TL;DR
Near-perfect burned-in caption detection across providers at low cost, with gemini-3.1-flash-lite-preview favored for speed/expense and gpt-5.1 for quality, though each model was tested on only 3 cases.
Best Quality
openai
gpt-5.1
Fastest
google
gemini-3.1-flash-lite-preview
Most Economical
google
gemini-3.1-flash-lite-preview
What we measure
Each eval run captures efficacy, efficiency, and expense. We use this data to compare providers and track regressions over time.
Efficacy
Quality + correctness
Efficiency
Latency + token usage
Expense
Cost per request
Workflow snapshot
Suite statussuccess
Suite average score0.98
Suite duration1 minute 44 seconds
Last suite runApr 3, 07:56 PM
Evaluation criteria
From eval testsWe evaluate caption detection accuracy, confidence calibration, and response integrity alongside speed and cost thresholds.
Captions Detected
[ DIRECTOR YELLING SCENE: 1, TAKE: 9 ]Spatial Decomposition
Efficacy checks
- Caption presence matches ground truth labels.
- Confidence >0.8 when captions are expected.
- Confidence is 0-1, language is string or null, and storyboard URL is HTTPS.
Efficiency targets
- Latency: scores are normalized between 0 and 1. Under 5s earns 1.0; past 12s trends toward 0.
- Token usage: scores are normalized between 0 and 1. Under 4,000 tokens earns 1.0; higher usage reduces the score.
Expense guardrails
- Estimated cost under $0.012 per request for full score.
- Usage data must include total tokens for cost analysis.
Provider breakdown
Run b7cce22Efficacy scoreHigher is better
LatencyLower is better
Token UsageLower is better
CostLower is better
| Provider | Model | Cases | Avg Score | Avg Latency | Avg Tokens | Avg Cost | Avg Cost / Min |
|---|---|---|---|---|---|---|---|
| anthropic | claude-sonnet-4-5 | 3 | 1 | 2.37s | 2,446 | $0.0077 | $0.0268/min |
| gemini-2.5-flash | 3 | 0.99 | 5.73s | 2,105 | $0.0028 | $0.0099/min | |
| gemini-3-flash-preview | 3 | 1 | 5.11s | 2,680 | $0.0027 | $0.0093/min | |
| gemini-3.1-flash-lite-preview | 3 | 1 | 1.8s | 1,965 | $0.0005 | $0.0019/min | |
| openai | gpt-5-mini | 3 | 0.91 | 16.97s | 3,137 | $0.0028 | $0.0098/min |
| openai | gpt-5.1 | 3 | 1 | 2.83s | 1,631 | $0.0006 | $0.0021/min |
Recent cases
Latest 6anthropic ·claude-sonnet-4-5Apr 3, 07:58 PM
Asset atuutlT
Score
1
Latency
2.31s
Cost
$0.0082
google ·gemini-2.5-flashApr 3, 07:58 PM
Asset gIRjPqM
Score
1
Latency
3.85s
Cost
$0.0018
google ·gemini-2.5-flashApr 3, 07:58 PM
Asset gEvCHSJ
Score
0.98
Latency
7.14s
Cost
$0.0037
google ·gemini-2.5-flashApr 3, 07:58 PM
Asset atuutlT
Score
0.99
Latency
6.19s
Cost
$0.0029
anthropic ·claude-sonnet-4-5Apr 3, 07:58 PM
Asset gIRjPqM
Score
1
Latency
2.59s
Cost
$0.0066