anthropic ·claude-sonnet-4-5Oct 16, 09:38 PM
Asset gIRjPqM
Score
1
Latency
1.7s
Cost
$0.0089
Workflow Eval Detail
Burned-in Captions
Analyzes video frames to detect hardcoded captions baked into the visual content—useful for compliance checks and accessibility audits.
Latest Runcompleted
muxinc/ai
main·d8d0122·@mux/ai v0.53.0
Cases
21
Avg Score
0.99
Avg Latency
4.46s
Avg Cost
$0.0031
Avg Cost / Min
$0.0109/min
Avg Tokens
2,961
TL;DR
Burned-in Captions achieved a 0.991 average score with strong integrity and validation results, while gpt-5.1 is recommended for quality and latency and gpt-5.6-luna for lowest cost.
Best Quality
openai
gpt-5.1
Fastest
openai
gpt-5.1
Most Economical
openai
gpt-5.6-luna
What we measure
Each eval run captures efficacy, efficiency, and expense. We use this data to compare providers and track regressions over time.
Efficacy
Quality + correctness
Efficiency
Latency + token usage
Expense
Cost per request
Workflow snapshot
Suite statussuccess
Suite average score0.99
Suite duration1 minute 33 seconds
Last suite runOct 16, 09:36 PM
Evaluation criteria
From eval testsWe evaluate caption detection accuracy, confidence calibration, and response integrity alongside speed and cost thresholds.
Captions Detected
[ DIRECTOR YELLING SCENE: 1, TAKE: 9 ]Spatial Decomposition
Efficacy checks
- Caption presence matches ground truth labels.
- Confidence >0.8 when captions are expected.
- Confidence is 0-1, language is string or null, and storyboard URL is HTTPS.
Efficiency targets
- Latency: scores are normalized between 0 and 1. Under 5s earns 1.0; past 12s trends toward 0.
- Token usage: scores are normalized between 0 and 1. Under 4,000 tokens earns 1.0; higher usage reduces the score.
Expense guardrails
- Estimated cost under $0.012 per request for full score.
- Usage data must include total tokens for cost analysis.
Provider breakdown
Run d8d0122Efficacy scoreHigher is better
LatencyLower is better
Token UsageLower is better
CostLower is better
| Provider | Model | Cases | Avg Score | Avg Latency | Avg Tokens | Avg Cost | Avg Cost / Min |
|---|---|---|---|---|---|---|---|
| anthropic | claude-sonnet-4-5 | 3 | 1 | 2.09s | 3,184 | $0.0099 | $0.0346/min |
| gemini-2.5-flash | 3 | 1 | 5.01s | 2,578 | $0.0026 | $0.009/min | |
| gemini-3-flash-preview | 3 | 1 | 4.54s | 3,346 | $0.0036 | $0.0125/min | |
| gemini-3.1-flash-lite | 3 | 1 | 1.98s | 2,624 | $0.0007 | $0.0024/min | |
| openai | gpt-5-mini | 3 | 0.94 | 10.57s | 3,816 | $0.0025 | $0.0088/min |
| openai | gpt-5.1 | 3 | 1 | 1.95s | 2,279 | $0.0022 | $0.0077/min |
| openai | gpt-5.6-luna | 3 | 1 | 5.09s | 2,902 | $0.0004 | $0.0013/min |
Recent cases
Latest run · 21 casesanthropic ·claude-sonnet-4-5Oct 16, 09:38 PM
Asset gEvCHSJ
Score
1
Latency
2.42s
Cost
$0.0104
anthropic ·claude-sonnet-4-5Oct 16, 09:38 PM
Asset atuutlT
Score
1
Latency
2.13s
Cost
$0.0104
google ·gemini-2.5-flashOct 16, 09:38 PM
Asset gIRjPqM
Score
1
Latency
5.22s
Cost
$0.0027
google ·gemini-2.5-flashOct 16, 09:38 PM
Asset gEvCHSJ
Score
0.99
Latency
5.56s
Cost
$0.0029
google ·gemini-2.5-flashOct 16, 09:38 PM
Asset atuutlT
Score
1
Latency
4.25s
Cost
$0.0022
google ·gemini-3-flash-previewOct 16, 09:38 PM
Asset gIRjPqM
Score
1
Latency
3.97s
Cost
$0.0033
google ·gemini-3-flash-previewOct 16, 09:38 PM
Asset gEvCHSJ
Score
1
Latency
4.77s
Cost
$0.0039
google ·gemini-3-flash-previewOct 16, 09:38 PM
Asset atuutlT
Score
1
Latency
4.88s
Cost
$0.0035
google ·gemini-3.1-flash-liteOct 16, 09:38 PM
Asset gIRjPqM
Score
1
Latency
1.37s
Cost
$0.0007
google ·gemini-3.1-flash-liteOct 16, 09:38 PM
Asset gEvCHSJ
Score
1
Latency
1.61s
Cost
$0.0007
google ·gemini-3.1-flash-liteOct 16, 09:38 PM
Asset atuutlT
Score
1
Latency
2.96s
Cost
$0.0007
openai ·gpt-5-miniOct 16, 09:38 PM
Asset gIRjPqM
Score
1
Latency
5.46s
Cost
$0.0014
openai ·gpt-5-miniOct 16, 09:38 PM
Asset gEvCHSJ
Score
0.94
Latency
11.2s
Cost
$0.0026
openai ·gpt-5-miniOct 16, 09:38 PM
Asset atuutlT
Score
0.89
Latency
15.04s
Cost
$0.0035
openai ·gpt-5.1Oct 16, 09:38 PM
Asset gIRjPqM
Score
1
Latency
1.57s
Cost
$0.0014
openai ·gpt-5.1Oct 16, 09:38 PM
Asset gEvCHSJ
Score
1
Latency
2.15s
Cost
$0.0019
openai ·gpt-5.1Oct 16, 09:38 PM
Asset atuutlT
Score
1
Latency
2.14s
Cost
$0.0033
openai ·gpt-5.6-lunaOct 16, 09:38 PM
Asset gIRjPqM
Score
0.99
Latency
6.16s
Cost
$0.0007
openai ·gpt-5.6-lunaOct 16, 09:38 PM
Asset gEvCHSJ
Score
1
Latency
5.21s
Cost
$0.0003
openai ·gpt-5.6-lunaOct 16, 09:38 PM
Asset atuutlT
Score
1
Latency
3.91s
Cost
$0.0002