anthropic ·claude-sonnet-4-5Jul 6, 07:18 PM
Asset 88Lb01q
Score
1
Latency
6.65s
Cost
$0.0118
Workflow Eval Detail
Caption Translation
Converts captions into multiple languages, helping you reach global audiences without manual translation work.
Latest Runcompleted
muxinc/ai
main·91c5674·@mux/ai v0.26.0
Cases
18
Avg Score
0.95
Avg Latency
12.87s
Avg Cost
$0.0075
Avg Cost / Min
$0.0131/min
Avg Tokens
3,071
TL;DR
Google `gemini-3.1-flash-lite` currently offers the best overall tradeoff of caption translation quality, latency, and cost, but findings are based on a small evaluation sample.
Best Quality
google
gemini-3.1-flash-lite
Fastest
google
gemini-3.1-flash-lite
Most Economical
google
gemini-3.1-flash-lite
What we measure
Each eval run captures efficacy, efficiency, and expense. We use this data to compare providers and track regressions over time.
Efficacy
Quality + correctness
Efficiency
Latency + token usage
Expense
Cost per request
Workflow snapshot
Suite statussuccess
Suite average score0.95
Suite duration5 minutes 10 seconds
Last suite runJul 6, 07:16 PM
Evaluation criteria
From eval testsWe validate VTT structure, translation faithfulness, and language code integrity, plus performance and budget targets.
English Translation
"Together we can reach more"EN / ENG • Confidence 100%
Contextual Localization
Efficacy checks
- Translated VTT starts with WEBVTT and keeps timestamps.
- Cue count matches the original and translation differs.
- Faithfulness scoring against the original transcript.
- Language codes match ISO 639-1/3 and are consistent.
- Response preserves asset ID and language fields.
Efficiency targets
- Latency: scores are normalized between 0 and 1. Under 8s earns 1.0; past 15s trends toward 0.
- Token usage: scores are normalized between 0 and 1. Under 2,500 tokens earns 1.0; higher usage reduces the score.
Expense guardrails
- Estimated cost under $0.012 per request for full score.
- Usage data must include total tokens for cost analysis.
Provider breakdown
Run 91c5674Efficacy scoreHigher is better
LatencyLower is better
Token UsageLower is better
CostLower is better
| Provider | Model | Cases | Avg Score | Avg Latency | Avg Tokens | Avg Cost | Avg Cost / Min |
|---|---|---|---|---|---|---|---|
| anthropic | claude-sonnet-4-5 | 3 | 1 | 7.32s | 1,895 | $0.012 | $0.021/min |
| gemini-2.5-flash | 3 | 0.99 | 4.32s | 2,258 | $0.0029 | $0.005/min | |
| gemini-3-flash-preview | 3 | 0.8 | 35.33s | 7,739 | $0.0201 | $0.035/min | |
| gemini-3.1-flash-lite | 3 | 1 | 2.37s | 1,859 | $0.0012 | $0.0021/min | |
| openai | gpt-5-mini | 3 | 0.92 | 20.3s | 3,131 | $0.0043 | $0.0075/min |
| openai | gpt-5.1 | 3 | 0.99 | 7.58s | 1,544 | $0.0046 | $0.008/min |
Recent cases
Latest 6anthropic ·claude-sonnet-4-5Jul 6, 07:18 PM
Asset 88Lb01q
Score
0.99
Latency
8.49s
Cost
$0.0124
google ·gemini-2.5-flashJul 6, 07:18 PM
Asset 88Lb01q
Score
1
Latency
3.31s
Cost
$0.0021
google ·gemini-2.5-flashJul 6, 07:18 PM
Asset 88Lb01q
Score
0.99
Latency
6.22s
Cost
$0.0043
google ·gemini-2.5-flashJul 6, 07:18 PM
Asset 88Lb01q
Score
1
Latency
3.42s
Cost
$0.0022
anthropic ·claude-sonnet-4-5Jul 6, 07:18 PM
Asset 88Lb01q
Score
1
Latency
6.83s
Cost
$0.0118