anthropic ·claude-sonnet-4-5Oct 16, 09:38 PM
Asset 88Lb01q
Score
0.88
Latency
6.56s
Cost
$0.0159
Workflow Eval Detail
Ask Questions
Answers natural-language questions about a video by retrieving relevant context and answering with a concise response.
Latest Runcompleted
muxinc/ai
main·d8d0122·@mux/ai v0.53.0
Cases
7
Avg Score
0.94
Avg Latency
4.14s
Avg Cost
$0.0041
Avg Cost / Min
$0.0071/min
Avg Tokens
4,048
TL;DR
Google delivered the strongest overall quality and fastest provider average, while GPT-5.6 Luna offered the lowest model cost; validate these findings with more runs.
Best Quality
google
gemini-2.5-flash
Fastest
google
gemini-3.1-flash-lite
Most Economical
openai
gpt-5.6-luna
What we measure
Each eval run captures efficacy, efficiency, and expense. We use this data to compare providers and track regressions over time.
Efficacy
Quality + correctness
Efficiency
Latency + token usage
Expense
Cost per request
Workflow snapshot
Suite statussuccess
Suite average score0.94
Suite duration28.98s
Last suite runOct 16, 09:37 PM
Evaluation criteria
From eval testsWe score answer accuracy and response integrity while tracking latency, token usage, and cost.
Question set3 checks
Question
Has on-screen text?
Answer: YesConf0.94
Question
Is the speaker visible?
Answer: NoConf0.88
Question
Scene indoors?
Answer: YesConf0.91
Accuracy
Match expected yes/no outputs.
Format
Required fields + response shape.
Integrity
Confidence and reasoning present.
Throughput
Latency and cost stay within targets.
Response validation
Efficacy checks
- Answers match expected yes/no outputs.
- Response includes required fields and answer structure.
- Confidence is 0-1 and reasoning strings are non-empty.
- Response preserves asset ID and storyboard URL.
Efficiency targets
- Latency: scores are normalized between 0 and 1. Under 8s earns 1.0; past 12s trends toward 0.
- Token usage: scores are normalized between 0 and 1. Under 2,900 tokens earns 1.0; higher usage reduces the score.
- Usage data must include total tokens for cost analysis.
Expense guardrails
- Estimated cost under $0.012 per request for full score.
Provider breakdown
Run d8d0122Efficacy scoreHigher is better
LatencyLower is better
Token UsageLower is better
CostLower is better
| Provider | Model | Cases | Avg Score | Avg Latency | Avg Tokens | Avg Cost | Avg Cost / Min |
|---|---|---|---|---|---|---|---|
| anthropic | claude-sonnet-4-5 | 1 | 0.88 | 6.56s | 4,739 | $0.0159 | $0.0277/min |
| gemini-2.5-flash | 1 | 0.99 | 3.7s | 3,127 | $0.0017 | $0.0029/min | |
| gemini-3-flash-preview | 1 | 0.95 | 4.2s | 3,978 | $0.0029 | $0.005/min | |
| gemini-3.1-flash-lite | 1 | 0.96 | 2.55s | 3,766 | $0.0011 | $0.002/min | |
| openai | gpt-5-mini | 1 | 0.91 | 5.37s | 4,905 | $0.0012 | $0.0021/min |
| openai | gpt-5.1 | 1 | 0.98 | 3.4s | 3,277 | $0.0054 | $0.0093/min |
| openai | gpt-5.6-luna | 1 | 0.93 | 3.19s | 4,541 | $0.0003 | $0.0005/min |
Recent cases
Latest run · 7 casesgoogle ·gemini-2.5-flashOct 16, 09:38 PM
Asset 88Lb01q
Score
0.99
Latency
3.7s
Cost
$0.0017
google ·gemini-3-flash-previewOct 16, 09:38 PM
Asset 88Lb01q
Score
0.95
Latency
4.2s
Cost
$0.0029
google ·gemini-3.1-flash-liteOct 16, 09:38 PM
Asset 88Lb01q
Score
0.96
Latency
2.55s
Cost
$0.0011
openai ·gpt-5-miniOct 16, 09:38 PM
Asset 88Lb01q
Score
0.91
Latency
5.37s
Cost
$0.0012
openai ·gpt-5.1Oct 16, 09:38 PM
Asset 88Lb01q
Score
0.98
Latency
3.4s
Cost
$0.0054
openai ·gpt-5.6-lunaOct 16, 09:38 PM
Asset 88Lb01q
Score
0.93
Latency
3.19s
Cost
$0.0003