anthropic ·claude-sonnet-4-5Jul 6, 07:18 PM
Asset 1XIUcA9
Score
0.97
Latency
2.69s
Cost
$0.012
Workflow Eval Detail
Chapters
Automatically segments long-form video content into navigable chapters with timestamps and titles—enabling viewers to jump to key moments instantly.
Latest Runcompleted
muxinc/ai
main·91c5674·@mux/ai v0.26.0
Cases
27
Avg Score
0.98
Avg Latency
5.36s
Avg Cost
$0.005
Avg Cost / Min
$0.0005/min
Avg Tokens
4,245
TL;DR
Google gemini-3.1-flash-lite currently looks best on quality, latency, and cost for the Chapters workflow, but each model was only run on 2 cases so results may not fully generalize.
Best Quality
google
gemini-3.1-flash-lite
Fastest
google
gemini-3.1-flash-lite
Most Economical
google
gemini-3.1-flash-lite
What we measure
Each eval run captures efficacy, efficiency, and expense. We use this data to compare providers and track regressions over time.
Efficacy
Quality + correctness
Efficiency
Latency + token usage
Expense
Cost per request
Workflow snapshot
Suite statussuccess
Suite average score0.96
Suite duration1 minute 29 seconds
Last suite runJul 6, 07:16 PM
Evaluation criteria
From eval testsWe evaluate chapter segmentation quality, timestamp accuracy, and title relevance alongside latency and cost metrics.
0:0020:00
Segmenting
→
↓
AI Chapters6 segments
1
Introduction & Welcome
0:00
2
Setting Up Your Environment
2:22
3
Core Concepts Deep Dive
6:25
4
Building the First Feature
10:12
5
Testing & Debugging
14:51
6
Deployment & Next Steps
18:44
Language: en
●Processing...
Temporal Segmentation
Efficacy checks
- Chapter titles are non-empty and descriptive.
- Start times are valid and monotonically increasing.
- Chapter count is reasonable for video duration.
- Language code matches expected ISO 639-1 format.
- Semantic coherence between chapter titles and content.
Efficiency targets
- Latency: scores are normalized between 0 and 1. Under 10s earns 1.0; past 25s trends toward 0.
- Token usage: scores are normalized between 0 and 1. Under 5,000 tokens earns 1.0; higher usage reduces the score.
Expense guardrails
- Estimated cost under $0.015 per request for full score.
- Usage data must include total tokens for cost analysis.
Provider breakdown
Run 91c5674Efficacy scoreHigher is better
LatencyLower is better
Token UsageLower is better
CostLower is better
| Provider | Model | Cases | Avg Score | Avg Latency | Avg Tokens | Avg Cost | Avg Cost / Min |
|---|---|---|---|---|---|---|---|
| anthropic | claude-sonnet-4-5 | 5 | 0.98 | 4.06s | 3,832 | $0.013 | $0.0015/min |
| gemini-2.5-flash | 5 | 0.99 | 5.96s | 4,905 | $0.0042 | $0.0005/min | |
| gemini-3-flash-preview | 5 | 0.98 | 7.21s | 4,930 | $0.0057 | $0.0006/min | |
| gemini-3.1-flash-lite | 5 | 0.99 | 1.16s | 3,786 | $0.0012 | $0.0001/min | |
| openai | gpt-5-mini | 4 | 0.96 | 11.53s | 4,322 | $0.0026 | $0.0003/min |
| openai | gpt-5.1 | 3 | 0.97 | 2.22s | 3,354 | $0.0017 | $0.0002/min |
Recent cases
Latest 6anthropic ·claude-sonnet-4-5Jul 6, 07:18 PM
Asset 1XIUcA9
Score
1
Latency
4.11s
Cost
$0.0133
anthropic ·claude-sonnet-4-5Jul 6, 07:18 PM
Asset 1XIUcA9
Score
1
Latency
5.01s
Cost
$0.013
anthropic ·claude-sonnet-4-5Jul 6, 07:18 PM
Asset 1XIUcA9
Score
1
Latency
4.21s
Cost
$0.0136
google ·gemini-2.5-flashJul 6, 07:18 PM
Asset 1XIUcA9
Score
0.97
Latency
6.82s
Cost
$0.0048
anthropic ·claude-sonnet-4-5Jul 6, 07:18 PM
Asset 1XIUcA9
Score
0.95
Latency
4.28s
Cost
$0.0134