anthropic ·claude-sonnet-4-5Jul 6, 07:18 PM
Asset 88Lb01q
Score
1
Latency
6.97s
Cost
$0.0129
Workflow Eval Detail
Summarization
Generates concise summaries and smart tags from your content—perfect for search, discovery, and quick recaps.
Latest Runcompleted
muxinc/ai
main·91c5674·@mux/ai v0.26.0
Cases
23
Avg Score
0.98
Avg Latency
6.61s
Avg Cost
$0.0041
Avg Cost / Min
$0.0072/min
Avg Tokens
3,418
TL;DR
Summarization quality is uniformly strong across providers, with Anthropic best on quality and Google `gemini-3.1-flash-lite` best on latency and cost, but results are only directional given the small (6-case) sample size.
Best Quality
anthropic
claude-sonnet-4-5
Fastest
google
gemini-3.1-flash-lite
Most Economical
google
gemini-3.1-flash-lite
What we measure
Each eval run captures efficacy, efficiency, and expense. We use this data to compare providers and track regressions over time.
Efficacy
Quality + correctness
Efficiency
Latency + token usage
Expense
Cost per request
Workflow snapshot
Suite statussuccess
Suite average score0.94
Suite duration45.82s
Last suite runJul 6, 07:16 PM
Evaluation criteria
From eval testsWe score summary quality, tag relevance, and semantic similarity while tracking latency, token usage, and cost.
...and so when we look at the numbers...
...the growth trajectory has been...
...our teams have worked incredibly hard...
...which brings me to the next point...
Analyzing
→
↓
AI Summary8:00 duration
6 tags extracted
●Complete
Semantic Extraction
Efficacy checks
- Title is non-empty, <=100 chars, and avoids filler starters.
- Description is non-empty, <=1000 chars, and avoids meta phrases.
- Tags are non-empty strings, unique, and <=10 items.
- Title, description, and tags are semantically similar to references.
- Response includes asset ID and HTTPS storyboard URL.
Efficiency targets
- Latency: scores are normalized between 0 and 1. Under 8s earns 1.0; past 20s trends toward 0.
- Token usage: scores are normalized between 0 and 1. Under 4,000 tokens earns 1.0; higher usage reduces the score.
- Usage data must include input and output tokens > 0.
Expense guardrails
- Estimated cost under $0.015 per request for full score.
Provider breakdown
Run 91c5674Efficacy scoreHigher is better
LatencyLower is better
Token UsageLower is better
CostLower is better
| Provider | Model | Cases | Avg Score | Avg Latency | Avg Tokens | Avg Cost | Avg Cost / Min |
|---|---|---|---|---|---|---|---|
| anthropic | claude-sonnet-4-5 | 4 | 0.99 | 6.47s | 3,833 | $0.0131 | $0.0227/min |
| gemini-2.5-flash | 4 | 0.98 | 6.97s | 2,846 | $0.0025 | $0.0037/min | |
| gemini-3-flash-preview | 4 | 0.98 | 6.76s | 3,503 | $0.0034 | $0.0065/min | |
| gemini-3.1-flash-lite | 4 | 0.99 | 2.13s | 2,983 | $0.0009 | $0.0016/min | |
| openai | gpt-5-mini | 4 | 0.94 | 12.73s | 4,602 | $0.0025 | $0.0058/min |
| openai | gpt-5.1 | 3 | 0.98 | 3.94s | 2,514 | $0.0017 | $0.0028/min |
Recent cases
Latest 6anthropic ·claude-sonnet-4-5Jul 6, 07:18 PM
Asset 88Lb01q
Score
1
Latency
4.75s
Cost
$0.0129
anthropic ·claude-sonnet-4-5Jul 6, 07:18 PM
Asset 88Lb01q
Score
1
Latency
5.24s
Cost
$0.0135
google ·gemini-2.5-flashJul 6, 07:18 PM
Asset 88Lb01q
Score
0.94
Latency
4.73s
Cost
$0.0021
google ·gemini-2.5-flashJul 6, 07:18 PM
Asset 88Lb01q
Score
0.99
Latency
9.93s
Cost
$0.0018
anthropic ·claude-sonnet-4-5Jul 6, 07:18 PM
Asset 88Lb01q
Score
0.97
Latency
8.93s
Cost
$0.013