anthropic ·claude-sonnet-4-5May 18, 06:01 PM
Asset 88Lb01q
Score
1
Latency
6.17s
Cost
$0.0132
Workflow Eval Detail
Summarization
Generates concise summaries and smart tags from your content—perfect for search, discovery, and quick recaps.
Latest Runcompleted
muxinc/ai
main·c15880c·@mux/ai v0.22.0
Cases
27
Avg Score
0.97
Avg Latency
6.89s
Avg Cost
$0.0042
Avg Cost / Min
$0.0079/min
Avg Tokens
3,495
TL;DR
High-quality, low-cost summarization across providers with Anthropic best on quality and Google best on speed/cost, but conclusions are tentative given only 7 cases.
Best Quality
anthropic
claude-sonnet-4-5
Fastest
google
gemini-3.1-flash-lite-preview
Most Economical
google
gemini-3.1-flash-lite
What we measure
Each eval run captures efficacy, efficiency, and expense. We use this data to compare providers and track regressions over time.
Efficacy
Quality + correctness
Efficiency
Latency + token usage
Expense
Cost per request
Workflow snapshot
Suite statussuccess
Suite average score0.95
Suite duration55.56s
Last suite runMay 18, 05:58 PM
Evaluation criteria
From eval testsWe score summary quality, tag relevance, and semantic similarity while tracking latency, token usage, and cost.
...and so when we look at the numbers...
...the growth trajectory has been...
...our teams have worked incredibly hard...
...which brings me to the next point...
Analyzing
→
↓
AI Summary8:00 duration
6 tags extracted
●Complete
Semantic Extraction
Efficacy checks
- Title is non-empty, <=100 chars, and avoids filler starters.
- Description is non-empty, <=1000 chars, and avoids meta phrases.
- Tags are non-empty strings, unique, and <=10 items.
- Title, description, and tags are semantically similar to references.
- Response includes asset ID and HTTPS storyboard URL.
Efficiency targets
- Latency: scores are normalized between 0 and 1. Under 8s earns 1.0; past 20s trends toward 0.
- Token usage: scores are normalized between 0 and 1. Under 4,000 tokens earns 1.0; higher usage reduces the score.
- Usage data must include input and output tokens > 0.
Expense guardrails
- Estimated cost under $0.015 per request for full score.
Provider breakdown
Run c15880cEfficacy scoreHigher is better
LatencyLower is better
Token UsageLower is better
CostLower is better
| Provider | Model | Cases | Avg Score | Avg Latency | Avg Tokens | Avg Cost | Avg Cost / Min |
|---|---|---|---|---|---|---|---|
| anthropic | claude-sonnet-4-5 | 4 | 0.99 | 6.06s | 3,838 | $0.0132 | $0.0229/min |
| gemini-2.5-flash | 4 | 0.99 | 7.41s | 3,041 | $0.0029 | $0.0064/min | |
| gemini-3-flash-preview | 4 | 0.97 | 7.82s | 4,339 | $0.005 | $0.011/min | |
| gemini-3.1-flash-lite | 4 | 0.99 | 2.37s | 2,986 | $0.0009 | $0.0016/min | |
| gemini-3.1-flash-lite-preview | 4 | 0.99 | 2.69s | 2,985 | $0.0009 | $0.0016/min | |
| openai | gpt-5-mini | 4 | 0.92 | 16.48s | 4,522 | $0.0024 | $0.0047/min |
| openai | gpt-5.1 | 3 | 0.99 | 4.91s | 2,509 | $0.0036 | $0.0072/min |
Recent cases
Latest 6anthropic ·claude-sonnet-4-5May 18, 06:01 PM
Asset 88Lb01q
Score
1
Latency
5.02s
Cost
$0.0129
anthropic ·claude-sonnet-4-5May 18, 06:01 PM
Asset 88Lb01q
Score
1
Latency
5.94s
Cost
$0.0133
google ·gemini-2.5-flashMay 18, 06:01 PM
Asset 88Lb01q
Score
0.95
Latency
7.82s
Cost
$0.0037
google ·gemini-2.5-flashMay 18, 06:01 PM
Asset 88Lb01q
Score
1
Latency
6.81s
Cost
$0.0025
anthropic ·claude-sonnet-4-5May 18, 06:01 PM
Asset 88Lb01q
Score
0.98
Latency
7.12s
Cost
$0.0131