anthropic ·claude-sonnet-4-5Apr 3, 07:58 PM
Asset 88Lb01q
Score
1
Latency
5.26s
Cost
$0.0109
Workflow Eval Detail
Summarization
Generates concise summaries and smart tags from your content—perfect for search, discovery, and quick recaps.
Latest Runcompleted
muxinc/ai
main·b7cce22·@mux/ai v0.13.1
Cases
23
Avg Score
0.98
Avg Latency
7.59s
Avg Cost
$0.0035
Avg Cost / Min
$0.0062/min
Avg Tokens
2,737
TL;DR
High-quality summarization across providers, with `claude-sonnet-4-5` best on quality and `gemini-3.1-flash-lite-preview` best on latency/cost, but results are only directional given the 6-case sample size.
Best Quality
anthropic
claude-sonnet-4-5
Fastest
google
gemini-3.1-flash-lite-preview
Most Economical
google
gemini-3.1-flash-lite-preview
What we measure
Each eval run captures efficacy, efficiency, and expense. We use this data to compare providers and track regressions over time.
Efficacy
Quality + correctness
Efficiency
Latency + token usage
Expense
Cost per request
Workflow snapshot
Suite statussuccess
Suite average score0.95
Suite duration52.87s
Last suite runApr 3, 07:56 PM
Evaluation criteria
From eval testsWe score summary quality, tag relevance, and semantic similarity while tracking latency, token usage, and cost.
...and so when we look at the numbers...
...the growth trajectory has been...
...our teams have worked incredibly hard...
...which brings me to the next point...
Analyzing
→
↓
AI Summary8:00 duration
6 tags extracted
●Complete
Semantic Extraction
Efficacy checks
- Title is non-empty, <=100 chars, and avoids filler starters.
- Description is non-empty, <=1000 chars, and avoids meta phrases.
- Tags are non-empty strings, unique, and <=10 items.
- Title, description, and tags are semantically similar to references.
- Response includes asset ID and HTTPS storyboard URL.
Efficiency targets
- Latency: scores are normalized between 0 and 1. Under 8s earns 1.0; past 20s trends toward 0.
- Token usage: scores are normalized between 0 and 1. Under 4,000 tokens earns 1.0; higher usage reduces the score.
- Usage data must include input and output tokens > 0.
Expense guardrails
- Estimated cost under $0.015 per request for full score.
Provider breakdown
Run b7cce22Efficacy scoreHigher is better
LatencyLower is better
Token UsageLower is better
CostLower is better
| Provider | Model | Cases | Avg Score | Avg Latency | Avg Tokens | Avg Cost | Avg Cost / Min |
|---|---|---|---|---|---|---|---|
| anthropic | claude-sonnet-4-5 | 4 | 0.99 | 5.79s | 3,108 | $0.011 | $0.0189/min |
| gemini-2.5-flash | 4 | 0.98 | 7.23s | 2,386 | $0.0029 | $0.0047/min | |
| gemini-3-flash-preview | 4 | 0.99 | 6.49s | 2,889 | $0.0026 | $0.0054/min | |
| gemini-3.1-flash-lite-preview | 4 | 0.99 | 3.57s | 2,319 | $0.0007 | $0.0013/min | |
| openai | gpt-5-mini | 4 | 0.93 | 17.27s | 3,633 | $0.002 | $0.0043/min |
| openai | gpt-5.1 | 3 | 0.99 | 4.36s | 1,870 | $0.0016 | $0.0026/min |
Recent cases
Latest 6anthropic ·claude-sonnet-4-5Apr 3, 07:58 PM
Asset 88Lb01q
Score
1
Latency
5.07s
Cost
$0.0108
anthropic ·claude-sonnet-4-5Apr 3, 07:58 PM
Asset 88Lb01q
Score
1
Latency
6.26s
Cost
$0.0115
google ·gemini-2.5-flashApr 3, 07:58 PM
Asset 88Lb01q
Score
0.95
Latency
6.76s
Cost
$0.0027
google ·gemini-2.5-flashApr 3, 07:58 PM
Asset 88Lb01q
Score
1
Latency
4.13s
Cost
$0.0017
anthropic ·claude-sonnet-4-5Apr 3, 07:58 PM
Asset 88Lb01q
Score
0.98
Latency
6.59s
Cost
$0.0108