How our team evaluates the Mux AI SDK in production-like conditions.

Eval structure

Our evals pair real inputs with expected outputs, then score the workflow against the 3 Es with consistent, repeatable metrics.

1. 1Test data with inputs and ground truth expectations.
2. 2Task function that runs the workflow and reports traces.
3. 3Scorers for efficacy, efficiency, and expense.

evalite("Workflow Name", {
  data: [{ input, expected }],
  task: async (input) => { /* run workflow */ },
  scorers: [
    { name: "accuracy", scorer: ({ output, expected }) => 0.9 },
  ],
});

Cross-provider testing

We run every eval across providers to compare quality, latency, and token usage side by side.

const providers = ["openai", "anthropic", "google"];
const data = providers.flatMap(provider =>
  testAssets.map(asset => ({ input: { assetId: asset.id, provider } })),
);

This helps reduce bias in our default recommendations by grounding decisions in comparable data across providers. It keeps our suggested workflows focused on measurable impact rather than preference.

Model pricing

We estimate costs using default model pricing and verify these numbers regularly against provider docs.