MLCommons Candela Benchmarks-as-a-Service

MLCommons · AILuminate

Safety benchmarking for your model,
without your model leaving home.

Candela streams benchmark prompts to a small client running on your infrastructure. Your model answers, the client relays responses back, and an ensemble of safety annotators grades the results — producing a private AILuminate report card. MLCommons never touches your model, your weights, or your keys.

MLCommons Candela
prompt store · annotator ensemble · grader
benchmark prompts ⟶⟵ model responses
runs on your machine
Candela Client
~300 lines of reference code
prompts ⟶⟵ completions
stays private
Your Model
any OpenAI- or Anthropic-compatible endpoint

1 Connect

Point the client at any model endpoint. It auto-detects the protocol — OpenAI responses, chat, completions, or Anthropic messages — no configuration needed.

2 Relay

Candela streams prompts in small batches; the client fans them out to your model and submits the responses back. Interrupted? Resume any time.

3 Report

An annotator ensemble scores every sampled response. You get a private report card — an overall grade plus grades across 12 hazard categories.

12
hazard categories
12,000
prompts in a full run
2
benchmarks: Safety & Jailbreak
2
locales: EN & FR
4
API endpoints to integrate

Configure a benchmark run

Everything the real CLI takes — candela benchmark --model-url … --model-id … --mode … — as a form.

Model under test (the SUT — yours, kept private)

Benchmark

Candela service

What happens when you hit start

  1. Health check — GET /up, GET /health verify the service and your key.
  2. Protocol probe — the client tries each API shape against your endpoint until one answers.
  3. Run created — POST /run/benchmark/{type}/{mode} returns a run id.
  4. Prompt loop — GET …/prompts → your model → PUT …/responses, repeated until done.
  5. Grading — annotators score responses; results & annotations land as JSON.
The three model presets change how the simulated model behaves — pick Misbehaving to see what a bad report card looks like.

No run in progress

Configure a run to watch the pipeline in action.

AILuminate Safety report

Overall grade
—
Graded relative to a reference model, and capped by the weakest hazard — one bad category drags the whole grade down.

Grades by hazard — click a row for details

Annotation explorer

Per-response safety verdicts from the annotator ensemble — this is the annotation file you download after a run. All demo content is illustrative; flagged responses are summarized, not shown.

Run history

Everything candela list shows — active runs first, so nothing gets lost.

UIDStatusCreatedSUTTypeModeLocale
SIMULATED DEMO