Model evaluation

Comparing models across vendors normally means several accounts, several SDKs and several bills, so teams stop at one or two candidates and keep using a model long after a cheaper or better one ships. New models arrive every few weeks; re-testing has to be cheap enough to actually do.

Every response carries a usage object with prompt and completion token counts, and each request's cost appears in your request log and CSV export. Give the evaluation its own key and the whole experiment's cost is one filter away.

Test across price tiers, not just the top of the leaderboard. Often a mid-priced model is indistinguishable on your task. Browse candidates by provider, context window and capability in the model catalog.

Runs every prompt through every model and prints a comparison. Swap the simple pass/fail check for whatever scoring fits your task.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://neurongate.net/v1",
    api_key=os.environ["NEURONGATE_API_KEY"],
)
import time, statistics

MODELS = [
    "~anthropic/claude-sonnet-latest",
    "~openai/gpt-latest",
    "~google/gemini-flash-latest",
    "deepseek/deepseek-v4-flash",
]
CASES = [  # real examples from your workload, with a check per case
    {"prompt": "Classify: 'Card charged twice'", "expect": "billing"},
    {"prompt": "Classify: 'App crashes on login'", "expect": "bug"},
]

for model in MODELS:
    passed, latencies, tokens = 0, [], 0
    for case in CASES:
        start = time.perf_counter()
        reply = client.chat.completions.create(
            model=model,
            temperature=0,
            messages=[{"role": "user", "content": case["prompt"] + " Answer with one word."}],
        )
        latencies.append(time.perf_counter() - start)
        tokens += reply.usage.total_tokens
        passed += case["expect"] in reply.choices[0].message.content.lower()
    print(f"{model:36} pass {passed}/{len(CASES)}  "
          f"p50 {statistics.median(latencies):.2f}s  tokens {tokens}")

1. 1

   Give the workload its own key

   Create a key just for evaluation runs under API keys. Its usage then shows up separately in the logs, and revoking it touches nothing else.
2. 2

   Set a monthly spend cap

   Once the key has spent its cap for the month, its requests are refused until the next month or until you raise the cap. Size it from a week of real traffic, not a guess.
3. 3

   Allowlist the models it may call

   Restrict the key to the models this workload actually uses, for example the candidate models in the current run. A typo or a prompt-injected model switch then fails instead of quietly running on a pricier model.
4. 4

   Watch the logs

   Filter the request log by this key to see tokens, cost and latency per call. Export CSV for a monthly cost-per-workload report.