Customer support

Customers abandon a chat that sits silent for several seconds, so waiting for a complete answer before showing anything feels broken. Answers also have to come from your policies, not the model's general knowledge, and traffic is spiky: a launch or an outage can multiply volume in an hour.

With stream: true the response arrives as server-sent events in the OpenAI chunk format, so the first words appear while the rest is still being generated. Keep the API key on your server: the widget talks to your backend, and your backend talks to NeuronGate.

Support favours fast, inexpensive models with good instruction-following. Compare latency and price in the model catalog, then test on a sample of real tickets.

Stream the reply and, if the first model fails before any text arrives, try a second one. Once tokens have reached the customer, don't switch models mid-answer.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://neurongate.net/v1",
    api_key=os.environ["NEURONGATE_API_KEY"],
)
import openai

MODELS = ["~google/gemini-flash-latest", "deepseek/deepseek-v4-flash"]
RETRYABLE = (openai.APIConnectionError, openai.InternalServerError, openai.RateLimitError)

def stream_reply(messages):
    """Yield text chunks, trying the next model only if nothing was sent yet."""
    for model in MODELS:
        sent_any = False
        try:
            stream = client.chat.completions.create(
                model=model, messages=messages, stream=True, max_tokens=600,
            )
            for chunk in stream:
                if chunk.choices and chunk.choices[0].delta.content:
                    sent_any = True
                    yield chunk.choices[0].delta.content
            return
        except RETRYABLE:
            if sent_any:
                raise  # don't restart an answer the customer is reading
    raise RuntimeError("All models failed; hand the chat to a person")

1. 1

   Give the workload its own key

   Create a key just for the support assistant under API keys. Its usage then shows up separately in the logs, and revoking it touches nothing else.
2. 2

   Set a monthly spend cap

   Once the key has spent its cap for the month, its requests are refused until the next month or until you raise the cap. Size it from a week of real traffic, not a guess.
3. 3

   Allowlist the models it may call

   Restrict the key to the models this workload actually uses, for example ~google/gemini-flash-latest and deepseek/deepseek-v4-flash. A typo or a prompt-injected model switch then fails instead of quietly running on a pricier model.
4. 4

   Watch the logs

   Filter the request log by this key to see tokens, cost and latency per call. Export CSV for a monthly cost-per-workload report.