RAG and search

A model on its own doesn't know your internal docs, and when it doesn't know, it tends to guess. Pasting every document into the prompt is expensive and still misses things. RAG narrows the prompt to the few passages that matter and asks the model to answer only from them.

Keep your embedding model and vector store where they are. NeuronGate takes the generation step, which is where the model choice matters most and where you'll want to try several models against each other.

The generator needs to follow "answer only from these sources" and to handle the size of your retrieved context. Context windows and prices are listed in the model catalog.

search() stands in for your retrieval layer. The prompt numbers each passage so the model can cite [1], [2] and your UI can link them.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://neurongate.net/v1",
    api_key=os.environ["NEURONGATE_API_KEY"],
)

def answer(question: str) -> str:
    passages = search(question, k=6)  # your vector store / search index
    sources = "\n\n".join(
        f"[{i}] ({p.doc_id}) {p.text}" for i, p in enumerate(passages, start=1)
    )
    reply = client.chat.completions.create(
        model="~anthropic/claude-sonnet-latest",
        temperature=0,
        messages=[
            {"role": "system", "content": (
                "Answer using only the numbered sources. Cite them like [1]. "
                "If the sources don't contain the answer, say you don't know."
            )},
            {"role": "user", "content": f"Sources:\n{sources}\n\nQuestion: {question}"},
        ],
    )
    return reply.choices[0].message.content

In RAG, retrieved passages are most of the input tokens. Send fewer, better passages: six well-chosen chunks usually beat twenty loosely related ones on both cost and accuracy.

1. 1

   Give the workload its own key

   Create a key just for search answers under API keys. Its usage then shows up separately in the logs, and revoking it touches nothing else.
2. 2

   Set a monthly spend cap

   Once the key has spent its cap for the month, its requests are refused until the next month or until you raise the cap. Size it from a week of real traffic, not a guess.
3. 3

   Allowlist the models it may call

   Restrict the key to the models this workload actually uses, for example ~anthropic/claude-sonnet-latest and deepseek/deepseek-v4-flash. A typo or a prompt-injected model switch then fails instead of quietly running on a pricier model.
4. 4

   Watch the logs

   Filter the request log by this key to see tokens, cost and latency per call. Export CSV for a monthly cost-per-workload report.