I was messing around with Hermes on AWS today, just doing some routine testing. I sent a pretty small prompt—nothing crazy, just a basic query to see how the model handled a specific logic puzzle. I hit enter, waited for the response, and then hopped over to check the metrics.
My first reaction was honestly: “Wait… what?” 😂
The logs showed that my tiny, one-sentence prompt had ballooned into an input of around 14,000 tokens. I actually rubbed my eyes for a second. I thought I’d accidentally piped my entire Downloads folder into the API request.
The mystery of the 14K tokens
I started digging into what was actually being sent under the hood. It turns out, when you're working with these advanced agentic setups, your prompt is just the tip of a very large, very expensive iceberg.
What I saw as a simple question was actually being wrapped in a massive layer of "contextual scaffolding." It's easy to forget that for an AI agent to be smart, it needs to carry its entire world on its back every time it talks to you.
What’s actually hiding in your payload?
So, where did all those tokens come from? After tearing apart the request, I found a few usual suspects that I hadn't properly accounted for:
Tool Definitions: Every single function or API the agent has access to is described in detail so the model knows how to use them.
System Instructions: Those "You are a helpful assistant" prompts are getting way longer and more specific to prevent the model from hallucinating.
Memory and History: To keep the conversation coherent, the system often injects a massive chunk of previous interactions.
RAG Context: If your setup is pulling from a knowledge base, it’s stuffing relevant documents into that prompt before it ever reaches the model.
It made me realize how much more an AI agent sees compared to what we actually type into the console. We’re typing a text message; the agent is receiving a research paper.
Why this made me stop and think
Seeing that 14K number was a good reality check. It’s a reminder that "simple" isn't always cheap or lightweight in the world of LLMOps. If you aren't careful with how you manage your context window and tool definitions, your AWS bill might give you a heart attack before your project even goes live.
I'm still playing around with Hermes—it’s a fantastic model—but this little "gotcha" moment was a great lesson in observability. Always check your logs, even when you think you know exactly what’s happening.
Have you guys looked at your raw input tokens lately? Was it what you expected, or are you also accidentally sending a novel’s worth of data every time you say "Hello"?
