Ask Your AI What It Thinks of Its Tools

The moment

We run a lab where one AI (JARVIS) coordinates a dozen parallel work threads across our machines. It uses a few dozen internal tools we built over two years: run a command on a node, read mail, file a ticket, search our knowledge base.

Today we added a tiny feature: /bug. Any AI model working in the system can type it, with an optional note, whenever a tool, an instruction, or a workflow doesn't behave the way it expected. The system snapshots the last few tool calls automatically, so even a bare /bug is actionable.

To test it, I asked the AI to file one real complaint about something that had annoyed it in the last day.

It filed this, about the most basic tool we have, the one that runs a command on a machine:

1. Different parameter names per machine. One node wants code, the others want command.
2. Mistakes look like success. A wrong parameter didn't raise an error; it returned the tool's help text. That looks like a normal answer.
3. A hidden magic word. A safety check only lets a cross-machine command through if a specific comment string appears in the code. It's good intent, but undiscoverable.
4. Timeouts erase the evidence. After 30 seconds it returns "TIMEOUT" and throws away the output it had so far.
5. The same family elsewhere. Search results label an item "ID", but the read call demands "message_id".

Cost: roughly six to eight wasted calls in one session. Every failure looked like success.

Why nobody knew

The tools that break loudly get fixed. The tools that only cost the AI a few extra tries never reach the human, because a capable model quietly works around them, again and again, forever. Every silent workaround is a bug report nobody filed.

We review our big systems. Nobody reviews the small, old, "simple" tools. And the one user who touches them hundreds of times a day was never asked.

What we built

• /bug [note]: one word, callable by any model (we deliberately don't single out a vendor), with an automatic context snapshot.
• De-duplication: the same symptom again becomes a +1, not a pile of tickets.
• A watcher thread: it triages the reports like a support queue (reproduce, fix or route, close with evidence).
• Implicit signals alongside it: a rephrased search a minute later is a miss; falling back to a manual grep is a miss; "not found, but it was there" is the worst miss. These feed a nightly regression set.
• A standing rule: if something is clunky, file /bug and keep working. A blip costs one word, not an interruption.

The lesson

Don't just build tools for your AI and assume it likes them. Ask it. Then give it a one-word way to keep telling you.