The Second Half of Every Decision Part 5 of the Cybersecurity Defense Lab series

1. Green Tests, Working System, Wrong Answers

At the end of the work described here, the system had 141 passing tests, a deterministic security report listing 20-plus active controls, and a health check showing every component green.

It also, at various points during that same period, did the following:

• Silently failed to answer six of its own commands

• Returned raw tool-call syntax as user-facing text

• Reported "research completed" on a page it never successfully read

• Fabricated a statistic ("a 68% increase in adversarial attacks") with no source

• Dated its own research output to 2023, three years in the past

Every one of these traced back to a decision that was correct in intent and incomplete in execution.

1.1 Why agents differ from conventional systems

In a conventional program, a broken HTTP client is loud. The request throws, the stack unwinds, a log line appears, and — if you are paying attention — an alert fires. The failure is structurally unable to hide, because nothing downstream can substitute for the missing data.

An agent has something downstream that can. When a tool returns an error string instead of page content, the model reads that error as context and continues. It has been trained to produce a helpful answer, and it does: coherent, well-organized, citing sources, formatted in Markdown.

The output looks like success. It is indistinguishable from success without checking the logs.

This is not the model behaving badly. It is the model doing exactly what it was built to do, applied to an input that should have stopped the process. The practical consequence:

In an agent, the model acts as a failure-masking layer. Hard errors become soft ones — and soft errors do not page anyone.

Everything that follows is a specific instance of this.

2. Six Decisions and Their Missing Halves

Each subsection below has the same shape: a decision that was reasonable, a second half that was skipped, and what happened as a result.

2.1 Adding a gateway → converting every call site

The decision. Route model calls through a proxy rather than calling the provider directly. Standard practice: it centralizes routing, enables provider swaps, and makes cost tracking possible.

The missing half. Every place that names a model must be converted to the new naming scheme. Not most places. Every one.

Introducing the gateway meant model identifiers changed from provider-native names (qwen3:30b) to gateway aliases (local-general). One code path — free-text chat — was converted. Nine command handlers still referenced the old constant directly.

What happened. Those nine commands sent an unrecognized identifier to the proxy, which correctly rejected it. Six user-facing features stopped working: idea capture, script generation, code help, code patching, reporting, and research-to-script. Free-text chat kept working, which is precisely why the breakage went unnoticed — the most-used path was the one path that had been converted.

Why tests missed it. Every test that touched model selection went through the converted function. No test asserted that the rest of the codebase also went through it.

The fix was mechanical. The guard is not:

def test_no_command_uses_the_raw_constant(): source = (REPO / "vasi.py").read_text() violations = re.findall(r'MODELS\[["\'](\w+)["\']\]', source) assert not violations

This is a static assertion about the whole file rather than a behavioral test of one function — and that difference is the point. Some invariants are properties of a codebase, not of a call.

2.2 Adding tool calling → sending the tools on every turn

The decision. Let the model call a small set of whitelisted tools. The loop is standard: call the model with a tool schema, execute any requested tools, append the results, call again for a final answer.

The missing half. The tool schema must accompany every call in that loop, including the one after tool results are appended.

The second call omitted it. The reasoning at the time was that the model had already made its tool selections, so the schema was no longer needed.

What happened. The model received a conversation containing tool-call syntax in its history but no tool definitions in its current request. It did the reasonable thing given that context: it imitated the pattern it saw. Users received raw markup as their answer.

<tool_call> {"name": "web_reader", "arguments": {"url": "https://..."}} </tool_call>

The tool was never executed. The model was not malfunctioning; it was pattern-matching on a conversation whose structure implied tool syntax was appropriate output.

The generalizable form. Tool schemas are not instructions consumed once. They are part of the conversation's frame, and removing them mid-conversation changes what the model believes it is being asked to produce.

2.3 Splitting into layers → carrying each module's imports

The decision. Break a 1,600-line file into layers with a one-directional dependency rule. Each function moves to the layer that owns its responsibility.

The missing half. A function's imports are part of the function. Moving one without the other produces a module that imports cleanly, compiles cleanly, and fails at runtime.

A web-fetching function was moved to the execution layer. Its import html stayed behind.

What happened. Every call raised NameError at the line performing HTML escaping. The function caught it in its own broad exception handler and returned "Radar error: internal error" — a string the model then read and worked around, without saying so, by answering from training data.

This is the clearest instance of the masking effect. The traceback existed. It was in the logs. But the user-facing signal was a confident research summary with invented figures, and nothing connected the two.

Why tests missed it. The function had a test for the security check at its top (URL validation) and nothing exercising the body. Coverage of a function's entry is not coverage of the function.

2.4 Blocking redirects → verifying them instead

The decision. Refuse HTTP redirects entirely. Sound reasoning, and the SSRF concern behind it is real: a permitted URL that redirects to an internal address defeats a check performed only against the original URL.

The missing half. Real-world URLs redirect constantly, and refusing all of them does not produce a safe tool — it produces an unusable one. The security goal is verification, not refusal.

What happened. Nearly every research attempt failed on a benign redirect. Two examples from the logs:

Blocked: https://www.example-domain.com/article -> https://example-domain.com/article (www dropped) Blocked: https://www.nist.gov/ai -> /artificial-intelligence (relative path)

Both are the same page. The tool refused both.

The corrected form. Follow redirects manually, re-validating each hop:

1. Request with automatic redirects disabled

2. On a 3xx, read Location

3. Resolve relative paths to absolute

4. Re-run the safety check on the new URL

5. Follow if safe, reject if not

6. Cap the chain at a small number of hops

This is more secure than enabling automatic redirects, which validates only the first URL and follows the rest unchecked. And it is more useful than blocking, which validates nothing because nothing succeeds.

The generalizable form. A control that makes its subject unusable is not a strict control; it is an absent feature with a security-shaped justification. The interesting question is rarely "block or allow" — it is "what must be true at each step."

2.5 Describing a tool → describing what it is not for

The decision. Give the model a page-reading tool with a description explaining what it does.

The description read: "Reads the text content of a web page. Required for doing research."

The missing half. A tool description defines a boundary, and boundaries need both sides. This one stated a purpose without stating a limit.

What happened. Asked to research a topic, the model reasoned that this tool was required for research, had no specific URL available, and constructed one:

https://www.google.com/search?q=...

Search engines return a JavaScript shell or a consent page to non-browser requests. The tool succeeded — HTTP 200, valid HTML, no error anywhere — and returned nearly no usable content. The model then produced a research report with fabricated figures and plausible-sounding source names.

Note the failure mode. Section 2.3 produced a visible error string that the model worked around. This produced no error at all. The request succeeded, the content was empty, and the model filled the gap.

The corrected description states the negative space:

"Reads the text content of a SPECIFIC web page. This is NOT a search engine; query URLs are rejected. Use only with a full page address the user provided. If no address is available, do not call this tool."

Plus an enforcement layer, because a description is a request rather than a constraint: search-engine hosts are rejected before any request is made.

The generalizable form. Every tool description is a small policy document. Stating only the purpose leaves the model to infer boundaries — and it will infer generously, because it is optimized to be helpful.

2.6 Using a model → telling it what day it is

The decision. Use a local model for general queries.

The missing half. The model does not know the current date. Its sense of "now" comes from training data, and it will reason from that date unless told otherwise.

What happened. Asked about recent developments in a fast-moving field, the model produced a well-structured summary dated to 2023 — roughly three years stale — citing 2023 regulations as current and 2023 statistics as recent.

This is worth separating from fabrication, because the remedy differs. The model was not inventing sources it knew to be false. It was reporting accurately from a snapshot it believed was current. The failure is one of context, not honesty.

Confirmation came from a control question about a document dated after the training cutoff. The model answered correctly and explicitly: "as of the current date (2023), a report from that period has not been published." It reasoned soundly from a false premise it had no way to check.

The fix is a single line of system prompt:

TODAY'S DATE: 03.09.2026. Your training data is considerably older. Information you believe to be current may be outdated. If you do not know the present state of a topic, say so rather than guessing.

After this change, the same question produced an explicit statement of uncertainty and a pointer to the system's live-search command, rather than a stale report.

The generalizable form. A model reasons from its context window. Anything not in it — including the date, the environment, and its own limitations — is filled in from training. Temporal grounding is close to free and its absence is close to invisible.

3. What the Six Have in Common

3.1 All six sit at integration points

None of these gaps is in an algorithm. Every one is in the seam between two components:

GapSeam2.1Application ↔ gateway2.2Agent loop ↔ model API2.3Layer ↔ layer2.4Tool ↔ open web2.5Model ↔ tool contract2.6Model ↔ environment

Unit tests are strongest inside components and weakest across them, which is exactly the wrong shape for this failure class.

3.2 The failures degrade in visibility

They are not equally detectable, and the pattern is instructive:

• 2.1 produced a visible error to the user (a command stopped responding)

• 2.2 produced visible nonsense (raw markup in the reply)

• 2.3 produced an internal error that the model worked around silently

• 2.4 produced a clean error message that read as a legitimate result

• 2.5 produced no error at all — the request succeeded and returned nothing useful

• 2.6 produced no error and no gap; the model had an answer it fully believed

Visibility declines monotonically as the model has more room to compensate. The last two are the dangerous ones, and they are dangerous because nothing went wrong in any component's own terms.

3.3 Tests exercised entry points, not paths

Every gap had adjacent test coverage:

• Model selection was tested — through the one converted function

• The tool loop was tested — with a mock that never checked what was sent

• The web tool was tested — for URL validation, not for execution

• Redirect blocking was tested — asserting the block worked, not that anything worked afterward

• The tool description was never asserted on at all

• The system prompt was never asserted on at all

Coverage of a function's guard clause is not coverage of the function. In each case the test verified that the door was locked without checking whether the room behind it was habitable.

4. What Actually Caught Them

Recorded honestly, since the answer is not encouraging for test-first approaches:

GapFound by2.1Reading the code while making a related change2.2Live use — visible nonsense in the reply2.3Running the function directly from a shell2.4Reading logs after output looked wrong2.5Reading logs after output looked right2.6Noticing a date in the model's own output

Four of six were found by looking at output that seemed fine. Not one was caught by an existing test.

Three practices did the work:

Reading logs when output looks correct. The 2.5 case surfaced only because a plausible report prompted a log check, which showed a search-engine URL and no real page fetch. The output gave no reason for suspicion.

Running functions directly. The NameError in 2.3 was invisible through the agent because a broad exception handler converted it to a string. One direct call from a shell showed the traceback immediately.

Reading the model's own claims about itself. The 2.6 case was found because the model wrote a research date into its output. Models frequently state their assumptions — dates, capabilities, what they believe they can access — and those statements are diagnostic.

Each gap now has a test written after the fact. Those tests protect against regression. They did not, and structurally could not, provide discovery.

5. Tests That Would Have Helped

The retroactive tests fall into three categories, and the categories are more portable than the specific assertions.

Codebase invariants. Assertions about the source as text, not about behavior:

def test_no_direct_constant_use(): source = (REPO / "vasi.py").read_text() assert not re.findall(r'MODELS\[["\'](\w+)["\']\]', source)

Cheap, brittle in the useful sense — they break loudly when someone reintroduces the pattern.

Full-path execution with realistic fakes. The web tool's test now runs the entire function against a mocked HTTP response, including the escaping step where the missing import lived:

def test_web_tool_processes_a_page(monkeypatch): monkeypatch.setattr(module, "is_safe_url", lambda u: True) monkeypatch.setattr(module.requests, "get", lambda a, *k: FakeResponse(...)) result = module.web_reader("https://example.com/") assert "Radar error" not in result assert "expected content" in result

The distinction from the original test is that this one goes past the guard clause.

Prompt and description assertions. Configuration expressed as English is still configuration:

def test_tool_description_states_what_it_is_not(): description = get_tool("web_reader")["function"]["description"].lower() assert "search engine" in description assert "required" not in description

Unusual, and worth the strangeness: two of six gaps lived in prose, and prose that shapes model behavior deserves the same regression protection as code.

6. Limitations

One system, one author. No claim is made about frequency in comparable systems, only that the mechanism is not system-specific.

The observer wrote the code. This biases toward finding what was findable from the inside and away from what requires an outside perspective.

Discovery remains unsolved. The tests added here prevent recurrence. Nothing here suggests a method for finding the next six gaps, which are presumably present and equally quiet.

Model behavior is not stable. Several observations — the tool-syntax imitation, the tendency to construct a search URL — are behaviors of one model family at one point in time. The mechanism generalizes; the specifics may not.

No adversarial evaluation. Every gap here arose from ordinary use. None was found by someone trying to break the system, and a determined attacker would likely find a different set.

7. Conclusion

Six decisions, each defensible. Six second halves, each easy to skip:

1. Add a gateway → convert every call site

2. Add tool calling → send the schema on every turn

3. Split into layers → carry each module's imports

4. Block redirects → verify them instead

5. Describe a tool → describe what it is not for

6. Use a model → tell it what day it is

The left column is where attention naturally goes: it is the interesting part, the part that appears in architecture diagrams and release notes. The right column is where the failures were.

What made these particular gaps hard was not their subtlety. Each is obvious once stated. It was that an agent's most useful property — producing a coherent answer from whatever context it has — is also what conceals them. The system did not go quiet when a component broke. It kept talking, fluently, and the fluency was the problem.

In a conventional system, a broken component produces an error. In an agent system, a broken component produces an answer.

That answer will be well-formatted. It will cite sources. It may be entirely wrong. And the tests, meanwhile, will be green.

Related Reading

• Part 1 — The Autonomy Axis — https://www.thecoremethodhowto.com/the-autonomy-axis-how-malware-independence-reshapes-defense

• Part 2 — The Deception Chain — https://www.thecoremethodhowto.com/the-deception-chain-what-changed-what-didnt

• Part 3 — The Verification Axis — https://www.thecoremethodhowto.com/the-verification-axis-when-nothing-breaks-and-nothing-is-deceived

• Part 4 — The Great Unification — https://www.thecoremethodhowto.com/the-great-unification-mapping-eight-threat-classes-to-a-working-system-flagship-article-in-the-cybersecurity-defense-lab-series

Video Companion

• DACE Gateway: Layered Design in Practice — https://youtu.be/9HkhKaKmqRA

  The system discussed here is documented, step by step, in the accompanying build series.

© 2026. All rights reserved.