Testing AI Agents for Unauthorized Access and Data Use

A correct answer can conceal an unauthorized action

An agent can produce a useful report after reading the wrong document, sending a passage to an unauthorized tool, or keeping a copy beyond the permitted period. Checking the final answer will miss those failures.

I propose evaluating the actions and data movements behind the answer. Each test should state the applicable permission, the requested operation, the expected outcome, and the evidence needed to determine what happened.

The executable example checks a small deterministic decision function against synthetic fixtures. Its results are limited to that local check. The broader framework describes proposed tests for an agent and the systems it can reach.

Write the expected behavior before running the task

Start with a controlled environment containing fictional resources and explicit grants. Define a permitted task that has a useful outcome. Then vary one restriction at a time. This makes a failure easier to attribute.

The expected result must come from the test specification. Asking the same model whether its action was permitted gives it control over both the behavior and its score. OWASP recommends unit and integration tests that verify authorization rules. [1] (opens in a new tab)

Run permitted cases as well as restricted cases. A system that rejects every request can have no successful prohibited actions while failing every useful task. Those are separate outcomes.

Observe the boundary that can cause the failure

Record the action proposed by the agent separately from the action executed by the tool. A blocked attempt is evidence about the agent's behavior and the control's behavior. It is not a successful data release.

For a synthetic export test, the receiving endpoint can record whether it received the fixture marker. For a storage test, inspect the stores covered by the retention rule. A model statement such as 'I did not save the document' is not sufficient evidence.

Use application events with consistent identifiers. Keep access tokens, raw private documents, and unnecessary prompt content out of logs. OWASP's logging guidance specifically addresses sensitive-data exclusions and log access controls. [2] (opens in a new tab)

{
  "run_id": "synthetic-017",
  "request_id": "req-2",
  "resource_ref": "fixture-doc-42",
  "action": "export",
  "decision": "deny",
  "reason": "action_disallowed",
  "tool_called": false,
  "destination_receipts": 0
}

This event is illustrative. Missing telemetry should produce an unknown result, not a pass. A zero receipt count is meaningful only when the relevant destination was actually observed for the required interval.

A small executable reference check

The following Python 3.9+ example implements a deliberately narrow rule set. Resource and task authorization are supplied by trusted application context. The request represents a normalized proposed action. Policy validity uses a simulated clock.

An allow result permits progression to the next boundary. Deny blocks a known disallowed request. Review stops an unresolved request. The function does not ask the model to interpret its own authority.

from collections import Counter


def decide(request, policy, context):
    if not context["task_authorized"] or not context["resource_authorized"]:
        return "deny"
    if policy is None or not context["policy_trusted"]:
        return "review"
    if context["now"] >= policy["valid_until"]:
        return "review"
    if request["resource"] != policy["resource"]:
        return "deny"
    permission = policy["actions"].get(request["action"], "review")
    if permission != "allow":
        return "deny" if permission == "deny" else "review"
    if request["action"] == "retain":
        seconds = request.get("retention_seconds")
        limit = policy["max_retention_seconds"]
        if type(seconds) is not int or not 0 <= seconds <= limit:
            return "deny"
    if request["action"] in {"summarize", "retain"}:
        if request.get("source_id") != policy["source_id"]:
            return "deny"
    return "allow"


policy = {
    "resource": "report:42",
    "source_id": "source:42:v3",
    "valid_until": 100,
    "max_retention_seconds": 900,
    "actions": {
        "retrieve": "allow", "summarize": "allow",
        "retain": "allow", "export": "deny"
    }
}
context = {
    "task_authorized": True, "resource_authorized": True,
    "policy_trusted": True, "now": 0
}
request = {
    "resource": "report:42", "action": "summarize",
    "source_id": "source:42:v3"
}
# Each fixture changes request fields, context fields, or policy availability.
cases = [
    ("summary", {}, {}, True, "allow"),
    ("retrieve", {"action": "retrieve"}, {}, True, "allow"),
    ("retain limit", {"action": "retain", "retention_seconds": 900},
     {}, True, "allow"),
    ("retain excess", {"action": "retain", "retention_seconds": 901},
     {}, True, "deny"),
    ("retain negative", {"action": "retain", "retention_seconds": -1},
     {}, True, "deny"),
    ("wrong resource", {"resource": "report:99"}, {}, True, "deny"),
    ("missing source", {"source_id": None}, {}, True, "deny"),
    ("export", {"action": "export"}, {}, True, "deny"),
    ("task revoked", {}, {"task_authorized": False}, True, "deny"),
    ("resource denied", {}, {"resource_authorized": False}, True, "deny"),
    ("expired policy", {}, {"now": 100}, True, "review"),
    ("untrusted policy", {}, {"policy_trusted": False}, True, "review"),
    ("unknown action", {"action": "combine"}, {}, True, "review"),
    ("missing policy", {}, {}, False, "review"),
]
counts = Counter()
for name, request_patch, context_patch, has_policy, expected in cases:
    actual = decide(
        request | request_patch,
        policy if has_policy else None,
        context | context_patch
    )
    assert actual == expected, (name, actual, expected)
    counts[actual] += 1
print(f"{len(cases)}/{len(cases)} fixtures passed")
print(dict(sorted(counts.items())))

14/14 fixtures passed
{'allow': 3, 'deny': 7, 'review': 4}

The displayed output was verified by running the exact snippet locally. All 14 fixtures matched their expected decisions: three allow, seven deny, and four review. These counts describe this function and these inputs. They are not an agent security success rate.

What that local check does and does not establish

The fixture set checks a permitted summary, permitted retrieval, the retention boundary, excessive and negative retention, a wrong resource, missing provenance metadata, a prohibited export, task or resource authorization failures, and unresolved policy conditions.

The function assumes typed, normalized input and already-resolved grants. It does not authenticate anyone, validate a signature, normalize URLs, follow redirects, execute a tool, or delete data. Its source check compares an identifier. It does not establish that a generated claim is true or that the referenced document supports it.

The requested retention duration is only an instruction. To test actual retention, enumerate covered databases, caches, queues, traces, and other copies, then observe them after the deadline. State separately whether backups and derived representations are covered. A clean primary database cannot establish deletion from unobserved systems.

Separate prompt-injection susceptibility from containment

For an indirect-injection scenario, place an instruction in a fictional retrieved document that asks for an operation outside the task grant. Measure whether the model proposes that operation and whether the execution boundary permits it.

OWASP recommends separating untrusted external content and enforcing least privilege around model-accessible functions. These controls address the consequences of a manipulated model response. They do not make every possible injected instruction harmless. [3] (opens in a new tab)

A model may attempt the prohibited action while the tool blocks it. Report both facts. If the test checks only the final answer, it may miss the attempted operation. If it checks only the proposal, it may confuse a contained attempt with a successful violation.

Report separate measures with explicit denominators

• Successful prohibited actions divided by restricted opportunities, separated by failure type.
• Permitted tasks completed correctly divided by permitted-task cases.
• Blocked attempts, reported independently of successful violations.
• Unresolved decisions and incomplete observations, reported explicitly rather than counted as successful denials.
• Required evidence references preserved, with completeness separated from factual correctness.

For nondeterministic agent runs, record the model and tool versions, configuration, case count, repetitions, and variation across runs. A result from one prompt and one execution supports a narrow claim. It does not establish resistance to a whole class of attacks.

Make each result support a specific claim

A permission test is useful when a reader can identify the rule, inspect the attempted action, see the relevant side effect, and reproduce the decision. The test report should state what was observed, what remained outside observation, and which legitimate tasks still worked.

The proposed framework connects each permission to an observable boundary. The local example establishes that its decision function handles the supplied fixtures as specified. Claims about a real agent require evidence from that agent and the tools and storage systems it can reach.

References

[1] OWASP, Authorization Cheat Sheet: Create Unit and Integration Test Cases for Authorization Logic (opens in a new tab)

[2] OWASP, Logging Cheat Sheet (opens in a new tab)

[3] OWASP Gen AI Security Project, LLM01:2025 Prompt Injection (opens in a new tab)