feat(agent): validate tool arguments against the schema
Browse filesHOSTED_TOOL_CONTRACTS was advisory. Dispatch went from the model's
arguments straight to capability.handler(**kwargs), so
`additionalProperties: False` enforced nothing and every undeclared
handler parameter stayed reachable from the agent.
That mattered most for valuation, which accepts base_growth_pct,
terminal_growth_pct, and beta without declaring them. An LLM that can set
its own growth rate, terminal growth, or beta can make a DCF agree with
whatever it already believed — the failure the locked-DCF rule in the
idea-loop design exists to prevent, and until now that rule held only as
long as a caller remembered to whitelist kwargs by hand.
Validation runs at the tool boundary: after alias normalisation, so
documented aliases still work, and before dispatch. It is keyed on the
capability name, not the tool name. A backgroundable capability is also
exposed as start_<capability>_task, which is not a HOSTED_TOOL_CONTRACTS
key, so keying on the tool name would leave all 24 task variants
validating vacuously — including start_valuation_task, i.e. the tilt
parameters would stay reachable through the one capability this check
exists to protect. _build_tool_definition hands the task variant the
capability's own input_schema, so the contract enforced here is the one
the model was shown. start_task's own `description` kwarg appears in no
contract, so a model passing it is rejected, which is what the task tool's
`additionalProperties: false` already advertises.
Rejections come back in the adapter's existing accepted:false shape with a
modelHint, matching how tool misuse is already reported, rather than
raising.
Only the agent surface is constrained. Routes, the service, and a future
controller keep the full signature by design — a deliberate internal
caller is trusted to choose inputs; a model is not.
The validator is hand-rolled rather than pulling in jsonschema, because
the contracts use a small fixed subset: type, enum, minimum, maximum,
minLength, maxItems, anyOf, items, required, additionalProperties. Three
details are load-bearing:
- It returns the arguments to dispatch, not just a verdict. An integral
float is accepted for an integer field, since models routinely emit
20.0, but it is also narrowed to int. Accepting without narrowing only
relocates the failure: relative_strength reaches `ordered[:top_n]` and
raises `TypeError: slice indices must be integers`, which
_classify_tool_error does not recognise, so the run aborts instead of
returning a tool error.
- A null-valued required argument is rejected. It satisfies "key present"
but not the handler, which goes on to call .upper() on it, producing the
same unclassified-abort path. Absent and null are different errors and
both are errors; a null *optional* stays allowed, as models commonly
send one for an omitted field.
- An anyOf property reports the matching branch's errors when exactly one
branch matches on type. `tickers` is anyOf[string, array], so an
over-long list otherwise produced "expected string or array, got list" —
unactionable, because the value is a list. The model would resend it,
hit the same error fingerprint, and burn the retry budget instead of
learning about maxItems.
Booleans are rejected for integer and string fields, since bool is an int
subclass in Python but a JSON boolean is neither.
Tests assert the property rather than a sample: every tool the default
agent is handed must resolve to a contract, so a future execution mode
with a new name shape cannot reintroduce the task-variant hole silently.
The hidden-parameter test runs against both the invoke and task surfaces.
Two parity tests keep the surface honest: declared properties must be
acceptable by the real handler, and any handler parameter absent from the
schema must be named as deliberately internal, so a new one fails rather
than quietly widening what the model could reach. Both build the registry
against the real service *and* the real chart opener — passing
fake_chart_opener measured a double that is wider than production
open_chart, which hid its undeclared `client` parameter. Measured drift is
3 capabilities of 34: valuation's three tilt inputs, open_chart's
in-process client, and the runtime-injected session_id.
One behaviour change: an empty consult question is now rejected by schema
validation as a retryable tool error instead of raising ValueError. The
contract already declared minLength 1; the raise predated validation. The
adapter's own guard still covers a whitespace-only question, which passes
minLength.
Not covered, deliberately: the argument-repair retry re-dispatches without
a second validation pass. It only rewrites the *value* of name/ticker and
adds no keys, and it now operates on the narrowed dict, so it cannot
introduce an undeclared or wrongly-typed argument.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
@@ -1,3 +1,4 @@
|
|
|
|
|
| 1 |
from copy import deepcopy
|
| 2 |
from typing import Any
|
| 3 |
|
|
@@ -464,3 +465,149 @@ def get_hosted_tool_contract(capability_name: str) -> dict[str, Any]:
|
|
| 464 |
return deepcopy(HOSTED_TOOL_CONTRACTS[capability_name])
|
| 465 |
except KeyError as exc:
|
| 466 |
raise KeyError(f"No explicit hosted tool contract registered for capability '{capability_name}'.") from exc
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from collections.abc import Mapping
|
| 2 |
from copy import deepcopy
|
| 3 |
from typing import Any
|
| 4 |
|
|
|
|
| 465 |
return deepcopy(HOSTED_TOOL_CONTRACTS[capability_name])
|
| 466 |
except KeyError as exc:
|
| 467 |
raise KeyError(f"No explicit hosted tool contract registered for capability '{capability_name}'.") from exc
|
| 468 |
+
|
| 469 |
+
|
| 470 |
+
# ---------------------------------------------------------------------------
|
| 471 |
+
# Argument validation
|
| 472 |
+
#
|
| 473 |
+
# The schemas above were advisory until this existed: the dispatch path went
|
| 474 |
+
# straight from the model's arguments to `capability.handler(**kwargs)`, so
|
| 475 |
+
# `additionalProperties: False` enforced nothing. That mattered because several
|
| 476 |
+
# handlers accept parameters the schema deliberately hides — `valuation` takes
|
| 477 |
+
# `base_growth_pct`, `terminal_growth_pct`, and `beta`, which let a caller tilt
|
| 478 |
+
# a valuation into agreeing with whatever it already believed.
|
| 479 |
+
#
|
| 480 |
+
# Validating here makes the tool surface the schema says it is. Internal Python
|
| 481 |
+
# callers (routes, the service, a controller) are unaffected by design: they
|
| 482 |
+
# hold the full signature, and locking the *agent* is the point.
|
| 483 |
+
#
|
| 484 |
+
# Hand-rolled rather than pulling in `jsonschema`, because the contracts use a
|
| 485 |
+
# small fixed subset: type, enum, minimum, maximum, minLength, maxItems, anyOf,
|
| 486 |
+
# items, required, additionalProperties.
|
| 487 |
+
# ---------------------------------------------------------------------------
|
| 488 |
+
|
| 489 |
+
_TYPE_NAMES = {
|
| 490 |
+
"string": str,
|
| 491 |
+
"boolean": bool,
|
| 492 |
+
"array": list,
|
| 493 |
+
"object": dict,
|
| 494 |
+
}
|
| 495 |
+
|
| 496 |
+
|
| 497 |
+
def _type_error(value: Any, expected: str) -> str | None:
|
| 498 |
+
"""None when `value` satisfies `expected`, else a human-readable reason."""
|
| 499 |
+
if expected == "integer":
|
| 500 |
+
# bool is an int subclass in Python; a JSON boolean is not an integer.
|
| 501 |
+
if isinstance(value, bool):
|
| 502 |
+
return "expected an integer, got a boolean"
|
| 503 |
+
if isinstance(value, int):
|
| 504 |
+
return None
|
| 505 |
+
# Models routinely emit 7.0 for an integer field; accept it losslessly.
|
| 506 |
+
if isinstance(value, float) and value.is_integer():
|
| 507 |
+
return None
|
| 508 |
+
return f"expected an integer, got {type(value).__name__}"
|
| 509 |
+
if expected == "number":
|
| 510 |
+
if isinstance(value, bool):
|
| 511 |
+
return "expected a number, got a boolean"
|
| 512 |
+
return None if isinstance(value, (int, float)) else f"expected a number, got {type(value).__name__}"
|
| 513 |
+
expected_type = _TYPE_NAMES.get(expected)
|
| 514 |
+
if expected_type is None:
|
| 515 |
+
return None # unknown type keyword: nothing to assert
|
| 516 |
+
if expected == "string" and isinstance(value, bool):
|
| 517 |
+
return "expected a string, got a boolean"
|
| 518 |
+
return None if isinstance(value, expected_type) else f"expected {expected}, got {type(value).__name__}"
|
| 519 |
+
|
| 520 |
+
|
| 521 |
+
def _check_value(name: str, value: Any, schema: dict[str, Any]) -> list[str]:
|
| 522 |
+
if "anyOf" in schema:
|
| 523 |
+
branches: list[dict[str, Any]] = schema["anyOf"]
|
| 524 |
+
reasons = [_check_value(name, value, branch) for branch in branches]
|
| 525 |
+
if any(not branch_errors for branch_errors in reasons):
|
| 526 |
+
return []
|
| 527 |
+
# Exactly one branch matching on type means the value's *shape* is right
|
| 528 |
+
# and something inside it is wrong — a bound, or an item's type. Report
|
| 529 |
+
# that branch's errors. The generic "expected string or array, got list"
|
| 530 |
+
# would restate the value's own type back at the model, which cannot act
|
| 531 |
+
# on it and would resend the same argument until the retry budget ends.
|
| 532 |
+
matched = [
|
| 533 |
+
branch_errors
|
| 534 |
+
for branch, branch_errors in zip(branches, reasons)
|
| 535 |
+
if branch.get("type") and not _type_error(value, branch["type"])
|
| 536 |
+
]
|
| 537 |
+
if len(matched) == 1:
|
| 538 |
+
return matched[0]
|
| 539 |
+
allowed = " or ".join(str(branch.get("type", "?")) for branch in branches)
|
| 540 |
+
return [f"{name}: expected {allowed}, got {type(value).__name__}"]
|
| 541 |
+
|
| 542 |
+
errors: list[str] = []
|
| 543 |
+
declared_type = schema.get("type")
|
| 544 |
+
if declared_type:
|
| 545 |
+
reason = _type_error(value, declared_type)
|
| 546 |
+
if reason:
|
| 547 |
+
return [f"{name}: {reason}"]
|
| 548 |
+
|
| 549 |
+
if "enum" in schema and value not in schema["enum"]:
|
| 550 |
+
errors.append(f"{name}: {value!r} is not one of {schema['enum']}")
|
| 551 |
+
if "minimum" in schema and isinstance(value, (int, float)) and value < schema["minimum"]:
|
| 552 |
+
errors.append(f"{name}: {value} is below the minimum of {schema['minimum']}")
|
| 553 |
+
if "maximum" in schema and isinstance(value, (int, float)) and value > schema["maximum"]:
|
| 554 |
+
errors.append(f"{name}: {value} is above the maximum of {schema['maximum']}")
|
| 555 |
+
if "minLength" in schema and isinstance(value, str) and len(value) < schema["minLength"]:
|
| 556 |
+
errors.append(f"{name}: must be at least {schema['minLength']} character(s)")
|
| 557 |
+
if "maxItems" in schema and isinstance(value, list) and len(value) > schema["maxItems"]:
|
| 558 |
+
errors.append(f"{name}: at most {schema['maxItems']} item(s), got {len(value)}")
|
| 559 |
+
if isinstance(value, list) and isinstance(schema.get("items"), dict):
|
| 560 |
+
for index, item in enumerate(value):
|
| 561 |
+
errors.extend(_check_value(f"{name}[{index}]", item, schema["items"]))
|
| 562 |
+
return errors
|
| 563 |
+
|
| 564 |
+
|
| 565 |
+
def validate_tool_arguments(
|
| 566 |
+
tool_name: str, arguments: Mapping[str, Any]
|
| 567 |
+
) -> tuple[list[str], dict[str, Any]]:
|
| 568 |
+
"""Validate model-supplied arguments against a tool's declared input schema.
|
| 569 |
+
|
| 570 |
+
Returns `(problems, arguments_to_dispatch)`. `problems` is a list of
|
| 571 |
+
human-readable strings, empty when the arguments are acceptable. The second
|
| 572 |
+
element is the dict the caller should actually dispatch: integral floats are
|
| 573 |
+
narrowed to `int` for integer-typed fields, because accepting `20.0` without
|
| 574 |
+
narrowing it only moves the failure downstream — `relative_strength(top_n=20.0)`
|
| 575 |
+
reaches `ordered[:top_n]` and raises `TypeError: slice indices must be
|
| 576 |
+
integers`, which no error classifier recognises, so the whole run aborts.
|
| 577 |
+
|
| 578 |
+
Unknown tools validate vacuously — an unregistered tool is a different error,
|
| 579 |
+
reported elsewhere. Callers must pass the *capability* name: task variants are
|
| 580 |
+
exposed as `start_<capability>_task` but are handed the capability's own
|
| 581 |
+
schema, so they validate against the same contract.
|
| 582 |
+
"""
|
| 583 |
+
contract = HOSTED_TOOL_CONTRACTS.get(tool_name)
|
| 584 |
+
if contract is None:
|
| 585 |
+
return [], dict(arguments)
|
| 586 |
+
schema = contract.get("input_schema") or {}
|
| 587 |
+
properties: dict[str, Any] = schema.get("properties") or {}
|
| 588 |
+
|
| 589 |
+
errors: list[str] = []
|
| 590 |
+
for required_name in schema.get("required") or []:
|
| 591 |
+
if required_name not in arguments:
|
| 592 |
+
errors.append(f"{required_name}: required")
|
| 593 |
+
elif arguments[required_name] is None:
|
| 594 |
+
# An explicit null satisfies "key present" but not the handler, which
|
| 595 |
+
# goes on to call `.upper()` on it. Optional nulls stay allowed below.
|
| 596 |
+
errors.append(f"{required_name}: required, but was null")
|
| 597 |
+
|
| 598 |
+
if schema.get("additionalProperties") is False:
|
| 599 |
+
unknown = sorted(set(arguments) - set(properties))
|
| 600 |
+
if unknown:
|
| 601 |
+
errors.append(
|
| 602 |
+
f"unknown argument(s) {unknown}; this tool accepts only {sorted(properties)}"
|
| 603 |
+
)
|
| 604 |
+
|
| 605 |
+
normalized = dict(arguments)
|
| 606 |
+
for name, value in arguments.items():
|
| 607 |
+
declared = properties.get(name)
|
| 608 |
+
if declared is None or value is None:
|
| 609 |
+
continue
|
| 610 |
+
errors.extend(_check_value(name, value, declared))
|
| 611 |
+
if declared.get("type") == "integer" and isinstance(value, float) and value.is_integer():
|
| 612 |
+
normalized[name] = int(value)
|
| 613 |
+
return errors, normalized
|
|
@@ -4,7 +4,11 @@ from collections.abc import Mapping
|
|
| 4 |
from typing import TYPE_CHECKING, Any
|
| 5 |
|
| 6 |
from ..contracts.definitions import TerraFinAgentDefinition, is_internal_agent_definition
|
| 7 |
-
from ..contracts.tool_contracts import
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
from ..runtime.capability import TerraFinCapability
|
| 9 |
from ..runtime.errors import TerraFinAgentApprovalRequiredError
|
| 10 |
from ..runtime.hosted import TerraFinHostedAgentRuntime
|
|
@@ -19,9 +23,9 @@ from .normalize import (
|
|
| 19 |
_repair_symbol_or_name,
|
| 20 |
)
|
| 21 |
from .types import (
|
| 22 |
-
ToolExecutionMode,
|
| 23 |
TerraFinToolDefinition,
|
| 24 |
TerraFinToolInvocationResult,
|
|
|
|
| 25 |
_ToolErrorDisposition,
|
| 26 |
)
|
| 27 |
|
|
@@ -75,6 +79,13 @@ class TerraFinHostedToolAdapter:
|
|
| 75 |
payload: dict[str, Any]
|
| 76 |
task: TerraFinTaskRecord | None = None
|
| 77 |
resolved_arguments = _normalize_common_alias_arguments(dict(arguments or {}))
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 78 |
preflight_result = self._preflight_tool_misuse(tool, session_id, resolved_arguments)
|
| 79 |
if preflight_result is not None:
|
| 80 |
return preflight_result
|
|
@@ -275,6 +286,71 @@ class TerraFinHostedToolAdapter:
|
|
| 275 |
},
|
| 276 |
)
|
| 277 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 278 |
def _preflight_tool_misuse(
|
| 279 |
self,
|
| 280 |
tool: TerraFinToolDefinition,
|
|
|
|
| 4 |
from typing import TYPE_CHECKING, Any
|
| 5 |
|
| 6 |
from ..contracts.definitions import TerraFinAgentDefinition, is_internal_agent_definition
|
| 7 |
+
from ..contracts.tool_contracts import (
|
| 8 |
+
HOSTED_TOOL_CONTRACT_VERSION,
|
| 9 |
+
get_hosted_tool_contract,
|
| 10 |
+
validate_tool_arguments,
|
| 11 |
+
)
|
| 12 |
from ..runtime.capability import TerraFinCapability
|
| 13 |
from ..runtime.errors import TerraFinAgentApprovalRequiredError
|
| 14 |
from ..runtime.hosted import TerraFinHostedAgentRuntime
|
|
|
|
| 23 |
_repair_symbol_or_name,
|
| 24 |
)
|
| 25 |
from .types import (
|
|
|
|
| 26 |
TerraFinToolDefinition,
|
| 27 |
TerraFinToolInvocationResult,
|
| 28 |
+
ToolExecutionMode,
|
| 29 |
_ToolErrorDisposition,
|
| 30 |
)
|
| 31 |
|
|
|
|
| 79 |
payload: dict[str, Any]
|
| 80 |
task: TerraFinTaskRecord | None = None
|
| 81 |
resolved_arguments = _normalize_common_alias_arguments(dict(arguments or {}))
|
| 82 |
+
# Validate AFTER alias normalisation (so documented aliases are accepted)
|
| 83 |
+
# and BEFORE dispatch (so nothing undeclared reaches a handler).
|
| 84 |
+
schema_result, resolved_arguments = self._preflight_argument_schema(
|
| 85 |
+
tool, session_id, resolved_arguments
|
| 86 |
+
)
|
| 87 |
+
if schema_result is not None:
|
| 88 |
+
return schema_result
|
| 89 |
preflight_result = self._preflight_tool_misuse(tool, session_id, resolved_arguments)
|
| 90 |
if preflight_result is not None:
|
| 91 |
return preflight_result
|
|
|
|
| 286 |
},
|
| 287 |
)
|
| 288 |
|
| 289 |
+
def _preflight_argument_schema(
|
| 290 |
+
self,
|
| 291 |
+
tool: TerraFinToolDefinition,
|
| 292 |
+
session_id: str,
|
| 293 |
+
arguments: Mapping[str, Any],
|
| 294 |
+
) -> tuple[TerraFinToolInvocationResult | None, dict[str, Any]]:
|
| 295 |
+
"""Reject arguments the tool's declared schema does not allow.
|
| 296 |
+
|
| 297 |
+
Returns `(rejection_or_None, arguments_to_dispatch)`; the second element
|
| 298 |
+
carries the validator's narrowing (see `validate_tool_arguments`) and must
|
| 299 |
+
be what the caller dispatches.
|
| 300 |
+
|
| 301 |
+
Without this the schema is decoration: dispatch went straight to
|
| 302 |
+
`capability.handler(**kwargs)`, so `additionalProperties: False` enforced
|
| 303 |
+
nothing and a handler's undeclared parameters stayed reachable. The
|
| 304 |
+
clearest case is `valuation`, whose hidden `base_growth_pct` /
|
| 305 |
+
`terminal_growth_pct` / `beta` let a caller tilt a valuation toward a
|
| 306 |
+
conclusion it already held.
|
| 307 |
+
|
| 308 |
+
Keyed on `capability_name`, not `name`: a backgroundable capability is also
|
| 309 |
+
exposed as `start_<capability>_task`, which is not a `HOSTED_TOOL_CONTRACTS`
|
| 310 |
+
key, so keying on `name` made all 24 task variants validate vacuously —
|
| 311 |
+
including `start_valuation_task`, leaving the tilt parameters reachable by
|
| 312 |
+
the one path this check exists to close. `_build_tool_definition` hands the
|
| 313 |
+
task variant the capability's own `input_schema`, so the model is shown the
|
| 314 |
+
contract validated here.
|
| 315 |
+
|
| 316 |
+
Only the agent surface is constrained. Internal Python callers keep the
|
| 317 |
+
full signature, which is the intent — a route or a controller is trusted
|
| 318 |
+
to choose inputs deliberately. `start_task`'s own `description` kwarg is
|
| 319 |
+
not in any contract, so a model that passes it is rejected; that matches
|
| 320 |
+
the `additionalProperties: false` schema the task tool advertises.
|
| 321 |
+
"""
|
| 322 |
+
problems, normalized = validate_tool_arguments(tool.capability_name, arguments)
|
| 323 |
+
if not problems:
|
| 324 |
+
return None, normalized
|
| 325 |
+
|
| 326 |
+
message = f"Invalid arguments for {tool.name}: " + "; ".join(problems)
|
| 327 |
+
return TerraFinToolInvocationResult(
|
| 328 |
+
tool_name=tool.name, # the name the model called, not the capability
|
| 329 |
+
capability_name=tool.capability_name,
|
| 330 |
+
session_id=session_id,
|
| 331 |
+
execution_mode=tool.execution_mode,
|
| 332 |
+
payload={
|
| 333 |
+
"accepted": False,
|
| 334 |
+
"error": {
|
| 335 |
+
"code": "tool_invalid_arguments",
|
| 336 |
+
"message": message,
|
| 337 |
+
"detail": problems,
|
| 338 |
+
"retryable": True,
|
| 339 |
+
"modelHint": (
|
| 340 |
+
"Retry with only the arguments this tool declares, using the types and "
|
| 341 |
+
"ranges in its schema. Parameters that are not in the schema are not "
|
| 342 |
+
"available through the tool surface, even if a related HTTP route or "
|
| 343 |
+
"Python method accepts them."
|
| 344 |
+
),
|
| 345 |
+
},
|
| 346 |
+
},
|
| 347 |
+
task=None,
|
| 348 |
+
is_error=True,
|
| 349 |
+
retryable=True,
|
| 350 |
+
error_code="tool_invalid_arguments",
|
| 351 |
+
error_message=message,
|
| 352 |
+
), normalized
|
| 353 |
+
|
| 354 |
def _preflight_tool_misuse(
|
| 355 |
self,
|
| 356 |
tool: TerraFinToolDefinition,
|
|
@@ -60,6 +60,27 @@ def capabilities():
|
|
| 60 |
return registry.list()
|
| 61 |
|
| 62 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 63 |
@pytest.fixture(scope="module")
|
| 64 |
def live_route_paths() -> set[str | None]:
|
| 65 |
"""Paths served by the assembled app.
|
|
@@ -197,6 +218,82 @@ def test_declared_cli_subcommands_are_wired(capabilities) -> None:
|
|
| 197 |
assert not unwired, f"capabilities declaring a CLI subcommand that is not wired: {unwired}"
|
| 198 |
|
| 199 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 200 |
def test_declared_http_routes_exist(capabilities, live_route_paths) -> None:
|
| 201 |
missing = sorted(
|
| 202 |
f"{c.name} -> {c.http_route_path}"
|
|
|
|
| 60 |
return registry.list()
|
| 61 |
|
| 62 |
|
| 63 |
+
@pytest.fixture(scope="module")
|
| 64 |
+
def real_capabilities():
|
| 65 |
+
"""Registry built against the REAL service *and* the real chart opener.
|
| 66 |
+
|
| 67 |
+
Signature checks must not measure a double. `BaseFakeService`'s stubs are
|
| 68 |
+
narrower than the handlers they stand in for, so a declared property the real
|
| 69 |
+
handler accepts (e.g. `market_snapshot`'s `force_refresh`) would read as a
|
| 70 |
+
mismatch. `fake_chart_opener` has the opposite problem: it is *wider* than
|
| 71 |
+
production `open_chart`, whose `client` parameter is undeclared and went
|
| 72 |
+
undetected here. Passing no `chart_opener` binds the real one.
|
| 73 |
+
|
| 74 |
+
Nothing is called — the handlers are only inspected — so this stays free of
|
| 75 |
+
network and env mutation.
|
| 76 |
+
"""
|
| 77 |
+
|
| 78 |
+
from TerraFin.agent.service import TerraFinAgentService
|
| 79 |
+
|
| 80 |
+
registry = build_default_capability_registry(TerraFinAgentService())
|
| 81 |
+
return registry.list()
|
| 82 |
+
|
| 83 |
+
|
| 84 |
@pytest.fixture(scope="module")
|
| 85 |
def live_route_paths() -> set[str | None]:
|
| 86 |
"""Paths served by the assembled app.
|
|
|
|
| 218 |
assert not unwired, f"capabilities declaring a CLI subcommand that is not wired: {unwired}"
|
| 219 |
|
| 220 |
|
| 221 |
+
# Handler parameters deliberately NOT in the tool schema. Each entry is a
|
| 222 |
+
# decision, not an oversight, and the tool boundary rejects them (see
|
| 223 |
+
# `tests/agent/test_tool_argument_validation.py`):
|
| 224 |
+
# valuation — the three tilt inputs. An LLM that can set its own growth rate,
|
| 225 |
+
# terminal growth, or beta can make a DCF agree with whatever it
|
| 226 |
+
# already believed, which is the failure the locked-DCF rule in
|
| 227 |
+
# the idea-loop design exists to prevent.
|
| 228 |
+
# macro_focus, open_chart — `session_id` is injected by the runtime
|
| 229 |
+
# (`_apply_defaults`), never supplied by a caller.
|
| 230 |
+
# open_chart — `client` is the `TerraFinAgentClient` the CLI passes in-process;
|
| 231 |
+
# a model naming its own HTTP client makes no sense.
|
| 232 |
+
INTERNAL_ONLY_HANDLER_PARAMS = {
|
| 233 |
+
"valuation": {"base_growth_pct", "terminal_growth_pct", "beta"},
|
| 234 |
+
"macro_focus": {"session_id"},
|
| 235 |
+
"open_chart": {"session_id", "client"},
|
| 236 |
+
}
|
| 237 |
+
|
| 238 |
+
|
| 239 |
+
def test_no_new_undeclared_handler_parameters(real_capabilities) -> None:
|
| 240 |
+
"""A handler parameter absent from the tool schema must be a deliberate choice.
|
| 241 |
+
|
| 242 |
+
Adding one silently widens what the model could reach if validation were ever
|
| 243 |
+
bypassed, so new ones have to be named here on purpose.
|
| 244 |
+
"""
|
| 245 |
+
|
| 246 |
+
import inspect
|
| 247 |
+
|
| 248 |
+
surprises: dict[str, list[str]] = {}
|
| 249 |
+
for capability in real_capabilities:
|
| 250 |
+
contract = HOSTED_TOOL_CONTRACTS.get(capability.name)
|
| 251 |
+
if contract is None:
|
| 252 |
+
continue
|
| 253 |
+
declared = set((contract.get("input_schema") or {}).get("properties") or {})
|
| 254 |
+
try:
|
| 255 |
+
parameters = inspect.signature(capability.handler).parameters
|
| 256 |
+
except (TypeError, ValueError): # pragma: no cover - builtins
|
| 257 |
+
continue
|
| 258 |
+
accepted = {
|
| 259 |
+
name
|
| 260 |
+
for name, parameter in parameters.items()
|
| 261 |
+
if parameter.kind in (parameter.POSITIONAL_OR_KEYWORD, parameter.KEYWORD_ONLY)
|
| 262 |
+
}
|
| 263 |
+
undeclared = accepted - declared - INTERNAL_ONLY_HANDLER_PARAMS.get(capability.name, set())
|
| 264 |
+
if undeclared:
|
| 265 |
+
surprises[capability.name] = sorted(undeclared)
|
| 266 |
+
|
| 267 |
+
assert not surprises, (
|
| 268 |
+
"handler parameters that are not in the tool schema and not listed as "
|
| 269 |
+
f"deliberately internal: {surprises}"
|
| 270 |
+
)
|
| 271 |
+
|
| 272 |
+
|
| 273 |
+
def test_declared_properties_are_acceptable_by_the_handler(real_capabilities) -> None:
|
| 274 |
+
"""A declared property the handler cannot take is a TypeError waiting to happen."""
|
| 275 |
+
|
| 276 |
+
import inspect
|
| 277 |
+
|
| 278 |
+
broken: dict[str, list[str]] = {}
|
| 279 |
+
for capability in real_capabilities:
|
| 280 |
+
contract = HOSTED_TOOL_CONTRACTS.get(capability.name)
|
| 281 |
+
if contract is None:
|
| 282 |
+
continue
|
| 283 |
+
declared = set((contract.get("input_schema") or {}).get("properties") or {})
|
| 284 |
+
try:
|
| 285 |
+
parameters = inspect.signature(capability.handler).parameters
|
| 286 |
+
except (TypeError, ValueError): # pragma: no cover - builtins
|
| 287 |
+
continue
|
| 288 |
+
if any(parameter.kind == parameter.VAR_KEYWORD for parameter in parameters.values()):
|
| 289 |
+
continue
|
| 290 |
+
missing = sorted(declared - set(parameters))
|
| 291 |
+
if missing:
|
| 292 |
+
broken[capability.name] = missing
|
| 293 |
+
|
| 294 |
+
assert not broken, f"schema declares properties the handler cannot accept: {broken}"
|
| 295 |
+
|
| 296 |
+
|
| 297 |
def test_declared_http_routes_exist(capabilities, live_route_paths) -> None:
|
| 298 |
missing = sorted(
|
| 299 |
f"{c.name} -> {c.http_route_path}"
|
|
@@ -1008,11 +1008,24 @@ def test_consult_tool_rejects_empty_question_argument() -> None:
|
|
| 1008 |
session_id="loop:consult-empty-question",
|
| 1009 |
)
|
| 1010 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1011 |
with pytest.raises(ValueError, match="question"):
|
| 1012 |
loop.tool_adapter.run_tool(
|
| 1013 |
conversation.session_id,
|
| 1014 |
"consult_warren_buffett",
|
| 1015 |
-
{"question": ""},
|
| 1016 |
)
|
| 1017 |
|
| 1018 |
|
|
|
|
| 1008 |
session_id="loop:consult-empty-question",
|
| 1009 |
)
|
| 1010 |
|
| 1011 |
+
# An empty string violates the contract's own `minLength: 1`, so schema
|
| 1012 |
+
# validation now rejects it at the tool boundary and hands the model a
|
| 1013 |
+
# retryable error instead of raising. The adapter's own guard still covers
|
| 1014 |
+
# what the schema cannot see: a whitespace-only question passes minLength.
|
| 1015 |
+
result = loop.tool_adapter.run_tool(
|
| 1016 |
+
conversation.session_id,
|
| 1017 |
+
"consult_warren_buffett",
|
| 1018 |
+
{"question": ""},
|
| 1019 |
+
)
|
| 1020 |
+
assert result.is_error is True
|
| 1021 |
+
assert result.error_code == "tool_invalid_arguments"
|
| 1022 |
+
assert "question" in result.error_message
|
| 1023 |
+
|
| 1024 |
with pytest.raises(ValueError, match="question"):
|
| 1025 |
loop.tool_adapter.run_tool(
|
| 1026 |
conversation.session_id,
|
| 1027 |
"consult_warren_buffett",
|
| 1028 |
+
{"question": " "},
|
| 1029 |
)
|
| 1030 |
|
| 1031 |
|
|
@@ -0,0 +1,253 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Argument validation at the tool boundary.
|
| 2 |
+
|
| 3 |
+
The declared `input_schema` used to be advisory: dispatch went straight from the
|
| 4 |
+
model's arguments to `capability.handler(**kwargs)`, so `additionalProperties:
|
| 5 |
+
False` enforced nothing and every undeclared handler parameter stayed reachable
|
| 6 |
+
from the agent. `valuation` is the case that matters — its hidden
|
| 7 |
+
`base_growth_pct` / `terminal_growth_pct` / `beta` let a caller tilt a valuation
|
| 8 |
+
toward a conclusion it already held, which is exactly what the locked-DCF rule
|
| 9 |
+
in the idea-loop design exists to prevent.
|
| 10 |
+
"""
|
| 11 |
+
|
| 12 |
+
import pytest
|
| 13 |
+
|
| 14 |
+
from TerraFin.agent.contracts.tool_contracts import validate_tool_arguments
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
def _problems(tool_name: str, arguments: dict) -> list[str]:
|
| 18 |
+
"""Just the problem list. `validate_tool_arguments` also returns the narrowed
|
| 19 |
+
arguments to dispatch; the tests that care about narrowing read it directly."""
|
| 20 |
+
|
| 21 |
+
return validate_tool_arguments(tool_name, arguments)[0]
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def test_valid_arguments_pass() -> None:
|
| 25 |
+
assert _problems("valuation", {"ticker": "AAPL", "projection_years": 10}) == []
|
| 26 |
+
assert _problems("market_data", {"name": "AAPL", "depth": "full"}) == []
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
def test_undeclared_valuation_tilt_parameters_are_rejected() -> None:
|
| 30 |
+
"""The whole point: these exist on the handler but not in the schema."""
|
| 31 |
+
|
| 32 |
+
for hidden in ("base_growth_pct", "terminal_growth_pct", "beta"):
|
| 33 |
+
problems = _problems("valuation", {"ticker": "AAPL", hidden: 42})
|
| 34 |
+
assert problems, f"{hidden} must not be reachable through the tool surface"
|
| 35 |
+
assert hidden in problems[0]
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
def test_missing_required_argument_is_reported() -> None:
|
| 39 |
+
assert _problems("valuation", {}) == ["ticker: required"]
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
def test_enum_violation_is_reported() -> None:
|
| 43 |
+
problems = _problems("market_data", {"name": "AAPL", "depth": "everything"})
|
| 44 |
+
|
| 45 |
+
assert len(problems) == 1
|
| 46 |
+
assert "not one of" in problems[0]
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
def test_numeric_bounds_are_enforced() -> None:
|
| 50 |
+
assert _problems("relative_strength", {"top_n": 101})
|
| 51 |
+
assert _problems("relative_strength", {"top_n": 0})
|
| 52 |
+
assert _problems("relative_strength", {"top_n": 50}) == []
|
| 53 |
+
|
| 54 |
+
|
| 55 |
+
def test_integral_floats_are_narrowed_not_merely_accepted() -> None:
|
| 56 |
+
"""Models routinely emit 20.0 for an integer field; 20.5 is a real error.
|
| 57 |
+
|
| 58 |
+
Accepting 20.0 without narrowing it to `int` only relocates the failure:
|
| 59 |
+
`relative_strength` reaches `ordered[:top_n]` and raises `TypeError: slice
|
| 60 |
+
indices must be integers`, which no classifier recognises, so the run aborts
|
| 61 |
+
instead of returning a tool error. The narrowed value is the contract.
|
| 62 |
+
"""
|
| 63 |
+
|
| 64 |
+
problems, normalized = validate_tool_arguments("relative_strength", {"top_n": 20.0})
|
| 65 |
+
assert problems == []
|
| 66 |
+
assert normalized["top_n"] == 20
|
| 67 |
+
assert isinstance(normalized["top_n"], int)
|
| 68 |
+
|
| 69 |
+
assert _problems("relative_strength", {"top_n": 20.5})
|
| 70 |
+
|
| 71 |
+
|
| 72 |
+
def test_narrowing_leaves_other_arguments_untouched() -> None:
|
| 73 |
+
_, normalized = validate_tool_arguments(
|
| 74 |
+
"relative_strength", {"top_n": 20.0, "universe": "sp500", "period": "6mo"}
|
| 75 |
+
)
|
| 76 |
+
|
| 77 |
+
assert normalized == {"top_n": 20, "universe": "sp500", "period": "6mo"}
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
def test_booleans_are_not_integers_or_strings() -> None:
|
| 81 |
+
"""bool is an int subclass in Python; JSON booleans are neither."""
|
| 82 |
+
|
| 83 |
+
assert _problems("relative_strength", {"top_n": True})
|
| 84 |
+
assert _problems("valuation", {"ticker": True})
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
def test_string_or_array_unions_accept_both_forms() -> None:
|
| 88 |
+
assert _problems("pattern_scan", {"tickers": "NVDA,AMD"}) == []
|
| 89 |
+
assert _problems("pattern_scan", {"tickers": ["NVDA", "AMD"]}) == []
|
| 90 |
+
assert _problems("pattern_scan", {"tickers": 123})
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
def test_array_bounds_and_item_types_name_the_actual_problem() -> None:
|
| 94 |
+
"""A union branch must not report the value's own type back at the model.
|
| 95 |
+
|
| 96 |
+
`tickers` is `anyOf: [string, array]`. Reporting "expected string or array,
|
| 97 |
+
got list" for an over-long list is unactionable — the value *is* a list — so
|
| 98 |
+
the model resends it, hits the same error fingerprint, and burns the retry
|
| 99 |
+
budget. Assert the wording, not mere truthiness.
|
| 100 |
+
"""
|
| 101 |
+
|
| 102 |
+
too_many = _problems("pattern_scan", {"tickers": ["X"] * 501})
|
| 103 |
+
assert too_many == ["tickers: at most 500 item(s), got 501"]
|
| 104 |
+
|
| 105 |
+
bad_item = _problems("pattern_scan", {"tickers": ["NVDA", 7]})
|
| 106 |
+
assert bad_item == ["tickers[1]: expected string, got int"]
|
| 107 |
+
|
| 108 |
+
# No branch matches on type, so the generic union message is the right one.
|
| 109 |
+
assert _problems("pattern_scan", {"tickers": 123}) == [
|
| 110 |
+
"tickers: expected string or array, got int"
|
| 111 |
+
]
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
def test_minimum_length_is_enforced() -> None:
|
| 115 |
+
assert _problems("valuation", {"ticker": ""})
|
| 116 |
+
|
| 117 |
+
|
| 118 |
+
def test_optional_nulls_are_left_to_the_handler() -> None:
|
| 119 |
+
"""An omitted-as-null optional is common from models and is not a schema error."""
|
| 120 |
+
|
| 121 |
+
assert _problems("news", {"ticker": "NVDA", "query": None}) == []
|
| 122 |
+
|
| 123 |
+
|
| 124 |
+
def test_a_null_required_argument_is_rejected() -> None:
|
| 125 |
+
"""`{"ticker": None}` satisfies "key is present" but not the handler.
|
| 126 |
+
|
| 127 |
+
`service.valuation` calls `ticker.upper()`, so a null that passes validation
|
| 128 |
+
becomes an `AttributeError` the classifier does not recognise, aborting the
|
| 129 |
+
run. Absent and null are different errors, and both are errors.
|
| 130 |
+
"""
|
| 131 |
+
|
| 132 |
+
assert _problems("valuation", {"ticker": None}) == ["ticker: required, but was null"]
|
| 133 |
+
assert _problems("valuation", {}) == ["ticker: required"]
|
| 134 |
+
|
| 135 |
+
|
| 136 |
+
def test_unknown_tool_validates_vacuously() -> None:
|
| 137 |
+
"""An unregistered tool is a different error, reported elsewhere."""
|
| 138 |
+
|
| 139 |
+
assert _problems("not_a_tool", {"anything": 1}) == []
|
| 140 |
+
|
| 141 |
+
|
| 142 |
+
@pytest.mark.parametrize("tool_name", ["consensus", "news", "pattern_scan", "relative_strength"])
|
| 143 |
+
def test_unknown_arguments_are_rejected_for_every_recent_tool(tool_name: str) -> None:
|
| 144 |
+
problems = _problems(tool_name, {"definitely_not_a_field": 1})
|
| 145 |
+
|
| 146 |
+
# Not problems[0]: a missing required argument is reported first for tools
|
| 147 |
+
# that have one.
|
| 148 |
+
assert any("unknown argument" in problem for problem in problems), problems
|
| 149 |
+
|
| 150 |
+
|
| 151 |
+
def _adapter_and_session():
|
| 152 |
+
"""A tool adapter over the real registry with a stub service.
|
| 153 |
+
|
| 154 |
+
Mirrors `tests/agent/test_tools.py::_adapter` — the runtime needs the service
|
| 155 |
+
as well as the registry, and the session id lives at
|
| 156 |
+
`session.session.session_id`.
|
| 157 |
+
"""
|
| 158 |
+
|
| 159 |
+
from fakes import BaseFakeService, fake_chart_opener
|
| 160 |
+
|
| 161 |
+
from TerraFin.agent.contracts.definitions import DEFAULT_HOSTED_AGENT_NAME
|
| 162 |
+
from TerraFin.agent.runtime import build_default_capability_registry
|
| 163 |
+
from TerraFin.agent.runtime.hosted import TerraFinHostedAgentRuntime
|
| 164 |
+
from TerraFin.agent.tools import TerraFinHostedToolAdapter
|
| 165 |
+
|
| 166 |
+
service = BaseFakeService()
|
| 167 |
+
registry = build_default_capability_registry(service, chart_opener=fake_chart_opener)
|
| 168 |
+
runtime = TerraFinHostedAgentRuntime(service=service, capability_registry=registry)
|
| 169 |
+
adapter = TerraFinHostedToolAdapter(runtime)
|
| 170 |
+
session = runtime.create_session(DEFAULT_HOSTED_AGENT_NAME, session_id="tool:schema")
|
| 171 |
+
return adapter, session.session.session_id
|
| 172 |
+
|
| 173 |
+
|
| 174 |
+
@pytest.mark.parametrize("tool_name", ["valuation", "start_valuation_task"])
|
| 175 |
+
def test_tool_path_rejects_a_hidden_valuation_parameter(tool_name: str) -> None:
|
| 176 |
+
"""End to end: the schema is enforced on BOTH surfaces the model can call.
|
| 177 |
+
|
| 178 |
+
`start_valuation_task` is the case that regressed. Validation keyed on
|
| 179 |
+
`tool.name`, and the task variant is named `start_valuation_task`, which is not
|
| 180 |
+
a `HOSTED_TOOL_CONTRACTS` key — so it validated vacuously and the tilt
|
| 181 |
+
parameters stayed reachable through the exact capability this check protects.
|
| 182 |
+
Every backgroundable capability had the same hole.
|
| 183 |
+
"""
|
| 184 |
+
|
| 185 |
+
adapter, session_id = _adapter_and_session()
|
| 186 |
+
|
| 187 |
+
result = adapter.run_tool(session_id, tool_name, {"ticker": "AAPL", "base_growth_pct": 40})
|
| 188 |
+
|
| 189 |
+
assert result.is_error is True
|
| 190 |
+
assert result.error_code == "tool_invalid_arguments"
|
| 191 |
+
assert "base_growth_pct" in result.error_message
|
| 192 |
+
assert result.payload["accepted"] is False
|
| 193 |
+
assert result.payload["error"]["retryable"] is True
|
| 194 |
+
assert "not available through the tool surface" in result.payload["error"]["modelHint"]
|
| 195 |
+
|
| 196 |
+
|
| 197 |
+
def test_every_exposed_tool_validates_against_a_contract() -> None:
|
| 198 |
+
"""No tool the model is handed may fall through validation.
|
| 199 |
+
|
| 200 |
+
The task-variant bypass was invisible because the tests only ever exercised
|
| 201 |
+
invoke-mode names. This asserts the property directly, over the full tool list
|
| 202 |
+
the default agent actually sees, so a future execution mode with a new name
|
| 203 |
+
shape cannot reintroduce it silently.
|
| 204 |
+
"""
|
| 205 |
+
|
| 206 |
+
from TerraFin.agent.contracts.definitions import DEFAULT_HOSTED_AGENT_NAME
|
| 207 |
+
from TerraFin.agent.contracts.tool_contracts import HOSTED_TOOL_CONTRACTS
|
| 208 |
+
|
| 209 |
+
adapter, _ = _adapter_and_session()
|
| 210 |
+
tools = adapter.list_tools_for_agent(DEFAULT_HOSTED_AGENT_NAME)
|
| 211 |
+
|
| 212 |
+
assert len(tools) > 40, "expected the full tool surface, not a filtered subset"
|
| 213 |
+
unvalidated = sorted(t.name for t in tools if t.capability_name not in HOSTED_TOOL_CONTRACTS)
|
| 214 |
+
assert not unvalidated, f"tools whose arguments no contract constrains: {unvalidated}"
|
| 215 |
+
|
| 216 |
+
|
| 217 |
+
def test_task_variants_still_start_with_valid_arguments() -> None:
|
| 218 |
+
"""The D1 fix must reject the undeclared argument, not the whole task path."""
|
| 219 |
+
|
| 220 |
+
adapter, session_id = _adapter_and_session()
|
| 221 |
+
|
| 222 |
+
result = adapter.run_tool(session_id, "start_valuation_task", {"ticker": "AAPL"})
|
| 223 |
+
|
| 224 |
+
assert result.is_error is False
|
| 225 |
+
assert result.payload["accepted"] is True
|
| 226 |
+
assert result.task is not None
|
| 227 |
+
assert result.task.input_payload["ticker"] == "AAPL"
|
| 228 |
+
|
| 229 |
+
|
| 230 |
+
def test_tool_path_still_accepts_declared_arguments() -> None:
|
| 231 |
+
adapter, session_id = _adapter_and_session()
|
| 232 |
+
|
| 233 |
+
result = adapter.run_tool(session_id, "valuation", {"ticker": "AAPL", "projection_years": 10})
|
| 234 |
+
|
| 235 |
+
assert result.is_error is False
|
| 236 |
+
assert result.payload["ticker"] == "AAPL"
|
| 237 |
+
|
| 238 |
+
|
| 239 |
+
def test_internal_callers_keep_the_full_signature() -> None:
|
| 240 |
+
"""Locking the agent must not lock a route or controller.
|
| 241 |
+
|
| 242 |
+
The service method deliberately still accepts the tilt parameters — a
|
| 243 |
+
deliberate internal caller is trusted; the model is not.
|
| 244 |
+
"""
|
| 245 |
+
|
| 246 |
+
import inspect
|
| 247 |
+
|
| 248 |
+
from TerraFin.agent.service import TerraFinAgentService
|
| 249 |
+
|
| 250 |
+
parameters = inspect.signature(TerraFinAgentService.valuation).parameters
|
| 251 |
+
|
| 252 |
+
for hidden in ("base_growth_pct", "terminal_growth_pct", "beta"):
|
| 253 |
+
assert hidden in parameters
|
|
@@ -162,7 +162,26 @@ class BaseFakeService:
|
|
| 162 |
"processing": processing(),
|
| 163 |
}
|
| 164 |
|
| 165 |
-
def valuation(
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 166 |
return {
|
| 167 |
"ticker": ticker,
|
| 168 |
"dcf": {"status": "ready", "intrinsic_value": 120.0},
|
|
|
|
| 162 |
"processing": processing(),
|
| 163 |
}
|
| 164 |
|
| 165 |
+
def valuation(
|
| 166 |
+
self,
|
| 167 |
+
ticker: str,
|
| 168 |
+
*,
|
| 169 |
+
projection_years: int | None = None,
|
| 170 |
+
fcf_base_source: str | None = None,
|
| 171 |
+
breakeven_year: int | None = None,
|
| 172 |
+
breakeven_cash_flow_per_share: float | None = None,
|
| 173 |
+
post_breakeven_growth_pct: float | None = None,
|
| 174 |
+
) -> dict[str, object]:
|
| 175 |
+
"""Accepts the schema's declared optionals, deliberately NOT the hidden
|
| 176 |
+
tilt parameters (`base_growth_pct`, `terminal_growth_pct`, `beta`) — those
|
| 177 |
+
must not be reachable from the tool surface."""
|
| 178 |
+
_ = (
|
| 179 |
+
projection_years,
|
| 180 |
+
fcf_base_source,
|
| 181 |
+
breakeven_year,
|
| 182 |
+
breakeven_cash_flow_per_share,
|
| 183 |
+
post_breakeven_growth_pct,
|
| 184 |
+
)
|
| 185 |
return {
|
| 186 |
"ticker": ticker,
|
| 187 |
"dcf": {"status": "ready", "intrinsic_value": 120.0},
|