mateclaw/mateclaw-server/src/test/resources/agent-evaluation

Offline Agent task policy replay (v1)

This harness replays synthetic evaluator replies against the production GoalEvaluationService and GoalCriteriaCodec. It executes neither a task-solving Agent nor workspace commands, and it cannot establish model success rates, artifact truth, prompt quality, online latency or cost. The mock ChatModel is the only provider; no credentials or network are required by this harness. Maven may need network access to resolve missing build dependencies.

From the repository root (Java 21 and Maven):

mvn -pl mateclaw-server -am -Dtest=OfflineGoalTaskReplayTest \
  -Dsurefire.failIfNoSpecifiedTests=false \
  -Dagent.eval.revision="$(git rev-parse HEAD)" test

The report is written to mateclaw-server/target/agent-evaluation/goal-baseline.json. -Dagent.eval.suite=/absolute/path/suite.json selects an alternate suite; -Dagent.eval.report=/absolute/path/report.json changes the output. Use separate output paths when comparing versions. Without a revision argument the report says unrecorded; do not use that report for revision comparisons. Record a dirty working tree separately; a supplied Git revision identifies committed source, not uncommitted modifications. revisionSource=caller_supplied_label makes that limitation explicit. Reports additionally include executedClassSha256 for the loaded GoalEvaluationService, GoalCriteriaCodec, GoalCriterion and GoalEvaluationResult class bytes. Compare under the same compiler/build: these hashes cover the named classes, not all dependencies, configuration or the OS. The historical baseline-v1.json remains unchanged and predates these hashes; its revision names production code from cycle-001. baseline-v2.json records a later build of the same ten cases with class hashes. Its working-tree label is explicit; it is not a new suite or ten additional Agent executions.

Schema v1 has schemaVersion, suiteId, and 1–100 tasks. Each task declares:

Field	Meaning
id	Unique nonblank case ID, stable across revisions
task	Human-readable user task
source	Regression/test/design provenance; synthetic inputs are identified
boundary	Distinct behavior or known limitation under inspection
persistent	Whether cumulative Goal semantics apply
criteria	Initial Goal checklist, or [] for bootstrap
terminalAnswer	Fixed assistant final answer (empty is a valid boundary)
evaluatorResponse	Exact synthetic model response string, including malformed JSON cases
expected	Required completed flag, score, decision, ordered remaining criterion IDs and fixture call count

Expected scores must be finite in [0,1], decisions are completed, continue, or fallback, and fixture call counts are 0 or 1. Empty suites, duplicate IDs, missing expected values, unknown fields and trailing JSON are rejected. The whole suite is validated before replay. Expectations are independent fixed inputs, not generated by copying the current runtime result. For a deliberate behavior change, review and explain each changed expectation.

Reports bind the exact suite bytes with SHA-256 and record the supplied source revision label and its source, executed class hashes, execution mode, every expected/actual result, and mismatches. A mismatch writes the report then fails the test/Maven command; all cases are still run. Invalid suites fail before a new report is written, so an existing report may be stale: always check the command exit status. matchedCases counts policy replay agreement, not successful Agent tasks. onlineModelCalls is zero; agentTaskSuccessRate and onlineCost are not_measured.

Ten initial cases cover distinct completion boundaries. The forged-text case intentionally expects semantic completion: no report file is created or checked. That known limitation demonstrates why nonblank evidence and Observe are not strong acceptance. A future trusted file recipe must use a separate execution mode and record its actual filesystem/version checks. A real model comparison must additionally pin runtime/skill/model versions and measure real calls; it cannot reuse these matched counts as its success rate.

Artifact tasks with actual temporary-file I/O

artifact-boundaries-v1.json is a separate schema/execution mode. Run it from the repository root:

mvn -pl mateclaw-server -am -Dtest=OfflineArtifactTaskReplayTest \
  -Dsurefire.failIfNoSpecifiedTests=false \
  -Dartifact.eval.revision="$(git rev-parse HEAD)" test

Output: mateclaw-server/target/agent-evaluation/artifact-baseline.json. Override artifact.eval.suite and artifact.eval.report with absolute paths for candidate runs. The initial committed artifact baseline's revision label refers to the production source before this harness was added. Reports additionally hash the actual loaded GeneratedFileCache, Entry, and ExecutionObservationSink class bytes, so a revision label alone is not treated as proof of which implementations were exercised. Class hashes depend on the compiler/build; compare revisions under the same build environment. They cover these named classes, not the entire transitive runtime or a signed attestation.

Each artifact task has id, task, source, operation, content, and expected. Operations are a closed set: REGISTER, MUTATE_INPUT, MUTATE_DOWNLOAD, RESTART_MUTATE_DOWNLOAD, STORAGE_UNAVAILABLE, DIRECT_RETURN, MISSING_OWNER, REWRITE_DISK. Input strings are bounded to 16,384 characters. The harness generates its own temporary paths and never executes fixture-provided paths, scripts or shell commands.

Expected results include hotContent, nullable coldContent, observations ([] or ["ARTIFACT_SNAPSHOT:OBSERVED"]), and nullable hotMatchesSnapshot/coldMatchesSnapshot. A match is null when no snapshot or readable version exists; it never silently becomes a pass. Every task uses the production cache and sink, writes/reads real temporary files, and reopens the cache to test restart behavior. The directory is removed by JUnit after the run. A mismatched expectation preserves all per-case results and fails the command. The checked-in artifact-baseline-v1.json documents these eight scenarios.

REWRITE_DISK intentionally observes that the hot cache keeps the original bytes while a new cache reads the externally replaced version, which no longer matches the historic snapshot digest. This is a documented limit of unmanaged storage, not an accepted strong-validation result. The report never emits a CHECK_RESULT/PASS. These eight tests plus ten evaluator replays are 18 fixed scenarios, not 18 Agent task executions. Actual task-solving models, their latency, cost and success rate remain unmeasured.

JSON artifact check pilot

The evidence details panel can explicitly check a registered JSON file with POST /api/v1/execution-evidence/{id}/json-check and a body such as {"requiredFields":["report","appendix"]}. The server uses the conversation and file permissions of the authenticated caller. Field names are exact top-level keys (1–16 unique keys, up to 128 characters each); values must be non-null. This does not validate value types, business correctness, or document quality.

The fixed json-required-fields recipe revision 1 reads at most the configured artifact-version budget, capped at 1 MiB. Zero disables reading. Durable bytes must match the registered digest before parsing. The result distinguishes MATCH, MISSING_FIELDS, INVALID_JSON, UNKNOWN, STALE, UNAVAILABLE and always has acceptanceEligible=false. Duplicate keys, multiple JSON documents, and nesting deeper than 32 are rejected. Responses contain no file values or parser excerpts. No Goal criterion, acceptance binding, or evidence PASS is written. The result describes the captured bytes at the displayed time; shared storage still has no managed generation/owner fence.

Run the separate six-case IO/recipe replay:

mvn -pl mateclaw-server -am -Dtest=OfflineJsonArtifactTaskReplayTest \
  -Dsurefire.failIfNoSpecifiedTests=false \
  -Djson.artifact.eval.revision="$(git rev-parse HEAD)" test

Default output: mateclaw-server/target/agent-evaluation/json-artifact-baseline.json. Override json.artifact.eval.suite and json.artifact.eval.report with absolute paths. json-artifact-boundaries-v1.json contains requirements and explicit expected status, missing fields, whether the recipe actually ran, and whether acceptance was granted. The harness validates all tasks before creating its own temporary paths. Operations are allowlisted: CHECK, REWRITE_DISK, TOO_SMALL_BUDGET, FOREIGN_OWNER. It never executes fixture paths or commands.

The report hashes the suite and loaded cache/read-result/recipe/result classes, records every mismatch, and fails the command if any case differs. Class hashes identify these loaded classes under this build, not every dependency. json-artifact-baseline-v1.json is the initial six-case baseline. Together with the earlier ten evaluator and eight platform IO scenarios there are 24 fixed scenarios, not 24 real Agent runs. The JSON replay uses real temporary durable files and the production recipe; it does not exercise HTTP authorization or a browser. Those boundaries have separate service/UI regressions. Its mode is offline_platform_fixture_jsoncheck; online calls are zero and Agent success rate/online cost remain not_measured.

H2 Goal service scenarios

goal-service-boundaries-v1.json adds six service workflows from the real completion/append/pause/bootstrap/definition-edit regressions. Run:

mvn -pl mateclaw-server -am -Dtest=OfflineGoalServiceTaskReplayTest \
  -Dsurefire.failIfNoSpecifiedTests=false \
  -Dgoal.service.eval.revision="$(git rev-parse HEAD)" test

Output: mateclaw-server/target/agent-evaluation/goal-service-baseline.json. goal.service.eval.suite and goal.service.eval.report accept absolute paths. The runner uses the actual Spring transactional Goal service and a fresh H2 MySQL-compatibility database with Flyway migrations. This is not a real MySQL/Kingbase run. It disables the autonomous Goal scheduler and supplies fixed evaluator results; no task-solving Agent or model is invoked. The external MemoryManager boundary is explicitly mocked so completion cannot sync to configured memory providers. Plugin loading is disabled and skill workspace reads use a temporary root; the Goal service, repositories and H2 remain real. The report lists this mocked boundary, rather than calling the entire app real.

Expected fields describe persisted status, score, criterion counts and first text, evaluation-definition revision, recorded evaluator usage and completion outcome. Completion code 0 means not attempted, 200 means the existing semantic completion service accepted the fixture, and 409 means it rejected the transition. A semantic completion is not execution-required acceptance. recordedEvalCallsUsed exercises bookkeeping with fixture deltas; it is not a count of online calls. The entire suite is validated before any service write; conversations use generated IDs and fixture input cannot supply SQL or paths.

Reports include suite and named production class hashes, the actual database product and latest applied Flyway version. These identify selected build/schema facts, not the entire runtime. Mismatches preserve all case results and fail the CLI. goal-service-baseline-v1.json is the initial six-case baseline.

The four suites now contain 30 fixed scenarios: ten synthetic evaluator responses, eight platform artifact IO scenarios, six JSON artifact checks and six H2 Goal service workflows. They are different execution modes and include related regression boundaries; do not treat them as 30 independent online Agent task attempts. Online calls remain zero, and online cost/Agent success rate remain not_measured. HTTP authorization, browser flows and distributed owner/scope fences are not exercised by the H2 replay.

All four suite parsers reject duplicate JSON keys at every nesting level, before running any case. This includes duplicate schemaVersion and keys inside expected, even if the duplicate values agree. Correct the input rather than relying on last-value-wins parsing. The four parser regression inputs are schema checks, not additional task scenarios; the fixed task count remains 30.