Skip to main content

Report statuses and evidence

Available in CLI v2.1.8

The status and evidence language described on this page is introduced in Testronaut CLI v2.1.8. Reports produced by earlier CLI versions remain supported and use the backwards-compatible presentation described below.

Testronaut reports separate execution activity from mission outcomes. A tool running normally does not, by itself, prove that the mission goal was achieved.

Tool and turn statuses​

  • 🛰️ Engaged means the tool invocation ran without a structured operation verdict.
  • ✅ Action confirmed or ⚠️ Action unconfirmed reflects an explicit tool result such as ok or passed.
  • 🚀 Progressing is an intermediate turn, not a pass verdict.
  • 🔄 Replotting course means Testronaut is retrying after an issue.
  • ⚠️ Navigation issue identifies a tool or browser-action problem.

For example, clicking Create Work Package can be an engaged action while creation remains unconfirmed. The report should not imply that the work package exists until later evidence supports that conclusion.

Mission outcomes​

The machine-readable mission status remains one of passed, recovered, or failed. Successful missions can also include an optional verification level:

  • evidence_recorded: the trace contains supporting browser observations, assertion results, screenshots, or matched expected dialogs.
  • verified: a separate verifier, such as the Jev completion gate or a launch-protocol probe, independently confirmed the outcome.
  • limited_evidence: the agent reported completion, but the trace did not contain enough supporting evidence.

Reports present these as:

  • ✅ Passed · evidence recorded
  • ✅ Passed · independently verified
  • ⚠️ Passed · limited evidence
  • ❌ Failed

Independent verification is a stronger confidence signal, not a requirement for an ordinary successful mission. A well-evidenced browser journey can be reported as accomplished without enabling Jev.

Backwards compatibility​

The verification fields are optional additions to report schema version 3. Older reports that do not contain verification, evidenceSources, or step-level verification metadata remain valid and display their original ✅ Passed, ⚠️ Passed after recovery, or ❌ Failed mission status.

Older intermediate turns may contain ✅ Passed. Current viewers treat that legacy label as neutral turn activity rather than proof of mission completion. Summary counts, status filters, comparisons, CI exit behavior, and upload compatibility continue to use the unchanged passed, recovered, and failed values.

Example mission data:

{
  "status": "passed",
  "verification": "evidence_recorded",
  "evidenceSources": ["browser_state", "matched_dialog"],
  "steps": [
    {
      "result": "✅ Mission accomplished · evidence recorded",
      "verification": {
        "level": "evidence_recorded",
        "sources": ["browser_state", "matched_dialog"]
      }
    }
  ]
}