Skip to main content

Jev guardrails

Jev guardrails provide a low-cost second opinion about whether a browser mission has enough evidence to finish. Your configured provider and model still own every reasoning turn. Guardrails never switch execution models.

Guardrails are optional and require your own Jev API key.

Create or manage the key in the TypeSafe AI API console, and use the Jev API reference for authentication and endpoint details.

Enable guardrails​

Set the key without committing it to source control:

export TESTRONAUT_JEV_API_KEY="your-key"

Then enable live guardrails for a run:

testronaut --guardrails missions/login.mission.js

To observe recommendations without allowing early completion:

testronaut --guardrails=shadow missions/login.mission.js

Project configuration​

Guardrails remain opt-in. Add this block to testronaut-config.json when every run in the project should use them:

{
  "guardrails": {
    "enabled": true,
    "mode": "live",
    "completionProbability": 0.8
  }
}

Use testronaut config --json to verify the mode, threshold, and whether a credential was detected. The key value is never printed.

Failure behavior​

Guardrails fail open. If Jev is unavailable, times out, returns an error, or lacks sufficient evidence, Testronaut continues with the primary model. A live guardrail completion also respects resource completeness checks and final screenshot requirements.

Reports​

Reports list the primary model together with Jev and separate model tokens from Jev tokens at run, mission, and turn levels. They also record candidate and triggered completions, errors, skipped evaluations, and latency.

An individual report does not claim tokens saved because it has no counterfactual run. Use observed token volume and triggered completion information when comparing repeated benchmark runs.

Model routing​

Model switching is a separate beta feature. It is not enabled by --guardrails, even if a fast model is configured. Beta routing requires an explicit --beta-model-routing opt-in.