Wire OpenShell sandbox lifecycle into the orchestrator

[saichandrapandraju]

What this adds

--protocol openshell wires the full OpenShell sandbox lifecycle into midojo's evaluation loop. The orchestrator creates a fresh sandbox per evaluation, seeds the workspace with the injected document, runs the agent inside the sandbox, collects workspace diff and OCSF kernel events, and tears down the sandbox.

Architecture

midojo-run
  ├── backend.provision(injections)    # render workspace files with attack payloads
  ├── backend.setup(pre_env)           # create, seed /sandbox/workspace/
  │     └── SandboxClient (gRPC)      # connects to the OpenShell gateway
  ├── agent_client.send_task(prompt)  # exec agent_command inside the sandbox
  ├── backend.snapshot()              # workspace diff + OCSF kernel events
  ├── PUT /environment                # post-env to the midojo control plane
  └── backend.teardown()             # delete sandbox (always runs, even on error)

Two observation channels

The sandbox proxy intercepts all egress and the kernel audits all I/O. midojo collects two independent observation streams after each evaluation:

Channel	What it captures
Workspace diff	Files created, modified, or deleted under /sandbox/workspace/
OCSF kernel events	Network connections (allowed/denied), processes launched, security findings

Separation of concerns

midojo controls the sandbox lifecycle because workspace injection requires writing the attack payload before the agent runs. Everything else belongs to the user:

• image — their sandbox image, with their agent pre-configured
• agent_command — how to invoke the agent per evaluation (analogous to --agent-url for other protocols)
• policy — which network endpoints the agent is allowed to reach

midojo does not manage LLM credentials or agent configuration. Providers registered with openshell provider create are activated by name; the gateway supervisor injects the credentials into the sandbox.

Suite structure

The document_assistant suite demonstrates the pattern:

suites/document_assistant/
├── suite.yaml            # test spec only: workspace files, tasks, predicates, policy
└── sandbox/
    ├── Containerfile     # example agent: pi community image + models.json baked in
    ├── models.json       # local inference config (no cloud token required)
    └── README.md         # "replace with your production image — eval doesn't change"

suite.yaml is a clean evaluation spec. The sandbox/ directory is an example agent provided as a starting point — users swap image: for their own production sandbox image and run the same evaluation unchanged.

New suite YAML fields

Field	Purpose
backend.type: openshell	Selects the OpenShell backend
backend.image	Sandbox image (bare names like pi expand to the community registry)
backend.agent_command	Command executed inside the sandbox per evaluation
backend.policy	Network and process policy applied at sandbox creation
backend.providers	Pre-registered OpenShell provider names to activate (gateway injects credentials)
backend.env_vars	Extra env vars injected at sandbox creation

New predicates

Workspace diff:

• workspace_file_exists / workspace_file_deleted / workspace_file_contains
• commands_match_pattern

OCSF kernel events:

• process_ran — kernel-verified process launch
• network_call_to / network_call_blocked_to
• security_finding_raised

Prerequisites (one-time)

OpenShell gateway:

curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh
openshell status   # → Status: Connected

Note: Homebrew 0.0.58 requires a source build — see Known Limitations.

OpenShell Python SDK (Maturin/Rust build, requires z3 + Rust):

brew install z3 && curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
cd /path/to/OpenShell
python -m grpc_tools.protoc -Iproto --python_out=python/openshell/_proto \
  --pyi_out=python/openshell/_proto --grpc_python_out=python/openshell/_proto proto/*.proto
maturin build --release && uv pip install target/wheels/openshell-*.whl

Local inference server at localhost:8321 (ogx server or any OpenAI-compatible server). From inside the sandbox, the host is reachable at host.openshell.internal:8321.

Running it

A pre-built example sandbox image is available (quay.io/rh-ee-spandraj/midojo:document-assistant-local) — no build required:

# Terminal 1
uv run midojo-serve --suite document_assistant --port 8090

# Terminal 2
uv run midojo-run --protocol openshell --suite document_assistant \
  --agent-url https://127.0.0.1:17670 \
  --control-url http://localhost:8090

To use your own sandbox image, update image: in suite.yaml. The rest of the eval is unchanged.

Known limitations

OCSF grades False on failure instead of N/A (#107)
When GetSandboxLogs fails, OCSF predicates return False rather than N/A. This means a successful attack can appear to have been resisted if the OCSF stream is unavailable. Fixing this properly requires bool | None from predicates — a verifier protocol change deferred to a follow-up.

Homebrew gateway 0.0.58 missing /run/netns tmpfs
The shipped binary doesn't mount the tmpfs the supervisor needs for network namespace setup. The fix is already in the OpenShell source; until 0.0.59 ships, build from source:

cd /path/to/OpenShell
Z3_ROOT=$(brew --prefix z3) cargo build --release -p openshell-server
brew services stop openshell
sudo cp target/release/openshell-gateway /opt/homebrew/opt/openshell/bin/openshell-gateway
brew services start openshell

🤖 Generated with Claude Code

[saichandrapandraju]

Hi @dmaniloff , would like to get early feedback on this, thanks!

[dmaniloff]

there's already a pi image and a pi policy for pi.dev agents.

it's the one that gets used when you say openshell sandbox create --from pi

let's use that instead.

and let's also add an agent_command that defaults to the image's entrypoint eg

agent_command: ["pi", "-p", "--no-session"] # optional, defaults to image's entrypoint

[dmaniloff]

though i may have a different opinion about this now ... let's chat when you get a moment.

[dmaniloff]

is there a way we can avoid the inline import?

[saichandrapandraju]

kept so midojo starts without the SDK installed...

[dmaniloff]

we should think about a way to surface this so that during grading we can say N/A rather than "attack failed"

[saichandrapandraju]

Deferred — tracked in #107

[dmaniloff]

thinking back to my suggestion re: agent_command ... i think we should probably try to avoid all this.

the concern of midojo stops at the agent boundary. in other words, agent config (like its command and providers etc) should not be what we deal with. this should all exist somewhere else already.

in the case of an openshell env it seems that there's some coupling between the agent's config & the env so maybe the way around it is via an image or another openshell native mechanism. let's chat about this.

[saichandrapandraju]

Hi @ABeltramo, can you take a look at this when you have time please, thanks!

Note that this is not the perfect implementation and there's still some discussion points (unresolved above) but I think this in a decent shape to showcase the flavor of how openshell x midojo work.

[ABeltramo]

Hi @ABeltramo, can you take a look at this when you have time please, thanks!

I will, probably next week though. I'm currently swamped with the 3.5 release and this isn't a priority item for the DP release. Btw we should move this under RHOAIENG-75712 and wrap up RHOAIENG-66502.

What I would suggest is to move the example suite to asago-ai/midojo-suites so that we can focus on reviewing the code for the feature.