Metadata-Version: 2.4
Name: evostudio-inference-sdk
Version: 0.7.0b3
Summary: Inference integration SDK for EvoStudio simulation benchmarks
Author: EvoMind Team
License-Expression: Apache-2.0
Project-URL: Homepage, https://studio.evomind-tech.com/
Project-URL: Documentation, https://studio.evomind-tech.com/docs/%E6%A8%A1%E5%9E%8B%E8%AF%84%E6%B5%8B%E5%AE%98%E6%96%B9%E9%95%9C%E5%83%8F%E4%B8%8E%E8%87%AA%E5%AE%9A%E4%B9%89%E9%95%9C%E5%83%8F
Keywords: robotics,simulation,evaluation,inference
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Dynamic: license-file

# EvoStudio Inference SDK

`evostudio-inference-sdk` is the model-integration boundary for EvoStudio's official simulation
benchmarks. Users implement one `Policy` lifecycle and do not install simulator packages or configure
benchmark scheduling and autotune parameters.

Install the SDK from PyPI:

```bash
python -m pip install evostudio-inference-sdk
```

```python
import numpy as np

from evostudio_inference_sdk import ActionChunk, BenchmarkName, Policy, PolicyCapabilities, run


class UserPolicy(Policy):
    def load(self, context):
        self.model = load_model(context.checkpoint_path)
        return PolicyCapabilities(
            supported_benchmarks=(BenchmarkName.LIBERO,),
            required_features=("images.primary", "state.proprioception"),
            supported_suites={BenchmarkName.LIBERO: ("libero_10",)},
            supported_action_chunk_sizes=(1,),
        )

    def act(self, observation):
        return ActionChunk(np.asarray(self.model.predict(observation.features), dtype=np.float32))


run(UserPolicy())
```

EvoStudio launches this entrypoint once per selected inference replica, passes the staged checkpoint and
benchmark contract to `Policy.load()`, and calls it through the local SDK protocol. The user process does
not start a simulator or select evaluator concurrency.

The SDK also contains EvoStudio's platform benchmark runtime, durable work queue, process lifecycle,
resource measurement, and automatic K/g selection under `evostudio_inference_sdk.runtime`. Those modules
are platform internals and are not exported from the user-facing package API. Installing the PyPI package
provides the public Policy API and protocol client; it does not install simulators, benchmark datasets,
model frameworks, or the platform image dependencies needed to execute those internal runtime modules.

Earlier source releases exposed a `libero` extra, but it installed only `imageio[ffmpeg]`; it never
installed a runnable LIBERO simulator or dataset. The extra has therefore been removed. Existing
environments keep their already-installed packages, but consumers that use `imageio` directly must now
declare `imageio[ffmpeg]>=2.34,<3.0` in their own image or application dependencies. This is only a video
dependency migration and is not a supported way to provision any Benchmark runtime.

The platform runtime uses three core scheduling concepts:

- **Server**: an inference process that loads one user `Policy` replica and serves action requests over a
  local socket. A Server does not run a simulator. Multiple Workers may share one Server, which can batch
  their concurrent requests. Autotune calls the selected Server replica count `g`.
- **Worker**: an evaluator process that owns a benchmark simulator lane. It claims Work Items, runs their
  environments and episodes, sends observations to a Server, applies the returned actions, and publishes
  results. A Worker does not load a separate copy of the user model. Autotune calls the selected Worker
  concurrency `K`.
- **Work Item**: the durable, atomically claimed unit of benchmark work. Its exact size is adapter-specific;
  for example, a RoboTwin Work Item is one task with its configured episodes. A failed Worker returns its
  claimed Work Item to the pending queue, while a completed Work Item is recorded idempotently so it cannot
  be scored twice.

RoboTwin and RoboCasa multi-episode work items publish each completed episode as an observable partial
result. Product progress therefore advances while the item is still running, while official aggregation
and durable completion continue to use only the item's final result. A non-OOM evaluator worker failure
restarts the worker and replays its claimed item, with one retry allowed per runtime pass. Observable
progress is recomputed from the durable queue snapshot and published through the atomic `testlog.txt`
status file.

Runtime errors are collected in the append-only `/root/testresult/errors.jsonl` journal. It includes
recoverable Worker failures as well as terminal checkpoint, Policy Server, simulator, configuration, and
other evaluation errors. Worker records include the original traceback excerpt, and a terminal Worker
failure carries that excerpt into the top-level error so Quick Test users see the simulator or Policy
failure instead of only the Worker exit code. Component logs remain available for detailed diagnosis.

`PolicyCapabilities` declares the supported benchmarks, maximum inference batch size, and safe
action-chunk sizes. Evaluator concurrency and replica count are platform-owned autotune parameters rather
than user-declared capabilities. SDK-native LIBERO scored evaluations measure real Policy-backed episodes
across K/g candidates and select the last candidate that improves throughput while resources remain safe.
Candidates that lose Workers, report a Worker failure or OOM, or leave probe episodes incomplete are not selected.

## API stability and versioning

The names exported by `evostudio_inference_sdk.__all__` are the public SDK API. Modules below
`evostudio_inference_sdk.runtime` are platform implementation details unless this document explicitly
states otherwise; applications should not depend on their internal classes or command-line arguments.

The package follows Semantic Versioning. During the current `0.x` Beta series, a minor release may make an
incompatible change to `__all__`, the public Policy lifecycle, or the wire protocol, while patch releases
remain compatible bug fixes. Every evaluation image pins one exact SDK version. User images should also
pin the version they validated, and review release notes before upgrading to another minor version. The
first `1.0.0` release will freeze the public Policy lifecycle and compatibility policy for stable use.

The installed version comes from distribution metadata and can be read with
`importlib.metadata.version("evostudio-inference-sdk")`; `evostudio_inference_sdk.__version__` is not part
of the public API. The release owner updates the version in `pyproject.toml` and `uv.lock`, moves the
`CHANGELOG.md` Unreleased entries into a dated release section, and records every user-visible API,
Policy-lifecycle, and wire-protocol change in the same release merge request.

`CHANGELOG.md` is included in the source distribution, but is intentionally not copied into the wheel's
runtime package. Wheel users can read the same release notes in the repository or PyPI source archive.

## Episode-scoped state

The runtime calls `reset(EpisodeContext)` before an episode, sends its `episode_id` on each observation,
and calls `end_episode(episode_id)` when the episode finishes. The SDK forwards this lifecycle but does not
store or synchronize mutable Policy state.

Stateful Policies should isolate mutable state by `episode_id` and release it from `end_episode()`.
Stateful and stateless Policy examples are provided in the EvoStudio user manual.

## Model-owned input preprocessing

Each Benchmark sends the canonical observation defined by its Benchmark contract without model-specific
resize, normalization, or channel/layout conversion. The user Policy performs those transformations before
calling its model. Model input dimensions therefore do not belong in the public Benchmark or Policy API.

## Current benchmark status

The repository contains the following platform execution chains. Their manifests describe dependencies
that EvoStudio's Benchmark images or shared storage must provide; they are not dependencies of the PyPI
distribution. A chain being present here does not guarantee that every deployment has provisioned its
external artifacts.

| Benchmark | Platform execution chain | Image/shared-storage dependencies |
| --- | --- | --- |
| LIBERO | SDK-native adapter and persistent LIBERO worker | Pinned LIBERO Python environment, simulator source, dataset and initial states |
| LIBERO+ | SDK-native adapter that reuses the LIBERO worker | Pinned LIBERO+ Python environment, simulator archive, dataset and initial states |
| RoboTwin | SDK-native adapter and worker that invoke the pinned RoboTwin launcher | RoboTwin environment and source, deploy policy, checkpoint assets and validated seed cache |
| RoboCasa | SDK-native adapter and persistent RoboCasa worker | Pinned RoboCasa environment, simulator/task source and dataset assets |
| CALVIN | SDK-native adapter and persistent CALVIN worker | Pinned CALVIN environment, benchmark/config source, dataset and evaluation-sequence manifest |

LIBERO and CALVIN convert their native observations to the canonical SDK feature names and send native
7-dimensional actions back to their simulators. CALVIN's task-D-to-task-D path implements the official
1,000 long-horizon sequences with static/gripper images, proprioception and language. The suite list and
runtime paths for every chain remain owned by its installed Benchmark image manifest.
- Evaluation has two product paths. Custom-image users provide their own SDK `Policy` and saved image.
  StarVLA users may instead upload a standard run snapshot and directly select the official StarVLA image.
  Both paths use platform Benchmark environments and evaluators from `/autodl-fs`.

The complete LIBERO entrypoint and scoring contract is documented in
`LIBERO_EVALUATION_PROTOCOL.md` in the Evaluation Kit repository.
The LIBERO document defines the `/root/startbenchmark.sh` arguments, checkpoint staging, canonical
observation/action mapping, episode initialization and horizons, success detection, output files, error
codes, and required validation sequence.
The SDK-native CALVIN manifest, observation/action mapping, official long-horizon evaluator, aggregation,
and validation procedure are documented in
`CALVIN_EVALUATION_PROTOCOL.md` in the Evaluation Kit repository.

The SDK integration, autotune, and benchmark-runtime throughput results are maintained in the Evaluation
Kit repository. The report keeps LIBERO and RoboTwin results separate and includes the StarVLA official
four-GPU RoboTwin launcher baseline used for the runtime comparison.

## Local policy HIL worker

`evostudio-inference-worker` is the independent model process used by
`policy_hil_continuous` collection. It reuses the SDK Unix-socket wire framing,
binds no network address, and never opens robot or camera devices. Local
checkpoints and all referenced base/tokenizer models must be inside explicitly
allowlisted roots:

```bash
python -m evostudio_inference_sdk.runtime.hil.worker \
  --socket /run/evostudio-inference/worker.sock \
  --checkpoint-root outputs=/home/dell/outputs \
  --dependency-root /home/dell/models \
  --hash-cache /var/lib/evostudio-inference/checkpoint-hashes.json \
  --device cuda:0 \
  --torch-threads 4 \
  --gpu-memory-fraction 0.8 \
  --worker-git-commit <evaluation-kit-commit> \
  --policy-runtime-version <pinned-lerobot-runtime>
```

The first runtime declares support only for complete Pi05 checkpoints. Catalog
discovery does not trust browser-provided paths. Loading verifies checkpoint and
dependency hashes, validates observation/action schemas, runs a real dry-run,
and freezes the selected session. The worker accepts one inference request at a
time and rejects stale policy epochs so an action chunk generated before an
intervention cannot be resumed afterward.

The policy environment is part of checkpoint compatibility. The Pi05 HIL
worker requires the pinned LeRobot 0.6.0 runtime used by the validated native
RTC deployment; archived `evostudio-engine` checkouts are not a supported
runtime source. Install that runtime into the worker environment and record its
immutable build identity with `--policy-runtime-version`. Startup verifies the
installed distribution version and native RTC module instead of trusting the
label or trying a second loader. A successful real dry-run freezes that runtime
version into the session provenance.

The HIL v4 contract runs Pi05 with real-time chunking only. It preserves the
pinned native LeRobot RTC timing semantics across the process boundary. The
Client reports the remaining action queue depth with each observation. The
Worker tracks model-local inference latency, uses its historical maximum for
RTC prefix guidance, and returns both the complete action chunk and the current
model-local merge delay. At the control-frame boundary, the Client drops that
reported number of actions before replacing its queue. End-to-end action age is
still measured independently for fail-closed safety. Takeover and handback
reset both sides of the queue, so actions generated before an intervention
cannot resume.

The example systemd unit and environment file are in `deploy/systemd`. The
unit gives the worker lower CPU/IO priority than the client, exposes only a
group-readable Unix socket, permits only Unix-domain networking, and restarts
the process after failure. A restarted worker returns to `idle`; robot motion
still requires a new client preflight and explicit episode start. Runtime logs
stay in journald (`journalctl -u evostudio-inference-worker`), readiness is
available only through the local socket, and hash/cache state stays under
`/var/lib/evostudio-inference`.

## Direct evaluation with the official StarVLA image

Custom images contain the user's SDK `Policy`, dependencies and `/workspace/evostudio-policy.json`.
The separate official StarVLA image path uses an EvoStudio-published image with a pinned StarVLA runtime
and a preconfigured evaluation entrypoint:

```text
/opt/evostudio/starvla/starvla_official_policy.py
```

Cloud stages the user-selected StarVLA run directory. The run keeps StarVLA's native layout:

```text
<run>/
├── config.yaml
├── dataset_statistics.json
├── checkpoints/
│   └── <selected full checkpoint>
└── <local resources referenced by config.yaml>
```

The staged evaluation snapshot contains one selected full checkpoint. `SessionContext.checkpoint_path`
may point to that checkpoint file or to the run directory. The official loader deterministically locates
the checkpoint and adjacent config/statistics, builds the framework from `config.yaml`, performs strict
state-dict loading, and initializes training-time normalization.

The official Policy follows the pinned StarVLA reference client for the active Benchmark. RoboTwin,
RoboCasa, and LIBERO+ use OpenCV `INTER_AREA`; LIBERO uses PIL bilinear resize; CALVIN uses the upstream
aspect-preserving resize-and-pad and uint8 conversion. All produce the pinned clients' `224x224` StarVLA
model input without changing the Benchmark's canonical Observation. The Policy does not derive this
client-side preprocessing from, or rewrite, the uploaded StarVLA configuration. Tokenization, inference
kwargs and dtype are read from the StarVLA configuration or
its framework defaults. Benchmark environments, datasets, evaluator workers and native action application
are provided by `/autodl-fs`. Both custom images and the official StarVLA image support an independent real
`K=1/g=1` quick test and full formal evaluation.

`config.yaml` must carry an `inference` section. It names the Observation features the Policy consumes and
declares the per-dimension action transforms:

```yaml
inference:
  dtype: bfloat16                    # required: bfloat16, float16 or float32
  unnorm_key: franka                 # optional: training normalization statistics key
  inputs:                            # required
    images:                          # required, non-empty; order is the camera order
      - images.primary
      - images.wrist
    language: language.instruction   # required
    state: state.proprioception      # optional, declare only if the model consumes it
  kwargs:                            # optional, forwarded to the StarVLA framework
    do_sample: false
    use_ddim: true
    num_ddim_steps: 10
  action_transforms:                 # optional, applied in order to the model output
    - index: 6
      operation: threshold
      threshold: 0.5
      above: -1.0
      below: 1.0
```

`inputs` values are Benchmark Observation feature names; LIBERO exposes `images.primary`, `images.wrist`,
`state.proprioception` and `language.instruction`. A missing `inputs` or malformed `action_transforms`
entry fails Policy startup. A syntactically valid but semantically wrong camera order or action transform
cannot be detected automatically, so both must match the training-time definitions.

`action_transforms` supports `threshold` (`index`, `threshold`, `above`, `below`) and `affine`
(`index`, `scale`, `offset`). The example above reproduces the LIBERO gripper binarization: dimension 6
becomes `-1.0` when the raw value exceeds `0.5`, and `1.0` otherwise.

The image ships an exporter that produces a portable run snapshot:

```bash
/root/miniconda3/envs/starVLA/bin/python \
  /opt/evostudio/starvla/export_starvla_eval_run.py \
  --run-dir <training run> \
  --checkpoint <training run>/checkpoints/<selected checkpoint> \
  --output <export directory> \
  --resource-root <root of the local construction resources>
```

The exporter copies every local resource referenced by the `framework` and `inference` sections into the
export directory, rewrites those configuration values to relative paths, and runs a strict-load preflight.
A hand-assembled snapshot that keeps training-host absolute paths (for example `framework.qwenvl.base_vlm`)
cannot be resolved inside the evaluation instance.

The official image Policy process contract is:

```json
{
  "schema_version": 1,
  "command": [
    "/root/miniconda3/envs/starVLA/bin/python",
    "/opt/evostudio/starvla/starvla_official_policy.py"
  ],
  "cwd": "/root/starVLA"
}
```

The LIBERO platform entrypoint loads the Policy process contract from the fixed image path
`/workspace/evostudio-policy.json`:

```json
{
  "schema_version": 1,
  "command": ["/opt/user/bin/python", "/workspace/user_policy.py"],
  "cwd": "/workspace"
}
```

EvoStudio selects the Policy runtime and validates the file independently during quick test and scored
evaluation. End users do not configure the path, K/g, or benchmark runtime flags.

## License

Copyright 2026 EvoMind Team. Licensed under the Apache License, Version 2.0. See `LICENSE`.
