The RoboNix Memory Action Retrieval Service lets existing embodied models use historical execution experience during online decision-making instead of leaving it as offline data. It retrieves candidate actions that match the current scene and instruction, then uses verification and policy fallback to contain retrieval errors, reducing repeated inference without replacing the original policy. The research implementation currently supports OpenVLA and π0, single-view and two-view Mix retrieval, and chunk-level hybrid verification. Across the evaluated LIBERO workloads, the complete project reaches up to 2.45× speedup with OpenVLA and 3.01× with π0 while retaining the original target-policy fallback boundary.

🎯 What this adds to RoboNix

This repository turns a research retrieval pipeline into an independently deployable RoboNix capability. The important boundary is not merely “Qdrant behind MCP”: the Service validates local observations, delays backend access until the first call, checks every returned trajectory, and preserves enough provenance for the caller to decide whether a candidate is safe to use.

The catalog identity is robonix.service.memory.action_retrieval; the public contract is robonix/service/memory/action_retrieval/retrieve.

🎬 Demo Video

The demo completes put the bowl on the plate in 101 simulator steps. An external benchmark driver made 26 real RoboNix retrieval calls, applied each returned 4 × 7 LIBERO action chunk, and recorded both camera streams. Mean retrieval similarity was 0.9672; P50/P95 Executor-to-result latency was 88.82/110.49 ms. The Service still returns candidates only: the benchmark driver, not the Service, owns simulation execution. No physical robot was commanded. Reproduce the rollout with a licensed LIBERO installation and a booted real deployment:

MUJOCO_GL=egl python benchmarks/target_server/run_robonix_rollout.py \
  --atlas 127.0.0.1:50251 \
  --provider action_retrieval \
  --output-dir "$RUN_ROOT/retrieval-rollout" \
  --task-suite libero_goal --dataset-type goal \
  --task-id 8 --initial-state 0

⚡ RoboNix release results

Release candidate 0.1.0 was exercised through the real path Executor → Atlas → MCP → Service → retrieval backend, rather than by importing the provider class directly.

Start by goal

RoboNix Service package

The repository root is directly publishable as robonix.service.memory.action_retrieval. It exposes robonix/service/memory/action_retrieval/retrieve, returns candidate trajectories only, and never commands robot hardware. Backend health is checked on the first request rather than during RoboNix boot.

python -m pip install -e '.[dev]'
python -m pytest -q
python scripts/release_audit.py
python -m build
python scripts/verify_distribution.py
rbnx validate .
rbnx build -p .

The deployment contract and safety boundary are documented in CAPABILITY.md. A mock deployment and a real HTTP backend configuration are in examples/minimal-deployment/README.md; the exact validation boundary is recorded in VALIDATION.md. Mock mode and a normal no-match both return success=false, no actions, and fallback_required=true. Network failures, malformed responses, and invalid trajectories fail the capability call. The complete research implementation and benchmarks below remain part of this repository.

Real RoboNix deployment

The release deployment is intentionally Service-only:

Executor -> Atlas -> MCP -> action_retrieval -> configured retrieval HTTP API

It does not start a robot driver or execute the returned trajectory. Prepare a dedicated environment and a large-volume data root; link existing datasets or models only after resolving and checking their targets. The validation host's concrete storage layout and safe-link procedure are recorded in benchmarks/target_server/README.md, while the public manifest remains portable.

python3.11 -m venv /path/to/environments/retrieval-service
source /path/to/environments/retrieval-service/bin/activate
python -m pip install --upgrade pip
python -m pip install -e '.[dev]'

rbnx validate .
rbnx build -p .

cd examples/real-deployment
cp .env.example .env
# Edit the untracked .env with this deployment's environment, runtime,
# input-root, and loopback backend paths.
set -a
source .env
set +a

rbnx build -f robonix_manifest.yaml
rbnx boot -v --no-update-check -f robonix_manifest.yaml
rbnx caps -v --server 127.0.0.1:50251
rbnx tools --server 127.0.0.1:50251
rbnx describe --server 127.0.0.1:50251 --provider action_retrieval
rbnx inspect --server 127.0.0.1:50251

Invoke the capability through Executor rather than importing the Python class:

python ../../benchmarks/target_server/invoke_executor.py \
  --atlas 127.0.0.1:50251 \
  --provider action_retrieval \
  --contract robonix/service/memory/action_retrieval/retrieve \
  --args-json '{"instruction":"pick up the cup","third_person_image_path":"/absolute/input/third.jpg","wrist_image_path":"/absolute/input/wrist.jpg","dataset_type":"goal","timeout_s":30}'

rbnx shutdown -f robonix_manifest.yaml

Boot does not contact the backend. The first retrieve performs the health check and request. Both input images must be JPEG, PNG, or WebP files below allowed_image_root. A genuine no-match is a normal fallback; an unavailable collection, network failure, malformed response, invalid trajectory shape, or non-finite action fails the MCP call. Configuration fields and defaults are defined in config.spec. The existing backend uses wire value dataset_type="10" for the LIBERO-Long suite.

📊 Performance snapshot

The project evaluates experience-backed hybrid execution on both token-based and diffusion-based VLA models. Results report task success rate (SR) and end-to-end speedup over the original target model.

📚 Table of Contents

• What this adds to RoboNix
• Demo Video
• RoboNix release results
• Real RoboNix deployment
• 📊 Performance snapshot
• 📰 News
• ⚡ System Capability and Results
• 🧠 Architecture Overview
• 🔌 RoboNix Integration and Outlook
• 🧪 Validated Release
• ⚙️ Requirements
• 🚀 Quick Start
• 📦 Dataset and Checkpoint Sources
• 🗄️ Build the Retrieval Index
• 🌐 Start the Retrieval Service
• 🩺 Troubleshooting
• 🗺️ Roadmap
• 📝 Citation
• 🤝 Contributors
• 📄 License

📰 News

• 2026-07-19: 🆕 Released the system-level experience-backed action retrieval Service with capability results, model support, and bilingual documentation.
• 2026-07-18: 🔥 Validated the complete two-view image → embedding → Qdrant → 4×7 action-trajectory request path from the independent repository root.
• 2026-07-18: 🗄️ Strictly checked 39 Mix collections containing 273,465 experience points with 4,352-dimensional cosine vectors and action payloads.

⚡ System Capability and Results

From the RoboNix runtime perspective, this Service is a stateful experience-memory provider. It converts historical observations and actions into reusable memory, retrieves candidate trajectories for the current task, and keeps verification and fallback between retrieval and physical execution.

Supported models

🧠 Architecture Overview

The validated two-view Mix path stores both searchable vectors and action payloads in Qdrant; it does not require MongoDB. MongoDB remains available for legacy collection workflows. The FastAPI embedding service generates OpenVLA vision features, while the Flask retrieval service selects a task-specific collection and returns a retrieved and optionally averaged action trajectory.

The single-view pipeline uses a third-person image. The Mix pipeline concatenates features from third-person and wrist views into a 4,352-dimensional representation:

🔌 RoboNix Integration and Outlook

This Service is an independently deployable RoboNix provider that maintains experience memory for embodied execution. Scene observations and task context form a structured retrieval request; Atlas discovers the provider, Nexus carries multimodal references, and Pilot consumes the retrieved trajectories without embedding database logic into the RoboNix core.

Looking forward, the service can evolve toward continuously updated robot memory with pluggable encoders, hierarchical indexes, redundancy compression, expiration policies, and safety-aware trajectory reuse across tasks and robot platforms.

🧪 Validated Release

Release candidate 0.1.0 was validated on 2026-08-01 with RoboNix commit 48af09190b99f7847dddf68457eec2db42d2c1a7, 20 real dual-view LIBERO observations, and the existing 39-collection Qdrant Mix index.

All direct and RoboNix responses matched, with zero parity failures. Mean wrapping overhead was 7.31 ms. Boot-time backend independence, first-call connection, real 4 by 7 trajectory retrieval, capability discovery, shutdown, and preservation of the shared retrieval and Qdrant processes were also checked.

Raw calls, input hashes, exact environment, failure semantics, and reporting limits are in benchmarks/target_server/ and VALIDATION.md. This Service benchmark does not create a new task-success claim.

🚀 Quick Start

conda create -n robonix-retrieval python=3.10 -y
conda activate robonix-retrieval
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip install -e . --no-deps

python -m pytest -q tests
python -m scripts.run --help

Prepare a large data root and download the pinned modified LIBERO RLDS dataset:

export DATA_ROOT=/data/robonix-retrieval
DOWNLOAD_FULL_DATASET=1 \
  scripts/data/download_libero_rlds.sh "$DATA_ROOT/datasets/libero_rlds"

Start Qdrant, then run the two-view embedding and retrieval services in separate terminals or tmux sessions. The detailed commands are provided below. The repository does not ship models, datasets, populated databases, or outputs.

📦 Dataset and Checkpoint Sources

The database is derived from third-person images, wrist images, language instructions, the current 7D action, and the next three 7D actions. A public release may publish a prebuilt index separately, but the repository itself contains only the reproducible builder.

⚙️ Requirements

The repository does not include model weights, datasets, populated databases, or robot-control software. Prepare the following before running the complete pipeline:

• an OpenVLA checkpoint and sufficient GPU memory;
• a Qdrant instance with persistent storage; MongoDB only for legacy workflows;
• robot demonstrations or LIBERO RLDS datasets;
• writable data, image, cache, log, and Qdrant backup directories;
• robot-side controllers when running real-hardware experiments.

🧰 Step 1: Installation

Clone the project and run all commands from the repository root:

git clone https://github.com/lusunn111/service-memory-action-retrieval-rbnx.git
cd service-memory-action-retrieval-rbnx

conda create -n rt-cache python=3.10 -y
conda activate rt-cache

python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip install -e . --no-deps

The root requirements.txt is the reproducible installation entry and delegates to requirements/requirements.txt. Flash Attention is CUDA- and compiler-sensitive; if installation fails, install the compatible PyTorch build first and then build Flash Attention separately:

python -m pip install packaging ninja
python -m pip install "flash-attn==2.5.5" --no-build-isolation

Run lightweight smoke checks after installation:

python -c "import service_bootstrap as s; print(s.activate_vendor())"
python -m scripts.run --help

🗄️ Step 2: Start Qdrant and Optional MongoDB

The validated two-view Mix path requires only Qdrant. Start it with persistent storage:

docker run -d \
  --name rtcache-qdrant \
  -p 6333:6333 \
  -p 6334:6334 \
  -v rtcache_qdrant:/qdrant/storage \
  qdrant/qdrant

Start MongoDB only when using a preserved legacy workflow:

docker run -d \
  --name rtcache-mongo \
  -p 27017:27017 \
  -v rtcache_mongo:/data/db \
  mongo:6

Check that both services are reachable:

curl http://localhost:6333/healthz
python -c "from pymongo import MongoClient; print(MongoClient('mongodb://localhost:27017/').admin.command('ping'))"

Use authenticated connections, private networks, access controls, backups, and externally managed secrets for shared or production deployments. The example configuration is intended for local development.

🔧 Step 3: Configuration

Copy the environment template and customize it:

cp configs/.env.example .env

Important settings include:

# Databases
MONGO_URL=mongodb://localhost:27017/
MONGO_DB_NAME=OpenVLACollection
QDRANT_HOST=localhost
QDRANT_PORT=6333

# Services
EMBEDDING_SERVER_HOST=0.0.0.0
EMBEDDING_SERVER_PORT=9020
EMBEDDING_SERVER_URL=http://127.0.0.1:9020/predict
RETRIEVAL_SERVER_HOST=0.0.0.0
RETRIEVAL_SERVER_PORT=5002

# Models and computation
OPENVLA_MODEL_PATH=/path/to/openvla
DEVICE=cuda:0
CUDA_VISIBLE_DEVICES=0

# Data
DATA_ROOT=/path/to/robot-datasets
LIBERO_DATASET_ROOT=/path/to/libero-rlds
QDRANT_BACKUP_ROOT=/path/to/qdrant-backups

The centralized configuration implementation is in configs/rtcache/rt_cache_config.py. CLI arguments may override some values. Prefer absolute paths for datasets, model checkpoints, and backups, and verify that no preserved upstream script still contains a source-machine path:

grep -R "/home/\|PATH_TO" scripts vendor/rtcache/scripts configs

🧠 Step 4: Start the Embedding Service

Single-view service

The standard FastAPI service accepts one image and an optional instruction. It generates OpenVLA and/or CLIP features and listens on port 9020 by default:

python -m scripts.run \
  scripts/embedding/embedding_server.py \
  --host 0.0.0.0 \
  --port 9020 \
  --device cuda:0 \
  --workers 1

Verify the service:

curl http://localhost:9020/health

curl -X POST http://localhost:9020/predict \
  -F "file=@/path/to/observation.png" \
  -F "instruction=pick up the red object" \
  -F "option=both"

Interactive FastAPI documentation is available at http://localhost:9020/docs.

Two-view Mix service

The Mix service requires third-person and wrist images and listens on port 9021 by default:

python -m scripts.run \
  scripts/embedding/embedding_server_mix.py \
  --host 0.0.0.0 \
  --port 9021 \
  --device cuda:0 \
  --workers 1

curl -X POST http://localhost:9021/predict \
  -F "third_person_image=@/path/to/third_person.png" \
  -F "wrist_image=@/path/to/wrist.png" \
  -F "instruction=place the object in the bowl" \
  -F "return_individual=false"

Do not run multiple model workers unless enough GPU memory is available. Each worker initializes its own model state.

🗄️ Step 5: Build the Retrieval Index

Start the embedding service before processing a dataset. The following example embeds LIBERO-Goal observations and inserts them into task-specific Qdrant collections:

python -m scripts.run \
  scripts/data_processing/process_libero_goal.py \
  --dataset_type goal \
  --base_dataset_path /path/to/libero-rlds \
  --embedding_server_url http://127.0.0.1:9020/predict \
  --qdrant_host localhost \
  --qdrant_port 6333 \
  --batch_size 50 \
  --max_episodes -1

Supported single-view dataset types are goal, 10, object, and spatial. Collections follow names such as libero_goal_task_0. Use --dataset_path to override the path derived from the dataset type and --use_benchmark when LIBERO benchmark task IDs should replace hash-based mapping.

For Mix indexing:

python -m scripts.run \
  scripts/data_processing/process_libero_goal_mix.py \
  --dataset_type goal \
  --base_dataset_path /path/to/libero-rlds \
  --embedding_server_url http://127.0.0.1:9021/predict \
  --qdrant_host localhost \
  --qdrant_port 6333 \
  --batch_size 50 \
  --max_episodes -1 \
  --backup \
  --backup_name mix_base

Mix collections use names such as libero_goal_mix_task_0. Options including --clear_db and --clear_all delete existing collections and are destructive; verify the Qdrant target and backup required data before using them.

🌐 Step 6: Start the Retrieval Service

Single-view LIBERO retrieval

python -m scripts.run \
  scripts/retrieval/retrieval_libero_goal.py \
  --host 0.0.0.0 \
  --port 5002 \
  --embedding-url http://127.0.0.1:9020/predict \
  --qdrant-host localhost \
  --qdrant-port 6333 \
  --dataset-types goal

Send a retrieval request:

curl -X POST http://localhost:5002/pipeline \
  -F "file=@/path/to/observation.png" \
  -F "instruction=put the butter in the bowl" \
  -F "dataset_type=goal"

Two-view Mix retrieval

python -m scripts.run \
  scripts/retrieval/retrieval_libero_goal_mix.py \
  --host 0.0.0.0 \
  --port 5003 \
  --embedding-url http://127.0.0.1:9021/predict \
  --qdrant-host localhost \
  --qdrant-port 6333 \
  --dataset-types goal

curl -X POST http://localhost:5003/pipeline \
  -F "third_person_image=@/path/to/third_person.png" \
  -F "wrist_image=@/path/to/wrist.png" \
  -F "instruction=put the butter in the bowl" \
  -F "dataset_type=goal"

Both variants expose GET /health, GET /stats, and POST /pipeline. A successful pipeline response includes retrieval metadata and fields such as rtcache_trajectory and averaged_trajectory. Treat returned actions as untrusted candidates: validate dimensions, limits, freshness, robot state, and collision constraints before execution on physical hardware.

🔌 Service Ports

Port 5002 is used by more than one preserved workflow. Assign distinct ports when data collection and retrieval services run concurrently.

📊 Benchmarking Guidelines

Compare RT-Cache with VINN, BehaviorRetrieval, and non-retrieval baselines using the same observations, task split, control frequency, and robot or simulator configuration. At minimum, report:

• task success rate and completed episodes;
• embedding, vector-search, and end-to-end latency;
• p50, p95, and p99 online latency after warm-up;
• retrieval top score, top-k setting, and similarity threshold;
• database collection count, point count, and embedding dimension;
• GPU memory, embedding throughput, and database resource usage;
• action-horizon length and fallback or rejection count.

Measure cold start separately from steady-state performance. The retrieval services preload collection payloads into memory, so startup time and host-memory usage scale with the database. Network transfer, image encoding, database placement, and robot control-loop latency must be included in end-to-end measurements.

🗂️ Repository Layout

.
├── modules/                       # Lazy module catalogs
│   ├── database/                  # MongoDB and Qdrant backends
│   ├── scene_encoding/            # OpenVLA/CLIP feature generation
│   ├── indexing/                  # Vector indexing views
│   ├── retrieval/                 # Online similarity retrieval
│   ├── memory_update/             # Backup, restore, and cleanup
│   └── verified_execution/        # SpecVLA and rebuttal implementations
├── scripts/
│   ├── data/                      # Dataset processing and acquisition
│   ├── serve/                     # Embedding and retrieval service views
│   ├── maintenance/               # Qdrant backup and restore utilities
│   └── run.py                     # Stable RT-Cache script runner
├── benchmarks/
│   ├── behavior_retrieval/        # BehaviorRetrieval baseline
│   ├── vinn/                      # VINN baseline
│   ├── specvla_validation/        # SpecVLA validation snapshot
│   └── rebuttal/                  # FLASH/OpenPI rebuttal snapshot
├── configs/                       # Environment, database, and RT-Cache config
├── requirements.txt               # Reproducible installation entry
├── requirements/                  # Python dependency pins
├── tests/                         # Layout and lazy-import tests
├── docs/assets/                   # Architecture assets and the web ImageGen prompt
├── utils/                         # Database, embedding, and image utilities
├── vendor/rtcache/                # Canonical RT-Cache source tree
└── service_bootstrap.py           # Vendor activation and guarded runner

vendor/rtcache/ is the canonical import-compatible retrieval implementation. Top-level directories provide an engineering-oriented view of the data, service, maintenance, and benchmark workflows. Validation and rebuttal sources are canonical under their respective benchmark directories.

🩺 Troubleshooting

RoboNix boots, but the first retrieval fails

Activation is deliberately offline; the first retrieve performs the backend health check. Confirm the configured loopback URL, call the backend /health endpoint, and verify that the Service environment can reach it. Do not make activation eagerly connect merely to hide a deployment error.

An input image is rejected before the backend is called

Resolve the file with readlink -f and confirm it stays below allowed_image_root. A filename extension is insufficient: the Service checks the JPEG, PNG, or WebP signature, file size, and decoded image limits. Put test inputs in the configured validation directory instead of broadening the root to the whole filesystem.

No match and backend failure look similar

A genuine no-match is a completed call with fallback_required=true; a missing collection, transport error, malformed response, invalid shape, or non-finite action fails the capability call. This distinction prevents infrastructure failure from being silently treated as ordinary policy fallback.

LIBERO-Long requests reach the wrong collection

The existing backend uses wire value dataset_type="10". The public Service configuration can map a readable suite name to that value, but the benchmark, Service, and backend must use the mapping consistently.

Port 5002 is already occupied

Preserved data-collection and single-view retrieval workflows both use 5002. Assign one a different port; never stop an unidentified process. The validated two-view path uses 5003.

Cleanup after validation

Use rbnx shutdown -f robonix_manifest.yaml, then check Atlas, Executor, and the provider process individually. Only stop processes started by the current deployment. Shared Qdrant, embedding, and retrieval services may belong to other experiments and must remain untouched.

🗺️ Roadmap

• [x] Publish an independently runnable source-only repository.
• [x] Validate OpenVLA embedding, Qdrant retrieval, and a 4×7 action response.
• [x] Preserve the π0 chunk-level retrieval and verification research path.
• [x] Adopt the RoboNix Mulan PSL v2 license and remove citation placeholders.
• [ ] Publish a small public two-view example dataset and prebuilt Qdrant index.
• [ ] Add container images and authenticated production API examples.
• [ ] Add incremental memory deduplication, compression, and expiration policies.
• [ ] Complete public end-to-end workflows for π0.5 and π0-FAST.
• [x] Provide the versioned RoboNix Service adapter and capability contract.

📝 Citation

If this Service supports your research, please consider giving the repository a star ⭐ and citing this software repository:

@software{mao2026robonix_memory_action_retrieval_service,
  author  = {Mao, Zhihao and He, Huiru and Zheng, Zihao},
  title   = {RoboNix Memory Action Retrieval Service},
  year    = {2026},
  version = {0.1.0},
  url     = {https://github.com/lusunn111/service-memory-action-retrieval-rbnx}
}

🤝 Contributors

We thank HuiruHe and zhengzihaoPKU for their contributions to the Service. See CONTRIBUTORS.md for the contributor policy.

📄 License

The project is licensed under the Mulan Permissive Software License, Version 2 (Mulan PSL v2); see LICENSE. Vendored components retain their included licenses.