robonix.primitive.orbbec.camera

primitive package v0.1.0

Orbbec RGBD camera primitive — multi-model wrapper over the vendored OrbbecSDK_ROS2 driver with config-selected launch file (Dabai DCW, Gemini 330-series), declaring rgb + depth + camera_info topic streams and JPEG camera/snapshot + camera/depth_snapshot MCP tools.

2 issues found while indexing this entry

  • capability robonix/primitive/camera/snapshot declares no transport (mcp, grpc or ros2)
  • capability robonix/primitive/camera/depth_snapshot declares no transport (mcp, grpc or ros2)
README primitive-orbbec-camera-rbnx@main

primitive-orbbec-camera-rbnx

Robonix package wrapping Orbbec RGBD cameras — a multi-model driver layer over the vendored OrbbecSDK_ROS2, with the launch file selected by config (launch_file). Validated on Dabai DCW and Gemini 336L (330-series). Owns the primitive/camera/* namespace. Exposes the camera's RGB + depth + camera_info streams under generic contracts plus JPEG snapshot / depth_snapshot MCP tools, so consumers (grasp / detection / scene / vision skills / Pilot / the web client) resolve names through atlas — no hardcoded /camera/color/image_raw paths on the consumer side.

Catalog name: robonix.primitive.orbbec.camera.

Capability surface

Contract Mode Transport Source / handler
robonix/primitive/camera/driver rpc gRPC Driver(CMD_INIT, config_json) — lifecycle gate
robonix/primitive/camera/rgb topic_out ROS 2 /<camera_name>/color/image_raw (sensor_msgs/Image)
robonix/primitive/camera/depth topic_out ROS 2 /<camera_name>/depth/image_raw (sensor_msgs/Image; HW-aligned to color when depth_registration=true)
robonix/primitive/camera/camera_info topic_out ROS 2 /<camera_name>/color/camera_info (sensor_msgs/CameraInfo)
robonix/primitive/camera/snapshot rpc MCP snapshot tool — latest RGB frame as JPEG (sensor_msgs/Image)
robonix/primitive/camera/depth_snapshot rpc MCP depth_snapshot tool — depth as 8-bit grey JPEG (preview only)

camera_info is package-locally defined (see capabilities/primitive/camera/camera_info.v1.toml) because the robonix global tree does not ship that contract yet; codegen + atlas merge package-level capabilities automatically. service-object-detect-rbnx uses it to synchronize RGB frames and, when depth is enabled, to back-project detections. The current vertical-grasp deploy skips depth and lets service-grasp-pose-rbnx map bbox pixels through a calibrated 2D homography.

The snapshot contracts are the on-demand, agent-facing form of the two streams: a persistent subscription keeps the latest raw frame per stream (JPEG encoding happens only when a call arrives), and a call older than one second is served by a one-shot re-read. camera/rgb / camera/depth remain the high-rate data plane for skills; snapshots add a JPEG view for Pilot (the LLM sees the frame as vision input) and the web client. extrinsics stays undeclared until a real handler exists — per the packaging rules, manifest entries must correspond to real declare_ros2_*/@cap.mcp(...) handlers.

Boot ordering

Boot this before any consumer of primitive/camera/*. In the current vertical-grasp pipeline, service-object-detect-rbnx resolves rgb and camera_info during its Driver(CMD_INIT) call. service-grasp-pose-rbnx does not consume camera topics directly; it receives the bbox from detection and uses the deploy's homography.

Driver-init lifecycle

start.sh brings up the atlas bridge — no ROS spawn. The bridge opens a gRPC server, registers the provider, declares only primitive/camera/driver (auto-emitted by the framework when codegen produces a Driver Servicer), then blocks on Driver(CMD_INIT, config_json).

When rbnx boot invokes Init it passes the manifest's config: block as JSON. The lifecycle then runs:

  1. CMD_INIT: parse and validate cfg (camera name, resolution / fps, depth_registration, USB pin, sentinel timeout);
  2. CMD_ACTIVATE: spawn ros2 launch orbbec_camera <launch_file> …;
  3. wait for the first sensor_msgs/Image on the configured RGB topic (sentinel);
  4. declare primitive/camera/{rgb, depth, camera_info} on atlas;
  5. subscribe both streams so the snapshot RPCs serve the latest frame.

The snapshot MCP tools are declared at startup (before CMD_INIT), the same way every Robonix provider advertises its capabilities; until the camera is activated their handlers report that no stream is available.

Atlas only ever advertises endpoints we've confirmed are publishing. CMD_DEACTIVATE drops the snapshot subscriptions and kills the orbbec subprocess; CMD_SHUTDOWN additionally stops the ROS context so the process exits cleanly. Idempotent.

If the camera is not connected or the driver stalls, the sentinel times out and Init returns state="error".

Layout

primitive-orbbec-camera-rbnx/
├── package_manifest.yaml
├── capabilities/
│   └── primitive/camera/camera_info.v1.toml   # package-local contract
├── orbbec_camera/
│   ├── __init__.py
│   ├── images.py                              # sensor_msgs/Image → JPEG (pure, unit-tested)
│   └── main.py                                # lifecycle + sentinel + snapshot tools
├── tests/
│   └── test_snapshot_images.py                # conversion unit tests (no ROS graph)
├── scripts/
│   ├── build.sh                               # colcon build src + rbnx codegen
│   └── start.sh                               # source ROS, exec main
└── src/
    └── OrbbecSDK_ROS2/                        # VENDORED orbbec/OrbbecSDK_ROS2
        ├── orbbec_camera/                     # ROS 2 driver pkg
        ├── orbbec_camera_msgs/
        └── orbbec_description/

Config (passed via Driver(CMD_INIT, config_json))

launch_file:        dabai_dcw.launch.py  # any launch file the vendored driver ships, per model —
                                     # e.g. gemini_330_series.launch.py (Gemini 335/336)
camera_name:        camera        # ROS namespace; topic prefix
depth_registration: true          # HW-align depth to color frame
enable_d2c_viewer:  false         # debug RViz; off in deploy
color_width:        640
color_height:       480
color_fps:          10            # dabai_dcw firmware default
depth_width:        640
depth_height:       400
depth_fps:          10
serial_number:      ""            # pin to specific device when multiple connected
usb_port:           ""
sentinel_timeout_s: 30.0          # max wait for first RGB frame in on_activate
snapshot_max_width: 640           # snapshot tools downscale to this width; 0 = sensor resolution
snapshot_jpeg_quality: 85         # JPEG quality, 1..100
snapshot_max_depth_m: 3.0         # depth preview: 0..this range → 8-bit grey (near = dark)
# Topic-name overrides (rarely needed):
# rgb_topic:          /<camera_name>/color/image_raw
# depth_topic:        /<camera_name>/depth/image_raw
# camera_info_topic:  /<camera_name>/color/camera_info

Build / run standalone

bash scripts/build.sh                           # colcon build vendored src + rbnx codegen
ROBONIX_ATLAS=127.0.0.1:50051 \
    bash scripts/start.sh                       # registers, awaits Init

To drive the lifecycle manually (without rbnx boot), call the camera's Driver service with CMD_INIT and a config JSON, then call CMD_ACTIVATE. Init only validates configuration; Activate returns after the first RGB frame is observed and the three topic streams are declared.

Verification

rbnx caps | grep camera
# Expected: orbbec_camera provider with
#   robonix/primitive/camera/{driver, rgb, depth, camera_info, snapshot, depth_snapshot}

ros2 topic hz /camera/color/image_raw           # ~10 Hz with default fps
ros2 topic hz /camera/depth/image_raw           # ~10 Hz
ros2 topic echo /camera/color/camera_info --once  # K, D, R, P intrinsics

Snapshot path (needs Pilot; the tool name is the contract leaf):

rbnx ask '只调用 provider_id=orbbec_camera 的 robonix/primitive/camera/snapshot 获取一帧,并描述你看到了什么;不要执行任何运动。'

The reply should describe the scene, which also proves Pilot attached the frame as vision input. Without Pilot, any MCP client can call the tool after resolving it through Atlas/ConnectCapability; the payload is a sensor_msgs/Image whose data is base64 JPEG.

Vendor / upstream

src/OrbbecSDK_ROS2/ is a verbatim copy of orbbec/OrbbecSDK_ROS2. The Dabai DCW firmware-blob SDK ships under orbbec_camera/SDK/{include,lib}/. If anything diverges from upstream, drop a *.patch alongside src/ documenting the diff.

License

This package: Apache-2.0. Vendored OrbbecSDK_ROS2: see its LICENSE file.