skill-wrist-vlm-grasp-rbnx

中文版

robonix.skill.wrist_vlm_grasp — returns an AgileX Piper arm to a saved observation pose, detects the requested tabletop object with a wrist-mounted RGB camera and a VLM, then grasps, lifts and holds it. put_down lowers the held object to the calibrated tabletop placement height before opening the gripper, fixing the former behavior that released objects from hover.

Call semantics, constraints and errors are in CAPABILITY.md, which Pilot retrieves through read_capability_doc. This README covers setup and builds, following the layout of skill-vertical-grasp-object-rbnx. That reference package targets a D1 arm/hand; this package targets a Piper arm and parallel gripper and consumes the existing robonix/skill/pick/* contracts.

Capabilities (5 MCP tools + lifecycle)

Use grab({"object_name": "a_pipe"}) for “抓取 A 管并保持握持” and put_down({}) for “放下”. grab and put_down are the primary wrist-camera workflow. The other tools retain the original sorting/demo interface and need their destination configuration. Use a single provider for these contracts in a deployment, or explicitly select the intended provider; this is an alternative implementation, not another D1 capability namespace.

Runtime dependencies

This skill discovers upstream providers through Atlas. It never opens the camera USB device or arm CAN interface itself. RGB images, joint feedback and joint commands use Atlas-resolved ROS 2 topics; pose computation and movement use gRPC.

The reset service must return to the same six observation joint angles configured in this skill. The grasp pose service must use the same homography and calibrated placement height. These services and the robot URDF are deployment dependencies, not bundled hardware drivers or a complete robot deployment. put_down computes link6 FK locally with kinpy.

Requirements: ROS 2 Humble (rclpy, sensor_msgs, cv_bridge), Robonix CLI rbnx with MCP codegen, the robonix_api Python library, and the Python dependencies in requirements.txt. ROS packages come from the robot runtime, not pip. Use an environment compatible with the ROS NumPy ABI.

Configuration

config.spec documents the runtime mapping. examples/skill-config.yaml is a deliberately unconfigured template; replace its calibration placeholders before use. examples/deployment.yaml shows a deployment entry.

• detect_scheme: vlm, grab_observe_mode: fixed: return to the saved pose for every grab, wait for feedback within tolerance, discard old frames and capture a fresh undistorted image. An API failure never falls back to YOLO.
• grab_vlm_base_url, grab_vlm_model, grab_vlm_api_key: image-capable, OpenAI-compatible chat-completions endpoint. GRAB_VLM_API_KEY is an optional environment fallback for the key. Pilot's planning model is independent and is not configured by this package.
• grab_vlm_reference_images: optional class-to-image-path mapping. Reference images are sent alongside the current image and are not shipped.
• grab_observe_joints_rad, camera_matrix, dist_coeffs, homography_matrix: calibrate for this mount, image resolution and table.
• desktop_height + grasp_offset: calibrated link6 reference height for grasping and placement, in base-frame metres. Account for fingers/tool/object geometry; this is not a universal physical table Z. travel_offset sets the clearance above the placement reference.
• kinematics_urdf_path: absolute path to the matching robot URDF with link6; needed for put_down at current XY/orientation.
• grasp_motion_block_reason: keep nonempty until calibration is ready. A nonempty value blocks grasping and tabletop placement before motion.

The intended profile does not require YOLO weights. Legacy YOLO/current/reset options remain in the implementation for compatibility; changing them requires a matching calibration. The wrist-camera profile defaults to VLM and fixed observation, and rejects missing observation joints at initialization.

Calibration and tabletop placement

See docs/calibration.md. The camera moves with the gripper; a planar homography is valid only for images captured at the saved observation pose and for the calibrated plane. This is not arbitrary-pose eye-in-hand calibration or a depth-based 3D grasp planner. Recalibrate after changing the mount, table, resolution, intrinsics or observation joints.

put_down uses fresh joint feedback, preserves XY/orientation and descends while holding. It opens only after the motion service confirms descent, then raises the empty gripper. Missing feedback, a gripper tilted more than 8° from down, or a failed descent blocks release. It does not use the sorting place_go_down_before_open override; that switch cannot turn “放下” into a drop. Upstream movement must require actual reach feedback. Collision checking, workspace supervision and correct tool height remain deployment responsibilities.

Build and test

python3 -m pip install -r requirements.txt
rbnx validate .
rbnx build -p .
python3 -m unittest discover -s tests -v

Build generates rbnx-build/codegen/{proto_gen,robonix_mcp_types} from the package-local TOML/IDL contracts. Generated files are not committed. Start via your robot deployment or rbnx start -p . --config /absolute/path/skill.local.yaml. The tests mock model/network/robot commands and do not move hardware. The full suite requires generated MCP types; the feedback and VLM tests run without ROS or codegen using the targeted command in the CI workflow.

The source workflow has been exercised on a Piper arm with a gripper-mounted camera for fixed-pose VLM grasp/hold and tabletop release. Those checks used robot-specific calibration, not the template in this repository. Gripper opening is a holding heuristic, not independent visual proof of grasp success. No B/C class success claim or universal accuracy guarantee follows from one setup.

License and provenance

MulanPSL-2.0. This package adapts the Piper vertical-grasp skill from the robonix_yolo_new deployment. Existing source attribution is retained; see NOTICE.md. Repository layout and documentation organization follow the published D1 reference skill; its hardware implementation is not claimed as this package's source. The source repository is syswonder/skill-wrist-vlm-grasp-rbnx. Listing in the Robonix package catalog requires a catalog pull request and maintainer merge; opening the request does not by itself publish a catalog listing.