Requesting movement#

MetaHuman can request named actions such as wave, bow, sit, and turn_left_45. It can also submit a motion_plan_v1: a limited sequence of joint positions and timings. The body core—the firmware component that owns command rules—turns an accepted request into calibrated movement.

Plans have limits on frame count, fields, duration, and joint ranges. They use the same stopping and failure behavior as named motions. Model output must pass these checks before it can affect the motors; it cannot write directly to the servo-control pulses.

The current contract describes eight joints. The planned V2 body has twelve, so joint names, calibration, motion assets, validators, and advertised features must be updated together before twelve-joint plans are accepted. The V2 roadmap covers that transition.

Accepted, finished, and successful are different steps#

Sending bytes into a network socket confirms only that the local transport accepted them. The caller still needs the robot’s acknowledgement and final result. Otherwise a broken connection could look like a successful action.

Even done has a limit: it describes the command. A future goal such as moving into better light would need a new camera image to establish whether the lighting improved. A completed walk alone cannot answer that question.

MetaHuman’s Action Result step interprets feedback. Goal Review then decides whether the requested outcome is complete, needs another action, or needs input. Command-only goals can use action_result evidence; visual goals require visual_observation. Other goals may depend on user input or environment state. The workflow saves this progress, and Robot Status displays it.

Only offer actions the connected body can perform#

The robot reports both its capabilities, meaning supported features, and its current readiness. A camera can be supported but unavailable right now. The gateway uses these reports to limit the actions MetaHuman can select:

• a snapshot requires a ready camera;
• wake-gated listening requires an accepted, ready wake-word model;
• a motion plan requires support advertised by the connected body.

The model’s permitted output uses those current action and command names. Ainekio owns the named-command catalog and advertises it to MetaHuman, whose selector schema and parser use that list. This keeps command coverage aligned and an unavailable feature from being offered as an executable choice.

Control messages use JSON. Images and audio travel as binary frames containing JPEG images or PCM audio samples, with a small type and counter header. Metadata identifies the originating snapshot request, movement, or utterance. When a movement requests an image, its metadata and JPEG can arrive before done, allowing the result and observation to be reviewed together.

Stops and connection loss#

Periodic application control pings show that the control connection is still active. A short gap in valid control traffic stops active motion without forcing a healthy idle session to disconnect. Loss of WebSocket, gateway, or Wi-Fi connectivity enters fail-safe/offline behavior and begins limited reconnect attempts.

The body also checks duplicate IDs, payload sizes, queue limits, joint calibration, and the ordering of stops and interruptions. Microphone and speech streams have explicit start/end states so stray media cannot be mistaken for a valid stream.

These checks do not give the robot navigation or measured balance. Its current movement is body-local and open-loop: it follows joint sequences without measuring its position relative to a target. Named turn angles are estimates.

Recover without accidentally repeating a movement#

A difficult case is a connection failing after the robot moves but before the result reaches MetaHuman. Re-sending the command might make it move twice.

September adapter work addresses this with saved action records, also called receipts. They retain an action’s ID, request content, dispatch record, final result, and feedback waiting to be delivered. Reusing an ID with different content is rejected. Dispatch also checks which Work Coordinator process is currently allowed to control the body, preventing an old process from sending new commands after a replacement takes over.

The September 8 repair removes the adapter’s fixed 30-second cutoff for waiting on a dispatched command. A late final result can still settle the original request. This changes result tracking; it does not remove firmware motion limits, control-connection checks, or direct stop.

Recovery matches the saved gateway instance, robot session, and command sequence without resending movement. A confirmed stop settles earlier asynchronous commands. Reconnection to a new host session can close the old control session as cancelled while recording that its earlier physical effect is unverified.

If the action may have run and no result can be established, the adapter retains outcome_unknown. This is an adapter result, separate from the robot’s protocol states. Elapsed time alone does not prove success or failure. Controlled tests cover late results, session changes, and cancellation; deployment, physical stop timing, and the cause of the original missing result remain unverified.

Tests still needed on the robot#

The next fault tests cover connection loss during each motion type, cancellation partway through a smooth transition, malformed or duplicate media metadata, and full camera or speaker queues while moving. Controller restarts and rollback of a failed over-the-air update must also be tested during interrupted sessions. Repeated reconnects should show that old commands are not replayed.

Source tests cover the protocol and shared body core. The status report distinguishes those results from physical tests. Read the gateway guide for the connection itself and MetaHuman integration for how a request is saved and reviewed.