A robotics platform that bridges simulation and reality through iterative learning. AI agents orchestrate NVIDIA Isaac Sim for simulation, control physical robots (SO-ARM101, XGO2, Zumi) via AWS IoT, and use agent memory to learn from task execution — tracking success rates, grasp accuracy, and sim-to-real transfer fidelity across iterations.
| Robot | Type | Connectivity | Use Case |
|---|---|---|---|
| SO-ARM101 | 6-DOF Arm | LeRobot (USB) | Pick-and-place in simulation + real |
| XGO2 | Quadruped Dog | IoT Greengrass | Vision navigation, grip, walking |
| Zumi | Wheeled Car | IoT Core (MQTT) | Perception, navigation, sensors |
Multiple agent backends — pick what fits your deployment:
| Agent | Interface | Best For |
|---|---|---|
| Bedrock Converse | Browser chat UI | IoT robot control (Zumi, XGO2) |
| OpenClaw | Telegram | Isaac Sim orchestration, Sim2Real |
| Hermes Agent | CLI / Telegram / Discord | Self-improving skills, local models |
See agent/README.md for details.
SO-ARM101 picks oranges from a kitchen counter in Isaac Sim, using LeIsaac assets.
# 1. Download scene assets
bash scripts/leisaac/download_assets.sh
# 2. Run interactive streaming
bash scripts/leisaac/run_streaming.sh
# 3. Connect: NVIDIA Streaming Client → localhostSee scripts/leisaac/README.md for full details.
Control real robots via natural language through a browser chat UI:
# 1. Provision IoT device
bash iot/provisioning/01-provision-iot-thing.sh my-zumi
# 2. Deploy device code
bash iot/provisioning/02-deploy-to-zumi.sh my-zumi
# 3. Start agent
cd agent/bedrock-converse && pip install -r requirements.txt
uvicorn app:app --reload --port 8000
# 4. Open http://localhost:8000 and chat
# "Turn on headlights", "Take a photo", "Navigate to the orange"├── agent/ # AI agent backends
│ ├── bedrock-converse/ # AWS Bedrock multi-agent (Act, Perception, Governance)
│ ├── openclaw/ # OpenClaw agent (Telegram + Isaac Sim)
│ ├── hermes/ # Hermes Agent (self-improving loop)
│ ├── aws-deployment/ # AWS deployment guide
│ └── mcp-articraft/ # Articraft MCP server (search + generate 3D assets)
├── scripts/
│ ├── leisaac/ # Kitchen orange picking (Isaac Sim demo)
│ ├── sim2real/ # Sim2Real memory pipeline
│ ├── telekinesis/ # Telekinesis perception-to-grasp
│ └── so101/ # SO-101 digital twin
├── example/
│ ├── xgo2/ # XGO2 robodog (Greengrass + vision nav)
│ └── zumi/ # Zumi car (IoT Core + sensors)
├── iot/
│ ├── provisioning/ # IoT Thing setup scripts
│ └── greengrass/ # Greengrass deployment
├── infra/
│ └── cloudformation.yaml # AWS memory pipeline stack
├── models/
│ └── so101/ # SO-ARM101 URDF
├── skill/
│ ├── SKILL.md # Isaac Sim skill
│ ├── LEISAAC_API.md # Isaac Sim 6.0 API reference
│ ├── SIM2REAL_MCP.md # MCP memory skill
│ ├── TELEKINESIS.md # Telekinesis skill
│ └── IOT_CONTROL.md # IoT robot control skill
├── docs/
│ ├── architecture.md
│ ├── iot-architecture.md # IoT connectivity design
│ └── THREAT_MODEL.md
└── configs/
└── docker_run.env
Generate and search articulated 3D assets via Articraft-10K through AWS Bedrock AgentCore Gateway (MCP protocol).
OpenClaw → AgentCore Gateway (MCP) → Lambda (dispatcher)
├── Bedrock KB (search 10K assets)
└── ECS Fargate (generate new assets)
→ Bedrock (Claude Opus) → CadQuery → URDF + STL meshes → S3
| Tool | Description |
|---|---|
search_assets |
Semantic search across 10,000 articulated 3D objects |
get_asset_urdf |
Download URDF package for any asset |
get_asset_metadata |
Detailed metadata (joints, parts, files) |
generate_asset |
Generate new URDF from text description (async) |
get_generation_status |
Poll generation job progress |
fork_asset |
Modify existing asset (async) |
list_categories |
Browse 10 object categories |
list_dataset_stats |
Dataset overview |
# Search for assets
mcporter call articraft.search_assets query="robot arm with gripper"
# Generate a new asset (~3 min)
mcporter call articraft.generate_asset description="desk lamp with two hinged arms"
# Check status
mcporter call articraft.get_generation_status job_id="<job-id>"See agent/mcp-articraft/README.md for deployment details.
| Service | Purpose |
|---|---|
| Bedrock | LLM reasoning + Converse API tool use + Knowledge Base (RAG) |
| IoT Core | MQTT device connectivity (Zumi) |
| IoT Greengrass | Edge ML + component deployment (XGO2) |
| DynamoDB | Simulation episode store |
| OpenSearch Serverless | Vector embeddings for RAG |
| S3 | Knowledge docs + photo upload (presigned URLs) |
| SageMaker Neo | Model compilation for edge devices |
| Bedrock AgentCore | MCP Gateway for tool orchestration |
| ECS Fargate | Async 3D asset generation (CadQuery) |
| ECR | Container registry for generator image |
Isaac Sim (simulation) ──▶ DynamoDB (episodes) ──▶ Real Robot
│ │
▼ ▼
S3 (knowledge) ──────▶ Bedrock KB (RAG) ──────▶ MCP Server (tools)
Record episodes:
from scripts.sim2real.episode_logger import EpisodeLogger
logger = EpisodeLogger()
eid = logger.start_episode(task="pick_orange_to_plate", robot_config={...})
logger.add_waypoint(1, "reach", {"shoulder_pan": -15, ...})
logger.end_episode(success=True, metrics={"time": 50.0})Transfer to real robot:
python scripts/sim2real/bridge.py --task pick_orange_to_plate --execute- Simulate — Run task in Isaac Sim (pick orange → place on plate)
- Evaluate — Agent reviews success/failure, logs to episodic memory
- Adapt — Modify strategy (grasp angle, approach vector, timing)
- Transfer — Deploy refined policy to real robot via Sim2Real bridge
- Learn — Real-world feedback updates agent's semantic memory
# Prerequisites: Docker, NVIDIA GPU driver 550+, nvidia-container-toolkit
git clone https://github.com/aws-samples/sample-self-improving-physical-AI.git
cd sample-self-improving-physical-AI
bash scripts/leisaac/download_assets.sh
bash scripts/leisaac/run_streaming.shcd agent/bedrock-converse
pip install -r requirements.txt
uvicorn app:app --reload --port 8000
# Open http://localhost:8000aws cloudformation create-stack \
--stack-name physical-ai-sim-memory \
--template-body file://infra/cloudformation.yaml \
--capabilities CAPABILITY_NAMED_IAM \
--region us-west-2- AWS Physical AI Blog — Embodied AI platform
- LightwheelAI/leisaac — Isaac Lab + SO-101 teleoperation
- AWS MCP Servers — Open source MCP servers for AWS
- Hermes Agent — Self-improving AI agent
- OpenClaw — Personal AI agent framework
- Telekinesis — Physical AI skill library
- Articraft-10K — 10,000 articulated 3D objects in URDF format
- HuggingFace LeRobot — Open-source robot learning
- NVIDIA Isaac Sim — Robot simulation
This project is licensed under the MIT-0 (MIT No Attribution) license.
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.