🖥️ DeskTwin

Your desk, understood.

Snap a photo of your desk → get an editable 3D twin → record what's plugged into what → fix problems with Meta AI.

Quick start · How it works · Meta models · Architecture · 한국어

_{Sample desk shown. Its devices and links are placeholders until you start your own desk.}

---

💡 Why DeskTwin?

One monitor goes black on a desk with a laptop, a dock, two screens and a pile of USB and Bluetooth gear. A photo shows what is there, but not what is plugged into what. A chatbot just guesses.

DeskTwin gives the AI a map of your actual setup and keeps facts and guesses apart:

	Comes from	Never treated as
📷 Photo interpretation	Muse Spark reading your desk photo	Proof that a device exists or is cabled
🙋 Your report	You confirm a link or a device status	Proof that the link works
💻 macOS observation	Optional, consented, local display/USB scan	Proof of cable routes or working parts

Nothing the model suggests is saved until you confirm it. Saving a cable never marks a device as working.

✨ How it works

1. 📷 Photo → 3D twin Muse Spark finds the objects on your desk (optionally outlined by SAM 3.1). You choose what to keep, review the 3D layout, and apply it.	2. 🔌 Record what connects to what Devices without a link show up as chips. Tap one, pick what it plugs into and the cable (or Unknown), and save.
3. 🛠️ Fix a problem with AI chat Pick the device that isn't working. AI chat opens with a draft question grounded in your recorded links. Nothing is sent until you press Send.	4. 🩺 Inspect & troubleshoot Every device shows its sources, ports and links. Monitors get a step-by-step check along the recorded monitor → dock → computer route.

Plus: ✏️ an Edit mode with collision-aware drag, surface placement and box-part shape editing · ↩️ 20-step Undo/Redo · 💻 optional local macOS display/USB matching · 📦 JSON import/export with no photos, chat or keys inside.

🧠 How Meta models power it

Model	Role in DeskTwin
Muse Spark muse-spark-1.3	📷 Reads the desk photo and proposes objects with kind, rough position, size, color and box parts. 💬 Answers questions about a selected device using your recorded setup.
SAM 3.1 sam-3.1 (optional)	✂️ With two-stage analysis on, outlines each chosen object in the photo so you can see exactly what was found.
Muse Image	🎨 Generated the demo desk image used in the video. Generated images are not committed.

Guardrails on every response:

• ✅ A strict JSON schema and existing device IDs are required before anything is shown.
• 🚫 Photo responses may describe visible objects only. Connection, status and executable fields are rejected.
• 🔑 The API key stays server-side in .env (never in VITE_*).
• 🧪 No key? The same workflow runs as a clearly labeled local demo that never pretends to be a model.

🚀 Quick start

Requires Node.js 22.12+

git clone https://github.com/junjunjunbong/DeskTwin.git
cd DeskTwin
npm ci
npm run dev

Open http://127.0.0.1:5173. Without a key this is the offline local demo, and no external calls are made.

🔑 Connect Meta models

cp .env.example .env

MODEL_API_KEY=your_key_here
META_BASE_URL=https://api.meta.ai/v1
META_MODEL=muse-spark-1.3
# Optional: two-stage photo analysis + SAM 3.1 outlines
META_PHOTO_TWO_STAGE=false
META_SAM_ENABLED=false
META_SAM_MODEL=sam-3.1

Restart the server, then open AI chat → Connect Meta API, review what will be sent, and enable it. Real calls may cost money. The local 20 requests/minute limit is not a spending cap.

💻 macOS hardware matching (optional)

Requires macOS 12+ and Xcode Command Line Tools.

npm run hardware:build
MODEL_API_KEY='' npm run dev

Open Hardware → Devices → Open Mac hardware details, agree to the local read, and press Start detection. You match each observation to a desk device yourself. Serial numbers become an installation-specific HMAC inside the native process and never leave the machine. Bluetooth live detection is not supported. See MACOS_HARDWARE.md.

🔧 Custom ports

MODEL_API_KEY='' DESKTWIN_WEB_PORT=5187 PORT=4337 npm run dev

🧩 Architecture

flowchart LR
  photo["📷 Desk photo"] -->|your consent| spark["Muse Spark<br/>structured output"]
  spark -.optional.-> sam["SAM 3.1<br/>outlines"]
  spark --> check{"Schema + ID<br/>validation"}
  check --> review["👀 You review<br/>the 3D scene"]
  review -->|Confirm and apply| ws[("Workspace<br/>source of truth")]
  you["🙋 You confirm<br/>connections"] --> ws
  mac["💻 macOS helper<br/>local only"] -.you match.-> ws
  ws --> chat["💬 AI chat<br/>Muse Spark"]
  ws --> diag["🩺 Monitor check"]

Loading

Path	What lives there
src/domain/workspace.ts	Source of truth: schema, commands, validation
src/components/	Three.js voxel scene, panels and dialogs
server/	Local API, origin checks, Meta client (server/meta.ts)
native/	macOS display/USB helper
tests/	Playwright end-to-end tests with mocked Meta and hardware

More in ARCHITECTURE.md.

🔒 Privacy by design

• 📷 Photos stay in memory unless you explicitly send one for analysis. They are never written to storage or JSON.
• 💬 Chat sends the current message, up to 12 recent Meta-mode turns and the shareable device and link records. macOS observations and hardware identities are never sent.
• 💾 Your workspace lives in your browser. Details in PRIVACY.md.

🧪 Development

npm run check            # TypeScript + unit/API tests + production build
npx playwright install chromium
npm run test:e2e         # Chromium UI tests (own servers, empty API key)
npm run test:production  # production server + browser test

Every test mocks Meta requests, so no API credit is spent. Busy ports: DESKTWIN_TEST_PORT=5187 DESKTWIN_TEST_API_PORT=4337 npm run test:e2e.

📋 What was verified, and what was not

• Live Meta calls (2026-11-04): a conversation request succeeded in 14.3 s. On a Muse Image demo desk, two-stage photo analysis reached a scene that could be applied. On a real desk photo, analysis returned 17 objects, but 4–5 overlaps (monitor, riser, stand) still need manual fixes before applying. Photo limits are 120 s on the server and 125 s in the browser. See PHOTO_TWO_STAGE.md and PHOTO_EVALUATION.md.
• Not claimed: guided step-by-step checks exist for monitors only; other devices go to AI chat. Physical Mac re-verification of the hardware helper, Bluetooth live discovery, Windows/Linux helpers, multi-user hosting and cloud sync are out of scope.
• This is a single-user local MVP. Don't expose the server to the public internet.

📚 More docs (some in Korean)

Scene editor · Storage · Photo review · Photo reference fitting · Hackathon notes · Submission script · Contributing

🏆 Built at the Meta AI Developer Hackathon

This repository was started during the Meta AI Developer Hackathon, and every commit is from 2026-10-26 or later. The workspace, photo-to-3D flow, connection recording, troubleshooting, local Mac matching boundary and Meta structured-output integration were all built during the event.

📄 License

MIT. Scene geometry and textures are generated by code, and no third-party 3D models or user photos are shipped. libheif-js (LGPL-3.0) is used for local HEIC conversion; see THIRD_PARTY_NOTICES.txt.

_{Made with 🧡 and a lot of cables.}