MR / Meta Quest, Pico

SketchAR for Meta Quest and Pico

SketchAR

A bright room with window shadows and a wooden easel holding, in a frame with highlighted corners, a drawing of a kitten in a kimono with flowers on its head; to the left a floating menu reading “Back to menu”, “Unlock image”, “Show Grid”, “Apply filter”, and a “Template opacity” slider below the drawing

Task

Bring an AR drawing app with an audience of more than 2 million people to headsets: the sketch hangs on a real wall in mixed reality and the person traces it by hand. Two platforms with different SDKs — without two separate products.

What we built

A shared Unity core package (425 files, about 1,800 tests) with thin hosts for Meta and Pico; platform SDKs plug in through abstract providers.
Our own 3D interface system with haptics for each hand.
Image deformation to fit the surface: trapezoid, cylinder, sphere, a fold with a draggable line.
Drawing together in one room: shared spatial anchors, Bluetooth discovery, our own network protocol, host failover.
Courses, templates, Google Drive and Dropbox, offline downloads, a subscription, 11 languages with automated text-fit checks.

Engineering decisions

Networking with no third-party engine.
CI/CD to the Meta Quest Store and Pico Store through ALPHA → BETA → STORE channels.
Errors from Sentry automatically turn into draft fixes.
An R&D prototype of AprilTag-based stabilisation through the Quest 3 camera, tested on the device.

Outcome and status

Meta at version 1.11, Pico at alpha 1.1.

Stack

Unity 6, OpenXR, Meta XR SDK (MRUK, Interaction SDK), PICO SDK, Sentry, GitHub Actions

A rough white wall overlaid with a large outline drawing of a woman’s face and hair; a hand with a black marker traces the line of the lips, with a round controller and a menu reading “Back to menu”, “Unlock image”, “Show grid” at the bottom left

All work Next caseRapty — a dance battle played with your body