Tourguide

A robot that gives the building tour.

Tourguide is an autonomous guide for office and work locations such as the lobby of Franklin Antonio Hall and the museum at Qualcomm. A Reachy Mini head rides on a Husarion ROSbot base. It builds its own map, localizes against AprilTag landmarks and navigates between exhibits with Nav2.

The assembled Tourguide robot: a white Reachy Mini head on an aluminum tower above a four-wheeled ROSbot base
The hardware

Hardware: Two robots integrated into one.

The base handles sensing and driving. The head handles talking to visitors. A custom aluminum mount joins them and puts the camera and lidar at useful heights. Tap a marker to see each part.

Front view of the robot with numbered markers on each component
Mapping

Building the lab map, one session at a time.

RTAB-Map fuses the OAK-D's RGB-D stream with wheel odometry and lidar into a 2D occupancy grid. It runs in multi-session mode, so each new drive extends the same map instead of starting over.

freeoccupiedunknowncamera obstacles
RTAB-Map occupancy grid of the lab

Session 2, after loop closure. Magenta marks obstacles seen by the camera; the map edges are now crisp and aligned.

How RTAB-Map fixes drift

Poses are nodes. Measurements are edges.

Wheel odometry drifts a little at every step. When the robot recognizes a place it has seen before, RTAB-Map adds a loop-closure edge and re-solves the whole graph, pulling every pose back into place.

min{Ti} Σ(i,j)∈ℰ ‖ Wij log( T̄ij−1 Ti−1 Tj )∨ ‖2

The log map takes each SE(3) error into 𝔰𝔢(3), where it can be minimized with Levenberg–Marquardt.

Landmarks

AprilTags pin the map to the real world.

Semantic landmarks

YOLO adds objects to the map.

AprilTags give fixed anchors. On top of those, the robot detects everyday objects like chairs with YOLO and stores them as landmarks in the same map frame. The map then records what things are, not only where the walls are.

YOLO runs on the OAK-D itself

A spatial-detection pipeline on the camera's VPU finds an object, lines its box up with the stereo depth map and averages the depth into an X, Y, Z centroid. The CPU only receives the result.

Same chair, or a new one?

Detections are moved into the map frame with TF. If a new one lands within τ of a stored object, it reinforces that landmark. Otherwise it gets a fresh ID ≥ 1000, which prevents false loop closures between identical-looking objects.

Trust position, not orientation

Bounding boxes give a decent position but no reliable heading, so each landmark carries this covariance:

0.0500000 00.050000 000.05000 000999900 000099990 000009999
Conversation

Gemini Powered Conversational Agent

The Reachy Mini head runs a conversation agent powered by Gemini. Visitors can ask it questions at each stop and it answers out loud. Currently, we are working on matching its movements to what it is saying.

Working

Gemini conversation agent

The head listens to visitors and replies with speech generated by a Gemini Live API powered agent, so the tour can go beyond a fixed script.

In progress

The LLM drives the head

We are building on a fork of the Reachy Mini conversation app so the LLM can also send turning and small movement commands, such as a head turn, a nod, a tilt or an antenna wiggle. The physical motion commands are built and via through tool calling: the LLM picks a movement tool and its parameters that are run on the robot.

In progress

Speech and motion in sync

The goal is to time those movements with the words, so the robot looks toward what it is describing and reacts with small, natural (and cute) gestures while it talks.