Panbot: A Perception-Triggered Autonomous Pancake-Cooking Robot

This post is the blog edition of the standalone technical article at ispaik06.github.io/Panbot — same content, nicer reading experience there.

Authors

Inseong Paik · NmDongQ

Published

March 2026

Code

Panbot · Panbot_vision

Stack

Python · LeRobot · PyTorch · Ultralytics · OpenCV

Video 1The full Panbot demonstration: batter pouring stopped by a YOLO segmentation trigger, a GRU cook-state estimator deciding the flip moment, and two ACT policies flipping and plating the pancake. Every stage transition in the video is decided by perception, not by a timer. (Click to play; loads from YouTube.)

Abstract

This article documents Panbot, an autonomous pancake-cooking system my teammate and I built on the LeRobot SO-ARM101 follower arm. Cooking is an unforgiving robotics benchmark: the medium (batter) is a liquid that sets irreversibly, the process (cooking) evolves on its own clock whether or not the robot is ready, and the manipulation (flipping a half-set pancake with a spatula) is contact-rich and hard to script. Our central design decision is a strict division of labor: perception decides when, control decides how. A single 4K trigger camera watches the pan through a calibrated perspective warp. During pouring, a YOLOv8 segmentation model measures batter coverage and stops the pour when the covered area ratio holds above threshold for a full second. During cooking, a ResNet18+GRU sequence classifier watches bubble dynamics over a 3-second temporal window and fires when the pancake is ready to flip — including through a deliberately engineered secondary trigger that detects the decay of the "almost ready" state rather than waiting for a confident "ready".

The motions themselves use the cheapest tool that works at each stage. Batter pouring is a deterministic 10-waypoint joint-space sequence with per-segment ramp times — dispensing demands repeatability, not adaptability. Flipping and plating are ACT imitation policies trained on 111 and 120 teleoperated episodes (about 160k and 170k frames each, four camera views), which exhibit implicit recovery: when the pancake lands badly or the spatula slips, the policy re-aligns and retries without any explicit recovery logic. A 30 Hz runtime orchestrator connects everything with watchdogs, ramped safe-pose returns, and a keyboard kill switch that degrades gracefully from any stage. The system cooks a pancake end-to-end; this article walks through every design decision, every tuned number, and the failure modes that shaped them.

• A perception-triggered runtime architecture. One shared trigger camera and two learned models gate all stage transitions; the policy observation cameras are kept strictly separate. Stage logic, trigger debouncing, and fail-safe paths live in a single 30 Hz orchestrator (§3).
• Metric batter measurement from a monocular camera. A four-point perspective warp normalizes the pan to a top-down view, making "fraction of pan covered" a stable, threshold-able quantity for a YOLOv8-seg model — with a hold-to-trigger debounce that survives segmentation flicker (§5–6).
• Temporal cook-state estimation. A ResNet18+GRU classifier over 16 frames sampled at stride 6 (a 3.03 s window) labels the pan as not_ready / almost_ready / ready, trained on 32 fully annotated cooking runs with a leakage-free run-level split (§7).
• A confidence-drop trigger. The flip moment is detected not only by the ready class but by a qualified drop in almost_ready confidence after sustained saturation — an empirical signature of bubble activity peaking and fading (§8).
• ACT policies with implicit recovery. Two chunked transformer policies (chunk 100 at 30 Hz) for flipping and plating, integrated to share one robot connection with the orchestrator, and demonstrably able to recover from misplaced pancakes and slipped grasps (§9–10).

1Introduction

Most tabletop manipulation demos share a convenient property: the world waits for the robot. A cube does not mind sitting on the table for another thirty seconds while the policy deliberates. Cooking removes that luxury. Once batter hits a hot pan, a physical process starts running on its own clock — the batter spreads, sets, bubbles, and eventually burns — and the robot's job is not merely to execute motions correctly but to execute them at the right moment. Pour too long and the pancake overflows the pan; flip too early and the half-liquid disc tears; flip too late and the bottom scorches. The task is a scheduling problem wearing a manipulation costume.

Panbot is our attempt to take that problem seriously on hobby-grade hardware: a LeRobot SO-ARM101 follower arm — six position-controlled Feetech STS3215 servos and 3D-printed links — a household electric griddle, a batter dispenser, and a 3D-printed spatula. Nothing in the hardware is exotic, which is precisely the point: the interesting engineering lives in when the robot acts, and that is a perception problem, not an actuation problem.

1.1 Perception decides when, control decides how

The pancake workflow decomposes into three motion tasks separated by two waiting decisions:

1. Pour batter into the pan — and decide when there is enough;
2. wait for the pancake to cook — and decide when it is ready to flip;
3. flip it, let the second side cook, and plate it.

The two decisions have nothing in common with the three motions. Deciding "enough batter" is a geometric measurement; deciding "ready to flip" is a judgment about the temporal texture of a cooking surface — bubbles forming, popping, and leaving pores as the batter sets. Neither decision benefits from being entangled with motor control, so Panbot does not entangle them: a single dedicated trigger camera watches the pan from above, two learned models turn its stream into boolean stage-transition events, and the motion controllers are swapped underneath whenever a trigger fires. Each motion task then uses the cheapest controller that does the job, summarized in Table 1.

Table 1Task decomposition. Each stage pairs a motion controller with the perception event that terminates it. Only the stages whose difficulty justifies learning use a learned policy.

Stage	Motion controller	Terminated by	Why this choice
1 · Batter pouring	Deterministic joint-waypoint stepper	YOLOv8-seg coverage trigger	Dispensing is a fixed, repeatable motion; precision and predictability beat adaptability
2 · Cook wait	Base-pose hold controller	GRU cook-state trigger	The robot should do nothing — reliably — while perception watches the pan
3 · Flipping	ACT policy (learned from 111 demos)	Configured rollout duration	Contact-rich, tool-mediated, hard to script; demonstrations capture the skill
4 · Plating	ACT policy (learned from 120 demos)	Configured rollout duration	Same: sliding a spatula under a floppy pancake resists waypoint scripting

This split has a second, subtler benefit: it makes failures diagnosable. When a stage transition misfires, the fault is in the trigger stack (camera, warp, model, threshold); when a motion goes wrong, the fault is in the controller stack (waypoints, policy, observation cameras). The two stacks share no state except the orchestrator's stage variable, and the troubleshooting table in the repository README is organized around exactly this boundary.

1.2 Scope, honestly stated

Panbot is a system-integration and perception-timing project, not a claim about general cooking autonomy. The workspace is fixed and instrumented for this one dish; the perception models are trained on this pan, this batter, and this kitchen's lighting; the ACT rollouts run for a configured duration rather than until a detected success. §13 catalogs these limitations without cosmetics. What the project does demonstrate is that with careful decomposition, a completely autonomous multi-stage cooking pipeline — liquid handling, process monitoring, and contact-rich manipulation — fits on a 3D-printed arm and a single GPU, and that the perception-triggered architecture holds up end-to-end in unedited runs (raw single-take footage).

2System Architecture

The system is organized around one invariant: the camera that decides is never the camera that acts. A single high-resolution trigger camera feeds the two perception models that gate stage transitions; four separate low-resolution cameras feed the ACT policies their multi-view observations. The two paths never mix — a trigger-camera problem shows up as a stage that never advances, a policy-camera problem shows up as bad actions, and neither can masquerade as the other. Figure 1 shows the full dataflow.

Figure 1Panbot's three-band architecture. The trigger perception band converts the shared 4K camera stream into two boolean events; the orchestration band swaps motion controllers when those events fire; the execution band is the LeRobot robot bus, which the ACT policies observe through four dedicated cameras. The trigger path and the policy observation path share no hardware.

2.1 Hardware

The arm is the standard LeRobot SO-ARM101 follower: six Feetech STS3215 serial-bus servos in a 3D-printed structure, driven over a single USB serial port. Around it we built the cooking cell visible in Video 1: an electric griddle pan, a squeeze-bottle batter dispenser resting in a 3D-printed cradle at a known pose, a 3D-printed spatula parked in a holder bowl, and a plating target. Every fixture has a fixed, calibrated location — Task 1's waypoints and the ACT policies' demonstrations all assume this cell layout. The five cameras are enumerated in Table 2.

Table 2Camera inventory from runtime.yaml. The trigger camera runs at 4K so that the warped pan crop retains enough resolution for segmentation; the policy cameras run at VGA because ACT resizes its inputs anyway and four simultaneous MJPG streams must share USB bandwidth.

Camera	Index	Resolution	Consumer	Purpose
Trigger	8	3840×2160 @ 30	YOLOv8-seg + GRU	Stage transitions
Right	2	640×480 @ 30	ACT policies	Right workspace view
Left	6	640×480 @ 30	ACT policies	Left workspace view
Global	4	640×480 @ 30	ACT policies	Overview of the cell
Wrist	0	640×480 @ 30	ACT policies	Eye-in-hand close-up

2.2 Rates and timing

Everything in the runtime is deliberately pinned to one clock. The main loop, the camera streams, and the policy rollouts all run at 30 Hz (task.hz = vision.fps = policy_fps = 30), so one loop tick consumes exactly one camera frame and emits at most one robot action. This alignment is not cosmetic: trigger debounce counts are specified in frames, so keeping frames and control ticks 1:1 makes "30 frames of stability" mean exactly one second of wall-clock stability, and makes log timestamps directly comparable across perception and control events.

Table 3Nominal and measured execution rates. Measured values are from the runtime logs of an instrumented end-to-end run (§12): the policy loop held 29.2–29.3 Hz against the 30 Hz target, including inference and camera reads.

Loop	Nominal	Measured	Notes
Main stage loop	30 Hz	frame-locked	sleeps to a fixed 33.3 ms period
YOLO trigger inference	every frame	30 Hz	640-px input, single class
GRU trigger inference	every frame	≈30 Hz	after a 91-frame warm-up buffer
ACT policy rollout	30 Hz	29.2–29.3 Hz	147 steps / 5.03 s · 5867 steps / 200.0 s
Base-pose hold refresh	every 10 ms gate	30 Hz effective	rate-limited inside the 30 Hz tick
Camera watchdog	2.0 s timeout	—	aborts the run if frames stop

Design note — why the trigger camera is not a policy camera

It is tempting to reuse the 4K overhead stream as a fifth policy view, or to reuse a policy camera for triggering. Both couplings are traps. The trigger models need the warped pan crop at high resolution and are useless during manipulation (the arm occludes the pan); the policies need exactly the camera poses they were trained with and would be perturbed by a view change. Keeping the paths disjoint also means the trigger camera can be released entirely before the policy stages begin — main_runtime closes it after the GRU fires, freeing USB bandwidth for the four policy streams.

3Runtime Orchestration

Panbot/control/main_runtime.py is the single process that owns everything: the robot connection, the trigger camera, both perception models, the stage machine, and every fail-safe. There is no middleware, no message bus, and no second process to fall out of sync with. The stage machine it implements is small enough to state exactly (Figure 2), and that smallness is a feature — every transition is either a perception trigger, a timer, or a fail-safe, and each appears as a single greppable line in the run log.

Figure 2The runtime stage machine. Solid edges are the nominal flow; dashed edges are fail-safe exits, reachable from every stage. The trigger camera is opened once before Stage 1 and released permanently after the GRU trigger — the policy stages never touch it.

3.1 The 30 Hz stage loop

Stages 1 and 2 share one loop body. Each tick reads a frame from the trigger camera, feeds it to whichever model is active for the current stage, steps whichever motion controller is active, and then sleeps the remainder of the 33.3 ms period:

while not stop_event.is_set():
    ok, frame = cap.read()                  # trigger camera, 4K
    if stage == "TASK1":
        trig, vis, info = yolo.step(frame)  # coverage ratio + debounce
        if trig: task1.interrupt_to_return()
        task1.step(now)                     # one send_action per tick
        if task1.is_return_done(): stage = "WAIT_GRU"
    elif stage == "WAIT_GRU":
        base_ctrl.tick()                    # rate-limited hold
        trig, vis, info = gru.step(frame)   # cook-state estimate
        if trig: break                      # → policy stages
    sleep(dt - elapsed)                     # lock to 30 Hz

Two properties of this loop matter more than its simplicity suggests. First, perception and motion advance in the same tick: the YOLO trigger does not pause pouring while it thinks, because a 640-px single-class segmentation runs comfortably inside the frame budget. Second, the trigger is an interrupt, not a phase boundary: when the YOLO event fires mid-ramp, interrupt_to_return() abandons the remaining pour waypoints immediately and starts the return sequence from the arm's current measured pose, so the pour stops within one tick of the decision rather than at the end of a waypoint.

3.2 Every motion is a ramp

The runtime contains exactly one motion primitive, used by the waypoint stepper, the base-pose controller, and every fail-safe path alike: linear interpolation in joint space from the robot's current measured configuration to a target, streamed as position commands at the loop rate. With start pose $q_0$ (read from the encoders at ramp entry), target $q_1$, and ramp duration $T$:

$$q(t) \;=\; (1-\alpha(t))\, q_0 + \alpha(t)\, q_1, \qquad \alpha(t) \;=\; \operatorname{clip}\!\Big(\tfrac{t - t_0}{T},\, 0,\, 1\Big),$$

sent at 30 Hz, i.e. $\lceil 30\,T \rceil$ intermediate commands. Reading $q_0$ from the observation rather than assuming the previous target is what makes the primitive safe to invoke from any state — after an interrupted pour, after a policy rollout that ended in an arbitrary configuration, or after an exception. The servos' own position loops handle the final tracking; the ramp exists to bound the commanded velocity, since a raw send_action jump to a distant pose would command a violent full-speed swing.

3.3 Fail-safe paths

Three independent mechanisms can stop a run, and all of them converge on the same exit ramp:

• Keyboard. A raw-terminal KeyWatcher thread reads stdin byte-by-byte in cbreak mode. A bare ESC byte sets the stop event; q only disables the preview windows. Because arrow keys also begin with ESC, the watcher waits 20 ms for a following byte and treats ESC-then-something as an escape sequence to ignore — a small detail that prevents an accidental arrow-key press from emergency-stopping a cooking run.
• Signals. SIGINT and SIGTERM are routed into the same stop event, so Ctrl+C and a supervisor kill behave identically to ESC.
• Camera watchdog. If the trigger camera delivers no frame for 2 s during a vision stage, the runtime raises rather than silently holding the last action — a frozen camera must never leave the arm mid-pour with no supervising perception.

The stop event is checked at every loop tick and inside the policy rollout loop, so an ESC during an ACT rollout exits within one 33 ms step. Every exit path then calls the same _best_effort_safe_pose(): ramp to the base pose over 2.5 s, hold it briefly under the base-pose controller, and disconnect. The finally block releases the camera, destroys the preview windows, and disconnects the robot even if the safe ramp itself throws.

Design note — the base pose is a contract

The base pose (arm folded up and back, gripper nearly closed, clear of the pan) appears in the config as six joint values and is the single well-known configuration that every stage starts from and returns to. Stages therefore compose: the ACT policies were trained from demonstrations that begin at this pose, the waypoint sequences begin and end at it, and any fail-safe can target it unconditionally because it is collision-free from everywhere in the task workspace. A BasePoseController re-sends the pose at a rate-limited interval while idle — the STS3215s hold position on their own, but periodic refresh makes the hold robust to a dropped packet or a manually disturbed joint.

4Task 1 — Deterministic Batter Pouring

Pouring is the one task in the pipeline where we deliberately chose not to learn. The batter bottle rests in a fixed cradle; the pan does not move; the grasp is the same every run. A scripted joint-space sequence executes this perfectly every time, is trivially inspectable, and — critically — can be interrupted at an arbitrary point by the perception trigger, which is much harder to guarantee for a learned policy mid-rollout. The adaptive part of pouring is not the motion; it is knowing when to stop, and that job belongs to the YOLO trigger (§6).

Video 2Task 1 executing: the arm grasps the batter bottle from its cradle, tilts it over the pan, and holds the pouring posture. The inset window is the live YOLO view on the warped pan crop; when the segmented coverage crosses threshold and holds, the runtime interrupts the pour and the arm re-docks the bottle.

4.1 A non-blocking waypoint stepper

Task1MotionStepper executes a list of joint-space waypoints as an explicit state machine with two phases per waypoint — RAMP (linear interpolation from the current measured pose, §3.2) and HOLD (re-send the target for a dwell time) — and advances one send_action per 30 Hz tick. Nothing blocks: the stepper is stepped by the main loop rather than running the sequence itself, which is what lets the YOLO trigger preempt it between any two ticks. The configured sequences are 10 waypoints out (approach → grasp → lift → tilt) and 11 back (un-tilt → travel → release → retreat), with the dwell time set to 10 ms — effectively continuous motion through the waypoints.

Figure 3Normalized joint values across the ten waypoints of the pouring sequence (x-axis: waypoint index; each panel scaled to its own range, shown at top right). The story is legible directly from the config: gripper (orange) opens wide at waypoint 5 and clamps to 18 at waypoint 6 — the grasp — and wrist_roll (orange) stays constant until the final three waypoints, where it rolls from +53.5 to −10.8: the pour tilt. The sequence ends in the pouring posture and stays there until the YOLO trigger interrupts.

4.2 Per-segment ramp times

A single global ramp speed cannot serve this sequence: the first move travels half the workspace from the folded base pose (want: slow), while the mid-sequence pours are short precision moves (want: brisk), and re-righting a bottle full of batter must not slosh (want: slow again). The config therefore supports per-waypoint ramp-time overrides keyed by target index:

task1_ramp_time_s: 1.0            # default per-waypoint ramp
task1_pose_hold_s: 0.01           # effectively continuous
task1_initial_ramp_overrides:
  0: 5.0        # base pose → first waypoint: long travel, go slow
  1: 1.0
task1_return_ramp_overrides:
  2: 3.0        # un-tilting the full bottle: slow to avoid slosh
task1_return_to_base_ramp_time_s: 1.0

The run logs confirm the schedule to the millisecond: the first ramp takes 5.01 s, ordinary waypoints tick by at 1.02–1.07 s including the dwell, and return-waypoint 3 — the bottle un-tilt — takes 3.01 s. The stepper also validates sequence lengths at construction (10 out, 11 back) so that a config edit that drops a waypoint fails at startup instead of mid-pour.

4.3 Interrupt semantics

When the coverage trigger fires, interrupt_to_return() switches the stepper's mode to RETURN and enters the first return waypoint's ramp from the current measured pose — mid-ramp, mid-hold, or in the terminal pouring posture, it does not matter. In the instrumented run of §12 the pour held its terminal posture for 15.2 s of steady flow before the trigger fired; in a second logged run, 8.1 s. That variation is the entire point of the architecture: the same fixed motion serves different batter amounts, temperatures, and flow rates because the stop decision floats free of the motion script. After the return sequence re-docks the bottle, the base-pose controller ramps the arm home over 1 s and Stage 2 begins.

5Top-View Normalization

Both perception models see the pan through the same preprocessing step: a fixed perspective warp that maps the pan region of the oblique 4K camera view onto an axis-aligned, top-down rectangle. This one transform does a surprising amount of work, and it is worth being precise about why it exists before describing the models it feeds.

Figure 4The calibrated warp, drawn to scale from the actual corners.json shipped in the repository. The pan's image is a skewed quadrilateral in the oblique camera view; the homography $H$ maps it onto an axis-aligned rectangle whose size is derived from the measured corner spans.

5.1 The transform

A one-time calibration tool streams the trigger camera and lets the operator click the pan's four inner corners in TL→TR→BR→BL order; the pixel coordinates are saved to corners.json. At runtime, the source quadrilateral $\{(x_i, y_i)\}_{i=1}^{4}$ and the destination rectangle $\{(0,0), (W{-}1,0), (W{-}1,H{-}1), (0,H{-}1)\}$ determine the unique homography — eight degrees of freedom from four point correspondences — via cv2.getPerspectiveTransform, and each frame is resampled as

$$\begin{pmatrix} x' \\ y' \\ 1 \end{pmatrix} \;\sim\; H \begin{pmatrix} x \\ y \\ 1 \end{pmatrix}, \qquad H = \begin{pmatrix} h_{11} & h_{12} & h_{13} \\ h_{21} & h_{22} & h_{23} \\ h_{31} & h_{32} & h_{33} \end{pmatrix},$$

where the output size is computed from the corner geometry itself — the width is the larger of the two horizontal edge lengths and the height the larger of the two vertical ones:

$$W = \max\big(\lVert \mathrm{BR}-\mathrm{BL} \rVert,\; \lVert \mathrm{TR}-\mathrm{TL} \rVert\big), \qquad H = \max\big(\lVert \mathrm{TR}-\mathrm{BR} \rVert,\; \lVert \mathrm{TL}-\mathrm{BL} \rVert\big).$$

With the shipped corners this yields a warped view of roughly 3156×1805 pixels — the pan fills the entire model input, at the camera's native resolution, with the background cropped away by construction.

5.2 Why warp before inference

• It makes area mean area. The pour-stop rule thresholds "fraction of the pan covered by batter". In the oblique view, a pixel near the far rim subtends more pan surface than one near the near rim, so raw mask area is a biased estimator that depends on where the batter sits. In the rectified view the pan-to-pixel scale is approximately uniform and the ratio $\text{mask px}/WH$ is a meaningful physical proxy.
• It deletes nuisance variation. Both models train and infer on pan-only crops: no table, no arm base, no background chairs to overfit to. For the GRU, whose training set is only 32 cooking runs, shrinking the input distribution to "the inside of this pan" is a large effective data-efficiency win.
• It decouples camera pose from the models. If the camera shifts, recalibrating means re-clicking four corners — not re-labeling data or retraining. The transform absorbs the viewpoint; the models only ever see the canonical view. (The flip side, honestly: if the camera shifts and nobody recalibrates, both models silently degrade — see §13.)

Both trigger models apply the warp independently with per-model config (use_warp, optional fixed output size), but in deployment they share the same corners file, and the batch tooling in Panbot_vision applies the identical transform when preparing training data — the warp seen at training time is bit-for-bit the warp seen at inference time.

6Stopping the Pour — Batter Segmentation

The pour-stop decision needs one number per frame: what fraction of the pan is covered by batter? Our first attempt computed it classically — a reference-frame difference against the empty pan with a binary threshold (measure_batter_area_1.py survives in the vision repository as the fossil of this approach). It worked until it met reality: specular highlights on the non-stick coating, the moving pour stream, steam, and the slow color shift of batter as it heats all punch holes in background subtraction. Batter-vs-pan is a semantic distinction, so we moved to a learned segmentation model and never looked back.

6.1 SAM-assisted labeling

The labeling pipeline in Panbot_vision is built for speed rather than scale. Warped pan frames are captured during real pours; an interactive labeling tool loads a Segment Anything ViT-B checkpoint and lets the annotator place positive clicks on batter and negative clicks on pan; SAM proposes candidate masks, the tool keeps the highest-scoring one live on screen, and a keystroke saves it as a binary PNG aligned to the frame — including explicit all-black masks for "no batter" frames, which matter as hard negatives. A converter then binarizes each mask, extracts external contours, simplifies them with Douglas–Peucker (tolerance $0.002 \times$ perimeter, components under 200 px dropped), and emits normalized polygon labels in YOLO-seg format with an automatic 80/20 train/val split. A few clicks per frame replace per-pixel polygon tracing; a usable dataset for this one-class problem took an afternoon, not a week.

6.2 Model and training

The segmentation model is an Ultralytics YOLOv8-seg fine-tuned from COCO weights on the single class batter at 640-px input (the training wrapper defaults to the nano variant and 80 epochs at batch 8; the deployed checkpoint is the run batter_seg_local_v1). This is deliberately the smallest model that solves the problem: one class, one pan, controlled lighting, and a hard 33 ms per-frame budget shared with the motion loop. The training wrapper can pull the dataset from and push best.pt to the Hugging Face Hub, which is how the vision repo and the runtime repo exchange artifacts without sharing a filesystem.

6.3 From masks to a trigger

At runtime, each warped frame passes through the model at confidence threshold 0.25. The trigger logic then reduces the prediction to one scalar and one counter. With $\mathcal{M}_t$ the largest predicted mask (upsampled to the warped resolution $W \times H$) at frame $t$:

$$r_t = \frac{\lvert \mathcal{M}_t \rvert}{W H}, \qquad c_t = \begin{cases} c_{t-1} + 1 & \text{if } r_t \ge \tau \\ 0 & \text{otherwise} \end{cases}, \qquad \text{trigger} \iff c_t \ge N,$$

with coverage threshold $\tau = 0.12$ and hold count $N = 30$ frames. Taking only the largest mask ignores stray droplets and the disconnected pour stream; the counter is a debounce: coverage must hold above threshold for one full second (30 frames at 30 fps), and a single frame that dips below — a flicker of the mask edge, a bubble of the stream occluding the pool — resets it to zero. The one-second debounce also has a physical meaning: batter keeps spreading after it lands, so demanding sustained coverage measures the settled pool rather than a transient splash.

Early · ratio 0.087 < τ · hit 0/30

Trigger · counter past 30 → fire

Settled · ratio 0.128 · pour stopped

Figure 5The live YOLO trigger view (real frames from Video 2, model overlay in green, telemetry in yellow). Left: the pool and the falling stream are segmented but coverage is far below threshold. Center: the ratio has held above threshold long enough to exhaust the 30-frame debounce — the moment the pour-stop interrupt fires (highlighted in the demo edit). Right: after the bottle re-docks, the settled pool reads a stable 0.128.

Across the five logged cook runs the trigger fired at measured ratios between 0.129 and 0.186 — past the 0.12 threshold by a margin, as expected, since the ratio keeps climbing during the debounce second. The 30-frame hold proved to be the load-bearing parameter: the module's built-in default is a stricter $\tau = 0.17$, and the deployment tuned it down to 0.12 for this pan and batter viscosity while relying on the debounce to reject transients. The README's troubleshooting table encodes the tuning grammar — fires too early: raise $\tau$ or $N$ or the model confidence; never fires: lower $\tau$, then audit the warp corners and checkpoint path before touching anything else.

Implementation note — the trigger latches by design

Strictly, the counter keeps counting past $N$ (the center panel of Figure 5 reads 43/30), but the runtime consumes only the first rising edge: the interrupt flips Stage 1 into the return sequence and the YOLO stepper is never consulted again. Keeping the trigger stateless-but-latched in the consumer, rather than in the model wrapper, means the same wrapper serves both the runtime and the offline evaluation scripts, which do want the raw per-frame signal.

7Knowing When to Flip — Temporal Cook-State Estimation

Deciding when a pancake is ready to flip is a fundamentally different perception problem from measuring batter coverage, and the difference dictates the architecture. A cooking pancake's readiness is written in its dynamics: bubbles nucleate at the surface, grow, pop, and leave open pores as the batter sets; the surface sheen dulls; the rim dries. A single frame is ambiguous — a still image of a bubbled surface cannot distinguish "bubbles actively forming" (too early) from "bubbles popped and set" (ready). So the estimator must watch a clip, not a frame.

7.1 Architecture: ResNet18 features, GRU dynamics

Figure 6The cook-state estimator. A rolling buffer keeps the last 91 warped frames; every tick, 16 of them (newest included, stride 6 backwards) are encoded independently by a shared ResNet18 and consumed by a single-layer GRU whose final hidden state feeds a 3-way classification head.

The design puts the capacity where the signal is. A frozen-architecture ResNet18 handles appearance — pores, sheen, browning at the rim — while a deliberately small GRU (hidden size 256, one layer) handles temporal evolution over the 16-step feature sequence. The sampling scheme buys a long effective time horizon at low compute: with sequence length $T = 16$ and stride $s = 6$, the buffer requirement is

$$L \;=\; (T-1)\,s + 1 \;=\; 91 \text{ frames} \;\approx\; 3.03 \text{ s at } 30 \text{ fps},$$

so the model sees three seconds of bubble dynamics for the inference cost of 16 encoder passes. The frame indices are anchored to the newest frame ($t, t{-}6, \dots, t{-}90$), and the identical $(T, s)$ scheme is used to generate the training sequences — train/test skew in the temporal geometry is ruled out by construction.

7.2 Dataset: 32 cooking runs, labeled by time segments

The training data is 32 complete real cooking runs recorded through the same 4K camera and the same warp. Rather than labeling frames, we labeled time segments: for each run, an annotations file gives frame ranges for not_ready (wet batter, bubbles forming), almost_ready (dense bubbling, surface beginning to set), and ready (bubbles popped, surface matte, flip window open). Each 16-frame training sequence inherits the label of the segment containing its final frame — the model's job is defined as "what is the state now, given the last three seconds", exactly matching its runtime use.

Table 4Cook-state dataset statistics. Frame counts from the 96 annotated segments of annotations.csv; sequence counts are the rows of the generated index files. The split assigns whole runs (24/4/4), so no cooking run contributes to two splits.

	not_ready	almost_ready	ready	Total
Labeled frames	33,711	17,415	13,259	64,385
Share	52.4%	27.0%	20.6%	—
Train sequences (24 runs)				30,188
Val sequences (4 runs)				7,551
Test sequences (4 runs)				7,737

The split deserves a paragraph, because it is where sequence datasets usually go wrong. Consecutive sequences from one run overlap by up to 15 of 16 frames; a random sequence-level split would put near-duplicates of training samples in the validation set and report a fantasy accuracy. The index generator instead splits at the run level, using a greedy assignment that balances the 0.8/0.1/0.1 ratio by sequence count (runs vary in length) while guaranteeing every label appears in every split, with a repair pass that moves runs from train if val or test ends up missing a class. The result: 24 train / 4 val / 4 test runs with zero overlap — validation measures generalization to new pancakes, not interpolation between neighboring frames.

7.3 Training and what the curves admit

Training is standard supervised classification: cross-entropy, AdamW at $3{\times}10^{-4}$ with weight decay $10^{-4}$, batch 32, AMP, ImageNet-normalized 224×224 inputs, best checkpoint selected by validation macro-F1 (the class shares in Table 4 make plain accuracy too forgiving of ignoring ready). The full TensorBoard history ships in the repository; Figure 7 replots it.

Figure 7Training history, replotted from the TensorBoard event files shipped in the repository. Left: macro-F1 — training saturates at ≈1.0 within four epochs while validation oscillates between 0.67 and 0.83, peaking at epoch 13 (the deployed checkpoint). Right: the corresponding losses. The gap is real overfitting and we report it as such: 24 training runs is a small world, and the model memorizes it. The deployed system compensates downstream with temporal smoothing and trigger hysteresis (§8) rather than pretending the classifier is better than it is.

Table 5Validation metrics of the deployed checkpoint (epoch 13): accuracy 0.871, macro-F1 0.830. The errors concentrate exactly where cooking is genuinely ambiguous — the almost_ready/ready boundary.

Class	Precision	Recall	F1
not_ready	0.903	1.000	0.949
almost_ready	0.840	0.748	0.791
ready	0.802	0.704	0.750

The structure of Table 5 is the honest summary of the problem. not_ready is essentially solved — wet batter looks like nothing else. The two later classes blur into each other because the underlying process is continuous and the label boundary is a human judgment call made once per run during annotation. A per-frame classifier this uncertain would make an unusable trigger if consumed naively; the next section is about consuming it non-naively.

8The Flip Trigger — Smoothing, Streaks, and the Confidence-Drop Path

The GRU emits a probability vector thirty times a second; the runtime needs a single, safe, irreversible boolean. Bridging that gap is a three-layer filter, and the third layer is the most interesting engineering decision in the perception stack.

8.1 Layer 1 — exponential smoothing

Raw per-tick probabilities flicker: consecutive 16-frame windows differ by one frame, yet can disagree noticeably near a class boundary. The runtime smooths the probability vector with an exponential moving average before taking any argmax:

$$\bar{p}_t \;=\; \lambda\, \bar{p}_{t-1} + (1-\lambda)\, p_t, \qquad \lambda = 0.7,$$

whose step response reaches 63% of a sustained change after $-1/\ln\lambda \approx 2.8$ inference ticks (roughly a tenth of a second) — fast enough to track cooking, slow enough to erase single-window flicker. The prediction consumed downstream is $\hat{y}_t = \arg\max_c \bar{p}_t[c]$ with confidence $\bar{p}_t[\hat{y}_t]$.

8.2 Layer 2 — the ready streak

The primary trigger path is a debounced streak, structurally identical to the YOLO rule: the smoothed prediction must be ready for 3 consecutive inference ticks. Because the EMA already lowpasses the signal, a short streak suffices where the YOLO trigger needed 30 frames. In the logged cook runs this path fired with smoothed confidences of 0.40–0.65 — modest numbers, which is expected: near the flip point the true state straddles the almost_ready/ready boundary, and the EMA blends across it.

8.3 Layer 3 — the almost-ready confidence drop

Relying on the ready class alone has a failure mode we hit repeatedly during bring-up: on some runs the classifier grows very confident in almost_ready and stays there — the smoothed distribution parks at $\bar{p}[\texttt{almost\_ready}] \approx 1.0$ and the ready class never wins the argmax for three straight ticks, while the physical pancake quietly overcooks. But watching those runs revealed a usable signature: as bubbling peaks and fades, the model's almost_ready confidence sags — the clip stops looking like prototypical peak-bubbling. The information is in the derivative of the confidence, not the argmax. The runtime encodes exactly that:

Figure 8The confidence-drop trigger as a state machine. Saturation (confidence pinned at 1.00, rounded to two decimals) must persist for 10 s to qualify the pattern as genuine peak bubbling; only then does a sag to ≤ 0.90 fire the flip. Any excursion out of almost_ready resets the qualification — the drop only counts if it happens from a sustained plateau.

The rule's parameters encode two distrusts. The 10-second qualification distrusts transient saturation: early in a cook the confidence can briefly touch 1.00 while bubbling ramps up, and a drop from a momentary peak means nothing. The two-decimal rounding of the confidence distrusts float jitter: "saturated" is defined as $\mathrm{round}(\bar{p}, 2) \ge 1.00$ and the drop as $\mathrm{round}(\bar{p}, 2) \le 0.90$, giving the comparison a built-in dead band of ten percentage points that EMA noise cannot cross by accident. The final trigger delivered to the stage machine is the disjunction of both paths:

$$\text{flip} \iff \underbrace{\big(\text{streak}(\hat{y}_t = \texttt{ready}) \ge 3\big)}_{\text{classifier is sure}} \;\lor\; \underbrace{\big(\text{qualified} \wedge \bar{p}_t[\texttt{almost\_ready}] \le 0.90\big)}_{\text{plateau decays}}$$

In the four logged cook waits, the streak path fired first every time — after watches ranging from 2 m 18 s to 6 m 36 s, at smoothed confidences between 0.40 and 0.65 (pancakes, it turns out, vary a lot). The drop path is the safety net for the runs where the argmax never commits, and it converts the model's most common failure mode (overconfident almost_ready) into its own detector.

Design note — the preview lies a little, on purpose

The live preview window shows almost_ready whenever ready is predicted but the streak is still building. During bring-up we found the flickering label actively misleading — an operator sees "ready", the robot does nothing (correctly — the debounce is still counting), and the system looks broken. Displaying the decision state rather than the raw argmax makes the UI truthful about what the robot will actually do. The raw prediction is still logged every tick for offline analysis.

9Flipping and Plating — ACT Imitation Policies

Flipping a pancake with a spatula is everything the pouring task is not: contact-rich, tool-mediated, tolerance-critical, and impossible to waypoint. The spatula must wedge under a disc that is set on top and still soft underneath, at an attack angle shallow enough not to plow it, then execute a flick whose outcome depends on friction, batter weight, and where the pancake actually sits — which varies, because the pour trigger controls amount, not placement. This is the regime where imitation learning currently earns its complexity, and we used ACT (Action Chunking with Transformers), as implemented in LeRobot.

Task 2 — flipping · ACT policy 1

Task 3 — plating · ACT policy 2

Video 3The two learned skills. Flipping: retrieve the spatula from its holder, wedge under the pancake, flip, and re-park the tool. Plating: retrieve the spatula again, extract the finished pancake from the pan, and deliver it to the plate. Both are single continuous policy rollouts at 30 Hz.

9.1 Demonstrations

Each task has its own dataset, teleoperated with a leader arm in the standard LeRobot workflow and recorded at 30 Hz with all four policy cameras. All demonstrations start from the base pose — the same base pose the runtime holds between stages — so the deployment-time initial condition is squarely inside the training distribution.

Table 6Demonstration datasets and policy training configuration (both public on the Hugging Face Hub). Observation: 6-D joint state + four 640×480 RGB views; action: 6-D target joint positions.

	Task 2 · flipping	Task 3 · plating
Episodes	111	120
Frames (30 fps)	161,933 (≈90 min)	169,983 (≈94 min)
Dataset	Panbot_task2_dataset_3	Panbot_task3_dataset_3
Checkpoint	act_panbot_task2_3	act_panbot_task3_3
Training steps	70,000 · batch 8 · lr 1×10⁻⁵
Backbone / model dim	ResNet18 per camera · d = 512, 8 heads
Transformer	4 encoder + 1 decoder layers
CVAE latent	32-D, VAE objective enabled
Chunk / executed steps	100 / 100 (3.33 s per chunk at 30 Hz)

9.2 Why chunking fits this task

ACT trains a conditional VAE whose decoder predicts a chunk of $k$ future actions from the current observation, optimizing L1 reconstruction plus a KL term over a latent "style" variable $z$:

$$\mathcal{L} \;=\; \big\lVert a_{t:t+k} - \hat{a}_{t:t+k}(o_t, z) \big\rVert_1 \;+\; \beta\, D_{\mathrm{KL}}\!\big(q(z \mid a_{t:t+k}, o_t)\,\Vert\, \mathcal{N}(0, I)\big).$$

With $k = 100$ and all 100 steps executed per prediction, the policy commits to 3.3-second motion segments — approximately the natural granularity of this task's primitives (one approach, one wedge, one flick). Chunking suppresses the compounding-error jitter of step-by-step imitation and gives the rollout its visibly smooth, decisive quality; the cost is reactivity between chunk boundaries, a limitation we return to in §10 and §13. At deployment the latent is set to its prior mean, making the policy deterministic given the observations.

9.3 Integration: policies as guests, not owners

In stock LeRobot examples, a policy script connects the robot, runs, and disconnects. That pattern breaks a multi-stage runtime — connecting the SO-ARM101 and four cameras takes seconds and re-homes the arm through unpredictable intermediate states. Panbot's common_policy_runner instead borrows the orchestrator's already-connected robot: it loads the pretrained config and weights from the Hub, reconstructs the observation/action processor pipelines and the feature spec from the live robot's own feature declarations, and then runs the standard predict-act loop —

obs   = robot.get_observation()               # joints + 4 cameras
frame = build_dataset_frame(features, obs)    # to policy feature spec
a     = predict_action(frame, policy, ...)    # ACT chunk head
robot.send_action(action_processor((a, obs)))
precise_sleep(dt - elapsed)                   # hold 30 Hz

— checking the shared stop event every step, so the keyboard kill switch preempts a rollout within 33 ms. Rollout durations are configured, not learned: 90 s per policy in the shipped config, with a 30 s base-pose wait between them while the pancake's second side cooks. The measured loop rate over a logged 200-second rollout was 29.3 Hz — the 5,867-step log line is the receipt.

Design note — one repo id is the whole deployment interface

Swapping a policy means editing one line of runtime.yaml (policies.policy1.repo_id). Datasets, checkpoints, and the runtime exchange everything through the Hub, so the machine that trains (a GPU workstation) and the machine that cooks share no filesystem. The config also carries optional per-policy overrides — task prompt, observation rename map, dataset stats source — none of which this deployment needed, but which cost nothing to plumb through and make the runner reusable for the next robot.

10Implicit Recovery

The most instructive behavior in the whole system was one we never programmed. During evaluation rollouts, when a flip landed the pancake half-folded against the pan rim, or the spatula lost the pancake mid-transfer, the ACT policies would frequently pause, re-approach, and try again — successfully. We call this implicit recovery: recovery behavior that emerges from the policy's training distribution rather than from any explicit failure detection or recovery logic. There is no "retry" branch anywhere in the code.

Flipping recovery · re-wedge after a failed first attempt

Plating recovery · re-approach after losing the pancake

Video 4Unscripted recovery during policy rollouts. In both clips the first manipulation attempt fails — the pancake sits badly or slips off the spatula — and the policy re-aligns and completes the task within the same rollout, with no external intervention and no recovery-specific code.

10.1 Where the behavior comes from

Three ingredients make this work, and all three are ordinary. First, the demonstrations contain recoveries: teleoperating 111 and 120 episodes of a genuinely fiddly task means the human operator botched wedges and re-tried, slid pancakes back off rims, and chased misplaced pancakes across the pan — and all of it went into the dataset. A demonstrator who never fails teaches a policy that cannot recover; our failure rate as teleoperators turned out to be a feature. Second, the policy is closed-loop at chunk granularity: every 3.3 s it re-observes through four cameras and predicts a fresh chunk conditioned on where the pancake actually is. A failed flip does not derail a plan, because there is no long-horizon plan — only the next chunk. Third, the rollout window leaves room to fail: 90 seconds is several attempts' worth of time for a skill whose nominal execution takes twenty, so a first attempt's failure is recoverable rather than terminal.

10.2 What implicit recovery is not

We want to be precise about the claim, because this class of demo is easy to oversell. The policy recovers from perturbations near its data manifold: a pancake displaced by a few centimeters, a slipped grasp, a half-completed flip — states that resemble states a human demonstrator visited and fixed. It does not recover from a pancake on the floor, a spatula knocked out of its holder sideways, or any state that requires understanding that something failed rather than simply acting from the current observation. There is no success detector: a rollout that silently accomplished nothing still ends when its timer ends, and the runtime proceeds regardless (§13). Within those boundaries, though, the robustness is real, repeatable on camera, and — for a system this small — genuinely useful: it converts the many small execution imperfections of hobby servos and a floppy 3D-printed spatula from run-enders into recoverable noise.

Safety note

Implicit recovery is a robustness property, not a safety guarantee. The arm operates in a constrained tabletop cell around a hot cooking surface, and every run kept a human within reach of the kill switch (§3.3) and the griddle's power. Nothing in this section should be read as the policy "knowing" about the hazard.

11Reliability Engineering

A robot that touches a hot pan for ten minutes per run, unattended, accumulates reliability requirements that a benchtop demo never meets. Most of main_runtime.py's bulk is not the stage logic of §3 but the scaffolding around it, and this section collects the pieces that earned their keep during bring-up.

11.1 Failure taxonomy and mitigations

Table 7Failure modes encountered or designed against during development, and where each mitigation lives. The two-camera-path architecture (§2) makes the first column's diagnosis nearly mechanical.

Failure	Symptom	Mitigation
Trigger camera stalls	Stage never advances	2 s frame watchdog → abort + safe ramp
Segmentation flicker	Premature pour stop	30-frame hold counter; largest-mask-only rule
GRU argmax never commits	Pancake overcooks in WAIT_GRU	Confidence-drop trigger path (§8.3)
Operator panic / bad behavior	—	ESC from any stage, checked every 33 ms, exits via ramp not freeze
Arrow key ≠ emergency stop	Spurious aborts	ESC-sequence disambiguation with 20 ms lookahead
Policy ends in odd pose	Next stage starts from out-of-distribution state	Ramped return to the shared base pose after every stage
Config drift (edited waypoints)	Wrong-length sequences	Startup validation: exactly 10 out / 11 back, unknown joint keys rejected
LeRobot version skew	Import errors on camera config	Multi-path best-effort import of OpenCVCameraConfig
Missing model files	Late crash mid-run	All paths (corners, YOLO, GRU) existence-checked before the robot moves
Any unhandled exception	—	Global handler: best-effort safe pose, then re-raise with full traceback in the log

11.2 One YAML to hold it all

Every number in this article — camera indices, thresholds, hold counts, EMA weight, ramp times, waypoint poses, policy repo ids, rollout durations — lives in a single runtime.yaml, normalized through a defaulting pass at load so that a sparse config still yields a fully-populated, type-coerced structure. Two consequences were worth the discipline. Tuning sessions never touched Python: adjusting the pour threshold between batters, or slowing the bottle un-tilt ramp, was a config edit visible in version control. And the config is the experiment record: since the log banner prints the loaded path, any logged run can be reproduced by pointing the runtime at the same file.

11.3 Logs as the primary instrument

The runtime logs to terminal and a timestamped file simultaneously, one line per event, with the trigger payloads serialized inline. Every claim in §12 is read directly off these files — which is the point. A line like

[YOLO] TRIGGER ✅ info={'ratio': 0.1857, 'mask_area': 1057740.0,
                        'hit_count': 30, 'hold_frames': 30, ...}
[GRU]  TRIGGER ✅ info={'pred_label': 'ready', 'conf': 0.5806,
                        'ready_streak': 3, 'buffer': 91, ...}

carries the entire decision context of a transition: not just that the trigger fired but the measured coverage, the debounce state, and the confidence it fired at. During tuning, diffing these payloads across runs replaced guesswork about thresholds with arithmetic; after tuning, they turned the demo videos into auditable experiments. The preview windows serve the same purpose live — the YOLO overlay and the GRU label render the trigger's internal state onto the camera stream (Figure 5) — and both can be disabled with a keypress without touching the run, since drawing is strictly a side effect of, never an input to, the decision path.

12Results — End-to-End Runs

The system's output is a cooked, flipped, plated pancake with no human input between "start" and "done". Video 1 (top of the article) shows the produced demonstration; Video 5 below is the unedited single-take counterpart. For the quantitative record, Table 8 reconstructs a complete instrumented run directly from its log file — every timestamp, trigger value, and step count below is copied from the log, not estimated. (This rehearsal run used shortened policy windows — 5 s flip, 200 s plate, 10 s wait — while exercising the full pipeline; the shipped configuration runs 90/90 with a 30 s wait.)

Table 8Timeline of a complete logged run (main_runtime_20260210_163726.log), from process start to clean shutdown: 7 m 32 s end to end, of which 2 m 56 s is the perception-gated cook wait.

Clock	Δt	Event	Logged detail
16:37:32	—	Stage 1 begins	robot + 4 cameras connected, trigger cam at 4K
16:37:52	20.3 s	Pour posture reached	10 waypoints, first ramp 5 s
16:38:08	15.2 s	YOLO trigger	ratio 0.186 ≥ 0.12 · hit 30/30 → pour interrupt
16:38:28	20.5 s	Bottle re-docked	11 return waypoints, 3 s un-tilt ramp
16:41:24	2 m 56 s	GRU trigger	ready · conf 0.581 · streak 3/3 · buffer 91/91
16:41:26	1.6 s	Policy 1 loaded	ACT flip checkpoint from Hub cache
16:41:31	5.0 s	Flip rollout ends	147 steps → 29.2 Hz measured
16:41:42	10.0 s	Inter-policy wait ends	base-pose hold while second side cooks
16:45:04	200.0 s	Plating rollout ends	5,867 steps → 29.3 Hz measured
16:45:05	1.0 s	Clean shutdown	ramp to base pose · “main_runtime finished OK ✅”

Across the four fully logged cook waits, the perception layer behaved consistently with its design: the YOLO trigger fired at coverage ratios of 0.129–0.186 (threshold 0.12 plus the climb during the one-second debounce), and the GRU streak path fired after waits of 2 m 18 s to 6 m 36 s at smoothed confidences of 0.40–0.65. The spread in cook-wait duration is the strongest argument for the architecture — a fixed timer tuned to any one of those runs would have burned or undercooked the other three.

Video 5The unedited evidence: one continuous take of the full pipeline with no cuts and no resets, including the real-time cook wait. Where Video 1 is the produced summary, this is the experiment log in video form.

12.1 Qualitative observations

Three observations from running the system that the numbers do not capture. First, the trigger architecture changes how failures feel: when something goes wrong, the run does not degrade into flailing — it stalls at a stage boundary with the preview window showing exactly which condition is unmet, and the fix (nudge a threshold, re-click the corners) is usually config-level. Second, the pour trigger is the quiet workhorse: batter viscosity varies between mixes and over time as it settles, so identical pour durations produce meaningfully different pancakes — the coverage trigger absorbed this without retuning across sessions. Third, chunked policies read as deliberate: at 3.3 s per committed chunk, the arm's behavior has a legible rhythm — approach, wedge, flip as distinct strokes — which made failures easy to attribute (a bad wedge angle versus a bad flick) and, frankly, makes the robot pleasant to watch.

13Limitations and Future Work

Everything below is a real limitation of the system as built. We list them in decreasing order of how much they would bite a user who tried to reproduce the results.

13.1 The perception models know one kitchen

Both trigger models are trained on this pan, this batter recipe, this camera, and this room's lighting. The GRU's training curves (Figure 7) show unambiguous overfitting — training macro-F1 saturates at 1.0 by epoch four while validation oscillates around 0.75–0.83 — and the honest reading is that 24 training runs cannot constrain an 11.8M-parameter model; the run-level split keeps the measurement honest, not the model general. Val macro-F1 of 0.830 is workable for a debounced trigger but would not survive a different pan coating or a window-lit kitchen without retraining. The same is true, more mildly, for the single-class YOLO. Scaling the cook-run corpus (recording is cheap; annotation is 96 segment boundaries per 32 runs) and adding augmentation across lighting are the obvious first steps.

13.2 The stage sequence is open-loop above the triggers

Perception decides when stages advance, but nothing verifies that a stage succeeded. The flip rollout ends on a timer whether or not the pancake flipped; plating ends whether or not the pancake reached the plate; there is no re-entry, no retry at the stage level, and no final "is there a pancake on the plate" check. Implicit recovery (§10) papers over small failures within a rollout, but a categorical failure — a pancake dropped outside all camera views — sails through to a cheerful clean shutdown. A success classifier per stage (the trigger-camera infrastructure would carry one; the plating stage even re-exposes the pan to it) is the highest-leverage missing component, and would also enable multi-pancake scheduling, which the single-shot stage machine currently cannot express.

13.3 Calibration is a single point of failure

The four warp corners are calibrated once and trusted forever. A bumped tripod silently shifts the warp; the models then see a distorted pan and degrade without any error being raised — during development this was our single most common "mystery failure", always fixed by re-clicking four corners. The trigger thresholds encode similar deployment-specific tuning ($\tau$ went from a default 0.17 to 0.12 for this batter; the 10 s/0.90 drop-trigger constants were hand-fit to observed confidence traces). None of this is self-checking. A startup sanity pass — projecting the corners back onto the live frame and demanding operator confirmation, plus telemetry that flags trigger statistics drifting outside historical bands — would convert silent degradation into loud degradation, which is most of the battle.

13.4 Hardware ceilings

The SO-ARM101 is position-controlled with no force or torque sensing: the spatula wedge relies on the compliance of a 3D-printed tool and the servos' tolerance of brief overload, not on measured contact. The policies compensate by having learned motions that usually avoid jamming, but a firmer pancake or a warped pan surface changes the contact physics in ways the system cannot feel. Kinematically, one 6-DOF arm bounds the achievable workspace and rules out bimanual strategies (stabilizing the pan while flipping). And the whole perception stack plus two ACT policies assume a CUDA GPU adjacent to the griddle — modest hardware by robot learning standards, but not nothing for a kitchen appliance.

13.5 Where this goes next

• Success detection and stage retry — close the loop above the triggers; the architecture already has the camera and the state machine seams for it.
• Learned pouring — the waypoint pour is placement-blind; a policy conditioned on the live coverage mask could center and shape the pour, subsuming the YOLO trigger into a continuous control signal.
• Continuous cooking — pancake $n{+}1$ pours while $n$ cooks; requires the success detector, a scheduling layer over the stage machine, and plating-side collision awareness.
• A single multi-task policy — flip and plate share the spatula-retrieval prefix; a task-conditioned ACT (the runner already plumbs a task string through) or a small VLA could merge the checkpoints and halve the model-management surface.
• Trigger self-calibration — replace the hand-fit drop-trigger constants with quantities estimated from the confidence trace itself (e.g., trigger on a sustained negative slope rather than absolute levels), removing the tightest coupling to this particular checkpoint's calibration.

14Closing Thoughts

Panbot began as a simple question: could a hobby-class arm cook something real, unattended, start to finish? The answer turned out to hinge on almost nothing we expected. The manipulation — the part that looks hard in the video — was largely solved by existing tools: LeRobot's teleoperation pipeline and a well-understood imitation architecture, applied carefully. The part that consumed our engineering attention was time: knowing when there is enough batter, knowing when a pancake is ready, and building trigger logic honest enough to act on models that are only 83% right.

Three lessons generalize beyond pancakes. Match the tool to the sub-problem, not the project. One scripted sequence, two learned policies, and two perception triggers is an unfashionable mixture, but each piece is the cheapest thing that solves its sub-problem, and the mixture is why the system is debuggable. Design for the model you have, not the model you want. The EMA, the streak debounce, and especially the confidence-drop trigger exist because the GRU is imperfect in a structured way; encoding that structure into the consumer bought more reliability than another month of training data would have. Instrument first. The trigger payloads in the logs and the live preview overlays were built before the first full run, and every tuning decision in this article traces back to them — a robot you can interrogate is a robot you can ship, even if only to your own kitchen.

And there is something quietly satisfying about the final artifact: a machine that watches a pancake with two neural networks, waits with the patience only a machine has, and flips at the right moment for reasons you can read off a log file. Everything — runtime, vision pipeline, datasets, checkpoints — is public; we would be delighted to see it cook in someone else's kitchen.

15References

1. T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, “Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware” (ALOHA / Action Chunking with Transformers), Robotics: Science and Systems (RSS), 2023.
2. R. Cadene, S. Alibert, A. Soare, et al., “LeRobot: State-of-the-art Machine Learning for Real-World Robotics in PyTorch,” Hugging Face, github.com/huggingface/lerobot, 2024.
3. TheRobotStudio and Hugging Face, “SO-ARM100 / SO-ARM101: Standard Open Arm,” github.com/TheRobotStudio/SO-ARM100.
4. G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics YOLOv8,” github.com/ultralytics/ultralytics, 2023.
5. A. Kirillov, E. Mintun, N. Ravi, et al., “Segment Anything,” IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
6. K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
7. K. Cho, B. van Merriënboer, C. Gulcehre, et al., “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation,” EMNLP, 2014.

Written by Inseong Paik · March 2026 · Panbot was built with NmDongQ. The runtime is at github.com/ispaik06/Panbot and the vision pipeline at github.com/ispaik06/Panbot_vision; the demonstration datasets and ACT checkpoints are public on the Hugging Face Hub. All quantitative claims trace to the repository: trigger values and timings to the shipped runtime logs, training curves to the TensorBoard event files, and configuration numbers to runtime.yaml. Figures are hand-drawn SVG; charts are replotted from the raw event data; math is pre-rendered with KaTeX and the page loads no external resources except the click-to-play YouTube embeds.