Panbot: A Perception-Triggered Autonomous Pancake-Cooking Robot

An end-to-end cooking system on a low-cost 6-DOF arm that pours batter, waits for the pancake to cook, flips it, and plates it — fully autonomously. One trigger camera and two learned perception models decide when each stage should happen; a deterministic waypoint sequence and two ACT imitation policies decide how. Built on the LeRobot stack with YOLOv8 segmentation, a ResNet18+GRU cook-state estimator, and a fail-safe runtime orchestrator.

▶ Watch on YouTube · 100% autonomous

• A perception-triggered runtime architecture. One shared trigger camera and two learned models gate all stage transitions; the policy observation cameras are kept strictly separate. Stage logic, trigger debouncing, and fail-safe paths live in a single 30 Hz orchestrator (§3).
• Metric batter measurement from a monocular camera. A four-point perspective warp normalizes the pan to a top-down view, making "fraction of pan covered" a stable, threshold-able quantity for a YOLOv8-seg model — with a hold-to-trigger debounce that survives segmentation flicker (§5–6).
• Temporal cook-state estimation. A ResNet18+GRU classifier over 16 frames sampled at stride 6 (a 3.03 s window) labels the pan as not_ready / almost_ready / ready, trained on 32 fully annotated cooking runs with a leakage-free run-level split (§7).
• A confidence-drop trigger. The flip moment is detected not only by the ready class but by a qualified drop in almost_ready confidence after sustained saturation — an empirical signature of bubble activity peaking and fading (§8).
• ACT policies with implicit recovery. Two chunked transformer policies (chunk 100 at 30 Hz) for flipping and plating, integrated to share one robot connection with the orchestrator, and demonstrably able to recover from misplaced pancakes and slipped grasps (§9–10).

1Introduction

Most tabletop manipulation demos share a convenient property: the world waits for the robot. A cube does not mind sitting on the table for another thirty seconds while the policy deliberates. Cooking removes that luxury. Once batter hits a hot pan, a physical process starts running on its own clock — the batter spreads, sets, bubbles, and eventually burns — and the robot's job is not merely to execute motions correctly but to execute them at the right moment. Pour too long and the pancake overflows the pan; flip too early and the half-liquid disc tears; flip too late and the bottom scorches. The task is a scheduling problem wearing a manipulation costume.

Panbot is our attempt to take that problem seriously on hobby-grade hardware: a LeRobot SO-ARM101 follower arm — six position-controlled Feetech STS3215 servos and 3D-printed links — a household electric griddle, a batter dispenser, and a 3D-printed spatula. Nothing in the hardware is exotic, which is precisely the point: the interesting engineering lives in when the robot acts, and that is a perception problem, not an actuation problem.

1.1 Perception decides when, control decides how

The pancake workflow decomposes into three motion tasks separated by two waiting decisions:

1. Pour batter into the pan — and decide when there is enough;
2. wait for the pancake to cook — and decide when it is ready to flip;
3. flip it, let the second side cook, and plate it.

The two decisions have nothing in common with the three motions. Deciding "enough batter" is a geometric measurement; deciding "ready to flip" is a judgment about the temporal texture of a cooking surface — bubbles forming, popping, and leaving pores as the batter sets. Neither decision benefits from being entangled with motor control, so Panbot does not entangle them: a single dedicated trigger camera watches the pan from above, two learned models turn its stream into boolean stage-transition events, and the motion controllers are swapped underneath whenever a trigger fires. Each motion task then uses the cheapest controller that does the job, summarized in Table 1.

This split has a second, subtler benefit: it makes failures diagnosable. When a stage transition misfires, the fault is in the trigger stack (camera, warp, model, threshold); when a motion goes wrong, the fault is in the controller stack (waypoints, policy, observation cameras). The two stacks share no state except the orchestrator's stage variable, and the troubleshooting table in the repository README is organized around exactly this boundary.

1.2 Scope, honestly stated

Panbot is a system-integration and perception-timing project, not a claim about general cooking autonomy. The workspace is fixed and instrumented for this one dish; the perception models are trained on this pan, this batter, and this kitchen's lighting; the ACT rollouts run for a configured duration rather than until a detected success. §13 catalogs these limitations without cosmetics. What the project does demonstrate is that with careful decomposition, a completely autonomous multi-stage cooking pipeline — liquid handling, process monitoring, and contact-rich manipulation — fits on a 3D-printed arm and a single GPU, and that the perception-triggered architecture holds up end-to-end in unedited runs (raw single-take footage).

2System Architecture

The system is organized around one invariant: the camera that decides is never the camera that acts. A single high-resolution trigger camera feeds the two perception models that gate stage transitions; four separate low-resolution cameras feed the ACT policies their multi-view observations. The two paths never mix — a trigger-camera problem shows up as a stage that never advances, a policy-camera problem shows up as bad actions, and neither can masquerade as the other. Figure 1 shows the full dataflow.

2.1 Hardware

The arm is the standard LeRobot SO-ARM101 follower: six Feetech STS3215 serial-bus servos in a 3D-printed structure, driven over a single USB serial port. Around it we built the cooking cell visible in Video 1: an electric griddle pan, a squeeze-bottle batter dispenser resting in a 3D-printed cradle at a known pose, a 3D-printed spatula parked in a holder bowl, and a plating target. Every fixture has a fixed, calibrated location — Task 1's waypoints and the ACT policies' demonstrations all assume this cell layout. The five cameras are enumerated in Table 2.

2.2 Rates and timing

Everything in the runtime is deliberately pinned to one clock. The main loop, the camera streams, and the policy rollouts all run at 30 Hz (task.hz = vision.fps = policy_fps = 30), so one loop tick consumes exactly one camera frame and emits at most one robot action. This alignment is not cosmetic: trigger debounce counts are specified in frames, so keeping frames and control ticks 1:1 makes "30 frames of stability" mean exactly one second of wall-clock stability, and makes log timestamps directly comparable across perception and control events.

3Runtime Orchestration

Panbot/control/main_runtime.py is the single process that owns everything: the robot connection, the trigger camera, both perception models, the stage machine, and every fail-safe. There is no middleware, no message bus, and no second process to fall out of sync with. The stage machine it implements is small enough to state exactly (Figure 2), and that smallness is a feature — every transition is either a perception trigger, a timer, or a fail-safe, and each appears as a single greppable line in the run log.

3.1 The 30 Hz stage loop

Stages 1 and 2 share one loop body. Each tick reads a frame from the trigger camera, feeds it to whichever model is active for the current stage, steps whichever motion controller is active, and then sleeps the remainder of the 33.3 ms period:

while not stop_event.is_set():
    ok, frame = cap.read()                  # trigger camera, 4K
    if stage == "TASK1":
        trig, vis, info = yolo.step(frame)  # coverage ratio + debounce
        if trig: task1.interrupt_to_return()
        task1.step(now)                     # one send_action per tick
        if task1.is_return_done(): stage = "WAIT_GRU"
    elif stage == "WAIT_GRU":
        base_ctrl.tick()                    # rate-limited hold
        trig, vis, info = gru.step(frame)   # cook-state estimate
        if trig: break                      # → policy stages
    sleep(dt - elapsed)                     # lock to 30 Hz

Two properties of this loop matter more than its simplicity suggests. First, perception and motion advance in the same tick: the YOLO trigger does not pause pouring while it thinks, because a 640-px single-class segmentation runs comfortably inside the frame budget. Second, the trigger is an interrupt, not a phase boundary: when the YOLO event fires mid-ramp, interrupt_to_return() abandons the remaining pour waypoints immediately and starts the return sequence from the arm's current measured pose, so the pour stops within one tick of the decision rather than at the end of a waypoint.

3.2 Every motion is a ramp

The runtime contains exactly one motion primitive, used by the waypoint stepper, the base-pose controller, and every fail-safe path alike: linear interpolation in joint space from the robot's current measured configuration to a target, streamed as position commands at the loop rate. With start pose $q_0$ (read from the encoders at ramp entry), target $q_1$, and ramp duration $T$:

sent at 30 Hz, i.e. $\lceil 30\,T \rceil$ intermediate commands. Reading $q_0$ from the observation rather than assuming the previous target is what makes the primitive safe to invoke from any state — after an interrupted pour, after a policy rollout that ended in an arbitrary configuration, or after an exception. The servos' own position loops handle the final tracking; the ramp exists to bound the commanded velocity, since a raw send_action jump to a distant pose would command a violent full-speed swing.

3.3 Fail-safe paths

Three independent mechanisms can stop a run, and all of them converge on the same exit ramp:

• Keyboard. A raw-terminal KeyWatcher thread reads stdin byte-by-byte in cbreak mode. A bare ESC byte sets the stop event; q only disables the preview windows. Because arrow keys also begin with ESC, the watcher waits 20 ms for a following byte and treats ESC-then-something as an escape sequence to ignore — a small detail that prevents an accidental arrow-key press from emergency-stopping a cooking run.
• Signals. SIGINT and SIGTERM are routed into the same stop event, so Ctrl+C and a supervisor kill behave identically to ESC.
• Camera watchdog. If the trigger camera delivers no frame for 2 s during a vision stage, the runtime raises rather than silently holding the last action — a frozen camera must never leave the arm mid-pour with no supervising perception.

The stop event is checked at every loop tick and inside the policy rollout loop, so an ESC during an ACT rollout exits within one 33 ms step. Every exit path then calls the same _best_effort_safe_pose(): ramp to the base pose over 2.5 s, hold it briefly under the base-pose controller, and disconnect. The finally block releases the camera, destroys the preview windows, and disconnects the robot even if the safe ramp itself throws.

4Task 1 — Deterministic Batter Pouring

Pouring is the one task in the pipeline where we deliberately chose not to learn. The batter bottle rests in a fixed cradle; the pan does not move; the grasp is the same every run. A scripted joint-space sequence executes this perfectly every time, is trivially inspectable, and — critically — can be interrupted at an arbitrary point by the perception trigger, which is much harder to guarantee for a learned policy mid-rollout. The adaptive part of pouring is not the motion; it is knowing when to stop, and that job belongs to the YOLO trigger (§6).

4.1 A non-blocking waypoint stepper

Task1MotionStepper executes a list of joint-space waypoints as an explicit state machine with two phases per waypoint — RAMP (linear interpolation from the current measured pose, §3.2) and HOLD (re-send the target for a dwell time) — and advances one send_action per 30 Hz tick. Nothing blocks: the stepper is stepped by the main loop rather than running the sequence itself, which is what lets the YOLO trigger preempt it between any two ticks. The configured sequences are 10 waypoints out (approach → grasp → lift → tilt) and 11 back (un-tilt → travel → release → retreat), with the dwell time set to 10 ms — effectively continuous motion through the waypoints.

4.2 Per-segment ramp times

A single global ramp speed cannot serve this sequence: the first move travels half the workspace from the folded base pose (want: slow), while the mid-sequence pours are short precision moves (want: brisk), and re-righting a bottle full of batter must not slosh (want: slow again). The config therefore supports per-waypoint ramp-time overrides keyed by target index:

task1_ramp_time_s: 1.0            # default per-waypoint ramp
task1_pose_hold_s: 0.01           # effectively continuous
task1_initial_ramp_overrides:
  0: 5.0        # base pose → first waypoint: long travel, go slow
  1: 1.0
task1_return_ramp_overrides:
  2: 3.0        # un-tilting the full bottle: slow to avoid slosh
task1_return_to_base_ramp_time_s: 1.0

The run logs confirm the schedule to the millisecond: the first ramp takes 5.01 s, ordinary waypoints tick by at 1.02–1.07 s including the dwell, and return-waypoint 3 — the bottle un-tilt — takes 3.01 s. The stepper also validates sequence lengths at construction (10 out, 11 back) so that a config edit that drops a waypoint fails at startup instead of mid-pour.

4.3 Interrupt semantics

When the coverage trigger fires, interrupt_to_return() switches the stepper's mode to RETURN and enters the first return waypoint's ramp from the current measured pose — mid-ramp, mid-hold, or in the terminal pouring posture, it does not matter. In the instrumented run of §12 the pour held its terminal posture for 15.2 s of steady flow before the trigger fired; in a second logged run, 8.1 s. That variation is the entire point of the architecture: the same fixed motion serves different batter amounts, temperatures, and flow rates because the stop decision floats free of the motion script. After the return sequence re-docks the bottle, the base-pose controller ramps the arm home over 1 s and Stage 2 begins.

5Top-View Normalization

Both perception models see the pan through the same preprocessing step: a fixed perspective warp that maps the pan region of the oblique 4K camera view onto an axis-aligned, top-down rectangle. This one transform does a surprising amount of work, and it is worth being precise about why it exists before describing the models it feeds.

5.1 The transform

A one-time calibration tool streams the trigger camera and lets the operator click the pan's four inner corners in TL→TR→BR→BL order; the pixel coordinates are saved to corners.json. At runtime, the source quadrilateral $\{(x_i, y_i)\}_{i=1}^{4}$ and the destination rectangle $\{(0,0), (W{-}1,0), (W{-}1,H{-}1), (0,H{-}1)\}$ determine the unique homography — eight degrees of freedom from four point correspondences — via cv2.getPerspectiveTransform, and each frame is resampled as

where the output size is computed from the corner geometry itself — the width is the larger of the two horizontal edge lengths and the height the larger of the two vertical ones:

With the shipped corners this yields a warped view of roughly 3156×1805 pixels — the pan fills the entire model input, at the camera's native resolution, with the background cropped away by construction.

5.2 Why warp before inference

• It makes area mean area. The pour-stop rule thresholds "fraction of the pan covered by batter". In the oblique view, a pixel near the far rim subtends more pan surface than one near the near rim, so raw mask area is a biased estimator that depends on where the batter sits. In the rectified view the pan-to-pixel scale is approximately uniform and the ratio $\text{mask px}/WH$ is a meaningful physical proxy.
• It deletes nuisance variation. Both models train and infer on pan-only crops: no table, no arm base, no background chairs to overfit to. For the GRU, whose training set is only 32 cooking runs, shrinking the input distribution to "the inside of this pan" is a large effective data-efficiency win.
• It decouples camera pose from the models. If the camera shifts, recalibrating means re-clicking four corners — not re-labeling data or retraining. The transform absorbs the viewpoint; the models only ever see the canonical view. (The flip side, honestly: if the camera shifts and nobody recalibrates, both models silently degrade — see §13.)

Both trigger models apply the warp independently with per-model config (use_warp, optional fixed output size), but in deployment they share the same corners file, and the batch tooling in Panbot_vision applies the identical transform when preparing training data — the warp seen at training time is bit-for-bit the warp seen at inference time.

6Stopping the Pour — Batter Segmentation

The pour-stop decision needs one number per frame: what fraction of the pan is covered by batter? Our first attempt computed it classically — a reference-frame difference against the empty pan with a binary threshold (measure_batter_area_1.py survives in the vision repository as the fossil of this approach). It worked until it met reality: specular highlights on the non-stick coating, the moving pour stream, steam, and the slow color shift of batter as it heats all punch holes in background subtraction. Batter-vs-pan is a semantic distinction, so we moved to a learned segmentation model and never looked back.

6.1 SAM-assisted labeling

The labeling pipeline in Panbot_vision is built for speed rather than scale. Warped pan frames are captured during real pours; an interactive labeling tool loads a Segment Anything ViT-B checkpoint and lets the annotator place positive clicks on batter and negative clicks on pan; SAM proposes candidate masks, the tool keeps the highest-scoring one live on screen, and a keystroke saves it as a binary PNG aligned to the frame — including explicit all-black masks for "no batter" frames, which matter as hard negatives. A converter then binarizes each mask, extracts external contours, simplifies them with Douglas–Peucker (tolerance $0.002 \times$ perimeter, components under 200 px dropped), and emits normalized polygon labels in YOLO-seg format with an automatic 80/20 train/val split. A few clicks per frame replace per-pixel polygon tracing; a usable dataset for this one-class problem took an afternoon, not a week.

6.2 Model and training

The segmentation model is an Ultralytics YOLOv8-seg fine-tuned from COCO weights on the single class batter at 640-px input (the training wrapper defaults to the nano variant and 80 epochs at batch 8; the deployed checkpoint is the run batter_seg_local_v1). This is deliberately the smallest model that solves the problem: one class, one pan, controlled lighting, and a hard 33 ms per-frame budget shared with the motion loop. The training wrapper can pull the dataset from and push best.pt to the Hugging Face Hub, which is how the vision repo and the runtime repo exchange artifacts without sharing a filesystem.

6.3 From masks to a trigger

At runtime, each warped frame passes through the model at confidence threshold 0.25. The trigger logic then reduces the prediction to one scalar and one counter. With $\mathcal{M}_t$ the largest predicted mask (upsampled to the warped resolution $W \times H$) at frame $t$:

with coverage threshold $\tau = 0.12$ and hold count $N = 30$ frames. Taking only the largest mask ignores stray droplets and the disconnected pour stream; the counter is a debounce: coverage must hold above threshold for one full second (30 frames at 30 fps), and a single frame that dips below — a flicker of the mask edge, a bubble of the stream occluding the pool — resets it to zero. The one-second debounce also has a physical meaning: batter keeps spreading after it lands, so demanding sustained coverage measures the settled pool rather than a transient splash.

Early · ratio 0.087 < τ · hit 0/30

Trigger · counter past 30 → fire

Settled · ratio 0.128 · pour stopped

Across the five logged cook runs the trigger fired at measured ratios between 0.129 and 0.186 — past the 0.12 threshold by a margin, as expected, since the ratio keeps climbing during the debounce second. The 30-frame hold proved to be the load-bearing parameter: the module's built-in default is a stricter $\tau = 0.17$, and the deployment tuned it down to 0.12 for this pan and batter viscosity while relying on the debounce to reject transients. The README's troubleshooting table encodes the tuning grammar — fires too early: raise $\tau$ or $N$ or the model confidence; never fires: lower $\tau$, then audit the warp corners and checkpoint path before touching anything else.

7Knowing When to Flip — Temporal Cook-State Estimation

Deciding when a pancake is ready to flip is a fundamentally different perception problem from measuring batter coverage, and the difference dictates the architecture. A cooking pancake's readiness is written in its dynamics: bubbles nucleate at the surface, grow, pop, and leave open pores as the batter sets; the surface sheen dulls; the rim dries. A single frame is ambiguous — a still image of a bubbled surface cannot distinguish "bubbles actively forming" (too early) from "bubbles popped and set" (ready). So the estimator must watch a clip, not a frame.

7.1 Architecture: ResNet18 features, GRU dynamics

The design puts the capacity where the signal is. A frozen-architecture ResNet18 handles appearance — pores, sheen, browning at the rim — while a deliberately small GRU (hidden size 256, one layer) handles temporal evolution over the 16-step feature sequence. The sampling scheme buys a long effective time horizon at low compute: with sequence length $T = 16$ and stride $s = 6$, the buffer requirement is

so the model sees three seconds of bubble dynamics for the inference cost of 16 encoder passes. The frame indices are anchored to the newest frame ($t, t{-}6, \dots, t{-}90$), and the identical $(T, s)$ scheme is used to generate the training sequences — train/test skew in the temporal geometry is ruled out by construction.

7.2 Dataset: 32 cooking runs, labeled by time segments

The training data is 32 complete real cooking runs recorded through the same 4K camera and the same warp. Rather than labeling frames, we labeled time segments: for each run, an annotations file gives frame ranges for not_ready (wet batter, bubbles forming), almost_ready (dense bubbling, surface beginning to set), and ready (bubbles popped, surface matte, flip window open). Each 16-frame training sequence inherits the label of the segment containing its final frame — the model's job is defined as "what is the state now, given the last three seconds", exactly matching its runtime use.

The split deserves a paragraph, because it is where sequence datasets usually go wrong. Consecutive sequences from one run overlap by up to 15 of 16 frames; a random sequence-level split would put near-duplicates of training samples in the validation set and report a fantasy accuracy. The index generator instead splits at the run level, using a greedy assignment that balances the 0.8/0.1/0.1 ratio by sequence count (runs vary in length) while guaranteeing every label appears in every split, with a repair pass that moves runs from train if val or test ends up missing a class. The result: 24 train / 4 val / 4 test runs with zero overlap — validation measures generalization to new pancakes, not interpolation between neighboring frames.

7.3 Training and what the curves admit

Training is standard supervised classification: cross-entropy, AdamW at $3{\times}10^{-4}$ with weight decay $10^{-4}$, batch 32, AMP, ImageNet-normalized 224×224 inputs, best checkpoint selected by validation macro-F1 (the class shares in Table 4 make plain accuracy too forgiving of ignoring ready). The full TensorBoard history ships in the repository; Figure 7 replots it.

The structure of Table 5 is the honest summary of the problem. not_ready is essentially solved — wet batter looks like nothing else. The two later classes blur into each other because the underlying process is continuous and the label boundary is a human judgment call made once per run during annotation. A per-frame classifier this uncertain would make an unusable trigger if consumed naively; the next section is about consuming it non-naively.

8The Flip Trigger — Smoothing, Streaks, and the Confidence-Drop Path

The GRU emits a probability vector thirty times a second; the runtime needs a single, safe, irreversible boolean. Bridging that gap is a three-layer filter, and the third layer is the most interesting engineering decision in the perception stack.

8.1 Layer 1 — exponential smoothing

Raw per-tick probabilities flicker: consecutive 16-frame windows differ by one frame, yet can disagree noticeably near a class boundary. The runtime smooths the probability vector with an exponential moving average before taking any argmax:

whose step response reaches 63% of a sustained change after $-1/\ln\lambda \approx 2.8$ inference ticks (roughly a tenth of a second) — fast enough to track cooking, slow enough to erase single-window flicker. The prediction consumed downstream is $\hat{y}_t = \arg\max_c \bar{p}_t[c]$ with confidence $\bar{p}_t[\hat{y}_t]$.

8.2 Layer 2 — the ready streak

The primary trigger path is a debounced streak, structurally identical to the YOLO rule: the smoothed prediction must be ready for 3 consecutive inference ticks. Because the EMA already lowpasses the signal, a short streak suffices where the YOLO trigger needed 30 frames. In the logged cook runs this path fired with smoothed confidences of 0.40–0.65 — modest numbers, which is expected: near the flip point the true state straddles the almost_ready/ready boundary, and the EMA blends across it.

8.3 Layer 3 — the almost-ready confidence drop

Relying on the ready class alone has a failure mode we hit repeatedly during bring-up: on some runs the classifier grows very confident in almost_ready and stays there — the smoothed distribution parks at $\bar{p}[\texttt{almost\_ready}] \approx 1.0$ and the ready class never wins the argmax for three straight ticks, while the physical pancake quietly overcooks. But watching those runs revealed a usable signature: as bubbling peaks and fades, the model's almost_ready confidence sags — the clip stops looking like prototypical peak-bubbling. The information is in the derivative of the confidence, not the argmax. The runtime encodes exactly that:

The rule's parameters encode two distrusts. The 10-second qualification distrusts transient saturation: early in a cook the confidence can briefly touch 1.00 while bubbling ramps up, and a drop from a momentary peak means nothing. The two-decimal rounding of the confidence distrusts float jitter: "saturated" is defined as $\mathrm{round}(\bar{p}, 2) \ge 1.00$ and the drop as $\mathrm{round}(\bar{p}, 2) \le 0.90$, giving the comparison a built-in dead band of ten percentage points that EMA noise cannot cross by accident. The final trigger delivered to the stage machine is the disjunction of both paths:

In the four logged cook waits, the streak path fired first every time — after watches ranging from 2 m 18 s to 6 m 36 s, at smoothed confidences between 0.40 and 0.65 (pancakes, it turns out, vary a lot). The drop path is the safety net for the runs where the argmax never commits, and it converts the model's most common failure mode (overconfident almost_ready) into its own detector.

9Flipping and Plating — ACT Imitation Policies

Flipping a pancake with a spatula is everything the pouring task is not: contact-rich, tool-mediated, tolerance-critical, and impossible to waypoint. The spatula must wedge under a disc that is set on top and still soft underneath, at an attack angle shallow enough not to plow it, then execute a flick whose outcome depends on friction, batter weight, and where the pancake actually sits — which varies, because the pour trigger controls amount, not placement. This is the regime where imitation learning currently earns its complexity, and we used ACT (Action Chunking with Transformers), as implemented in LeRobot.

9.1 Demonstrations

Each task has its own dataset, teleoperated with a leader arm in the standard LeRobot workflow and recorded at 30 Hz with all four policy cameras. All demonstrations start from the base pose — the same base pose the runtime holds between stages — so the deployment-time initial condition is squarely inside the training distribution.

9.2 Why chunking fits this task

ACT trains a conditional VAE whose decoder predicts a chunk of $k$ future actions from the current observation, optimizing L1 reconstruction plus a KL term over a latent "style" variable $z$:

With $k = 100$ and all 100 steps executed per prediction, the policy commits to 3.3-second motion segments — approximately the natural granularity of this task's primitives (one approach, one wedge, one flick). Chunking suppresses the compounding-error jitter of step-by-step imitation and gives the rollout its visibly smooth, decisive quality; the cost is reactivity between chunk boundaries, a limitation we return to in §10 and §13. At deployment the latent is set to its prior mean, making the policy deterministic given the observations.

9.3 Integration: policies as guests, not owners

In stock LeRobot examples, a policy script connects the robot, runs, and disconnects. That pattern breaks a multi-stage runtime — connecting the SO-ARM101 and four cameras takes seconds and re-homes the arm through unpredictable intermediate states. Panbot's common_policy_runner instead borrows the orchestrator's already-connected robot: it loads the pretrained config and weights from the Hub, reconstructs the observation/action processor pipelines and the feature spec from the live robot's own feature declarations, and then runs the standard predict-act loop —

obs   = robot.get_observation()               # joints + 4 cameras
frame = build_dataset_frame(features, obs)    # to policy feature spec
a     = predict_action(frame, policy, ...)    # ACT chunk head
robot.send_action(action_processor((a, obs)))
precise_sleep(dt - elapsed)                   # hold 30 Hz

— checking the shared stop event every step, so the keyboard kill switch preempts a rollout within 33 ms. Rollout durations are configured, not learned: 90 s per policy in the shipped config, with a 30 s base-pose wait between them while the pancake's second side cooks. The measured loop rate over a logged 200-second rollout was 29.3 Hz — the 5,867-step log line is the receipt.

10Implicit Recovery

The most instructive behavior in the whole system was one we never programmed. During evaluation rollouts, when a flip landed the pancake half-folded against the pan rim, or the spatula lost the pancake mid-transfer, the ACT policies would frequently pause, re-approach, and try again — successfully. We call this implicit recovery: recovery behavior that emerges from the policy's training distribution rather than from any explicit failure detection or recovery logic. There is no "retry" branch anywhere in the code.

10.1 Where the behavior comes from

Three ingredients make this work, and all three are ordinary. First, the demonstrations contain recoveries: teleoperating 111 and 120 episodes of a genuinely fiddly task means the human operator botched wedges and re-tried, slid pancakes back off rims, and chased misplaced pancakes across the pan — and all of it went into the dataset. A demonstrator who never fails teaches a policy that cannot recover; our failure rate as teleoperators turned out to be a feature. Second, the policy is closed-loop at chunk granularity: every 3.3 s it re-observes through four cameras and predicts a fresh chunk conditioned on where the pancake actually is. A failed flip does not derail a plan, because there is no long-horizon plan — only the next chunk. Third, the rollout window leaves room to fail: 90 seconds is several attempts' worth of time for a skill whose nominal execution takes twenty, so a first attempt's failure is recoverable rather than terminal.

10.2 What implicit recovery is not

We want to be precise about the claim, because this class of demo is easy to oversell. The policy recovers from perturbations near its data manifold: a pancake displaced by a few centimeters, a slipped grasp, a half-completed flip — states that resemble states a human demonstrator visited and fixed. It does not recover from a pancake on the floor, a spatula knocked out of its holder sideways, or any state that requires understanding that something failed rather than simply acting from the current observation. There is no success detector: a rollout that silently accomplished nothing still ends when its timer ends, and the runtime proceeds regardless (§13). Within those boundaries, though, the robustness is real, repeatable on camera, and — for a system this small — genuinely useful: it converts the many small execution imperfections of hobby servos and a floppy 3D-printed spatula from run-enders into recoverable noise.

11Reliability Engineering

A robot that touches a hot pan for ten minutes per run, unattended, accumulates reliability requirements that a benchtop demo never meets. Most of main_runtime.py's bulk is not the stage logic of §3 but the scaffolding around it, and this section collects the pieces that earned their keep during bring-up.

11.1 Failure taxonomy and mitigations

11.2 One YAML to hold it all

Every number in this article — camera indices, thresholds, hold counts, EMA weight, ramp times, waypoint poses, policy repo ids, rollout durations — lives in a single runtime.yaml, normalized through a defaulting pass at load so that a sparse config still yields a fully-populated, type-coerced structure. Two consequences were worth the discipline. Tuning sessions never touched Python: adjusting the pour threshold between batters, or slowing the bottle un-tilt ramp, was a config edit visible in version control. And the config is the experiment record: since the log banner prints the loaded path, any logged run can be reproduced by pointing the runtime at the same file.

11.3 Logs as the primary instrument

The runtime logs to terminal and a timestamped file simultaneously, one line per event, with the trigger payloads serialized inline. Every claim in §12 is read directly off these files — which is the point. A line like

[YOLO] TRIGGER ✅ info={'ratio': 0.1857, 'mask_area': 1057740.0,
                        'hit_count': 30, 'hold_frames': 30, ...}
[GRU]  TRIGGER ✅ info={'pred_label': 'ready', 'conf': 0.5806,
                        'ready_streak': 3, 'buffer': 91, ...}

carries the entire decision context of a transition: not just that the trigger fired but the measured coverage, the debounce state, and the confidence it fired at. During tuning, diffing these payloads across runs replaced guesswork about thresholds with arithmetic; after tuning, they turned the demo videos into auditable experiments. The preview windows serve the same purpose live — the YOLO overlay and the GRU label render the trigger's internal state onto the camera stream (Figure 5) — and both can be disabled with a keypress without touching the run, since drawing is strictly a side effect of, never an input to, the decision path.

12Results — End-to-End Runs

The system's output is a cooked, flipped, plated pancake with no human input between "start" and "done". Video 1 (top of the article) shows the produced demonstration; Video 5 below is the unedited single-take counterpart. For the quantitative record, Table 8 reconstructs a complete instrumented run directly from its log file — every timestamp, trigger value, and step count below is copied from the log, not estimated. (This rehearsal run used shortened policy windows — 5 s flip, 200 s plate, 10 s wait — while exercising the full pipeline; the shipped configuration runs 90/90 with a 30 s wait.)

Across the four fully logged cook waits, the perception layer behaved consistently with its design: the YOLO trigger fired at coverage ratios of 0.129–0.186 (threshold 0.12 plus the climb during the one-second debounce), and the GRU streak path fired after waits of 2 m 18 s to 6 m 36 s at smoothed confidences of 0.40–0.65. The spread in cook-wait duration is the strongest argument for the architecture — a fixed timer tuned to any one of those runs would have burned or undercooked the other three.

▶ Raw footage · single take, no cuts

12.1 Qualitative observations

Three observations from running the system that the numbers do not capture. First, the trigger architecture changes how failures feel: when something goes wrong, the run does not degrade into flailing — it stalls at a stage boundary with the preview window showing exactly which condition is unmet, and the fix (nudge a threshold, re-click the corners) is usually config-level. Second, the pour trigger is the quiet workhorse: batter viscosity varies between mixes and over time as it settles, so identical pour durations produce meaningfully different pancakes — the coverage trigger absorbed this without retuning across sessions. Third, chunked policies read as deliberate: at 3.3 s per committed chunk, the arm's behavior has a legible rhythm — approach, wedge, flip as distinct strokes — which made failures easy to attribute (a bad wedge angle versus a bad flick) and, frankly, makes the robot pleasant to watch.

13Limitations and Future Work

Everything below is a real limitation of the system as built. We list them in decreasing order of how much they would bite a user who tried to reproduce the results.

13.1 The perception models know one kitchen

Both trigger models are trained on this pan, this batter recipe, this camera, and this room's lighting. The GRU's training curves (Figure 7) show unambiguous overfitting — training macro-F1 saturates at 1.0 by epoch four while validation oscillates around 0.75–0.83 — and the honest reading is that 24 training runs cannot constrain an 11.8M-parameter model; the run-level split keeps the measurement honest, not the model general. Val macro-F1 of 0.830 is workable for a debounced trigger but would not survive a different pan coating or a window-lit kitchen without retraining. The same is true, more mildly, for the single-class YOLO. Scaling the cook-run corpus (recording is cheap; annotation is 96 segment boundaries per 32 runs) and adding augmentation across lighting are the obvious first steps.

13.2 The stage sequence is open-loop above the triggers

Perception decides when stages advance, but nothing verifies that a stage succeeded. The flip rollout ends on a timer whether or not the pancake flipped; plating ends whether or not the pancake reached the plate; there is no re-entry, no retry at the stage level, and no final "is there a pancake on the plate" check. Implicit recovery (§10) papers over small failures within a rollout, but a categorical failure — a pancake dropped outside all camera views — sails through to a cheerful clean shutdown. A success classifier per stage (the trigger-camera infrastructure would carry one; the plating stage even re-exposes the pan to it) is the highest-leverage missing component, and would also enable multi-pancake scheduling, which the single-shot stage machine currently cannot express.

13.3 Calibration is a single point of failure

The four warp corners are calibrated once and trusted forever. A bumped tripod silently shifts the warp; the models then see a distorted pan and degrade without any error being raised — during development this was our single most common "mystery failure", always fixed by re-clicking four corners. The trigger thresholds encode similar deployment-specific tuning ($\tau$ went from a default 0.17 to 0.12 for this batter; the 10 s/0.90 drop-trigger constants were hand-fit to observed confidence traces). None of this is self-checking. A startup sanity pass — projecting the corners back onto the live frame and demanding operator confirmation, plus telemetry that flags trigger statistics drifting outside historical bands — would convert silent degradation into loud degradation, which is most of the battle.

13.4 Hardware ceilings

The SO-ARM101 is position-controlled with no force or torque sensing: the spatula wedge relies on the compliance of a 3D-printed tool and the servos' tolerance of brief overload, not on measured contact. The policies compensate by having learned motions that usually avoid jamming, but a firmer pancake or a warped pan surface changes the contact physics in ways the system cannot feel. Kinematically, one 6-DOF arm bounds the achievable workspace and rules out bimanual strategies (stabilizing the pan while flipping). And the whole perception stack plus two ACT policies assume a CUDA GPU adjacent to the griddle — modest hardware by robot learning standards, but not nothing for a kitchen appliance.

13.5 Where this goes next

• Success detection and stage retry — close the loop above the triggers; the architecture already has the camera and the state machine seams for it.
• Learned pouring — the waypoint pour is placement-blind; a policy conditioned on the live coverage mask could center and shape the pour, subsuming the YOLO trigger into a continuous control signal.
• Continuous cooking — pancake $n{+}1$ pours while $n$ cooks; requires the success detector, a scheduling layer over the stage machine, and plating-side collision awareness.
• A single multi-task policy — flip and plate share the spatula-retrieval prefix; a task-conditioned ACT (the runner already plumbs a task string through) or a small VLA could merge the checkpoints and halve the model-management surface.
• Trigger self-calibration — replace the hand-fit drop-trigger constants with quantities estimated from the confidence trace itself (e.g., trigger on a sustained negative slope rather than absolute levels), removing the tightest coupling to this particular checkpoint's calibration.

14Closing Thoughts

Panbot began as a simple question: could a hobby-class arm cook something real, unattended, start to finish? The answer turned out to hinge on almost nothing we expected. The manipulation — the part that looks hard in the video — was largely solved by existing tools: LeRobot's teleoperation pipeline and a well-understood imitation architecture, applied carefully. The part that consumed our engineering attention was time: knowing when there is enough batter, knowing when a pancake is ready, and building trigger logic honest enough to act on models that are only 83% right.

Three lessons generalize beyond pancakes. Match the tool to the sub-problem, not the project. One scripted sequence, two learned policies, and two perception triggers is an unfashionable mixture, but each piece is the cheapest thing that solves its sub-problem, and the mixture is why the system is debuggable. Design for the model you have, not the model you want. The EMA, the streak debounce, and especially the confidence-drop trigger exist because the GRU is imperfect in a structured way; encoding that structure into the consumer bought more reliability than another month of training data would have. Instrument first. The trigger payloads in the logs and the live preview overlays were built before the first full run, and every tuning decision in this article traces back to them — a robot you can interrogate is a robot you can ship, even if only to your own kitchen.

And there is something quietly satisfying about the final artifact: a machine that watches a pancake with two neural networks, waits with the patience only a machine has, and flips at the right moment for reasons you can read off a log file. Everything — runtime, vision pipeline, datasets, checkpoints — is public; we would be delighted to see it cook in someone else's kitchen.

15References

1. T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, “Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware” (ALOHA / Action Chunking with Transformers), Robotics: Science and Systems (RSS), 2023.
2. R. Cadene, S. Alibert, A. Soare, et al., “LeRobot: State-of-the-art Machine Learning for Real-World Robotics in PyTorch,” Hugging Face, github.com/huggingface/lerobot, 2024.
3. TheRobotStudio and Hugging Face, “SO-ARM100 / SO-ARM101: Standard Open Arm,” github.com/TheRobotStudio/SO-ARM100.
4. G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics YOLOv8,” github.com/ultralytics/ultralytics, 2023.
5. A. Kirillov, E. Mintun, N. Ravi, et al., “Segment Anything,” IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
6. K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
7. K. Cho, B. van Merriënboer, C. Gulcehre, et al., “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation,” EMNLP, 2014.