π0.5/so-101 arm policy for making sandwiches (wip)

I recently got an SO-101 arm to play around with. After some time I wanted to make it do something actually useful for me, so I thought I might as well put it to work on making me lunch.

Note: This project is still a work-in-progress, so I'm writing new stuff as new milestones are hit. Last update: 12 sep 2026.

Approach/Setup

Making a sandwich is a relatively long horizon task composed of multiple subtasks, so the way I thought about setting it up is roughly as follows:

1. Bread handling (sliced bread, baguette, etc)
2. Slice handling (charcuterie, cheese)
3. Vegetable handling (leaves, tomatoes, etc)

The main reason for this specific separation is mostly about the differences in physical properties between the different parts of the sandwich. They are all soft, deformable objects that need significantly different grasps, and although within each category there are subcategories, they make for the most natural grouping. They also make for a pretty clear sequence of milestones.

The idea then with these different subtask behaviors is to have an orchestrator LLM harness that decides what command to execute at each step to build the sandwich end to end.

I have a standard leader/follower setup, connected to a Jetson Orin Nano.

Initial data collection

After going grocery shopping for some bread, it was time to get to work!

I broke down bread handling into the following actions:

• pick up bread from a container / anywhere on the table
• place the bread on a plate - i.e. the sandwich's foundation
• close the sandwich

sliced bread is probably the easiest, given it has a uniform shape, and both sides are the same (both crumb) - however, crusty bread requires more complex handling, given at different stages of the process, the crust/crumb have to face a specific way (crust down on foundation, crust up on closing) - and the elongated shape makes closing a more difficult task as it requires a bit more alignment work given the direction it's facing, whereas for sliced bread it's much easier given all 4 sides are equal.

so after some time in the data collection mines, the distribution of this initial dataset ended up being as follows

I varied the table layouts and the containers and surfaces used for bread handling: a basket, a wooden bowl, two different plates, a wooden board, and the table itself.

fun note, at some point in the middle of recording these demos, one of the feeble 3d printed casings of the so-101 arm broke due to a very sudden movement I executed programmatically. in order to fix it I had to unscrew and take apart the top part of the arm (gripper and wrist), use some good old hot silicon glue, and reassemble. little did I know, this completely messed up the calibration settings the arm had due to new homing offsets on the motors. but luckily after recalibrating + recording a few more specific poses, Claude was able to remap the older data to the new calibration settings so everything was the same.

According to the timestamps, I spent about 4 hours recording these demos for about ~90 minutes of actually recorded motion.

Training first π0.5 checkpoint & evals

The training run was a full fine-tune of lerobot/pi05_base for 30k steps with an effective batch size of 32 on two parallel 96 gb rtx pro 6000 blackwell gpus on Runpod. It took about nine hours to complete.

Because my 8gb Jetson can't fit π0.5 into memory, I hosted the model on a serverless modal container/endpoint and executed action chunks locally at ~20 hz, pausing between chunks for new predictions. for the most part I made it execute 25 actions before requesting a fresh prediction, given 50 seemed more likely to get it stuck in the wrong trajectory, with a typical cloud round trip of about 1.3 seconds.

I set up a basic evals matrix to determine the success rate for the different commands across the different bread types, the setup is ~30 trials per task/type combination with a timeout of 60s.

doing the evals took me about ~3hrs for 123 trials.

failure modes

the first thing that stands out is obviously the 0% success rate for closing baguette sandwiches, this is because I intentionally hadn't included training data with baguette closings, as I wanted to check if it would generalize, it didn't generalize very well, it couldn't tell which way the crust should face and the placement would just be very ungraceful or misaligned

apart from this, one of the main issues was a very weird "down-grasp" the model had developed, in which it would try to grab the bread from below, when it was perfectly pickable from above given it was protruding from the basket (on purpose to make it easier to pick up)

this was not something that was taught specifically in the demos, and the weird angle would usually make it fail given there was not a very clean grasping point, but usually it would just hover indecisively near the bottom of the bread.

for placing/closing the main issues were either poor placement/centering of the bread (cases in which the bread was half-outside the plate) which I did not count as successful, or funnily enough, cases in which the model had trouble letting the bread go because the gripper was below the bread and it was trying to go up after release instead of back and then up to let it fall nicely, also sometimes it accidentally dropped the bread mid-way and it failed to recover in time before the timeout, I had recorded in the training data a few recovery examples in case the bread fell, so it seems this behaviour was at least showing up.

behaviors that turned out nicer than expected

placing bread and closing the sandwich in certain circumstances was really clean, it would even tidy up after initial placement to make sure it was a bit more aligned!! and for placing baguette, it learned that the crust had to go down and it would execute the appropriate wrist rolls given its position, which I had expected it would have way more trouble picking up.

it also seemed to manage similar types of bread pretty well, like dark sliced bread, or other crusty types, even though they were not in the training data.

Collecting recovery data, and training second model checkpoint

I wasn't super happy with these success rates, so it was time to see if we could pump up the success rate by fixing some of the main behavioral failure modes that were very prevalent like the down-grasp by using the dAgger method of basically running inference, seeing where the model starts to fail, interrupting and taking control, finishing the trajectory and collecting the data, such that the model knows what to do if it finds itself in that specific position.

I wanted to see if we could achieve this with relatively little data, i.e. at least one or two orders of magnitude less data than what we’d originally trained on.

The ergonomics of collecting recovery data were very very annoying, I had to hold the leader arm with one hand while keeping the other on my laptop to stop it at the specific time and then come back to save the data, this was very slow and just really uncomfortable, so I asked Claude to build me a voice-controlled system in which I could be totally hands-off and just shout "stop!" when the model was starting to mess up, this was really nice to have and made the task just way less mind numbing.

overall this is the recovery data I collected

According to the timestamps, I spent about 1.5 hours collecting only 11 (!) minutes of recorded motion, so collecting recovery demos was about 3x slower than collecting the original demos, mainly due to waiting for the initial inference before taking control.

you'll notice that this covers only some of the tasks/failure modes, the reason was to first get a sense of how well the recovery data approach would work with this amount of data.

also while I was at it I added just 8 baguette closing demos, which we didn't have any for, to see if maybe this was just enough to make it work at least some of the time with just a few examples + behavior transfer from the other tasks.

evals / improvements

in a nutshell: sliced bread pickup improved significantly, baguette only very marginally, and baguette closing still was completely unsuccessful.

the eval setup was the same, 60s timeout, ~30 trials.

here you can see an inference example of the exact same table layout & task on the v1 (left) vs v2 (right) checkpoint of the model, in which the down grasp behavior was improved

also something really interesting I found out is that the improvements in this grasp did not transfer that much to other prompts that involve the same task, for example, in some demos of "close the sandwich" instead of starting with the bread already being held, it is in the basket, and we teach the model to basically grab the nearest slice from wherever it might be, and even when it's in the same circumstances (vertically stacked in the basket) it showed basically no signs of having learned this correction mechanism, even though it did show on the direct "pick up" prompt.

but overall this approach shows a decent path to address a lot of the failures given enough data, weirdly enough baguette seems to resist this, and I can't quite figure out why yet, maybe it's just that it has less data in general but I would have expected it to generalize better.

Testing out-of-distribution generalization to slices

I thought I could maybe have a free lunch (pun intended) by using the latest bread-only checkpoint to handle slices but as I predicted at the beginning it didn't go very well.

on cheese for example, although it sometimes does recognise where it is on the table, approaches it, and does try to grab it, in a rather ok way, it grabs too little and it moves in a way that causes the slice, which is significantly more tear-prone than bread, to break.

however I noticed that if I help it get a better grip by moving the gripper manually so it takes a bigger chunk, it manages to move it where it needs to go, and placement is somewhat fine although it could be vastly improved, but it looks like it's something that could be learned with relatively little data, I think. TBD.