In robot model announcements, “generalization” can mean a new object, a new instruction, a new room or a new robot. Those are separate tests with very different difficulty. Figure’s Helix 2.5 release on September 17, 2026 is one of the more detailed attempts to measure one of them, and its definitions are worth borrowing.

This is a closer look at that September evaluation. For the broader set of model releases, see the current model comparison.

What Figure tested

Figure pretrained Helix 2.5 on Index, its dataset of recorded human behavior, then adapted it to three tasks: tidying a living room, folding towels and making a bed. It then took robots into 30 Bay Area homes where it had collected no data. Test objects were set aside before experiments began and checked against the training data. Each task used one fixed model checkpoint across every home.

Success meant finishing the whole task: all 13 to 15 toys in the basket, every towel folded and placed, or pillows and comforter set at the top of the bed. “We give no partial credit,” Figure wrote. Each toy had a one-minute timeout, and any trial needing a human to step in for safety counted as a failure.

The Index-pretrained model succeeded on 56 percent of these trials. The same setup trained from random weights succeeded on 9 percent. Figure says architecture, training, task data and evaluation were held fixed, so the gap measures pretraining alone.

Two caveats come from Figure’s own definitions. “Zero-shot” refers to the homes and objects; the tasks themselves were taught with fine-tuning data collected elsewhere. And 56 percent means the robot did not finish a little under half the time.

How others have defined it

Google DeepMind’s RT-2 (2023) tested “previously unseen objects, backgrounds, and environments” and reported success rising from 32 percent with RT-1 to 62 percent. Figure’s first Helix (February 2025) claimed generalization to “thousands of novel test objects,” with every training item excluded from evaluation. That is object generalization in a familiar setting.

Physical Intelligence’s π0.7 (April 2026) describes another kind, “compositional generalization”: combining skills to do something with no direct training data. Its examples are candid about limits. Asked with a single prompt to load a sweet potato into an air fryer, the robot made “a reasonable attempt” after a few false starts. Step-by-step spoken coaching worked much better. The company also says π0.7 got a two-arm UR5e system to fold laundry consistently even though it had collected no laundry data on that robot.

Questions to ask of any claim

  • What was new: the object, the layout, the building, the task or the robot?
  • Was any data collected in the test location?
  • Was the model fixed before testing began?
  • Does the score count partial completion?
  • How many trials were run, and how many failed?