Anyone who has trained a pet or a new colleague knows the frustration of correcting the same mistake in slightly different forms. A robot planning a path across a cluttered table has the same problem: tell it to avoid the laptop and it will avoid the laptop, then happily swing a full coffee cup over the phone beside it. Researchers at MIT and New York University have built a system, FLEX, that tries to read the intent behind such a correction, and will present it at the Conference on Robot Learning this month.
FLEX works in three steps. In the first, Add, an object the user mentions joins the robot's list of things that matter for the task, which is how earlier methods already worked. The second, Generalise, is the new part: a large language model, here GPT-4o, looks at everything on that list and infers the broader category, so a laptop becomes electronics, a mug becomes fragile things and a stack of paper becomes flammable objects. In the third, Mask, the robot's vision system checks which listed objects are no longer in the scene and drops them, so the list does not grow without end. The whole loop takes about five seconds.
The team tested it in simulation with a robot arm serving coffee, moving a candle and carrying a bowling ball, with volunteers typing commands such as "Don't spill on my laptop". In each case the robot followed the literal instruction and the inferred rule, steering around a phone it had never been warned about.
The idea is to borrow the common sense a model such as the one behind ChatGPT picks up from vast amounts of text, rather than asking people to spell out every rule. It fits a wider push to let people talk to household and assistive robots instead of programming them.
The evidence is thin so far. The results come from three simulated tasks described in a university press release, with no success rates or comparison numbers given there. A language model that over-generalises could make a robot timid, avoiding every object it decides might be precious, and the work does not yet show how often users would need to undo such inferences. Real kitchens, with their lighting and clutter, will test the vision step far harder than a simulator.