Enhancing the LLM-Based Robot Manipulation Through Human-Robot Collaboration

Large language models (LLMs) are gaining popularity in robotics, but LLM-based robots are limited to simple, repetitive motions due to the poor integration between language models, robots, and the environment. This paper proposes a novel approach to enhance the performance of LLM-based autonomous manipulation through human-robot collaboration (HRC). A prompted GPT-4 language model decomposes high-level language commands into sequences of motions executable by the robot, and a YOLO-based perception algorithm provides visual cues to the LLM for planning feasible motions within the specific environment. Furthermore, an HRC method combining teleoperation and dynamic movement primitives (DMP) allows the LLM-based robot to learn from human guidance.

IEEE Robotics and Automation Letters, 9(8), 6904–6911