This new system not only designs Lego models that match text descriptions (prompts), but also ensures they can be built brick by brick in the real world, whether by hand or with robotic assistance, as reported by Ars Technica.
How does LegoGPT work?
The tool leverages large language model (LLM) technology, similar to that used by chatbots like ChatGPT. Similar to OpenAI's AI solution that predicts the next word, LegoGPT anticipates the next brick to be assembled to generate step-by-step building plans. The team used Meta's Llama-3.2-1B-Instruct model to achieve this, training LegoGPT on a dataset of over 47,000 Lego structures, including 28,000 distinct 3D components.
The real feat of LegoGPT lies in its adherence to the laws of physics through a method called "physics-aware rollback." If the system detects that a piece could compromise the structure's stability, it identifies the unstable brick, removes it and all the bricks placed after it, and then explores a new approach. This approach allows the AI model to claim a stability rate of over 98%.
In addition to strength, LegoGPT also takes aesthetics into account, offering texture and color options. To validate their virtual creations, the researchers put LegoGPT to the test in the real world using dual-arm robots equipped with force sensors. These robots assembled the models according to the AI's instructions, and human testers also built some of the designs by hand. This confirmed that the models are not only stable, but also human-like.
LegoGPT still has limitations
While promising, the current version of LegoGPT is limited to a 20x20x20 build space and uses only eight standard brick types from the 21 object categories it has been trained on. However, the Carnegie Mellon team plans to expand the brick library (including slopes and tiles) and enrich its training database.
The good news for all brick enthusiasts is that LegoGPT is freely available. The researchers (Ava Pun, Kangle Deng, Ruixuan Liu, Deva Ramanan, Changliu Liu, and Jun-Yan Zhu) shared their dataset, code, and models on their project website and on GitHub.
0 Comments