Embedded Files
Introduction
This project consists of building an end-to-end solution of an AI system with the capabilities of assembling an IKEA furniture using UR5e robot and a 2-finger Robotiq gripper. The AI model learns from a video demonstration of the IKEA furniture assembly process and generates an assembly plan. The assembly plan consists of a relationship between furniture parts (table leg, shelf) and the action (pick up, attach, insert) that needs to be performed on them, in a tree structure ending in an assembled furniture. This assembly plan is then fed into the robot and the robot autonomously assembles the product on the workbench.
My key responsibility consisted of development of a Proof of Concept (PoC) of our robotic system that assembles an IKEA furniture based on the assembly plan generated by our AI model. For the PoC, we used the IKEA lack side table as the furniture to be assembled. We used a Universal Robots UR5e robot attached with a Robotiq 2F-140 Gripper and a Microsoft Kinect v1 depth camera.
I first simulated the operation in Gazebo (pickup, align). Tighten could not be simulated in Gazebo due to physics engine constrains for simulation of screwing.
The Kinect camera was used for two roles –
To identify and extract pose, and size information of the items placed on the workbench
To provide depth information via point cloud to MoveIt for collision avoidance
The computer vision module was used only in Gazebo simulation. I did not get the time to implement it on the physical system.
For Motion Planning of the UR5e arm MoveIt was used.
In the above demo the UR5e robot is performing assembly of an IKEA table. The actions include grasp --> align --> tighten.
Issues Debugged
Gazebo Simulation
The robot grasping in the simulation was not working well which made the object held by the gripper to slip. I solved this issue by using Gazebo plugins which were a work-around Gazebo’s physics engine. Finding these plug-ins was difficult and I had to go through a lot of forums to get the solution.
The robotiq gripper’s official ROS package had issues such as lack of color information (gripper appeared white) and gripper joints being broken when spawned in Gazebo. I solved this issue by investigating a plethora of other repositories that contained spin- offs of the official robotiq ROS package and using the one that solved both the issues.
Real Robot
Created ROS driver and joint publisher nodes for the Robotiq gripper since official nodes to do so were not available.
The IKEA table legs are uneven in dimension. When placed upright on two black columns, the edges of the lower face on top of the columns aren't level, causing the legs to tilt slightly. This affects the robot's grasp position, as each leg has a different tilt, requiring individual grasp adjustments for accurate handling.
Papers and Patents
Reddy, G., et al. “Act2Code: Generating Sequential Robotic Instructions from Video Demonstration.” 11th ACM IKDD CODS and 29th COMAD
Upadhyay, A., et al., (2023). “Generating Robotic Assembly Instructions from 3D CAD Models.” U.S. Patent Application No. 18208968, filed June 13, 2023. Patent Pending
Upadhyay, A., et al., (2022). “Robotic Assembly Instruction Generation from a Video.” U.S. Patent Application No. 17/950021, filed September 21, 2022. Patent Pending
Upadhyay, A., et al. “P3DAttnNet: Automated Assembly Plan Generation from Video Demonstration,” ICPR Workshop: Deep Learning for Visual Detection and Recognition (DLVDR2022)
Page updated
Google Sites
Report abuse