Coders Packet

Gesture Controlled Robot-Using Python and CopeliaSim Edu

By Rithish Kumar E

Hand Gesture recognition is made using Python(OpenCV) which acts as a client and Robot in CoppeliaSim acts as the server.

In this Project, the CopeliaSim Edu environment interacts with the Python program using remote dll, the robot acts as the server and the script acts as a client. Once the connection is established, OpenCV is used to get the webcam footage, the first 30 frames are sets to read the average of the background and after which placing our hand in the region of interest, our hand gets subtracted from the background using the threshold method.

Now, the hand has been segmented, next, the number of fingers is counted in the frame using hand gesture recognition as proposed by Malima Calculate the convex hull of the hand segment, the convex hull will have at least 4 most outward points, on the top, bottom, left, and right.

In theory, the center of the hand is halfway between the top and bottom halfway between left and right, Now find the center of the palm using these extremes points in the convex hull, Using the palm’s center, construct a circle with the maximum Euclidean Distance (between the palm’s center and the extreme points) as radius, Perform bitwise AND operation between the thresholded hand image (frame) and the circular ROI (mask). This reveals the finger slices, which could further be used to calculate the number of fingers shown.

Given the number of fingers, the script commands the robot to Move Forward on detecting one finger, Backward on detecting four, left on detecting two, and Right on detecting three, or halt otherwise.

The threshold is mapped for every frame and hand is detected. The instruction is simultaneously sent to the robot.

Requirements: Python IDE with OpenCV installed and CopeliaSim Edu software for robot simulation.

Start the simulation in CopeliaSim Environment and Run the script in Python IDE.

Download Complete Code


No comments yet