Skip to main content

CakeRobot - A Gesture Driven Robot That Follows Your Hand Movements Using Arduino, C# and Kinect

Over the last few weekends I’ve spend some time building a simple robot that can be controlled using Kinect. You can see it in action below.
Ever since I read this Cisco paper that mentions Internet Of Things will crate a whooping 14.4 Trillion $ at stake, I revamped my interests in hobby electronics and started hacking with DIY boards like Arduino and Raspberry Pi. That turned out to be fun, and ended up with the robot. This post provides you the general steps, and the github code may help you build your own.
Even if you don’t have a Kinect for the controller, you can easily put together a controller using your phone (Windows Phone/Android/iOS) as we are using blue tooth to communicate with the controller and the robot.


Now, here is a quick start guide to build your own. We’ve an app running in the laptop that is communicating to the Robot via blue tooth in this case that pumps the commands based on input from Kinect, you could easily build a phone UI as well.
And if you already got the idea, here is the code – You may read further to build the hardware part.

1 – Build familiarity

You need to build some familiarity with Arduino and/or Netduino – In this example I’ll be using Arduino.

Arduino

Explore Arduino. The best way is to
Mainly you need to understand the Pins in the Arduino board. You can write simple programs with the Arduino IDE (try the Blink sample to blink an LED in IDE File->Samples). PFB the pins description, from SparkFun website.
    • GND (3): Short for ‘Ground’. There are several GND pins on the Arduino, any of which can be used to ground your circuit.
    • 5V (4) & 3.3V (5): The 5V pin supplies 5 volts of power, and the 3.3V pin supplies 3.3 volts of power. Most of the simple components used with the Arduino run happily off of 5 or 3.3 volts. If you’re not sure, take a look at Spark Fun’s datasheet tutorial then look up the datasheet for your part.
    • Analog (6): The area of pins under the ‘Analog In’ label (A0 through A5 on the UNO) are Analog In pins. These pins can read the signal from an analog sensor (like a temperature sensor) and convert it into a digital value that we can read.
    • Digital (7): Across from the analog pins are the digital pins (0 through 13 on the UNO). These pins can be used for both digital input (like telling if a button is pushed) and digital output (like powering an LED).
    • PWM (8): You may have noticed the tilde (~) next to some of the digital pins (3, 5, 6, 9, 10, and 11 on the UNO). These pins act as normal digital pins, but can also be used for something called Pulse-Width Modulation (PWM). We have a tutorial on PWM, but for now, think of these pins as being able to simulate analog output (like fading an LED in and out).
    • AREF (9): Stands for Analog Reference. Most of the time you can leave this pin alone. It is sometimes used to set an external reference voltage (between 0 and 5 Volts) as the upper limit for the analog input pins.

Kinect

Protocols


2 – Get The Components

Again, you could find an online store to buy these components. Also, you could try the nearby local electronic store and buy some bread boards, jumper wires (get some mail to mail & female to female wires) etc as well. Here is the list of components you need to build CakeRobot.
  • A Chassis – I used Dagu Magician Chassis  – From Spark Fun, Rhydolabz – it comes with two stepper motors that can be controlled by our driver board.
  • A Arduino board with Motor Driver – I used Dagu Mini Motor Driver – bought from Rhydolabz in India. For other countries you need to search and find. Some description about the board can be found here – It also has a special slot to plug in dagu blue tooth shield. You could also use Micro Magician
    • You also need a Micro USB cable to connect your PC to the motor driver to upload code.
  • A Bluetooth Shield – Get the Dagu blue tooth module if you can find it. I’ve purchased a Class 2 Rn 42 blue tooth shield From Rhydolabz
  • Few mini modular bread boards
  • Jumper wires – Get a mixed pack with M/M, F/F, M/F – like this one
  • A Tiny blue tooth dongle for your PC/Laptop, like this one, to communicate with the blue tooth shield in the robot (if you don’t have built in blue tooth)
  • Few sensors if you want to have more fun – I had an ultra sonic distance sensor to avoid collisions in my final version. A better alternative is Ping sensor.
  • A battery pack and battery holder that should supply around 6V to the Mini Motor Driver
  • Other components for your later creativity/exploration
    • LEDs
    • Resistors
    • More Sensors
  • Tools
    • Few star screw drivers
    • Duct tapes/rubber bands (yea, we are prototyping so no soldering as of now)
And I found these reads pretty useful

3 – Programming the components

You need to spend some hours figuring out how to program each of the components.
  • To start with, play with Arduino a bit, connecting LEDs, switches etc. Then, understand a bit about programming the Digital and Analog pins. Play with the examples
  • Try programming the ultrasonic sensor if you’ve one using your Arduino, using serial sockets. If you are using Ping sensor, check out this
  • Try programming the blue tooth module (Code I used for the distance sensor and blue tooth module are in my examples below, but it’ll be cool if you can figure things out yourself).

4 – Put the components together

Assemble the Dagu Magician Chassis, and place/screw/mount the mini motor driver and Bluetooth module on top of the same. Connect the components using jumper wires/plugin as required. A high level schematic below.
image

Here is a low resolution snap of mine, from top.
image

5 – Coding the Arduino Mini Driver

You can explore the full code in the Github repo  - How ever, here are few pointers. According to the Dagu Arduino Mini driver spec, the following digital pins can be used to control the motors
  • D9 is left motor speed
  • D7 is left motor direction
  • D10 is right motor speed
  • D8 is right motor direction

To make a motor move, first we need to set the direction by doing a digitalWrite of HIGH or LOW (for Forward/Reverse) to the direction pin. Next set the motor speed by doing an analogWrite of 0~255 to the speed pin. 0 is stopped and 255 is full throttle.
In the Arduino code, we are initiating communication via blue tooth, to accept commands as strings. For example, speedl 100 will set the left motor speed to 100, and speedr 100 will set the right motor speed to 100. Relevant code below.
        //Setting up the communication with Bluetooth shield over serial 

        Serial.begin(115200);  // rn42 bt

       .......more


      //Read the input In getSerialLine (shortened for brevity)

      while(serialIn != '\n')
 {
  if (!(Serial.available() > 0)) 
  {
   return;
  }

  serialIn = Serial.read();
  if (serialIn!='\n') {
   char a = char(serialIn);
   strReceived += a;
  }
 }


        ....more

        //Process the command (shortened for brevity)
        ....
      
 else if (command=="speedl")
 {
  val=getValue(input,' ',1).toInt();
  analogWrite(leftMotorSpeed,val);
 } 
 else if (command=="speedr")
 {
  val=getValue(input,' ',1).toInt();
  analogWrite(rightMotorSpeed,val);
 } 


        ......more

       
Have a look at the full code of the quick Arduino client here. Then, compile and upload the code to your mini driver board.

6 – Coding the Controller & Kinect

Essentially, what we are doing is just tracking the Skeletal frame, and calculating the distance of your hand from your hip to provide the direction and speed for the motors. Skeletal tracking details here
We are leveraging http://32feet.codeplex.com/ for identifying the Blue tooth shield to send the commands. Please ensure your blue tooth shield is paired with your PC/Laptop/Phone – you can normally do that by clicking the blue tooth icon in system tray in Windows, and clicking Add Device.
       //For each 600 ms, send a new command
       //_btCon is our instance variable for a blue tooth connection, built over the cool 32Feet library
      
       internal void ProcessCommand(Skeleton skeleton)
        {

            var now = DateTime.Now;
            if (now.Subtract(_prevTime).TotalMilliseconds < 600)
                return;

            _prevTime = DateTime.Now;

            Joint handRight = skeleton.Joints[JointType.HandRight];
            Joint handLeft = skeleton.Joints[JointType.HandLeft];
            Joint shoulderRight = skeleton.Joints[JointType.ShoulderRight];
            Joint shoulderLeft = skeleton.Joints[JointType.ShoulderLeft];
            Joint hipLeft = skeleton.Joints[JointType.HipLeft];
            Joint hipRight = skeleton.Joints[JointType.HipRight];
            Joint kneeLeft = skeleton.Joints[JointType.KneeLeft];

            if (handRight.Position.Y < hipRight.Position.Y)
            {
                _btCon.SetSpeed(Motor.Left, 0);
            }

            if (handLeft.Position.Y < hipLeft.Position.Y)
            {
                _btCon.SetSpeed(Motor.Right, 0);
            }

            if (handRight.Position.Y > hipRight.Position.Y)
            {
                var speed = (handRight.Position.Y - hipRight.Position.Y) * 200;
                if (speed > 230) speed = 230;
                _btCon.SetSpeed(Motor.Left, (int)speed);
            }

            if (handLeft.Position.Y > hipLeft.Position.Y)
            {
                var speed = (handLeft.Position.Y - hipLeft.Position.Y) * 200;
                if (speed > 230) speed = 230;
                _btCon.SetSpeed(Motor.Right, (int)speed);
            }

        }
And so, it sets the speed based on your hand movements. Explore the ConnectionHelper and BluetoothConnector classes I wrote.

Conclusion

The code is here in Github. Fork it and play with it, and expand it.

Popular posts from this blog

Top 7 Coding Standards & Guideline Documents For C#/.NET Developers

Some time back, I collated a list of 7 Must Read, Free EBooks for .NET Developers, and a lot of people found it useful. So, I thought about putting together a list of Coding Standard guidelines/checklists for .NET /C# developers as well.As you may already know, it is easy to come up with a document - the key is in implementing these standards in your organization, through methods like internal trainings, Peer Reviews, Check in policies, Automated code review tools etc. You can have a look at FxCop and/or StyleCop for automating the review process to some extent, and can customize the rules based on your requirements.Anyway, here is a list of some good Coding Standard Documents. They are useful not just from a review perspective - going through these documents can definitely help you and me to iron out few hidden glitches we might have in the programming portion of our brain. So, here we go, the listing is not in any specific order.1 – IDesign C# Coding StandardsIDesign C# coding stand…

5 Awesome Learning Resources For Programmers (To help you and your kids to grow the geek neurons)

Happy New Year, this is my first post in 2012. I’ll be sharing few awesome learning resources I’ve bookmarked, and will be pointing out some specific computer/programming related courses I've found interesting from these resources.Also, thought about saving this blog post for my kids as well - instead of investing in these Child education schemes (though they are too small as of today, 2 years and 60 days respectively ). Anyway, personally my new year resolution is to see as much videos from this course collections (assuming I can find some free time in between my regular job && changing my babies diapers).1 – Khan AcademyAs I mentioned some time back, you and your kids are missing some thing huge if you havn’t heard about Khan Academy.  It is an awesome learning resource, especially if you want to re-visit your basics in Math, Science etc.With a library of over 2,600 videos covering everything from arithmetic to physics, finance, and history and 268 practice exercises, th…

Hack Raspberry Pi – How To Build Apps In C#, WinForms and ASP.NET Using Mono In Pi

Recently I was doing a bit of R&D related to finding a viable, low cost platform for client nodes. Obviously, I came across Raspberry Pi, and found the same extremely interesting. Now, the missing piece of the puzzle was how to get going using C# and .NET in the Pi. C# is a great language, and there are a lot of C# developers out there in the wild who are interested in the Pi.In this article, I’ll just document my findings so far, and will explain how develop using C# leveraging Mono in a Raspberry Pi. Also, we’ll see how to write few minimal Windows Forms & ASP.NET applications in the Pie as well.Step 1: What is Raspberry Pi?Raspberry Pi is an ARM/Linux box for just ~ $30. It was introduced with a vision to teach basic computer science in schools. How ever, it got a lot of attention from hackers all around the world, as it is an awesome low cost platform to hack and experiment cool ideas as Pi is almost a full fledged computer.  More About R-Pi From Wikipedia.The Raspberry Pi