Turtlebot Tele Presence Robot

Introduction:

The goal for my project is to use the Turtlebot Robot that the Computer Science department owns and use it as a tele-presence robot. A tele-presence robot is one that is controlled from a distance and conveys the image and voice of the operator to the area that the robot is in. This way, though the operator could be far away, his or her presence is brought to the area of operation and they can participate to some degree.

A tele-presence robot typically has a mobile base and a screen with a webcam. The screen shows the operators likeness, and the webcam allows the operator to see the robot's surroundings. The robot operator could control the robot from a desktop computer located at any spot. In my case the Turtlebot provides the mobile base and the laptop that comes with the Turtlebot provides the viewing screen and the webcam. Typically when operating the Turtlebot the laptop is closed and placed on one of the low shelves that sit on the Turtlebot's base. For my project the laptop is left open and mounted on top of the top shelf.

Procedure:

The project requires that the robot be attached to another computer via the internet so that video information can be passed from the movable base to the operators computer and also back from the operators computer to the movable base. Instructions to the movable base must also be transmitted. These instructions would originate with the operator and would simply tell the base how to move.

The tool that is used to transmit information to and from the base is the Google Hangout. Google Hangouts can be started by anyone with a Google account. Google provides a framework for enhancing the Google Hangout experience called the Google Hangout Extension. Google encourages developers to write their own Extensions and info on writing Extensions can be found easily on line. Extensions can be programmed in Javascript.

Also, the mobile base must be programmable. The Turtlebot robots come with ROS installed on them. ROS stands for 'Robot Operating System'. ROS has lots of information on line regarding how to program their robots using Python or C++. For the purpose of this project I used Python.

I should mention here that there is a requirement that the Hangout Extension communicate with the mobile base. This would be a difficult thing to program but is handled nicely by a ROS library called 'Rosbridge Server'. This group of programs allows you to send info from a javascript enabled browser, through a web socket, to the ROS robot. This functionality is not supported by the Firefox browser, but is supported currently by the Google Chrome browser. For this reason I use the Chrome browser.

Below is a list of objectives, and following them a description for some and where the project stands on them. After completing the first few goals more were added in an attempt to have the robot navigate an area for itself.

• Causing the Mobile Base to Move
• Establishing a Google Hangout Extension
• Establishing Project Resource Management
• Directing the Mobile Base from the Hangout Extension
• Arranging Elements of the Hangout Extension Visually
• Add ROS Functions for Mapping
• Add ROS Function for Navigation of a Map
• Incorporate Mapping and Navigation in the Layout of the Extension
• Allow for Remote Control of Mapping and Navigation
• Write Library Type Software for Storing Maps
• Map Some Areas and Navigate them Autonomously

Description of the items in the list are below.

Causing The Mobile Base to Move:

ROS has a concept called a 'topic' and 'messages' which are passed along topics. It was possible to write messages in Javascript and pass them to the robot along topics. These messages would direct the robot to move. There are two wheels under the base of the Turtlebot, and the Turtlebot is round. To cause the robot to turn 'Angular' messages are passed. To cause the robot to move forward or back, 'Linear' messages are passed. It is possible to direct the robot to turn and move forward at the same time, but the Hangout Extension never does this.

Establishing A Google Hangout Extension:

Google Hangout Extensions are relatively easy to program. You do have to host the extension somewhere, and for this I use Google App Engine. The biggest problem with the Hangout Extension is that it's slower transmitting information than I would like. The Extension is called 'Tele NP' and is set up as a developer extension. This way it is only visible to project members. The App Engine instance is called 'awesometelenp'. This name is obscure, but I needed to find an instance name that was descriptive on the one hand but original on the other hand.

Establishing Project Resource Management:

Google Code is used for the code repository. The name of the project on the Google Code site is 'telenp'.

Directing the Mobile Base From the Hangout Extension:

This was a matter of employing the rosbridge server software with the Hangouts Extension, the Chrome web browser and the ROS operating system on the robot. As I mention above, under good conditions the feedback from the robot to the operator is OK, but under poor conditions the feedback is too slow.

Arranging Elements of the Hangout Extension Visually:

I am not by any stretch of the imagination a visual artist. I have created several screens in the hangout to accommodate different functionality. I have tried to give each function its own separate space.

Add ROS Functions for Mapping and Navigation of a Map:

This is the stage that I'm at now. ROS provides for something called an 'Occupancy Grid' which is a sort of map. ROS also provides mapping software that allows you to make your own maps. ROS also provides navigation software that lets the robot navigate through an already made map. I'm working on making it so that you can use this software all remotely from the operator's computer.

Incorporate Mapping and Navigation in the Layout of the Extension:

This is an ongoing process. One big achievement was to allow the user to see the map when that map was created and stored on the remote computer. The mobile base collects information that goes into a map (when the mapping software is running) but the user really needs to see this map in order to tell the robot where to start and stop when the navigation software is running.

Allow for Remote Control of Mapping and Navigation:

It turns out that mapping software and navigating software does not coexist well. When one is running the other must not be running. A way must be found to start one when it is desired and then stop it when desired, and start the other. This was not easy but has recently been accomplished by employing ROS software for 'RAPP' management. Rapps are 'Robot Applications'. ROS uses these tools to control robots from Android phones, and I reuse that for my application.

Write Library Type Software for Storing Maps:

While some Library type software exists in the ROS system, it is partially incomplete and it is also poorly documented. The ROS system uses 'Mongo' for its map storage in a package called 'map_store'. The on line documentation for 'map_store' is poor, and the software itself is used by only one program that I know of. For this reason it didn't meet my needs very well. I used Python and Mongo to write my own map library software. I felt that the map storage software that I was using was saving improper values, so if I make my own, maybe the values will come out right. That step was successful.

Map Some Areas and Navigate them Autonomously:

So far I have mapped different areas, and recently have navigated them autonomously and from remote control.

Conclusion

The Turtlebot 2 robot is a mobile base type robot that can be used for tele-presence.

The Robot Operating System (R.O.S.) is a good tool for robot development. It can be used for manipulating robots in many ways. It can be used to direct the movements of a mobile base type platform.

Google Hangouts is also a good tool for the transfer of video, audio, and even text. In our example we use it to transfer instructions for the Turtlebot 2 robot.

There were some shortcomings to the project. One was that teleoperation of the robot was slow if the wifi connection was bad or the hangout was slow, which sometimes happens.

Firefox 26 Breaks Google Hangouts Code

Around the 11th of January, 2014, Firefox 26 was released. This release almost went unnoticed by me but I found that it made using Firefox in my Hangouts extension impossible. Luckily Google Chrome still works for me. The problem is subtle. When Firefox is being used the Hangout extension loads as it normally does, but javascript that would change the content of html or the source for img tags doesn't work.

For example, this javascript code would normally work, but doesn't in Firefox 26:

document.getElementById("someImgID").src = "some_img_url.png"

And this code, which is really simple javascript, doesn't work either.

document.getElementById("someDivID").innerHTML = "some text"

I've been trying many things to see if I cannot get the extension so that it works in firefox again. The error in the 'someImgID' code is a 404 NOT FOUND error, so I've been moving the png file around in the different folders of the project to see if I cannot find a place where the image can be read. No luck so far. In the case of the 'someDivID' code, the error is noticed as a 'undefined' block. Truly I'm not sure I even know what this means under these circumstances. I don't even know what to try in order to test this problem. The text in question is hard coded into the javascript. It's not being loaded from anywhere as any kind of file or anything.

As I said, Chrome still works. I fear, though, that chrome will be updated some day soon and my hangout extension will be rendered totally unusable. This is my fear.

App.XML - Hangouts

OVERVIEW
In the sample zip file, in the 'static' folder, there is a file called 'app.xml'. This file holds the html code for your Hangouts Extension. It is, strictly speaking, not an html file. It has a header and footer that are xml code, and then a CDATA section that holds html styled code. There is no 'body' tag and no 'head' tag, but there's html code that the Hangout presents inside the browser window beside the Hangout video screen. It is challenging to edit this file, as it is organized in two types of markup. Also in this file is all the javascript that you want to include in your project.

I wanted to edit my html in a separate file. That way I could load it into a browser locally and find out what my page was going to look like to some degree. This would keep the number of unnecessary deployments to a minimum. I also wanted to edit my javascript in a separate file. I wanted the advantages of using an editor that had context highlighting. What I ended up doing was to edit the files separately, and then concatenating them together into one file using a simple bash script. The bash script needed to be run every time the files were edited, or just before a deployment.

The app is basically contained in the app.xml file. It does reference several other files in the same directory. These are javascript libraries like the one for ros, and graphics files in the png format. The app.xml file, and the path to that file, are referenced in the Hangouts API console. This is the crucial reference that the Hangout uses to display the extension. You need an App Engine account and you need to create an App Engine project before you can enable the Hangout extension or even see the screen where the app.xml file url goes. You can, however, read about this console here.

Details

GIT FILE STRUCTURE
I'm using git and debian 'jessie' for my development environment. The git repository is google code. The repository name is telenp. The organization of the repository is not standard, but it's exceedingly simple. There is one folder called 'telenp'. Inside that folder are the different project-folders that the project might call for. One is the 'awesometelenp' folder. The 'awesometelenp' folder holds the code that is both the Google Hangouts Extension and the App Engine App. This is because the Google App Engine is used in this project to host the Hangouts extension code.

It is anticipated that there will be at least one more folder in the telenp folder. This will be for the ROS node that will pass instructions off from the Hangouts screen to the Turtlebot motors. This node, though not strictly necessary, would make sure that each instruction to the Turtlebot came with sequence numbers. This sub-project should be called something like 'turtlebot_tele_presence' as the robot is meant to act as a telepresence robot.

Another prospective folder might be the 'turtlebot_tele_presence_kinect' folder. This node might be used to filter kinect data. A diagram for the file structure follows.

• telenp/
• awesometelenp/
• turtlebot_tele_presence/
• turtlebot_tele_presence_kinect/

When you check out the telenp project in your home directory using git, you get all the projects. If you are working on a ros computer and you want to compile, for example, the 'turtlebot_tele_presence' node, you would add this path to the ros package path variable - not deleting the old path. This would be 'ROS_PACKAGE_PATH'.

/home/<username>/telenp/turtlebot_tele_presence/

This way you can work on the project, commit changes etc., and have the working node in your ros setup.

APP ENGINE
To edit the code from the Google Hangout Extension, on the same computer change directory to the folder '~/telenp/awesometelenp/'. There you can edit the various files used in the Extension. To upload them to the App Engine instance, type the following:

$ appcfg.py update .

There is a space and a dot at the end of the command line. This works if you're in the 'awesometelenp' folder. You must have the google python package installed on your system. This is found here.

RE-USE
To use the ROS node, you would download the code and compile it and make sure it's on the path (ROS_PACKAGE_PATH) and then compile and use it. There is nothing special about it except that it is for use with the Turtlebot.

To use the other part of the project, especially the contents of the 'awesometelenp' folder, for your own project, you must change the name of the project to something else, and then edit the app.yaml file to reflect the name change. The 'awesometelenp' project is something that I set up with Google App Engine, but you can change the name to something else and register it yourself as your own project.

Start Coding

ROSBRIDGE JAVASCRIPT
One component of rosbridge is a javascript library. The library makes it easier to post topics and messages in the ROS environment. I began to work on the javascript part of the project before I actually had the turtlebot robot at all. I'm not entirely comfortable with javascript, so the process was challenging for me.

BACKGROUND
The first thing I did was to create the appengine project, as suggested by the google links here and here. I downloaded the python google appengine management package. This package has 'appcfg.py' in it, which is an important python file that uploads your appengine project to the web. (You can get that here.) Then I downloaded the sample hangout project, which I would use as a template in for my project. This sample file can be found here.

Before deploying I edited my app.yaml file. Then I began editing the files in the 'static' sub directory. During this process I deployed my extension several times to see that it was working. I began editing the html and javascript files that came with the sample project.

MISTAKE
Here is where I first used the rosbridge javascript library. Without thinking I included that library in the html using 'script' tags. Then I went back and edited my javascript file. I included the file I had written in the html file, with the assumption that it would be able to reference the rosbridge library that had already been included.

Here's the point. I didn't know enough javascript and html to know that I would have to do more than just include the two files in the same html file. What I had to do was first reference the javascript rosbridge library in the top of the file, and then copy and paste the text from my personal javascript library into the html file - between script tags - in a position below the rosbridge reference. This was something of a revelation to me, and only after doing that did my use of the rosbridge library begin to work.

new blog

GENERAL
I wanted an independent study project for Spring 2014 in the Computer Science department at my college. They had a Turtlebot robot outfitted with ROS (Robot Operating System). I proposed that I implement a telepresence robot with that hardware and using the Google+ Hangout platform to transmit audio and video. This robot was supposed to allow a user to navigate a remote area while the operator was physically located some distance away. The project idea was accepted, and this blog has been created to chronicle the progress I make.

The Spring 2014 semester has not started yet, but I've made some small progress on the project.

There is a Google Hangouts API that allows you to develop 'Extensions' for the Hangout platform. I investigated this. The Hangouts API requires that the code for the Extension be hosted on some web server. In the documentation they suggest Google App Engine, so I use App Engine as my serving location.

I also investigated Google Code Project Hosting for my code repository. For the code repository I use Git version control.

The names of these various components follow. The app Extension on Google Hangouts is called 'Tele NP'. As of this writing it is in the 'development' stage, so it is not available to any app user. The App Engine serving component is called 'awesometelenp.appspot.com'. If you navigate to this URL you see a cryptic message on the screen because the server is configured to work with Google Hangouts exclusively. The project hosting is called 'https://code.google.com/p/telenp/' or 'telenp'.

CODING
The Google Hangouts Extensions are written in Javascript, and the App Engine can host code in Python or Java. ROS code is written in C++ or Python, so I'm hoping to use the languages Javascript and Python almost exclusively. I'm hoping not to program in Java or C++.

The Javascript in Hangouts is run locally on the computer end node for that hangout but there is a provision for passing info from one computer to the group in the hangout. Also there's a provision for monitoring information that is broadcast by any one computer in the group to the rest of the group. This is accomplished by including a specific Javascript library from Google in your code.

The Hangout will automatically transmit the video and audio information from one computer to another, so all that is left to do is to transmit instructions from one computer to another regarding how to move the turtlebot. The Google Javascript library can take care of getting the instructions from a remote computer, and all that is left to do is to transmit those instructions to the hardware.

ROS is installed on the turtlebot, and that should be able to control the wheels of the robot. What we need to do then is to get the instructions from the javascript in the Extension to the ROS framework for robot movement. We hope to do this with a ROS library called 'rosbridge'. At this writing I have not tested rosbridge, but it uses web sockets to interact between the Extension (or any javascript) and the ROS operating system.

This is a simplified picture of how I want to proceed with the project. I may want to add my own ROS node to control more finely how ROS uses the Hangout data, but if I do this it will be after testing the simpler situation where no special ROS node is required.