Telenp Turtlebot Project

A video:

https://www.youtube.com/watch?v=3RrPJSoMG9I

Using the Tele NP Hangout with the Turtlebot

Appendix C:

Using the Tele NP Hangout with the Turtlebot

This section is meant to describe the actual operation of the robot with the Hangout extension. We start by describing the startup procedure. Then we give tips for actually using the extension html interface. As before we outline the information that we are about to present.

1. Start The Robot and Launch the ROS Package (and Google Chrome)
2. Start The Controlling Computer (and Google Chrome)
3. Launch Google Hangouts and The Hangout Extension on Both Computers
4. Establish for the Hangout Extensions which computer is the Turtlebot
5. Test Teleop
6. Start Gmapping
7. Save Map
8. Start Navigation (Load A Map)
9. Tell the Navigation Software Where the Turtlebot Is
10. Tell the Navigation Software Where the Desired Goal Is
11. Specify Additional Goals

Though there are several parts to the Hangout Extension and they do not all look the same, the entire Extension is confined to a band along the right side of the Hangout screen. On the central portion of the screen is the video from the robot. Along the top of that gray band on the right is a set of three tabs. The tabs are 'controls' , 'setup', and 'map'. You will be presented with the 'controls' tab first.

1. Start The Robot and Launch the ROS Package (and Google Chrome)
   Plug the kinect device in to the Turtlebot's laptop computer. Remember that the kinect must be plugged into the adapter which itself plugs in to the Turtlebot for power and the laptop for data. Plug the Turtlebot itself into the laptop via the USB cable. Turn on the Turtlebot and the laptop, and make sure you are booting into the Ubuntu 12.04 LTS/ROS partition. The password for the laptop is usually 'turtlebot'. When the Turtlebot is on there will be a bright green light on its control panel. Check network connectivity on the laptop. You need access to the internet in general. 
    
   On the turtlebot computer start the Google Chrome Browser. Also start a terminal window. Type the following in the terminal window:
    
   $ roslaunch tele_presence app_manager.launch
   
   
2. Start The Controlling Computer (and Google Chrome)
   Start the computer that you want to use as the controller. Your computer must have Google Chrome and Google Hangouts installed. Start Google Chrome at this time.
   
   
3. Launch Google Hangouts and The Hangout Extension on Both Computers
   At this time you need to use the credentials that you set up for the tele presence robot. The two users that you choose to use at this time must be established as developers for the App Engine project. For the turtlebot computer log in with one user, and for the controller computer log in with the other user. Start Hangouts and the TeleNP extension on both computers. The TeleNP extension can be found in the left bar that shows up on Hangouts when you hover over that area with the mouse. You may have to choose the 'more extensions' button and then go to the 'developers' section to find your own project. You should then see the video connection between the computers now easily.
   
   
4. Establish for the Hangout Extensions which computer is the Turtlebot
   On the screen of the laptop connected to the Tutlebot you should see the direction buttons for the extension, and also some options. One of the options, under the heading 'Start' is a checkbox that is labled 'Turtlebot Node'. On the Turtlebot laptop, click the 'Turtlebot Node' checkbox. The screen on both computers should change slightly. The direction pad will disappear from the screen on the Turtlebot Computer. At the same time the 'Turtlebot Node' checkbox will disappear from the controller computer's screen. You should hear a series of tones from the Turtlebot itself. There will be a checkbox on the controller computer's screen that says 'Controller Node'. On the controller computer, click this checkbox. 
    
5. Test Teleop
   On the controller computer, click the arrow pictures on the screen. The turtlebot should move.
   
   
6. Start Gmapping
   On the controller computer, click on the tab marked 'MAP'. You will see a panel of buttons arranged vertically. The top option, under the heading 'Do This:' will be the 'Make Test Map' button. Click this button. The sidebar will change to the 'Make Test Map' screen. There are two main buttons here, one marked 'NEW TEST MAP' and one marked 'NEW G MAP'. Click the 'NEW G MAP' button. At this time the turtlebot will reconfigure itself for g-mapping. This reconfiguration takes some time. The turtlebot will emit a tone, seem to pause, and then emit a second tone. After the second tone there will be a wait, and then a yellow box will appear on the controlling computer's screen that will tell you that the reconfiguration was completed successfully. Click the 'Done' button. You can now go to the tabs at the top of the sidebar and click 'CONTROLS'. From the 'Controls' tab you can direct the turtlebot around the area to be mapped. The robot will use its laser to record information about solid objects in its path. You cannot see the map as it is being made, so try to be thorough. You cannot go back and add to a map that is already made. If a map is incomplete you must start g-mapping over again and make an entirely new map.
   
   
7. Save Map
   Go back to the 'MAP' tab, at the top of the sidebar, when you are done. You should see a series of buttons again. (If you are not seeing the vertical button panel, look for a 'Done' or 'Return' button.) The 'Save Map' button, second from the top, should be ready for you to click. DO NOT click the 'STOP/SWITCH SERVICE' button at the bottom of the row as you are not done g-mapping. When you click the 'Save Map' button you will be given a screen for naming the map in the library. Use simple names without spaces or special characters. Click the 'SAVE' button. Wait again for the yellow box that tells you that the process completed without error. Once you see the yellow box, you can click 'Done'. Your map should now be part of the turtlebot map library.
   
   
8. Start Navigation (Load A Map)
   Back at the 'MAP' sidebar the 'Load Map' function is disabled. Click 'STOP/SWITCH SERVICE', and wait while the turtlebot reconfigures from g-mapping to teleop. The switch takes about a minute. When it is done you will notice that the 'Load Map' button is enabled. (There is no yellow box to tell you that the operation is finished.) Click the 'Load Map' button. The sidebar will change to show the Load Map options. The Load Map screen is used to launch the 'navigation' configuration for the turtlebot. There are many buttons on this screen. There is a horizontal line on the screen which you should note the position of. Try to use the buttons below the horizontal line. Next to the text that reads 'Load this file:' is a dropdown box that will allow you to select your map. After selecting your map, click the 'LOAD' button. At this time the turtlebot will reconfigure itself for the navigation function. This configuration takes some time. You should hear a series of tones, and a pause, and then another series of tones. After another short wait a yellow box with a confirmation message appears in the sidebar. When you see the yellow box you know you can proceed. Do not click DONE at this time. Click 'RUN NAV TOOLS'. This will take you to the screen where you can specify starting and ending coordinates for your navigation. After clicking the 'RUN NAV TOOLS' button the nav tools are visible on the 'View Current Map' page. You are automatically taken to that page now.
   
   
9. Tell the Navigation Software Where the Turtlebot Is
   Click 'Refresh Current Map' at the top of the sidebar. Wait for a period of time while the map is loaded. When the map is ready an alert box appears on the screen with the xy location of the origin for your map. The numbers you see should not be zeros, but they are not user modifiable. Click OK on the alert and you should see the map appear in the window of the sidebar. Scroll until you see the main part of the map in the center of the window. Mentally figure out where the turtlebot robot is and what direction it is facing. The orientation of the map is determined by what direction the turtlebot was facing when the g-mapping process was started. The map cannot be rotated in the window. Click the 'START COORDS' button. Hover the cursor over the map, and specifically over the spot where the turtlebot is to start from. This should be it's location in the room. Click the mouse button and a small blue square is placed on the map. You can move it by hovering over another part of the map, and clicking again. Now you should click on the button that allows you to specify the starting angle. With every click the angle of the button changes. Twelve positions are available. Choose the one that best reflects the turtlebot's orientation on the map. Click the 'SEND START' button.
   
   
10. Tell the Navigation Software Where the Desired Goal Is
    Click the 'STOP COORDS' button. Hover the cursor over the map in the spot you want the turtlebot to end up in. Click the mouse button. A red square is placed on the screen. Choose an angle using the button on the right side of the sidebar. This is the angle you want the turtlebot to be facing when it stops moving. If you click 'SEND GOAL' now the turtlebot will attempt to navigate from the starting location to the stop location. The process will be abandoned if it cannot figure a path, or if the start location as defined by the map does not match what it sees around it.
    
    
11. Specify Additional Goals
    You can repeat step 10 as often as you like. Disregard the blue square after the robot has reached its destination. Place the red square in the next destination, and click the 'SEND GOAL' button.

Installing The Hangout Extension

Appendix A:

Installing The Hangout Extension

You can install the Google Hangout Extension used in this project by following the instructions below.

1. Establish an account with Google for their App Engine service.
2. Make a new App Engine Instance.
3. Download and install the App Engine tools.
4. Download the Google Hangouts project from the Google Code site.
5. Configure the Hangout code
6. Upload the Hangout Extension.
7. Edit the list of approved users
8. Start the Hangout Extension.
9. Use the Hangout.

You could probably install the Google Hangout extension using a Windows or Macintosh Operating system, but for this explanation we will assume that you are using Ubuntu Linux, 12.04 LTS.

1. You might want to set up a Google user just for working with the Hangout Extension. Setting up a Google account is easy. Go to any google sign in page and click on the 'create account' button.

2. See the following link for information about getting your own account registered for Google App Engine. https://console.developers.google.com/start/appengine Google has tons of howto's about their products. When you are presented with options about what programming language you want to use, select python.
   
   At this stage you may be required to choose a unique name for your google App Engine instance. This name is important. For development the name 'awesometelenp' was chosen. Yours must be different. At this stage you must also create your own Hangout Extension. The following link shows how to publish your app: https://developers.google.com/+/hangouts/running . The following link goes through the process of creating a project. https://developers.google.com/console/help/#creatingdeletingprojects . After this you want to enable the specific API for Google Hangouts Extensions. ‎

3. The url for the python tools for working with App Engine follow: https://commondatastorage.googleapis.com/appengine-sdks/featured/google_appengine_1.9.1.zip .
   
   Follow instructions on this page for installing the appengine software on your computer : https://developers.google.com/appengine/downloads#Google_App_Engine_SDK_for_Python

4. As of this writing the code for the Hangout Extension and also the ROS software is all located on a google code site called 'Telenp'. The url for the google code site is: https://code.google.com/p/telenp/ . Instructions for using 'git' is beyond the scope of this document, but the code is stored in a git repository and you will need to use git to download it. The repository is divided into two sections, one called 'awesometelenp' -- which is the hangout extension -- and one called 'turtlebot_tele_presence' -- which is the ROS package. When you download the software you get both at once. I recommend that people create a folder in their home directory called 'workspace' and then download the telenp code into that folder. Then the hangout code could be found at '~/workspace/telenp/awesometelenp/' and the ROS code could be found at '~/workspace/telenp/turtlebot_tele_presence/' .

5. At '~/workspace/telenp/awesometelenp/' there is a file called 'app.yaml'. On line 2 is the spot for you to put your unique app engine instance name. Replace the word 'awesometelenp' with your name. Save and close the file. In the actual html and java code of the project, replace the word 'awesometelenp' with your name. It is used in several image names and url's. A simple find/replace operation in your favorite text editor should work. The four files you should be concerned with are called 'app.js' , 'app-map.js' , 'app.html' , and 'app-map.html' .

6. After completing step 3 above you should be able to upload the App Engine app to the App Engine instance. Open a terminal and perform the command '

$ cd ~/workspace/telenp/awesometelenp

$ appcfg.py update .

(Notice the period after the word 'update'). This will update your instance. You can also perform the commands from another directory, as is outlined in the google site at this address: https://developers.google.com/appengine/docs/python/tools/uploadinganapp#Python_Uploading_the_app

7. This document assumes that you are using the developers version of the Hangouts Extension. Using this setup, if you go to your app engine dashboard at: https://appengine.google.com/ and click on the link representing your app engine instance, you will be taken to a page that has 'Permissions' as one of the many links on the left side of the page. Here you can edit the email addresses that are used to access your project. These email addresses will be able to access the Hangout Extension in the developer section of your Google Hangouts. You must have at least two addresses in your list, one for the controller computer, and one for the turtlebot computer.

8. Log in to Google as one of the email addresses that you set up in part 7. Open Google Plus and start a hangout. Invite the users from the list. You need to invite the user that is attached to the Turtlebot computer. Open the hangout extension that you've renamed by hovering over the left side of the screen and choosing 'Add Apps' from the bar that appears. Choose 'Developers' and then choose yours from the list presented. Do this on the turtlebot computer too.

9. Here we describe the last part of the installation of the Hangout Extension. We assume that you have also installed the ROS operating system and the tele-presence software developed for this project. Log in to Google as one of the email addresses from section 7 using Google Chrome as your web browser. You must have the Hangout Plugin installed. See the link above for installing the plugin: https://www.google.com/tools/dlpage/hangoutplugin . Go to the Google Plus page. Start a hangout and invite others to join you. You must invite someone from the list you made in section 7 to join you. One of these emails should be on the turtlebot computer attached to the turtlebot itself. At this point you should start the ROS software on the turtlebot computer by typing 'roslaunch tele_presence app_manager.launch' in a terminal. Finally, with the ROS software running on the turtlebot and the hangout extensions running on both the turtlebot computer and the controlling computer, the last step of the setup can be performed. Click the 'Turtlebot Node' checkbox on the screen on the turtlebot computer. Go to the controller computer and click on the 'Controller Node' checkbox. Now you should be able to direct the Turtlebot using the arrow buttons on the controller computer's screen.

Installing the ROS Nodes

Appendix B

Installing the ROS Nodes:

You can install the custom ROS Nodes used in this project by following the instructions below.

1. Install ROS On Your Computer.
2. Setup ROS In Your '.bashrc' File.
3. Locate the On-Line Code Resource.
4. Download the ROS and Google Sources.
5. Identify the ROS Workspace.
6. Prepare the Workspace For Inclusion In ROS.
7. Build the ROS Workspace.

Though programming Google Hangouts can be done on Macintosh or Windows computers, ROS should be used on a Ubuntu Linux computer. In this document we assume that you are using Ubuntu Linux 12.04 LTS.

1. A good resource for installation of ROS can be found at the ROS Wiki, at the following URL: http://wiki.ros.org/hydro/Installation/Ubuntu . You should follow these instructions first before trying to install the ROS nodes from this project. For this project we used ROS Hydro. There is also some special software to install for this project. Execute the following code from a terminal:

$ sudo apt-get install ros-hydro-rosbridge-server
$ sudo apt-get install python-pip
$ sudo pip install pymongo

2. Open a terminal window. Add the following line to your '.bashrc' file:

source /opt/ros/hydro/setup.bash

Next exit the terminal you are in and open a new terminal.

3. Code for this project is all located at the following URL: https://code.google.com/p/telenp/ . The code is stored in a git repository.

4. At this time the authors use a directory in their home directory called 'workspace' after the fashion of the Eclipse style workspace. In this directory is placed the git repository. Open a terminal and create an Eclipse style workspace by typing:

$ mkdir workspace

$ cd workspace

and finally to download all tele_np sources, type:

$ git clone https://david.c.liebman@code.google.com/p/telenp/

5. The repository is divided into two sections, one called 'awesometelenp' -- which is the hangout extension -- and one called 'turtlebot_tele_presence' -- which is the ROS package. When you download the software you get both at once.

6. You have to let ROS know that you're including your own material in your own development workspace. This is accomplished by adding another line to your '.bashrc' file. If you have been following closely, this is the line:

source ~/workspace/telenp/turtlebot_tele_presence/devel/setup.bash

Another line you should include is:

export ROS_PACKAGE_PATH=~/workspace/telenp/turtlebot_tele_presence/:$ROS_PACKAGE_PATH;

That makes three lines that are important in the '.bashrc' file. Close the terminal you are in and open another.

7. Because my home folder is named differently from yours, you need to delete the 'build' and 'devel' folders from the 'turtlebot_tele_presence' folder before you build. The following commands should allow you to build the ROS software:
   
   

$ cd ~/workspace/telenp/turtlebot_tele_presence/src/

$ catkin_init_workspace

Then type:

$ cd ..

$ rm -fr build devel

Then type:

$ catkin_make

After that you should exit your terminal and open another. On subsequent builds you should only need to execute the 'rm -fr build devel' and 'catkin_make' .

Turtlebot Tele Presence Robot

Introduction:

The goal for my project is to use the Turtlebot Robot that the Computer Science department owns and use it as a tele-presence robot. A tele-presence robot is one that is controlled from a distance and conveys the image and voice of the operator to the area that the robot is in. This way, though the operator could be far away, his or her presence is brought to the area of operation and they can participate to some degree.

A tele-presence robot typically has a mobile base and a screen with a webcam. The screen shows the operators likeness, and the webcam allows the operator to see the robot's surroundings. The robot operator could control the robot from a desktop computer located at any spot. In my case the Turtlebot provides the mobile base and the laptop that comes with the Turtlebot provides the viewing screen and the webcam. Typically when operating the Turtlebot the laptop is closed and placed on one of the low shelves that sit on the Turtlebot's base. For my project the laptop is left open and mounted on top of the top shelf.

Procedure:

The project requires that the robot be attached to another computer via the internet so that video information can be passed from the movable base to the operators computer and also back from the operators computer to the movable base. Instructions to the movable base must also be transmitted. These instructions would originate with the operator and would simply tell the base how to move.

The tool that is used to transmit information to and from the base is the Google Hangout. Google Hangouts can be started by anyone with a Google account. Google provides a framework for enhancing the Google Hangout experience called the Google Hangout Extension. Google encourages developers to write their own Extensions and info on writing Extensions can be found easily on line. Extensions can be programmed in Javascript.

Also, the mobile base must be programmable. The Turtlebot robots come with ROS installed on them. ROS stands for 'Robot Operating System'. ROS has lots of information on line regarding how to program their robots using Python or C++. For the purpose of this project I used Python.

I should mention here that there is a requirement that the Hangout Extension communicate with the mobile base. This would be a difficult thing to program but is handled nicely by a ROS library called 'Rosbridge Server'. This group of programs allows you to send info from a javascript enabled browser, through a web socket, to the ROS robot. This functionality is not supported by the Firefox browser, but is supported currently by the Google Chrome browser. For this reason I use the Chrome browser.

Below is a list of objectives, and following them a description for some and where the project stands on them. After completing the first few goals more were added in an attempt to have the robot navigate an area for itself.

• Causing the Mobile Base to Move
• Establishing a Google Hangout Extension
• Establishing Project Resource Management
• Directing the Mobile Base from the Hangout Extension
• Arranging Elements of the Hangout Extension Visually
• Add ROS Functions for Mapping
• Add ROS Function for Navigation of a Map
• Incorporate Mapping and Navigation in the Layout of the Extension
• Allow for Remote Control of Mapping and Navigation
• Write Library Type Software for Storing Maps
• Map Some Areas and Navigate them Autonomously

Description of the items in the list are below.

Causing The Mobile Base to Move:

ROS has a concept called a 'topic' and 'messages' which are passed along topics. It was possible to write messages in Javascript and pass them to the robot along topics. These messages would direct the robot to move. There are two wheels under the base of the Turtlebot, and the Turtlebot is round. To cause the robot to turn 'Angular' messages are passed. To cause the robot to move forward or back, 'Linear' messages are passed. It is possible to direct the robot to turn and move forward at the same time, but the Hangout Extension never does this.

Establishing A Google Hangout Extension:

Google Hangout Extensions are relatively easy to program. You do have to host the extension somewhere, and for this I use Google App Engine. The biggest problem with the Hangout Extension is that it's slower transmitting information than I would like. The Extension is called 'Tele NP' and is set up as a developer extension. This way it is only visible to project members. The App Engine instance is called 'awesometelenp'. This name is obscure, but I needed to find an instance name that was descriptive on the one hand but original on the other hand.

Establishing Project Resource Management:

Google Code is used for the code repository. The name of the project on the Google Code site is 'telenp'.

Directing the Mobile Base From the Hangout Extension:

This was a matter of employing the rosbridge server software with the Hangouts Extension, the Chrome web browser and the ROS operating system on the robot. As I mention above, under good conditions the feedback from the robot to the operator is OK, but under poor conditions the feedback is too slow.

Arranging Elements of the Hangout Extension Visually:

I am not by any stretch of the imagination a visual artist. I have created several screens in the hangout to accommodate different functionality. I have tried to give each function its own separate space.

Add ROS Functions for Mapping and Navigation of a Map:

This is the stage that I'm at now. ROS provides for something called an 'Occupancy Grid' which is a sort of map. ROS also provides mapping software that allows you to make your own maps. ROS also provides navigation software that lets the robot navigate through an already made map. I'm working on making it so that you can use this software all remotely from the operator's computer.

Incorporate Mapping and Navigation in the Layout of the Extension:

This is an ongoing process. One big achievement was to allow the user to see the map when that map was created and stored on the remote computer. The mobile base collects information that goes into a map (when the mapping software is running) but the user really needs to see this map in order to tell the robot where to start and stop when the navigation software is running.

Allow for Remote Control of Mapping and Navigation:

It turns out that mapping software and navigating software does not coexist well. When one is running the other must not be running. A way must be found to start one when it is desired and then stop it when desired, and start the other. This was not easy but has recently been accomplished by employing ROS software for 'RAPP' management. Rapps are 'Robot Applications'. ROS uses these tools to control robots from Android phones, and I reuse that for my application.

Write Library Type Software for Storing Maps:

While some Library type software exists in the ROS system, it is partially incomplete and it is also poorly documented. The ROS system uses 'Mongo' for its map storage in a package called 'map_store'. The on line documentation for 'map_store' is poor, and the software itself is used by only one program that I know of. For this reason it didn't meet my needs very well. I used Python and Mongo to write my own map library software. I felt that the map storage software that I was using was saving improper values, so if I make my own, maybe the values will come out right. That step was successful.

Map Some Areas and Navigate them Autonomously:

So far I have mapped different areas, and recently have navigated them autonomously and from remote control.

Conclusion

The Turtlebot 2 robot is a mobile base type robot that can be used for tele-presence.

The Robot Operating System (R.O.S.) is a good tool for robot development. It can be used for manipulating robots in many ways. It can be used to direct the movements of a mobile base type platform.

Google Hangouts is also a good tool for the transfer of video, audio, and even text. In our example we use it to transfer instructions for the Turtlebot 2 robot.

There were some shortcomings to the project. One was that teleoperation of the robot was slow if the wifi connection was bad or the hangout was slow, which sometimes happens.

Firefox 26 Breaks Google Hangouts Code

Around the 11th of January, 2014, Firefox 26 was released. This release almost went unnoticed by me but I found that it made using Firefox in my Hangouts extension impossible. Luckily Google Chrome still works for me. The problem is subtle. When Firefox is being used the Hangout extension loads as it normally does, but javascript that would change the content of html or the source for img tags doesn't work.

For example, this javascript code would normally work, but doesn't in Firefox 26:

document.getElementById("someImgID").src = "some_img_url.png"

And this code, which is really simple javascript, doesn't work either.

document.getElementById("someDivID").innerHTML = "some text"

I've been trying many things to see if I cannot get the extension so that it works in firefox again. The error in the 'someImgID' code is a 404 NOT FOUND error, so I've been moving the png file around in the different folders of the project to see if I cannot find a place where the image can be read. No luck so far. In the case of the 'someDivID' code, the error is noticed as a 'undefined' block. Truly I'm not sure I even know what this means under these circumstances. I don't even know what to try in order to test this problem. The text in question is hard coded into the javascript. It's not being loaded from anywhere as any kind of file or anything.

As I said, Chrome still works. I fear, though, that chrome will be updated some day soon and my hangout extension will be rendered totally unusable. This is my fear.

App.XML - Hangouts

OVERVIEW
In the sample zip file, in the 'static' folder, there is a file called 'app.xml'. This file holds the html code for your Hangouts Extension. It is, strictly speaking, not an html file. It has a header and footer that are xml code, and then a CDATA section that holds html styled code. There is no 'body' tag and no 'head' tag, but there's html code that the Hangout presents inside the browser window beside the Hangout video screen. It is challenging to edit this file, as it is organized in two types of markup. Also in this file is all the javascript that you want to include in your project.

I wanted to edit my html in a separate file. That way I could load it into a browser locally and find out what my page was going to look like to some degree. This would keep the number of unnecessary deployments to a minimum. I also wanted to edit my javascript in a separate file. I wanted the advantages of using an editor that had context highlighting. What I ended up doing was to edit the files separately, and then concatenating them together into one file using a simple bash script. The bash script needed to be run every time the files were edited, or just before a deployment.

The app is basically contained in the app.xml file. It does reference several other files in the same directory. These are javascript libraries like the one for ros, and graphics files in the png format. The app.xml file, and the path to that file, are referenced in the Hangouts API console. This is the crucial reference that the Hangout uses to display the extension. You need an App Engine account and you need to create an App Engine project before you can enable the Hangout extension or even see the screen where the app.xml file url goes. You can, however, read about this console here.