1. Prelude

1.1 Purpose

This tutorial shows you how to develop a fully functional example of a web service using Google App Engine. This example manages books and borrowing at a public library, to serve a librarian and several members. The example evolves in 8 steps and grows to over 600 lines of code. At each step, you can copy and paste the code into appropriate files and try that version. You can also see the changes highlighted from the previous version to easily notice how the new features were implemented.

You can complete this tutorial within a few hours. You can modify the code in this example to learn more about Google App Engine using the exercises provided in various sections. You can also use the code here as a starting point for developing other similar services (video checkout, simple online store, employee surveys etc.). You can create simple web services within a few days.

Google App Engine makes it really easy to create and deploy substantial web services. Author and his company have implemented a light version of their novel workgroup collaboration software (kamune) on Google App Engine very quickly. It is called k-lite and has a variety of features from asynchronous and real time communication to workflow management. You can request an account to try k-lite beta to see a real world example of what's possible with Google App Engine.

1.2 Prerequisites

You need to have a PC (Windows XP or Vista, OS/X or LInux) and an Internet connection. You also need a mobile phone to receive the activation code as a text message from Google before you can host your service there. Even without the mobile phone, you can create and test the web service locally on your PC, you just won't be able to upload it to Google. Google App Engine SDK is free. During the beta, you can create up to 10 services. You get a decent amount of resources for your service (CPU, bandwidth, databases and storage) to serve several million page views for certain classes of applications. At this time, pricing has not been announced for resource consumption beyond those limits.

You need to be proficient in some modern programming language such as JavaScript or C++. You need to have a general idea of how web services work and have a working knowledge of HTML, CSS and HTTP client-server interactions. To use Google App Engine, you need to program in Python. However, for this tutorial, you are not expected to know Python. A few tips and pointers are provided here for you to get a jump start in Python to be comfortable enough in getting this example working and in trying variations. You can learn more later.

In essence, with Google App Engine, you create the web service and Google will run it for you for potentially large numbers of users worldwide.

2. Designing the Library Web Service

We'll create a simple web service for a public library in this tutorial. Before we start coding, we'll need to figure out how we want this service to work. We'll walk through a few user scenarios, figure out the user interface and the network interface and interactions. We'll get to coding in the next chapter.

2.1 Scenario

For a typical scenario, let's say a library member Adam walks into the library. He logs into this service from a computer in the library. He browses through various books. He selects a book, picks it up from the bookshelf and takes it to the librarian, Beth. Beth logs into this service from a different computer, records that Adam is checking out this book. (To simplify, we'll assume that each book in the library has a barcode with a unique serial number, that there is no due date for borrowed books and that there is only one copy for each book). A few days later, Adam logs in to the service from his home computer, finds out what books he must return and returns them to the library. Librarian Beth records those books as returned. In between, when new books arrive, Beth adds them to the database through the web service. Beth also handles new memberships.

Exercises

(Exercises purpose: This tutorial is designed for readers to try the code as they follow along. Exercises are provided to experiment with variations in the code or with design at each stage to gain a better understanding of the concepts covered).

• 2.1.a Create scenario for handling multiple copies of a book.
• 2.1.b Create a scenrario for reserving a checked out book.
• 2.1.c Introduce due dates and late fees.

2.2 Features

We'll let users login (with their gmail accounts) and logout. We'll let the librarian manage the books (add books and list books), manage the members (add members and list members) and manage the borrowing (checkout books, handle returns and list books that are borrowed). We'll let members browse the books and list the books they borrowed.

Exercises

• 2.2.a Elaborate on features for automatic reminders.

2.3 User Interface

We'll provide a login screen for all users and let them login with their gmail credentials. We'll predefine a specific gmail account as that of the librarian. We'll provide two menus — one for the librarian and one for the members. Once any user is logged in, we'll always give them a link to logout.

For the librarian menu, we'll provide links to add a book, list books, add a member, list members, checkout a book, and return a book. For the members, we'll provide links to list books and list the member's outstanding books.

Exercises

• 2.3.a Add features for deleting books and member accounts.
• 2.3.b Incorporate ability to search books by title or by author name into this user interface.

2.4 Data Structures

For each book, we'll store its barcode (serial number), title and author name. For each member, we'll store their gmail address and their full name. For each book, if it was borrowed, we'll add a reference to the member that borrowed it.

Exercises

• 2.4.a Extend the data structures to facilitate computing due dates and late fees.
• 2.4.b Figure out a way for the service to record the member's email address but prevent the librarian from knowing it.
• 2.4.c Define data structures to record the recent activity.
• 2.4.d Define data structures to track popular books.

2.5 Logic or Algorithms

If the user visits the site and is not logged in, we'll have the service show a welcome page with a link to login. Once the user logs in, the web service will show them a web page with the librarian's menu or member's menu. When the user selects a menu item, the service will process the request and send the results in a standard template with the menu included.

When the Adam checks out a book in the library, for example, we'll have Beth click on checkout in the user interface. The service will then send a page with the standard menu and the checkout form. When Beth chooses the member and the book in that form, we'll have the server process that request (make the necessary database writes) and send another page with the standard menu and the result.

We'll use this pattern of sequential responses. User clicks on a link, server processes the request, and sends a whole new page with the results and the standard menu. (We could use a different pattern that sends the user requests in the background with AJAX, but that is left for another tutorial another time). For this sequential pattern, you can craft each page with Python or use Django templates. We won't use Django templates for this tutorial, to avoid having to learn yet another thing.

2.6 User Interface Design

We'll use list elements for showing the menu. We'll use CSS to style the elements. We'll show the application title, users email address and a link to logout in the top line. We'll show a list of menu items in the next line. We'll show a heading for what the page is about and show the results or form.

Exercises

• 2.6.a Draw up on paper a few example screens on how you want the page to look like.
• 2.6.b Create 2 different layouts for how you want the page to look like.
• 2.6.c Experiment with a few fonts and colors and choose what you want for this service.
• 2.6.d Design a footer for privacy, copyright and terms.

Exercises

• 2.7.a Do the iterative development and testing on Google App Engine instead of the local server.

3. Getting Started

3.1 Deferring Sign Up

To run your service on Google, you'll need to register by clicking on Sign up link at http://code.google.com. You'll need to provide your mobile phone number and receive the activation code on it before you can upload and host your service on Google. If you have multiple gmail accounts, decide beforehand which of those you want to bind to your mobile phone number. The users of your web service won't know which email account you used to register for Google App Engine, except when you send email from the system account and it goes out from the gmail account you registered with.

To develop applications locally on your desktop you don't need to provide your mobile number. You can create, run and test your web service on the local web server provided by the Google App Engine SDK. In that case, however, only only browsers on your local PC can connect to the service. This typically limits to testing your service with only one user at a time. You can work around this by running multiple browsers (Firefox, Internet Explorer, Google Chrome, Safari etc.) with a different user logged into each of these browsers. This is also helpful to make sure that your service runs well on different browsers.

For this tutorial you do not need to register with Google. You can download the SDK, develop and test locally. You can register and upload your test service to Google later.

3.2 Installation

This tutorial covers the development of the service in the Windows environment using a command prompt. You can run Google App Engine on OS/X or Linux also. OS/X is a little tricky. You need to run Google App Engine Launcher before the directories where app engine is installed become visible from a terminal window. If you are not using the Windows environment, you can easily map the few command prompt actions to appropriate shell commands in the other environments.

Download and install :

1. Python : http://python.org/download Version 2.6.1 will work.
2. Python Imaging Library (PIL) : http://www.pythonware.com/products/pil/ You only need this if you plan to use images in your service. Library 1.1.6 for Python 2.6 will work. We don't use this for this tutorial.
3. Google App Engine SDK : http://code.google.com/appengine/downloads.html

A good development environment will speed up the process. The following items are optional but can be handy. You likely have your favorites already installed.

1. Firefox http://getfirefox.com
2. Install Firebug add-on
3. Internet Explorer 8 beta http://www.microsoft.com/ie8. It has a good development environment.
4. Google Chrome http://www.google.com/chrome
5. Notepad will do, but a text editor like vim http://www.vim.org can be very powerful if you invest the time to learn it.
6. A source code revision control system like Git http://git-scm.com can help you track multiple versions and revert back as you iterate through the development. Git Magic is a good introduction to Git. It can also make it easy if you develop on multiple desktops or if you are working on a web service with a few other people.

3.3 Hello World

3.3.1

C:\> mkdir library
C:\library\> cd library
C:\library\> mkdir html

3.3.2

3.3.3

3.3.4

C:\library\> dev_appserver.py .

If that didn't work, go to the directory where appengine was installed and launch dev_appserver.py with the library directory as the argument.

C:\Program Files\Google\google_appengine\> python dev_appserver.py c:\library

If dev_appserver.py ran but gave you Python syntax errors, please ensure that the lines are aligned properly.

3.3.5

from any browser on your PC. You should see the welcome message.

3.4 How did this work?

a. When you visit your local server, the browser sends a HTTP GET request for /.

b. On the server side, app.yaml tells the appengine to route all requests starting with / to main.py.

c. main.py in turn sends the GET request to the get method MainPage.

d. get method of MainPage class in turn sets content type to text/html and sends the welcome string as the response, which displays the welcome page.

Here are some tips on string handling in Python.

3.6 Echo Test

Let us create an echo handler that sends back what it receives from a form, to get a better understanding of how things come together for this service. We'll create a web page called echo.html with two forms — one that sends the form data as a GET and one that sends the form data as a POST. We'll create a EchoPage handler on the server that writes back what it receives.

3.6.1

There are three key changes to main.py from the previous version:

welcome buffer now has a link to echo test.
1. EchoPage is a new handler.
2. /echo is registered with application

3.6.2

3.6.3

From your browser, visit the home page:

and click on echo test.

You can also launch echo directly with

These should show you the form. Type various strings ie each text box and click send to see how the server sees the data. You can hit back to go back and try another string.

Exercises

• 3.6.a Edit echo.html and change the form to send multiple input strings.
• 3.6.b Change the form to send different input types — radio buttons, checkboxes and drop down lists.
• 3.6.c Include the link to do another test when sending the echo.

When you put files such as echo.html in html sub directory, those files are sent directly to the client when requested (instead of being dynamically generated). You can place images and standard files like privacy and terms in this directory as static html files.

4. Membership

4.1 Authentication

We'll now start creating the framework for the public library web service. When users visit the web page, if they are not logged in, we'll ask them to login. If they are, we'll show them their email address and provide a link to logout.

Bookmark Google App Engine's Getting Started Guide: http://code.google.com/appengine/docs/python/gettingstarted/ In this guide, you can refer to the users service for what we'll use below.

Replace main.py with this

We use users class to find out if the user is logged in and if so, what their email address is. We also use this class to create URLs that go through Google's authentication code for logging in or logging out users and bring them back to the URL we specify.

Exercises

• 4.1.a Create a separate page when the user logs out and thank them for using this service.

4.2 Librarian and Members

If the user's email is test@example.com, (the default email address), we'll assume that the user is the librarian. Otherwise, we'll assume that they are a member of the library. We'll show different pages for the librarian and for the members.

Replace main.py with this file.

4.3 Member Database

We'll create a members database. When a user that is not the librarian logs in, we'll check to see if they are in the database. If not, we'll show them a signup form asking them for their full name. Once they submit it, we'll add them to the member database.

Replace main.py with this

This section covered defining a database, adding a record, creating a form to collect the data, using a new handler to handle this data.

Exercises

• 4.3.a Collect the address for new users and store it in the database.

4.3.1 Testing member creation

Connect to the site and login with say, adam@city.com. You should see the signup form. Fill in Adam Smith. You should then see a member welcome page listing Adam's name.

For debugging, you can view the raw database records by going to the admin console at:

You can use the buttons in Datastore Viewer to see the database entries.

4.3.2 Clearing the database

If you want to start over with your service and clear all the old database entries, you can

1. Use the admin interface above and delete the reccords manually
2. When you start dev_appserver.py you can use the -c option to clera the datastore.
3. You can write custom code to clear all the records.

4.4 Member Database Listing

We'll provide a way for the librarian to list all the members. We'll create a menu for the librarian with a command to list members. We'll create a new handler to handle this request. In that handler, we'll fetch all the member records and display only the names.

We'll use a null query that matches all the records. You can refer to GQL Reference to find out how to formulate specific queries.

Replace main.py with this:

5. Books

5.1 Book Database Creation

Similar to member database creation, we'll create a database for books. We'll create a menu item for the librarian to add a book. We'll create a form to collect the information about the book and add it to the database. We'll use the same AddBookPage handler to send the form (with GET) and process the data (with POST).

Replace main.py with

5.2 Listing Books

We'll add list books to both librarian and member menus. When we get this request, we'll walk through all the books and list them, just as we listed the members.

Here is the updated main.py to do that.

We restructured member_page to show men and include the argument as result. We also added some styling for the table display.

Exercises

• 5.2.a Sort the book list by title during GQL query

6. Checkout

6.1 Book Checkout

We'll create a menu item for the librarian to checkout a book to a member, by selecting the member and the book from drop down lists. If the book was already checked out, we won't include it in the dropdown list. We'll add a field to the database to point the book to the member that checked it out.

Here is the updated main.py:

There are several key points to note in this version. You can use ReferenceProperty field to link different database entities, as we did by pointing borrwer in Book.

We added the borrower field to Book definition after a few records for Book were already created in previous versions. When we add additional fields to a database midway like this, the records that were created earlier won't have the new field, but the later ones do. You could clear the datastore for the test server and start over again to make sure that all the records have all the fields. Or you could write custom code to migrate earlier entries to new entries. Or you could check to see if the field is None in Python, which covers both the cases of the field not existing or existing but pointing to None. This is what we do here.

In situations like checking out a book, we need to avoid a race condition. When hosted on Google, multiple librarians could be logged into this service from different computers, even with the same email. We need to make sure that two librarians don't checkout the book to two different members at the exact same time. We won't run into that situation in this tutorial because we assume that the member physically carries the book over to the librarian, so only one member could be holding the book at any time. However, if the members were reserving a book online, this could happen. You can look into transactions to avoid such timing issues.

Another thing to note is that when we send the checkout form, we use the barcode and member email, as unique keys, to send back to the server, even though what we display is the title and member's full name. This ensures that even if two books have the same title or two members have the same name, we choose the right ones.

Exercises

• 6.1.a When all the books are checked out, send an error message, instead of the form with empty dropdown boxes.
• 6.1.b Let the librarian type the barcode and member's email in the form, instead of using the drop down boxes.
• 6.1.c Use a member ID, so we don't have to send the email address of members to the librarian, even if it is hidden in the form.

6.2 List Books Due

We'll add links to show the books due. For the Librarian, we'll show all books due from all the members. For the member, we'll only show the books they are due to return.

Here is the updated main.py to do that.

To find out all the books due from a given member, we can use the pointer to retrieve all the book records that point to a given member record. This comes in really handy to connect and filter through various database entities. If you use this technique for other problems, please make sure that you don't have two database entities each pointing to the other one. This is a common database problem. When such a need arises, you'll need to create a third database entity that can point to both these entities.

Exercises

• 6.2.a Add a new field for recording when the book is checked out. Show that time when listing books that are due.
• 6.2.b Create a separate database for authors and point the books to the right author record. Provide commands to list all the books available from a given author.

6.3 Returning Books

To return books, we'll provide a link for the librarian, next to each book that is due. When the librarian clicks that link, we'll record that book as returned, by setting the borrower field for that book to none.

Here is the updated main.py to do that.

Here we reuse books duehandler for the librarian to craft a link that invokes the return handler.

Exercises

• 6.3.a Provide an alternative way to return books by typing in the barcode.
• 6.3.b List books due by a given member and choose one of them as returned.

7. Conclusion

7.1 Uploading your service

We developed a working version of a web service iteratively progressively adding some of the features of Google App Engine on the local server provided by Google App Engine SDK. You can choose a unique URL by signing in to Google App Engine and creating a new service. You can then edit app.yaml to change the application name to what you chose, and upload the service with

you'll be asked for your gmail user name and password that you registered with for the SDK. If this didn't work, as before with dev_appserver.py, run this from the directory app engine was installed in and specify the directory of your service instead of the current directory.

7.2 Scaling

The current UI does not deliberately scale when there are thousands of books and members — listing books and members or checking books out or returning them may fail, if not be inefficient. Addressing this is not hard though and will be left as an exercise:

Exercises

• 7.2.a Whenever a list of items is over 20, say, show that list in multiple pages, with 20 items per page, and with links to go to next or previous pages as appropriate.
• 7.2.b Avoid showing lists altogether and let users search for item. Show the search results 20 per page, as above.
• 7.2.c Let the user specify the barcode and member ID for checking books out and returing them, instead of using drop down boxes or links.

7.3 Performance

As you add features, if a particular web request takes too long, Google suspends that request. This can be particularly annoying when a nicely working service suddenly stops working when you add a small new feature that pushes it over the edge. You can check the administrative logs by logging in to Google App Engine again, as in the previous section, to see which of your requests are taking too long and redesign your service to do the computation in smaller chunks and to use caching. Memcache's inc can also be handy for building your synchronization framework, since you don't know which request goes to which server.

Exercises

• 7.3.a Find out which web request leads to the most amoutn of database hits.
• 7.3.b Use memcache to map member's email to member record.

7.4 Next Steps

You can use this example as a starting point to create other services. You can modify this to create a video rental service, or an employee survey service, or a wedding registry.

You can walk through the online documentation for Google App Engine and try the other features. As you become familiar with the various features and techniques for using them, you can apply that knowledge to create and deploy creative solutions for real world problems for businesses or consumers, in an expedient manner.

7.5 Followup

Please send your feedback, questions, comments or requests to Raja Abburi at tutorial@navaraga.com.

Here is the gtalk chatback badge for the author: