Status Update on Google-driven Web Development Book

After posting a link to a conference on building web apps with Google this morning, I'm reminded that I haven't written here much lately about my book. So here's the latest news for those who might be wondering, "Hey, what ever happened to Deryck's Google book project?"

The short story is that a complete book is just not a possibility for me. While I lack no dedication to my work, this just wasn't the season for me to do a book. My career took off right about the time I signed the book agreement, and now two job changes, several projects, and two years of aggressive development while also trying to write a book have taken their toll on my enthusiasm to see this book completed. I am also at a different place now, with different technical and intellectual interests, so turning back to the topic of Google-driven development is a little like turning back in time.

I have done quite a bit of work on the book in the last couple years, though. To keep this from being lost work, I'm working on turning the book material into 3 different shortcut chapters as part of Prentice Hall's digital short cuts program. I am really excited about this. The material will see some published form, and the work is not a complete loss.

I'm finishing the first of these this week, and expect to move fairly quickly on the other two (on the order of a couple months, I hope.) I'll certainly update here as these shortcuts are released.

Link | Posted by deryck on March 12, 2008 | 0 comments

What happened to the book previews?

For those who may have noticed -- yes, the book previews have stopped appearing here on my site. I'd like to say this will have no effect on my getting this book done, but the reality is that the book's future looks very bleak. I've been discussing this some with my editor who has been both understanding and a complete pleasure to work with. The short point of this is that the book is on hold until I sort through some things.

For those who want more information, I may write a longer post in the future or continue the previews later, depending on how all this works out. I've just got several major projects looming at work between now and May 15, and these projects are taking all my time and attention. I certainly hope the load at work won't sneak up on me like this again, but until I clear out some of these projects, that element in this equation isn't going to change. No use pretending it will.

I have to say, too, that writing a tech book takes much more attention than other activities or even other types of writing I've done. I could write 300 words a day on fiction or for this blog if I really put my mind to it. I find this hard to do with a technical book, though. I'm sure this has something to do with my own writing process, but it seems that writing a page in a tech book requires more from me in terms of ramp up time, writing, and wind down than does writing some code, a short story, or a blog post. This may have something to do with my own paranoia at the quality of the content and its accuracy more than anything else. I have found the process quite paralyzing as compared with other types of wirting.

At any rate, I'll be sure to post here when something definitive happens with the book. Thank you, my imaginary-ideal-blog-reader friend, for understanding!

Link | Posted by deryck on March 9, 2007 | 2 comments

Google-driven Web Development, Chapter 4

Google Maps

For the Google Maps preview portion of our tour of Google-driven Web Development, I thought I'd give you a bit longer section this time. I want, too, to highlight some of the things about the Maps API that can be useful outside of just building mapping applications. The Google Maps script can be used as a library without creating a full-blown mapping application. Geocoding and XMLHttpRequest handling are two uses that quickly spring to mind.


Until very recently, Google Maps did not provide any geocoding service. Developers were forced to call out to other web site APIs, or worse, look up an address by hand, to convert a postal address into a set of geocoded latitude and longitude points. The Maps API now provides this much needed functionality with GClientGeocoder().

  var client = new GClientGeocoder();
  client.getLatLng('1600 Amphitheatre Parkway, Mountain View, CA', 

  function doSomething(point) {
      map.setCenter(point, 3);

Here, we're using a callback function to center our map on the GPoint returned by getLatLng. getLatLng is pretty tied to Google Maps, but for more detailed gecoding data, which could be used independently of Google Maps, use getLocations.

  var client = new GClientGeocoder();
  client.getLocations('1600 Amphitheatre Parkway, Mountain View, CA', 

  function myParser(data) {
      //Do something with the returned data here

getLocations returns a JavaScript object, in the form of JSON, with extended address information. The data returned for our Google headquarters search looks like this.

    "name": "1600 Amphitheatre Parkway, Mountain View, CA, USA",
    "Status": {
      "code": 200,
      "request": "geocode"
    "Placemark": [
        "address": "1600 Amphitheatre Pkwy, Mountain View, CA 94043, USA",
        "AddressDetails": {
          "Country": {
            "CountryNameCode": "US",
            "AdministrativeArea": {
              "AdministrativeAreaName": "CA",
              "SubAdministrativeArea": {
                "SubAdministrativeAreaName": "Santa Clara",
                "Locality": {
                  "LocalityName": "Mountain View",
                  "Thoroughfare": {
                    "ThoroughfareName": "1600 Amphitheatre Pkwy"
                  "PostalCode": {
                    "PostalCodeNumber": "94043"
          "Accuracy": 8
        Point: {
          coordinates: [-122.083739, 37.423021, 0]

This bit of JSON contains a lot of additional information, but it also makes parsing the latitude and longitude points easier when using this feature outside of Google Maps. This could be useful indeed, and hopefully, you're starting to get a sense for how Google's tools and APIS can be used for web development more generally, rather than just in a single application context.

One of the most interesting and useful cases for this is when dealing with XMLHttpRequest objects.

XMLHttpRequest Handling

In earlier chapters, we've been through how to create and use XMLHttpRequest objects. This is not difficult, but requires a few lines of code to get working properly. The maps API can act as an XMLHttpRequest library, abstracting out the functionality into a simple object and methods.

To request a file from your server:

  GDownloadUrl('/files/testpage.html', handleData);

  function handleData(data) {

GDownloadUrl is even smart about the data that is returned. For HTML or text, the data is returned as plain text as sent by the responseText. If the file were XML, the data is XML as returned by the responseXML of XMLHttpRequest. The XML would then have to be parsed, but this can be done by GXml, also provided by the maps API. Consult the docs for complete details, but these methods do provided easy access to XMLHttpRequest even if you're not working on a map application per se.

All you need to do to take advantage of this and the other special features mentioned above is link the maps API script into to your current web page. You don't have to be creating a map to take advantage of all the useful features of the Google Maps API.

That's all the maps goodness we have time for this week. Next preview: this coming Wednesday. We'll look at open source software from Google. Until then...

Link | Posted by deryck on February 20, 2007 | 0 comments

Google-driven Web Development, Chapter 3

Reader and GData

Is it just me, or does AJAX seems to be less about the X in its name these days? Developers are embracing pure JavaScript implementations, favoring JSON as a transport for data rather than XML. Would that be AJAJ? Doesn't sound as catchy, I know. JSON does make a lot of sense for the common case in web development -- passing string representations of data between client and server -- but XML is not dead by any means. Where you need to send meta-data alongside your data, structured documents still rule.

Google Reader is an application that deals in structured documents. It's a feed reader, so obviously, XML is its native tongue. Not only does reader do the parsing for you, but the web app also provides access to itself via a Reader-specific version of Google's GData format. In the spirit of last week's post, I'll again post the intro from this chapter:

Google Reader is Google's RSS and ATOM feed parser. Like Bloglines and Rojo, Reader is meant to make reading blogs and other syndicated content from a web browser easier. Reader also provides a way to share reading lists and feeds with others, making Reader one-part Bloglines, one-part Google Reader, like Gmail, makes use of XMLHttpRequest and has a rich JavaScript-driven user interface. Because RSS and ATOM feeds are forms of XML, Reader has a nicely developed internal XML format and feed fetching system. Once we have taken a brief tour of Reader's features and layout, we'll study it's use of XML and the GDATA XML format and API. We will also looking closely at the JavaScript and CSS used to create Reader's UI and look at how this application rich UI compares to Gmail's.

We'll cover the following topics:

  • Using Google Reader
  • From Login to onLoad
  • Google Data API Protocol
  • View vs. Finder
  • Reusable UI Elements
  • Styling the Chrome

After a tour of Reader for those new to the app, we begin to dive right into the heart of how Reader works.

You'll notice one of the first things that happens is that the user is given a User ID based on his or her Google account. It is important. This User ID is used throughout Reader to match the user's reading lists and other data to the particular Google account. The User ID is used as a unique identifier in the URLs passed to the XMLHttpRequest calls for feeds. We can tell by looking at a section of Reader's web page source, viewed just after the page finished loading, that the User ID is dynamically written by server-side code.

  <script type="text/javascript">
      var _USER_ID;
      var _USER_EMAIL;
      var _COMMAND_TOKEN;
      var _LOGIN_URL = "\
      var _AUDIO_PLAYER_URI = "/reader/ui/2806928869-audio-player.swf";

  <script type="text/javascript">
  _COMMAND_TOKEN_EXPIRES = new Date().getTime() + 1000 * 60 * 20;

  _USER_EMAIL = '';

Notice there are two sets of JavaScript script tags in the source, both of which have the variable _USER_ID. In the first set of tags, _USER_ID is declared but not initialized to any value. In the second block the variable is given a value. I've used X's above to hide any actual user IDs. Notice, too, that the first script tag has the JavaScript indented and formated. The second block runs flush, indicating this section was likely auto-generated after the user's credentials, mine in this case, were authenticated. This _USER_ID is unique to each user, as the variable name implies, and serves as an access point to a user's feed data.

Feed data is accessed entirely via XMLHttpRequest, all while the page is loading. Feed data may also be requested by the user -- by clicking on parts of the page -- after the page has loaded. At any rate, the _USER_ID is an identifier used by the Reader program and not visible to the user. The only way to discover your own User ID is to look at the page source from the Reader web page, or use one of numerous JavaScript and web application debugging tools available online.

The chapter, then, offers a sample of the URLs that are fetched via XMLHttpRequest. A lot can be learned about what Reader is doing just by installing Firebug in Firefox and watching the XMLHttpRequest traffic. From the sample URLs, we look at a bit of the GData that Reader is using.


      <generator uri="">Google   

      <title>Items labeled "prefs" by deryck in Google Reader</title>

      <link rel="self" 



        <id gr:original-id="queue-sorting">,2005:reader/item/
        <link rel="via"  
          title="Items labeled "prefs" by deryck in Google 
        <category term="user/xxxxxxxxxxxxxxxxxxxx/pref/"
          scheme="" label="prefs"/>
        <title type="text">date</title>
        <content type="html">date</content>

        <author gr:unknown-author="true">
          <name>(author unknown)</name>

        <source gr:stream-id="user/xxxxxxxxxxxxxxxxxxxx/pref/
          <title type="text">(title unknown)</title>

I've X'ed out USER_IDs above as well, but you can get the idea of how Reader is using the IDs within it's own internal API. There's a lot more to go into, obviously. :-) There is a whole chapter on the subject. With Gmail, we look at how the program translates JavaScript arrays into UI elements. With Reader, we're dealing with XML elements. JavaScript is still the engine driving the program, but the data format has changed. I know someone will say, "You could represent the same thing in JavaScript object form." This would look something like:

  <script type="text/javascript">
    var feed = {
      'title' : 'tems labeled "prefs" by deryck in Google Reader',
      'continuation' : 'CMHnp_HSi4IC',
      'author' : {
          'name' : 'deryck'
      'entry' : {
          'link' : 'user/xxxxxxxxxxxxxxxxxxxx/pref/',
          'published' : '2005-12-21T02:09:51Z',

Naturally, the data can be similarly expressed in JavaScript, but XML does allow for more information to be passed around. XML can be criticized for being verbose, but in this case, the verbosity does add meaning. Another critical point about all this is that the feeds themseleves are returned in this Reader GData form, so one parser can be used to rule them all -- feeds and application data -- if you'll allow that terrible, geek-abused allusion.

Next post: Google Maps.

(NOTE: I had some issues at work that came up unexpectedly on Friday, so this posting was delayed. I'm uploading both this one and the Maps chapter preview today. See all of the Google book weblog entries.)

Link | Posted by deryck on February 20, 2007 | 0 comments

Google-driven Web Development, Chapter 2


Here in part two of this preview of Google-driven Web Development, we'll look at Gmail.

There's no way to state the importance of Gmail enough. It was the first of it's kind in webmail programs. All other webmail sites are playing catch up now, and really, Gmail helped take AJAX mainstream. Or at least, offered the first large scale application of AJAX techniques as core technology. My intro for this chapter in the book is short, but I hope it suggests some of the same.

Gmail has been unique among web mail services from the moment of its release. The service offers over 2 GB and counting of storage, uses a JavaScript-driven rich user interface, and provides Google's powerful search capabilities for email. Web mail accounts are often thought of as throw away accounts, something for signing into other online systems to prevent Spam collecting in your real email account's Inbox. Gmail is meant to change your impression of web mail. Gmail is meant to be a web mail service that you'll want to use.

Web mail suffers this reputation for good reason. Prior to Gmail, web mail was slow and awkward to use. Login. Wait. Select Inbox. Wait Select mail to read. Wait. And, finally, read your mail. To make a web mail service that people would want to use, the developers of Gmail made extensive use of JavaScript. This browser-side language makes Gmail quicker and more responsive and gets web mail closer to a desktop mail program's capabilities. JavaScript, like web mail, suffers from a bad reputation, mostly due to early security vulnerabilities and the proliferation of poorly implemented JavaScript sites. Gmail stands as a testament to what good, smart JavaScript development can produce. This chapter will detail how JavaScript creates such a unique web mail experience.

In this chapter, we'll cover:

  • The Gmail Difference
  • Framing Up the Page
  • Data Parsing and HTML Generation
  • XMLHttpRequest and AJAX
  • Document Object Model Control

Framing Up The Page

After a brief intro to how Gmail works, hopefully touching on some of the lesser known features for advanced Gmail usage, we get into the internals of the app's architecture. The Gmail UI is all in JavaScript, so it's easy to look under the hood and see how the app works. My intent here is not to reveal any "secrets" of Gmail but rather to just give developers a glimpse into the site's architecture and design and to show how JavaScript-based web development offers possibilities for all the limitations that the language may have (though obviously that last line is a debatable point).

Gmail uses a series of frames, a mixture of both traditional frameset frames and inline iframes, to create a framework -- no pun intended -- around which to manipulate Gmail's user interface. When you think of frames, you may think of those awkward sites with a frame used for navigational links on one side of the browser window and another frame for content on the other side. This is certainly common, but frames can also serve as a proxy between the server and the user. Frames can load data and the JavaScript within one frame can prepare that data for presentation in another, without the user being privy to (or slowed down by) the mechanics of this data loading and manipulation.

In Gmail there are several frames in use at any one time, but the average user is completely unaware of this. The user only sees one of the frames, the one that contains mail client interface. There is one more primary frame hidden away from view, and several iframes hidden in the UI.

Generally, there are two frames — main and js. I think the names say it all. main is the page that you see when using Gmail. js is the frame that loads the JavaScript that makes up the Gmail UI engine.

To build the initial interface, Gmail reads a series of JavaScript arrays that contain the various bits of the page that are displayed, plus some additional meta data concerning the display. These arrays are almost analogous to the html in a static web page. Once a user logs on, the main frame is loaded with a series of these JavaScript arrays that represent the user's Inbox. HTML is created from these arrays.

From there, we spend some time in the book going through the JavaScript arrays, what the possible points in the data arrays could/do represent, and the function that builds the page from these arrays. Everything in Gmail is built from JavaScript, which makes for an interesting application design.

For a sample array, see this view of my Inbox in JavaScript:

  D(["t",["109cf7e500109440",0,1,"6:18 am","
      "» ", "Search Syntaxes",
      "Also, don't forget to check out the special search 
          syntaxes and operators. Gmail gives you …",
      ["^t"], "", "109cf7e500109440", 0, 
      "Mon Mar 6 2006_6:18 AM",0,"",0,0],
      ["109cf7d208f6fb43", 0, 0, "6:16 am",
      "» ",    "Liking Gmail?",
      "Hey, dude. How do you like gmail? kewl, huh?!? Cheers, 
          deryck -- Deryck Hodge http://www …", [],
      "", "109cf7d208f6fb43", 0, "Mon Mar 6 2006_6:16 AM",0,"",0,0],
      ["109cf79a5a9ec48c", 0, 0, "6:13 am",
      » ", 
      "It\'s easy to switch to Gmail!","Did you know that Gmail was voted 
          #2 in PC World's Top 100 products of 2005, right after 
      [], "", "109cf79a5a9ec48c", 0, "Mon Mar 6 2006_6:13 AM", 0, "", 
      0, 0], ["109cf79a5491ae6a", 1, 0, "6:13 am",
          Team","» ",
      "Gmail is different. Here\'s what you need to know.","Auto-
          save, keyboard shortcuts, customized addresses and more. Sound 
          cool? We think so too …",
      [], "", "109cf79a5491ae6a", 0, "Mon Mar 6 2006_6:13 AM", 0, "", 
      0 ,0]

No privacy concerns from me. :-) And I think you can get the idea of how each of the parts of the UI is represented in JavaScript, in a series of D arrays. The Gmail Agent API offers a more indepth look at these arrays, though it's based on an older version of Gmail.


I also try to offer an introduction to AJAX and creating a basic XMLHttpRequest. I also use some obfuscated code from Gmail to illustrate how the simple XMLHttpRequest setup and request cycle is used almost verbatim in Gmail. There are some interesting choices made in Gmail, too, like populating an array with a few initial XMLHttpRequest objects which can be pulled from the array at any time. Gmail also makes nice use of JavaScript closures and passing the current XMLHttpRequest around as a function context. This makes handling multiple XMLHttpRequest objects clean and neat.

Gmail really is interesting and import in more ways the one.

On Friday, I'll preview the next chapter, a look at Google Reader and GData. We'll be putting the X in AJAX next time. Until then...

Link | Posted by deryck on February 12, 2007 | 0 comments

Google-driven Web Development, Chapter 1

Accessible and Useful Design

It's a question I've asked myself numerous times in the last year. Why write a book on Google?

There are no shortage of books on the Search giant. O'Reilly has the excellent Google Hacks and Google Maps Hacks; Syngress has Johnny Long's Google Hacking for Penetration Testers. There are books on AdWords, Analytics, and PageRank, and as many or more on Google's business and history. I particularly like The Search for a tour of Google's early days. But all of these books fall cleanly in one of three categories.

  1. Tips and tricks with Google's sites for developers
  2. Tips and tricks with Google's sites for regular users
  3. How to understand Google's business and technology roots

There are clearly parts of Google that are not covered in any of these topics. One such area is the extensive list of APIs Google makes available for programmers and web developers. In fact, my interest with Google really originated with the release of their SOAP-based search API. About the time I was writing my first command-line search tool, Gmail started taking off, and then Google Maps, and there was a whole world of possibilities in these JavaScript-based sites. Namely, that you could see the code and hack at the sites yourself. And then, the list of APIs began to grow as Google launched it's Open Source program. So just for fun, I started doing a tutorial at LinuxWorld on using these APIs, how Google used JavaScript to create rich browser-based apps, and other related ideas.

This book, obviously, grew out of that tutorial and tries to fill a different space than the other Google books available. As the beginning of Chapter 1 goes:

Google's mission is to organize the world's information and make it universally accessible and useful. Google's efforts at organizing information on the web are well documented. There are many books and websites concerned with Google's server cluster, its search index, and company policy, and indeed, these elements have helped Google create an index of the World Wide Web that is second to none. This is, however, only one-half of Google's mission, and I would argue, only one-half of what attracts users to Google. The other-half is the web sites that make that amazing index accessible and useful to others. In this chapter, we begin by examining how it is that Google makes information accessible and useful. We'll look generally at Google and its history and specifically at the design of itself.

After including an image of an early version of Google's home page, I note:

From the very beginning, Google set itself apart for the simplicity of its own web page. The emphasis was meant to be on the information to which Google provided access. Remember, too, that the system was originally designed to just rank keywords against the index. There was no need for anything else, so why add it to the web page?

Notice that there are no categories, no services, no news headlines, no weather. Nothing but a simple text box and prominent logo. Nothing but search. Figure 1-2 shows an example of the Google homepage today. Although there are links to alternate services and searches, the homepage remains remarkably uncluttered. To begin a search, just type in the box and click “Google Search."

And from there we talk a bit about Google's roots in Academia and some of the philosophies that underpin Google's choices in design and development. While this is not a book on design philosophy or aesthetics, I do argue that Google puts its users first, which is the key to creating useful and usable web sites and applications. Several of the elements -- like a minimal, uncluttered interface -- on continue to reappear in other forms on other sites.

But this is a tech book, right? So we do get into technical matters, too, even as we lay the foundation for the rest of the book. We talk about Google's search operators and special syntaxes and also how Google uses CSS and JavaScript to enhance search, not as technologies for their own sake. For Google's homepage, JavaScript is used minimally.

  function qs(el)
      if (window.RegExp && window.encodeURIComponent) {
          var ue=el.href;
          var qe=encodeURIComponent(document.f.q.value);
          if(ue.indexOf("q=")!=-1) {
              el.href=ue.replace(new RegExp("q=[^&$]*"),"q="+qe);
      return 1;

This script may look quite cryptic, but its job is simple. The function qs() is called when any of the links other than the Google search submit buttons are clicked. The el passed to the function is the element that was clicked. If the RegExp and encodeURIComponent objects are available the query is encoded to be used for a URL and the element that was clicked's href attribute -- the link itself -- is replaced with the encoded query string now added to the link. Certain charters are not allowed in URLs, like spaces and ampersands, so this function makes encoding this within the browser easy. In fact, without JavaScript there is no way to ensure a properly encoded URL from within the browser.

This function also exists to further the goal of accessible and useful design. If you have already entered your query terms into the main Google search text box, and then decide you want to search for images instead of web sites. When you click the images link above the text box, your query is encoded and passed along to the image search page, without you having to re-enter the search later. A useful feature indeed! And one made possible by a simple use of JavaScript.

Contrast this with JavaScript usage in Gmail and Maps, and you'll see two very different approaches to JavaScript usage. Although, both are done from the same prespective, "to organize the world's information and make it universally accessible and useful". In Chapter 2, which I'll preview in a week, we really get under the hood of Google more, looking at the JavaScript UI engine that is Google's successful webmail site.

Link | Posted by deryck on February 5, 2007 | 0 comments

Google-driven Web Dev Book Previews

Since my New Year's resolutions to write more and finish my book, I can safely say I've done a terrible job at each. If I'm going to finish this book — which I *really* want to do — I'm going to have to just force myself to make time for it. Time is hard to come by these days, what with all I've got going on at WPNI, and I'm generally not one to seek out more pressure for myself. So... I'm calling on you dear reader and friend to help me see this task through.

I've been impressed with the way the Django book has been shaping up online, and so, I thought I'd take a cue from Adrian and Jacob. As much as I liked the open nature of the process for their book, I always thought posting a public release schedule was a nice way to put a bit of pressure on themselves to finish it. (Whether this was intended or not, I don't know, but it seems to me a natural side effect of the public posting.) I'm going to follow suit here.

My book is not released under an open source license like the Django book, but I had previously received permission from my editor, Mark Taub, to publish small samples of the book as I worked on it. I'll be publishing here, then, a sample from each chapter with some comments from me on related ideas — the writing process, choices in material, significance of the topics, etc. I also won't be putting the preview samples up in the same chapter order of the book. I'll start with what I have finished, and then work around my other material in an order that allows me time to finish the longer or more complicated chapters in a timely manner.

The schedule will be as follows:

  • Monday, February 5
    • Chapter 1 — Accessible and Useful Design
  • Monday, February 12
    • Chapter 2 — Gmail
  • Friday, February 16
    • Chapter 3 — Reader and GData
  • Monday, February 19
    • Chapter 4 — Google Maps
  • Wednesday, February 21
    • Chapter 9 — Open Source Google
  • Wednesday, February 28
    • Chapter Chapter 8 — Webmaster Tools
  • Monday, March 5
    • Chapter 5 — Office and Productivity Sites
  • Wednesday, March 14
    • Chapter 6 — Personalized Google
  • Friday, March 23
    • Chapter 7 — Server-side Development
  • Friday, March 30
    • Chapter 10 — A Complete Google-driven App

In return for all this free, previewy goodness, all I ask in return is that you all call me out in great public shame and humiliation if I miss a deadline. Call me out in the comments here (or comments for other book related posts.) Call me out in your blogs. And any comments you have on the book, how to make it better, and what you like or don't like would be appreciated to. :-)

Here's to a New Year and finishing this book together! Cheers.

Link | Posted by deryck on February 4, 2007 | 0 comments

Google-Driven Web Development Book

I started work in late December on a book for Addison-Wesley/Prentice Hall based on my Google-Driven Web Development tutorial. The basic idea of the book is to use Google as both a source of inspiration and a tool for development. The first half of the book will be spent examining the design of Google's more popular apps — Gmail, Maps, the personalized homepage, Google Suggest, and others — and the second half will give examples of how to develop your own Google-like application or an application built with Google's available APIs and open source code.

Sound interesting?

Link | Posted by deryck on January 25, 2006 | 2 comments