February archive
February 22, 2007
Okay, so I'm a bit behind as usual on the book previews. The chapter preview due for yesterday
will now be bumped to Monday, February 26. I was in Arlington working from the office so far this
week which got me off my schedule a bit. Today I'm catching up on email and trying to clear some
hanging issues at work. Tomorrow I'm celebrating my birthday with my wife when I get off work. Yes,
it's 35 for me on Saturday! The mid-thirties! Which is nice actually.
So look for the next preview on Monday and please forgive the delay.
February 20, 2007
Google Maps
For the Google Maps preview portion of our tour of
Google-driven Web Development, I thought I'd give you
a bit longer section this time. I want, too, to highlight some of the things
about the Maps API that can be useful outside of just building mapping applications.
The Google Maps script can be used as a library without creating a full-blown
mapping application. Geocoding and XMLHttpRequest handling are two uses that
quickly spring to mind.
Geocoding
Until very recently, Google Maps did not provide any geocoding service. Developers were forced to call out to other web site APIs, or worse, look up an address by hand, to convert a postal address into a set of geocoded latitude and longitude points. The Maps API now provides this much needed functionality with GClientGeocoder().
var client = new GClientGeocoder();
client.getLatLng('1600 Amphitheatre Parkway, Mountain View, CA',
doSomething);
function doSomething(point) {
map.setCenter(point, 3);
}
Here, we're using a callback function to center our map on the GPoint returned by getLatLng. getLatLng is pretty tied to Google Maps, but for more detailed gecoding data, which could be used independently of Google Maps, use getLocations.
var client = new GClientGeocoder();
client.getLocations('1600 Amphitheatre Parkway, Mountain View, CA',
myParser);
function myParser(data) {
//Do something with the returned data here
}
getLocations returns a JavaScript object, in the form of JSON, with extended address information. The data returned for our Google headquarters search looks like this.
{
"name": "1600 Amphitheatre Parkway, Mountain View, CA, USA",
"Status": {
"code": 200,
"request": "geocode"
},
"Placemark": [
{
"address": "1600 Amphitheatre Pkwy, Mountain View, CA 94043, USA",
"AddressDetails": {
"Country": {
"CountryNameCode": "US",
"AdministrativeArea": {
"AdministrativeAreaName": "CA",
"SubAdministrativeArea": {
"SubAdministrativeAreaName": "Santa Clara",
"Locality": {
"LocalityName": "Mountain View",
"Thoroughfare": {
"ThoroughfareName": "1600 Amphitheatre Pkwy"
},
"PostalCode": {
"PostalCodeNumber": "94043"
}
}
}
}
},
"Accuracy": 8
},
Point: {
coordinates: [-122.083739, 37.423021, 0]
}
}
]
}
This bit of JSON contains a lot of additional information, but it also makes parsing the latitude and longitude points easier when using this feature outside of Google Maps. This could be useful indeed, and hopefully, you're starting to get a sense for how Google's tools and APIS can be used for web development more generally, rather than just in a single application context.
One of the most interesting and useful cases for this is when dealing with XMLHttpRequest objects.
XMLHttpRequest Handling
In earlier chapters, we've been through how to create and use XMLHttpRequest objects. This is not difficult, but requires a few lines of code to get working properly. The maps API can act as an XMLHttpRequest library, abstracting out the functionality into a simple object and methods.
To request a file from your server:
GDownloadUrl('/files/testpage.html', handleData);
function handleData(data) {
alert(data);
}
GDownloadUrl is even smart about the data that is returned. For HTML or text, the data is returned as plain text as sent by the responseText. If the file were XML, the data is XML as returned by the responseXML of XMLHttpRequest. The XML would then have to be parsed, but this can be done by GXml, also provided by the maps API. Consult the docs for complete details, but these methods do provided easy access to XMLHttpRequest even if you're not working on a map application per se.
All you need to do to take advantage of this and the other special features mentioned above is link the maps API script into to your current web page. You don't have to be creating a map to take advantage of all the useful features of the Google Maps API.
That's all the maps goodness we have time for this week. Next preview: this coming Wednesday. We'll look at open source software from Google. Until then...
February 20, 2007
Reader and GData
Is it just me, or does AJAX seems to be less about the X in its name these
days? Developers are embracing pure JavaScript implementations, favoring JSON
as a transport for data rather than XML. Would that be AJAJ? Doesn't sound as
catchy, I know. JSON does make a lot of sense for the common case in web
development -- passing string representations of data between client and server --
but XML is not dead by any means. Where you need to send meta-data alongside
your data, structured documents still rule.
Google Reader is an application that deals in structured documents. It's a feed
reader, so obviously, XML is its native tongue. Not only does reader do the
parsing for you, but the web app also provides access to itself via a Reader-specific
version of Google's GData format. In the spirit of last week's post, I'll again
post the intro from this chapter:
Google Reader is Google's RSS and ATOM feed parser. Like Bloglines and Rojo, Reader is meant to make reading blogs and other syndicated content from a web browser easier. Reader also provides a way to share reading lists and feeds with others, making Reader one-part Bloglines, one-part del.icio.us. Google Reader, like Gmail, makes use of XMLHttpRequest and has a rich JavaScript-driven user interface. Because RSS and ATOM feeds are forms of XML, Reader has a nicely developed internal XML format and feed fetching system. Once we have taken a brief tour of Reader's features and layout, we'll study it's use of XML and the GDATA XML format and API. We will also looking closely at the JavaScript and CSS used to create Reader's UI and look at how this application rich UI compares to Gmail's.
We'll cover the following topics:
- Using Google Reader
- From Login to onLoad
- Google Data API Protocol
- View vs. Finder
- Reusable UI Elements
- Styling the Chrome
After a tour of Reader for those new to the app, we begin to dive right into the
heart of how Reader works.
You'll notice one of the first things that happens is that the user is given a User ID based on his or her Google account. It is important. This User ID is used throughout Reader to match the user's reading lists and other data to the particular Google account. The User ID is used as a unique identifier in the URLs passed to the XMLHttpRequest calls for feeds. We can tell by looking at a section of Reader's web page source, viewed just after the page finished loading, that the User ID is dynamically written by server-side code.
<script type="text/javascript">
<!--
var _USER_ID;
var _USER_EMAIL;
var _COMMAND_TOKEN;
var _COMMAND_TOKEN_EXPIRES;
var _LOGIN_URL = "https://www.google.com/accounts/ServiceLogin?nui\
u003d1&service\u003dreader&continue\u003dhttp%3A%2F%2F
www.google.com%2Freader%2F";
var _AUDIO_PLAYER_URI = "/reader/ui/2806928869-audio-player.swf";
//-->
</script>
<script type="text/javascript">
<!--
_USER_ID = 'XXXXXXXXXXXXXXXXXXXX';
_COMMAND_TOKEN = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';
_COMMAND_TOKEN_EXPIRES = new Date().getTime() + 1000 * 60 * 20;
_USER_EMAIL = 'deryckh@gmail.com';
//-->
</script>
Notice there are two sets of JavaScript script tags in the source, both of which have the variable _USER_ID. In the first set of tags, _USER_ID is declared but not initialized to any value. In the second block the variable is given a value. I've used X's above to hide any actual user IDs. Notice, too, that the first script tag has the JavaScript indented and formated. The second block runs flush, indicating this section was likely auto-generated after the user's credentials, mine in this case, were authenticated. This _USER_ID is unique to each user, as the variable name implies, and serves as an access point to a user's feed data.
Feed data is accessed entirely via XMLHttpRequest, all while the page is loading. Feed data may also be requested by the user -- by clicking on parts of the page -- after the page has loaded. At any rate, the _USER_ID is an identifier used by the Reader program and not visible to the user. The only way to discover your own User ID is to look at the page source from the Reader web page, or use one of numerous JavaScript and web application debugging tools available online.
The chapter, then, offers a sample of the URLs that are fetched via XMLHttpRequest.
A lot can be learned about what Reader is doing just by installing
Firebug in Firefox and watching the XMLHttpRequest traffic.
From the sample URLs, we look at a bit of the GData that Reader is using.
<feed
xmlns="http://www.w3.org/2005/Atom"
xmlns:gr="http://www.google.com/schemas/reader/atom/"
xmlns:media="http://video.yahoo.com/mrss">
<generator uri="http://www.google.com/reader">Google
Reader</generator>
<id>tag:google.com,2005:reader/user/xxxxxxxxxxxxxxxxxxxx/pref/
com.google/prefs</id>
<title>Items labeled "prefs" by deryck in Google Reader</title>
<gr:continuation>CMHnp_HSi4IC</gr:continuation>
<link rel="self"
href="http://www.google.com/reader/atom/user/xxxxxxxxxxxxxxxxxxxx/
pref/com.google/prefs?complete=true&ck=1148848167297"/>
<author>
<name>deryck</name>
</author>
<updated>2005-12-21T02:09:51Z</updated>
<entry>
<id gr:original-id="queue-sorting">tag:google.com,2005:reader/item/
d4a7953aa59779fc</id>
<link rel="via"
href="user/xxxxxxxxxxxxxxxxxxxx/pref/com.google/prefs"
title="Items labeled "prefs" by deryck in Google
Reader"/>
<category term="user/xxxxxxxxxxxxxxxxxxxx/pref/com.google/prefs"
scheme="http://www.google.com/reader/" label="prefs"/>
<title type="text">date</title>
<published>2005-12-21T02:09:51Z</published>
<updated>2005-12-21T02:09:51Z</updated>
<content type="html">date</content>
<author gr:unknown-author="true">
<name>(author unknown)</name>
</author>
<source gr:stream-id="user/xxxxxxxxxxxxxxxxxxxx/pref/com.google/
prefs">
<id>tag:google.com,2005:reader/user/xxxxxxxxxxxxxxxxxxxx/pref/
com.google/prefs</id>
<title type="text">(title unknown)</title>
</source>
</entry>
</feed>
I've X'ed out USER_IDs above as well, but you can get the idea of how Reader is
using the IDs within it's own internal API. There's a lot more to go into,
obviously. :-) There is a whole chapter on the subject. With Gmail, we look
at how the program translates JavaScript arrays into UI elements. With Reader,
we're dealing with XML elements. JavaScript is still the engine driving the
program, but the data format has changed. I know someone will say, "You could
represent the same thing in JavaScript object form." This would look
something like:
<script type="text/javascript">
var feed = {
'title' : 'tems labeled "prefs" by deryck in Google Reader',
'continuation' : 'CMHnp_HSi4IC',
'author' : {
'name' : 'deryck'
},
'entry' : {
'link' : 'user/xxxxxxxxxxxxxxxxxxxx/pref/com.google/prefs',
'published' : '2005-12-21T02:09:51Z',
}
};
</script>
Naturally, the data can be similarly expressed in JavaScript,
but XML does allow for more information to be passed around. XML can be criticized
for being verbose, but in this case, the verbosity does add meaning. Another
critical point about all this is that the feeds themseleves are returned in this
Reader GData form, so one parser can be used to rule them all -- feeds and
application data -- if you'll allow that terrible, geek-abused allusion.
Next post: Google Maps.
(NOTE: I had some issues at work that came up unexpectedly on
Friday, so this posting was delayed. I'm uploading both this one and the Maps
chapter preview today. See all of the Google
book weblog entries.)
February 15, 2007
While moaning about terrible weather sites and my desire for a Google-like
type my zip and get instant results page, Aly Hurt
mentioned that Google was doing weather now. I didn't know this. Can someone confirm
is this new or something I've just overlooked in the past?
Try something like Arlington, VA weather or 36853 weather
for an example. Aly, Levi, and I all had varying
results while trying this out.
Kind of cool, though.
February 12, 2007
Gmail
Here in part two of this preview of Google-driven
Web Development, we'll look at Gmail.
There's no way to state the importance of Gmail enough. It was the first of
it's kind in webmail programs. All other webmail sites are playing catch up now,
and really, Gmail helped take AJAX mainstream. Or at least, offered the first
large scale application of AJAX techniques as core technology. My intro for this
chapter in the book is short, but I hope it suggests some of the same.
Gmail has been unique among web mail services from the moment of its release. The service offers over 2 GB and counting of storage, uses a JavaScript-driven rich user interface, and provides Google's powerful search capabilities for email. Web mail accounts are often thought of as throw away accounts, something for signing into other online systems to prevent Spam collecting in your real email account's Inbox. Gmail is meant to change your impression of web mail. Gmail is meant to be a web mail service that you'll want to use.
Web mail suffers this reputation for good reason. Prior to Gmail, web mail was slow and awkward to use. Login. Wait. Select Inbox. Wait Select mail to read. Wait. And, finally, read your mail. To make a web mail service that people would want to use, the developers of Gmail made extensive use of JavaScript. This browser-side language makes Gmail quicker and more responsive and gets web mail closer to a desktop mail program's capabilities. JavaScript, like web mail, suffers from a bad reputation, mostly due to early security vulnerabilities and the proliferation of poorly implemented JavaScript sites. Gmail stands as a testament to what good, smart JavaScript development can produce. This chapter will detail how JavaScript creates such a unique web mail experience.
In this chapter, we'll cover:
- The Gmail Difference
- Framing Up the Page
- Data Parsing and HTML Generation
- XMLHttpRequest and AJAX
- Document Object Model Control
Framing Up The Page
After a brief intro to how Gmail works, hopefully touching on some of the
lesser known features for advanced Gmail usage, we get into the internals of
the app's architecture. The Gmail UI is all in JavaScript, so it's easy to
look under the hood and see how the app works. My intent here is not to
reveal any "secrets" of Gmail but rather to just give developers a glimpse
into the site's architecture and design and to show how JavaScript-based
web development offers possibilities for all the limitations that the language
may have (though obviously that last line is a debatable point).
Gmail uses a series of frames, a mixture of both traditional frameset frames and inline iframes, to create a framework -- no pun intended -- around which to manipulate Gmail's user interface. When you think of frames, you may think of those awkward sites with a frame used for navigational links on one side of the browser window and another frame for content on the other side. This is certainly common, but frames can also serve as a proxy between the server and the user. Frames can load data and the JavaScript within one frame can prepare that data for presentation in another, without the user being privy to (or slowed down by) the mechanics of this data loading and manipulation.
In Gmail there are several frames in use at any one time, but the average user is completely unaware of this. The user only sees one of the frames, the one that contains mail client interface. There is one more primary frame hidden away from view, and several iframes hidden in the UI.
Generally, there are two frames — main and js. I
think the names say it all. main is the page that you see when using
Gmail. js is the frame that loads the JavaScript that makes up the
Gmail UI engine.
To build the initial interface, Gmail reads a series of JavaScript arrays that contain the various bits of the page that are displayed, plus some additional meta data concerning the display. These arrays are almost analogous to the html in a static web page. Once a user logs on, the main frame is loaded with a series of these JavaScript arrays that represent the user's Inbox. HTML is created from these arrays.
From there, we spend some time in the book going through the JavaScript arrays, what
the possible points in the data arrays could/do represent, and the function
that builds the page from these arrays. Everything in Gmail is built from
JavaScript, which makes for an interesting application design.
For a sample array, see this view of my Inbox in JavaScript:
D(["t",["109cf7e500109440",0,1,"6:18 am","
DeryckHodge",
"» ", "Search Syntaxes",
"Also, don't forget to check out the special search
syntaxes and operators. Gmail gives you …",
["^t"], "", "109cf7e500109440", 0,
"Mon Mar 6 2006_6:18 AM",0,"",0,0],
["109cf7d208f6fb43", 0, 0, "6:16 am",
"Deryck
Hodge",
"» ", "Liking Gmail?",
"Hey, dude. How do you like gmail? kewl, huh?!? Cheers,
deryck -- Deryck Hodge http://www …", [],
"", "109cf7d208f6fb43", 0, "Mon Mar 6 2006_6:16 AM",0,"",0,0],
["109cf79a5a9ec48c", 0, 0, "6:13 am",
"Gmail
Team","
» ",
"It\'s easy to switch to Gmail!","Did you know that Gmail was voted
#2 in PC World's Top 100 products of 2005, right after
…",
[], "", "109cf79a5a9ec48c", 0, "Mon Mar 6 2006_6:13 AM", 0, "",
0, 0], ["109cf79a5491ae6a", 1, 0, "6:13 am",
"Gmail
Team","» ",
"Gmail is different. Here\'s what you need to know.","Auto-
save, keyboard shortcuts, customized addresses and more. Sound
cool? We think so too …",
[], "", "109cf79a5491ae6a", 0, "Mon Mar 6 2006_6:13 AM", 0, "",
0 ,0]
]
);
No privacy concerns from me. :-) And I think you can get the idea of how
each of the parts of the UI is represented in JavaScript, in a series of
D arrays. The Gmail
Agent API offers a more indepth look at these arrays, though it's based
on an older version of Gmail.
AJAX
I also try to offer an introduction to AJAX and creating a basic XMLHttpRequest. I also use some obfuscated code from Gmail to
illustrate how the simple XMLHttpRequest setup and request
cycle is used almost verbatim in Gmail. There are some interesting choices
made in Gmail, too, like populating an array with a few initial
XMLHttpRequest objects which can be pulled from the array at any
time. Gmail also makes nice use of JavaScript closures and passing the current
XMLHttpRequest around as a function context. This makes handling
multiple XMLHttpRequest objects clean and neat.
Gmail really is interesting and import in more ways the one.
On Friday, I'll preview the next chapter, a look at Google Reader and GData.
We'll be putting the X in AJAX next time. Until then...
February 5, 2007
Accessible and Useful Design
It's a question I've asked myself numerous times in the last year. Why
write a book on Google?
There are no shortage of books on the Search giant. O'Reilly has the excellent
Google Hacks and Google Maps Hacks;
Syngress has Johnny Long's Google Hacking for Penetration
Testers. There are books on AdWords, Analytics, and PageRank, and
as many or more on Google's business and history. I particularly like
The Search for a tour of Google's early days. But
all of these books fall cleanly in one of three categories.
- Tips and tricks with Google's sites for developers
- Tips and tricks with Google's sites for regular users
- How to understand Google's business and technology roots
There are clearly parts of Google that are not covered in any of these topics.
One such area is the extensive list of APIs Google makes available for programmers
and web developers. In fact, my interest with
Google really originated with the release of their SOAP-based search API. About the time
I was writing my first command-line search tool, Gmail started taking off, and then
Google Maps, and there was a whole world of possibilities in these JavaScript-based
sites. Namely, that you could see the code and hack at the sites yourself. And
then, the list of APIs began to grow as Google launched it's
Open Source program. So just for fun, I
started doing a tutorial at
LinuxWorld on using these APIs,
how Google used JavaScript to create rich browser-based apps, and other related
ideas.
This book, obviously, grew out of that tutorial and tries to fill a different
space than the other Google books available. As the beginning of Chapter 1 goes:
Google's mission is to organize the world's information and make it universally accessible and useful. Google's efforts at organizing information on the web are well documented. There are many books and websites concerned with Google's server cluster, its search index, and company policy, and indeed, these elements have helped Google create an index of the World Wide Web that is second to none. This is, however, only one-half of Google's mission, and I would argue, only one-half of what attracts users to Google. The other-half is the web sites that make that amazing index accessible and useful to others. In this chapter, we begin by examining how it is that Google makes information accessible and useful. We'll look generally at Google and its history and specifically at the design of google.com itself.
After including an image of an early version of Google's home page, I note:
From the very beginning, Google set itself apart for the simplicity of its own web page. The emphasis was meant to be on the information to which Google provided access. Remember, too, that the system was originally designed to just rank keywords against the index. There was no need for anything else, so why add it to the web page?
Notice that there are no categories, no services, no news headlines, no weather. Nothing but a simple text box and prominent logo. Nothing but search. Figure 1-2 shows an example of the Google homepage today. Although there are links to alternate services and searches, the homepage remains remarkably uncluttered. To begin a search, just type in the box and click “Google Search."
And from there we talk a bit about Google's roots in Academia and some of the
philosophies that underpin Google's choices in design and development. While this
is not a book on design philosophy or aesthetics, I do argue that Google puts its users first, which
is the key to creating useful and usable web sites and applications. Several of
the elements -- like a minimal, uncluttered interface -- on google.com continue
to reappear in other forms on other sites.
But this is a tech book, right? So we do get into technical matters, too, even
as we lay the foundation for the rest of the book. We talk about Google's search
operators and special syntaxes and also how Google uses CSS and JavaScript to
enhance search, not as technologies for their own sake. For Google's homepage,
JavaScript is used minimally.
function qs(el)
{
if (window.RegExp && window.encodeURIComponent) {
var ue=el.href;
var qe=encodeURIComponent(document.f.q.value);
if(ue.indexOf("q=")!=-1) {
el.href=ue.replace(new RegExp("q=[^&$]*"),"q="+qe);
}else{
el.href=ue+"&q="+qe;
}
}
return 1;
}
This script may look quite cryptic, but its job is simple. The function qs() is called when any of the links other than the Google search submit buttons are clicked. The el passed to the function is the element that was clicked. If the RegExp and encodeURIComponent objects are available the query is encoded to be used for a URL and the element that was clicked's href attribute -- the link itself -- is replaced with the encoded query string now added to the link. Certain charters are not allowed in URLs, like spaces and ampersands, so this function makes encoding this within the browser easy. In fact, without JavaScript there is no way to ensure a properly encoded URL from within the browser.
This function also exists to further the goal of accessible and useful design. If you have already entered your query terms into the main Google search text box, and then decide you want to search for images instead of web sites. When you click the images link above the text box, your query is encoded and passed along to the image search page, without you having to re-enter the search later. A useful feature indeed! And one made possible by a simple use of JavaScript.
Contrast this with JavaScript usage in Gmail and Maps, and you'll see two very
different approaches to JavaScript usage. Although, both are done from the same
prespective, "to organize the world's information and make it universally accessible
and useful". In Chapter 2, which I'll preview in a week, we really get under the hood
of Google more, looking at the JavaScript UI engine that is Google's successful
webmail site.
February 4, 2007
Since my New Year's resolutions to write more and finish
my book,
I can safely say I've done a terrible job at each. If I'm going to finish
this book — which I *really* want to do — I'm going to have to just force
myself to make time for it. Time is hard to come by these days, what with
all I've got going on at
WPNI, and I'm
generally not one to seek out more pressure for myself. So... I'm calling on
you dear reader and friend to help me see this task through.
I've been impressed with the way the Django
book has been shaping up online, and so, I thought I'd take a cue from
Adrian and Jacob. As much as I liked the open nature of the process for their
book, I always thought posting a public release schedule was a nice way
to put a bit of pressure on themselves to finish it. (Whether this was
intended or not, I don't know, but it seems to me a natural side effect
of the public posting.) I'm going to follow suit here.
My book is not released under an open source license like the
Django book, but I had previously received permission from my editor, Mark
Taub, to publish small samples of the book as I worked on it. I'll
be publishing here, then, a sample from each chapter with some comments
from me on related ideas — the writing process, choices in material,
significance of the topics, etc. I also won't be putting the preview
samples up in the same chapter order of the book. I'll start with
what I have finished, and then work around my other material in an
order that allows me time to finish the longer or more complicated
chapters in a timely manner.
The schedule will be as follows:
- Monday, February 5
- Chapter 1 — Accessible and Useful Design
- Monday, February 12
- Friday, February 16
- Chapter 3 — Reader and GData
- Monday, February 19
- Wednesday, February 21
- Chapter 9 — Open Source Google
- Wednesday, February 28
- Chapter Chapter 8 — Webmaster Tools
- Monday, March 5
- Chapter 5 — Office and Productivity Sites
- Wednesday, March 14
- Chapter 6 — Personalized Google
- Friday, March 23
- Chapter 7 — Server-side Development
- Friday, March 30
- Chapter 10 — A Complete Google-driven App
In return for all this free, previewy goodness, all I ask in return is
that you all call me out in great public shame and humiliation if I miss
a deadline. Call me out in the comments here (or comments for other
book related posts.) Call me out in your blogs. And any comments you
have on the book, how to make it better, and what you like or don't like
would be appreciated to. :-)
Here's to a New Year and finishing this book together! Cheers.
February 2, 2007
After hearing about ESPN's new addition of comments
from Levi just now and how
the site uses JavaScript to fill in comments, I feel I must ask..... why?!? I love JavaScript as much
as the next guy. Really, I do. I lay up many nights worried about whether I'll ever finish my book on Google,
which is filled with info on JavaScript and their use of. But if you're publishing system can't load comments --
or anything else for that matter -- without the use of JavaScript, please get a new one!
If you can't find one, find a good Django developer,
and let her or him build one for you. It's really not rocket science. :-)
