The Yahoo! Geo Technologies Blog

On October 28, 2008, Yahoo! announced the launch of several components of the Yahoo! Open Strategy (Y!OS). Among these was the Yahoo! Query Language (YQL), a web service that enables developers to query, filter, and combine data across multiple Yahoo! web services using a SQL-like syntax. The GeoPlanet Web Service has been integrated into YQL, allowing users to take advantage of place in a number of different ways. Web Services that return place names can be used to generate WOEIDs. The GeoPlanet Web Service can be used to get place names needed by other web services from WOEIDs. The possibilities are endless!

A great place to start exploring YQL is the YQL Console. This web application allows developers to experiment with YQL queries and create URLs they can then embed in their own web applications. More information about YQL is available on the Yahoo! Developer Network YQL website.

YQL in a nutshell

YQL represents web service collections as tables with primary and secondary keys representing collection filters and data fields representing each of the collection’s response elements. The YQL DESC verb describes a table, including its primary key, other supported input fields, the output fields, and an example query. The YQL SELECT verb selects specific data fields (response elements) from a table (collection) using a key (filter).

GeoPlanet collections and resources are represented by eight YQL tables.

Examples

The following are some examples showing how information from GeoPlanet can be generated using YQL and used in various ways. The basic examples use only the geo tables to return data. The intermediate examples combine various geo tables to return data. The advanced examples combine geo data with data from other Yahoo! web services.

Basics

Learn about the geo.places table

desc geo.places

Determine the WOEID for a place (e.g. SFO)

select woeid from geo.places where text = “SFO”

Determine the name of a place known by its WOEID

select name from geo.places where woeid = 12521721

Determine the name of the country a place is located in (e.g. Sunnyvale, CA)

select country from geo.places where text = “Sunnyvale, CA”

Intermediate

Return just place name and place type name for administrative areas containing a place (Sunnyvale, CA)

select name,placeTypeName from geo.places.ancestors where descendant_woeid in (select woeid from geo.places where text=”Sunnyvale, CA”)

Determine the timezone of a place (e.g. SFO)

select name from geo.places.belongtos where member_woeid in (select woeid from geo.places where text = “SFO”) and placetype = “Time Zone”

Determine the names of neighboring places (e.g. Sunnyvale, CA)

select name from geo.places.neighbors where neighbor_woeid in (select woeid from geo.places where text = “Sunnyvale, CA”)

Determine the Zip Codes in a town (e.g. Sunnyvale, CA)

select name from geo.places.children where parent_woeid in (select woeid from geo.places where text = “Sunnyvale, CA”) and placetype = “Postal Code”

Advanced

Determine the WOEID for a user (e.g. me)

select woeid from geo.places where text in (select location from social.profile where guid = me)

Determine the timezone for a user (e.g. me)

select name from geo.places.belongtos where member_woeid in (select woeid from geo.places where text in (select location from social.profile where guid = me)) and placetype = “Time Zone”

Get weather forecast for a specific place known by its WOEID

select item.description from weather.forecast where location in (select postal from geo.places where woeid = 12797568)

Summary

YQL is a powerful tool that combines and filters the data provided by many Yahoo! web services in a form that allows developers already familiar with SQL to be productive quickly and effectively. By integrating GeoPlanet with YQL, Yahoo! Geo Technologies has added the value of GeoPlanet’s geo-permanent named place management system to an already powerful service.

Eddie Babcock, Yahoo! Geo Technologies

On Wednesday September 24th, the British Computer Society, North London Branch, in association with the BCS Geospatial Specialist Group, hosted an event on Mobile Location Based Services at BCS headquarters in London.

I was invited to attend this event and presented a talk entitled The Open Location Ecosystem and the Mobile Internet, together with Justin Davies from NinetyTen and Buddyping and Andrew Grill, a mobile advertising evangelist.

Mobile Location-Based Services (LBS) have been ‘the next big thing’ for years now, but have not materialised due to a mixture of technical and business constraints. With increasing numbers of mobile phones being equipped with Wi-Fi and GPS receivers, this looks set to change.

What are the main location technologies and how do they work? What technical, business and social challenges do companies face when developing LBS for mobiles? What LBS can we expect to see on mobiles in the next 2-3 years? How will current favourites such as social networking and online advertising evolve in a location-aware world? This talk will address these questions and more.

Join us at this dynamic free event to learn more and share your views.

Our expert presenters are:

* JUSTIN DAVIES Founder & CTO, NinetyTen/Buddyping,
http://www.ninetyten.com http://www.buddyping.com
* GARY GALE Head of UK Engineering, Yahoo! Geo Technologies,
http://www.ygeoblog.com, http://developer.yahoo.com/geo/
* ANDREW GRILL Mobile Advertising Evangelist
http://www.andrewgrill.com/blog/

Yahoo! received a very warm welcome from the BCS, from the other presenters and from the audience; the hot topics for Yahoo! at the Q&A session afterwards were GeoPlanet, use of WOEIDs and Fire Eagle. The deck I presented is available for download here.

The resources referred to in the Yahoo! presentation are listed below.

• Yahoo! Geo Technologies blog - http://www.ygeoblog.com
• Fire Eagle: http://fireeagle.yahoo.net/
• Fire Eagle Developer APIs: http://developer.yahoo.com/fireeagle/
• Yahoo! GeoPlanet: http://developer.yahoo.com/geo/
• Yahoo! BluePrint: http://mobile.yahoo.com/developers/roadmap

More outreach and talks of this type are being planned and you’ll be able to read about them here on the Yahoo! Geo Technologies blog first.

Gary Gale, Head of UK Engineering, Yahoo! Geo Technologies

Y! Geo is pleased to announce an update to the Yahoo! GeoPlanet web service. We’ve listened to your feedback and added several useful features to help you develop interesting applications using GeoPlanet.

First, we added a new query parameter called “callback” that allows GeoPlanet to return JSON data wrapped in a JavaScript function call (aka JSONP), so you can more easily process the GeoPlanet response within a JavaScript application and avoid cross-site serving issues.

Second, we added support for a new response format called GeoJSON. This is an evolving standard for adding geographic content to JSON.

Third, we added a new filter for the /places collection called “$and”. This filter allows you to apply two other filters (such as .q and .type) to the same request. This means that you can now limit results to just the place types you are interested in. The syntax for filters may seem strange, but it offers a lot of functionality.

Fourth, we added a /placetype resource that allows you to get information about a place type. This is handy if you want to find the name and/or description (see the next item) for a single place type.

Last, we added support for a long representation for the /placetype resource (and /placetypes collection) that includes a one line description of the place type. This is handy if you want a better understanding of a place type.

More detailed information about these new features is available in the GeoPlanet documentation, and examples are available on our GeoPlanet General Discussion Forum.

We also fixed several bugs reported by our developer community. We continue to view GeoPlanet as an evolving product, so let us know how we can make it better! Post a message to one of our GeoPlanet developer forums and tell us what you think.

Eddie Babcock, Principal Engineer, Yahoo! Geo Technologies

How else could one title this post? Here’s the story: Birmingham City Council in the UK recently sent out 720,000 leaflets advertising their services; the picture on the leaflet, however, depicts a different city with the same name: Birmingham, Alabama (US). Classic local authority snafu.

It’s clear that the individual charged with illustrating the pamphlet searched for ‘Birmingham’, found what looked like a nice skyline, and failed to fact-check. It is also possible that this individual never realized that there are multiple towns called ‘Birmingham’ (eighteen, actually). Whatever the cause, the story highlights some fundamental-but-oft-overlooked challenges in geoparsing that we embrace at Yahoo! Geo Technologies.

Geoparsing is of course the process of identifying places referenced in free- or unstructured text, and is the essential ingredient of any system where we want to geolocate content with machine analysis. The two steps of successful geoparsing are (1) token identification, and (2) geographic disambiguation. Let’s take a look at each briefly:

The first step in geoparsing is token identification: identifying place-names, such as ‘Wayne’ or ‘The Bay Area’, in unstructured content like newspaper articles or web pages, while ensuring at the same time that one does not falsely identify terms like ‘New England Clam Chowder’ as a place (a post on our fun with these potential false-positives will follow).

But token identification is the easy half of the battle; many entity-recognition applications, like the otherwise excellent OpenCalais are not capable of geotagging the above BBC article on ‘Birmingham’, for example, as it — correctly — identifies seven ‘Birminghams’, but does not tell us whether those referred to within are the UK city, one of its seventeen US namesakes, or a mix of both. (You can try this yourself with any text using the Calais Viewer.) Human cognition can certainly determine this with a quick read-through, but we’re looking at machine parsing specifically here.

To do this properly, we first require the means to refer to a place in a permanent, unambiguous, and machine-friendly manner: usually this is attempted by expanding the geographic context so that the token ‘Wayne’, when found in text, can be indexed as ‘Wayne, PA, USA’; this works sometimes but is hardly machine-friendly. (Furthermore, there are ten towns called ‘Wayne’ in Pennsylvania, so the above string gets us no closer to our goal.) In truth, string-based indexing will always have its exceptions, so we have opened GeoPlanet, our gazetteer of places and their unique Where-on-Earth Identifiers (WOEIDs), to provide the vocabulary to describe the world’s places without ambiguity.

So, now that we’ve found the correct tokens (’Wayne’) in our hypothetical text, and dismissed misleading, place-sounding terms (’Yorkshire Pudding’), we then determine which place, of all the places with that name, is specifically being referenced. This is geographic disambiguation (or geodisambiguation for the portmanteau-inclined). Let’s take for example ‘Rome’, of which there are over thirty: there is of course ‘the’ Rome, in Italy (WOEID: 721943), and for many of us, this is the only Rome we know. However, residents of Rome, Georgia (WOEID: 2484261) would argue otherwise. This highlights the problem: how can we be certain which place is being referenced when we have only ‘Rome’ in the text? Obviously the language helps in some instances, as does context (is ‘Georgia’ or ‘Italy’ mentioned elsewhere in the document?). But when geodisambiguating at Yahoo! (and this is the fun bit), we take into account the location of the user (or publisher) to capture the ‘locality’ of the term, and really put geography in the first-person. For example, although ‘Rome’ by itself will usually refer to ‘Rome, Italy’, the probability of its referring to ‘Rome, Georgia’ increases as we move geographically towards the latter. This approach ensures that Yahoo! returns the ‘correct’ city when a search for ‘Birmingham’ is performed in the UK, compared to the same search in the US. This approach ensures that content originating from Rome, Georgia will be geoparsed and disambiguated correctly to the correct and local ‘Rome’.

Acknowledging that geography is in the eye of the beholder is just one way that Yahoo! Geo Technologies provides our users with the most personally georelevant results. Shame Birmingham Council did not come to us first.

Tyler Bell, Advanced Products Manager, Yahoo! Geo Technologies

On Tuesday the talented team at Brickhouse launched Fire Eagle, Yahoo!’s user location management platform, to loud acclaim.

I’ve been a huge fan of Fire Eagle since its inception — as a business driver it was conceived to slice horizontally through the vertical towers that now dominate the Location Based Services landscape. Its launch not only returns ownership of User Location to the hands of the user, but undoubtedly triggers the tearing out of hair and significant re-writing of business plans; this can only be good — opening user location ensures that new businesses are built on the opportunities this affords, and not on the ‘captive audience, closed service’ concepts that currently dominate.

The product ethos is focused wholly on protecting user privacy while exposing the power of location; this is what really is most impressive. While developers usually tout the ‘heavy lifting’ that Fire Eagle does to make geolocation appear easy (more on this below), I would suggest that Fire Eagle’s greatest success is the care and attention evident in the product to ensure that users have complete control over who has access to their location, and at what granularity this is exposed. Far from shying away from the complex and at times intractably confused technical and policy issues surrounding user location, privacy, and geolocation, the Fire Eagle Team has carefully met them head-on and delivered a well-conceived, innovative, and enabling technology. This is delicate ground, certainly, and each subsequent step must be taken with similar diligence, but I am very excited to see what new ideas, products, and businesses emerge from Open User Location.

I am, of course, hardly an unbiased observer: the Yahoo! Geo Technologies team provides the machinery that performs the aforementioned ‘heavy lifting’. Our tech helps Fire Eagle determine where on earth its users are, assists with its geographic granularity protection, and ensures that developers can integrate the geographic data returned by Fire Eagle with other systems via ‘where on earth’ IDs (WOEIDs) and our GeoPlanet Web service. In a similar manner we also power the geoinformatics underlying Flickr’s new geotagging service. Combined with the significant geo wizardry and craft of the Fire Eagle, Flickr, and other teams at Yahoo!, we are continuing to provide the tools and platforms to spatially enable the Web and provide our users with the most personally georelevant experience possible.

Tyler Bell, Advanced Products Manager, Yahoo! Geo Technologies

Thanks to the good folk of the Yahoo! Developer Network we now have a set of forums dedicated to discussing all things associated with Yahoo! GeoPlanet.

You’ll find forum categories for requesting enhancements, showcasing applications or demos which make use of GeoPlanet, requests for using GeoPlanet in a commercial environment and general discussions, conversations and Geo related chat.

Members of the Yahoo! Geo Technologies group will be on the forums and we look forward to meeting and chatting with you all there.

You can find out more by pointing your browsers to the Yahoo! Developer Network GeoPlanet forums at http://developer.yahoo.net/forum/index.php?showforum=31 now.

Cheers,

Gary Gale, Head of UK Engineering, Yahoo! Geo Technologies