An image annotation tool - using Javascript to read and create RDF

libby.miller@bristol.ac.uk, 2003-06-15

use the tool (requires Mozilla or IE windows)

Aim: to have a tool that would

How to use the tool

The assumption is that you wish to annotate a photo or picture which is somewhere on the internet. You can say things like:

using a forms-based interface.

As the image is annotated, the information from the forms is displayed as RDF. If you email Libby Miller you can ask for a password to use my server to store your data; otherwise you will need to put it on a server and add its url to a database yourself.

Related work includes

Code used:

The requirements of simplicity and minimal install meant that the RDF data could not be installed on the user's machine. For this reason a number of remote services were defined, allowing access to remote RDF data about people, locations and keywords over http.

The language was also important. This started as a Java application; however, Java on Debian is not straightforard to install; and Java on windows machines now usually requires administrator access. We required a language that was multiplatform, and did not require a download or install.

Javascript was chosen because of the good support available in two major browsers - the multiplatform Mozilla and IE for windows. Javascript is a fully functional programming language running inside browsers. It doesn't require installation; however each browser runs a different variant of the language, so interoperability is difficult. In particular, the capacity to download and parse XML using the browser's native parser is restricted to IE on Windows and Mozilla. This functionality is used by the javascript RDF parser used to access the remote services, and so the application only runs on these browsers.

Functionality

Form-based user intereface

A previous version of the tool used a text-based interface for speed of use.

Applications such as Matt Biddulph's IRC bot-based conversational interface to image annotation, and Damian Steer's foaffinger rendezvous-based foaf creation tool have successfully used stateful, text-based interfaces to data creation. However, a significant issue for image annotation is that as you catalogue you need to be able to see the image. It is also useful to be able to pick from a list of thumbnail images and then annotate several; this means that commandline or bot interfaces are limited in their usefulness in this case.

It was hoped that the use of javascript might be overcome these issues, however, user feedback of the text-input tool showed that many people found the text-based interface confusing from within a web page, and so a clickable version was produced. The visual cues this gives makes cataloging images faster, although there are several significant problems with layout of the information.

Viewing the images to be catalogued

The tool uses a proxy to download a thumbnails page, or a page with images in it, or a single image, into an iframe. The images are accessed using the DOM, and displayed. Clicking on an image triggers a download of the image or html page linked to in the initial thumbnails page, and then the tool figures out if it is an image or an html page. If the latter, it makes a guess about which is the correct image, and makes that the main item to be catalogued. At this stage we have something like this:

screenshot of image annotation tool showing a
parrot

and the RDF generated looks like this:

<rdf:RDF
xmlns='http://xmlns.com/foaf/0.1/'
xmlns:rdf='http://www.w3.org/1999/02/22-rdf-syntax-ns#'
xmlns:rdfs='http://www.w3.org/2000/01/rdf-schema#'
>
<rdf:Description rdf:about="">
<annotates
rdf:resource="http://swordfish.rdfweb.org/photos/2003/06/12/2003-06-12-Images/4.jpg"/>
</rdf:Description>
<Image
rdf:about="http://swordfish.rdfweb.org/photos/2003/06/12/2003-06-12-Images/4.jpg">
<thumbnail
rdf:resource="http://swordfish.rdfweb.org/photos/2003/06/12/2003-06-12-Thumbnails/4.jpg"/>
</Image>
</rdf:RDF>
Using wordnet remotely

Dan Brickley has produced a service whereby appending a noun to the namespace http://xmlns.com/wordnet/1.6/ gives you the wordnet heirarchy for that noun, if it exists. The image annotating tool uses this trick, so if you type 'parrot' into the 'keyword' box, the tool uses Jim Ley's RDF parser to fetch the RDF associated with http://xmlns.com/wordnet/1.6/Parrot, and display it in a useful way so that the tool user can check that it displays the term they are interested in, and also see if a sublcass of the main term might be more appropriate.

results of a search for wn:Parrot feedback to the user when 'parrot' is
searched for

The wordnet term is then added to the generated RDF by clicking on it, for example:

illusttration of some of the subclasses of
wn:Parrot
<rdf:RDF
xmlns='http://xmlns.com/foaf/0.1/'
xmlns:rdf='http://www.w3.org/1999/02/22-rdf-syntax-ns#'
xmlns:rdfs='http://www.w3.org/2000/01/rdf-schema#'
>
<rdf:Description rdf:about="">
<annotates
rdf:resource="http://swordfish.rdfweb.org/photos/2003/06/12/2003-06-12-Images/4.jpg"/>
</rdf:Description>
<Image
rdf:about="http://swordfish.rdfweb.org/photos/2003/06/12/2003-06-12-Images/4.jpg">
<thumbnail
rdf:resource="http://swordfish.rdfweb.org/photos/2003/06/12/2003-06-12-Thumbnails/4.jpg"/>
   <depicts>
      <wn:Parrot/>
   </depicts>
</Image>
</rdf:RDF>
Finding people to catalogue

For images containing people, it's useful to be able to say that the image depicts a particular, identified person. See the codepiction experiment for more information about this approach.

One issue is a convenient way of finding people's sha1'ed email addresses (or their actual email addresses and converting them using a tool). This is where a remote service from a database which already contains this information is useful. This could be, for example, a private address book with a remote interface which produces RDF. In this case, we use an interface to a harvested RDF database.

a list of people and their images from a
search for the name 'Dan'

Sha1ed mailboxes and images are shown in response to a query on a substring of a name. Clicking on the image or the name produced adds the person to the RDF. If the person is not in the database, they can be added manually using the forms. At no time is an email address made public.

Location

Ideally the location of a photo would come from the EXIF data the camera produces; the same goes for date. The first of these is not yet available so we have had to come up with something else. The latter is readily available, but I have not come across EXIF parsers in javascript as yet.

We have chosen to use the nearestAirport property to associate an image with location data at this time. This is because information linking airports with latitude and longitude is freely available. A an added bonus this method preserves privacy.

The key issue in terms of accessing geo data is human-readable to lat/long mappings. As a rough pass, the airports data works well because there is a human-readable name for the airport which includes the nearest town or city. This means we can search on the airports data using user-inputted names of places and get out the lat/longs. A similar (and more finegrained) approach would be to use the spacenamespace data; at the moment this is UK-only however.

Modelling the nearestAirport information was difficult. It is not the nearestAirport to the picture as an artifact (the picture may be held on one or more servers, well away from the location). Nor is it necessarily a picture of a location. Instead, it's the location the camera was in when the picture was taken. Similar arguments apply to the date the picture was taken. A new property, creationEvent was created to test this out. The use of creationEvent masks a hidden resource - an object representing the event, to which nearestAirport and date can be attached.

Description

You can add a textual description to the photo.

Uploading

Issues

Some notes on the use of FOAF properties

This tool uses several properties which are not in the official FOAF schema. These are foaf:annotator and foaf:creationEvent. These are here as an experiment and will be probably moved into another namespace.