Now that I have reached a major milestone on the software side of the application, I will now turn my attention to the visualization of the semantic information presented, to make it much more intuitive. Lots of exciting developments are forthcoming, stay tuned!

• Lightning-fast searches!: I managed to get search times down from over 30 seconds to less than 200 milliseconds, quite the dramatic improvement! This is fast enough for instant search! Try it! Just type a term you’d like to know more about, and without even pressing return, you’ll get a list of search results.



• Complete architectural redesign: Pretty much everything you see has been redesigned, as well as everything under the hood. This release represents a major architectural overhaul of the core software, to support plugins. This means that it should be much easier for me and others to develop new visualizations and reasoners for semantic information. In other words, this release represents the beginning of the Exxis Semantic platform! Stay tuned!
• Back button works!: This is surprisingly tricky to get right, in the world of web applications, but the back and forward browser buttons should work as expected.
• No support for Internet Explorer: I’ve decided to drop support for Internet Explorer, as version 8 is very lacking in its HTML5 support. You’ll get a much better experience when viewing the site on Firefox or Chrome.

• No more pizza!: I have done away with the basic Pizza ontology that I was using for testing, and have replaced it with the OpenCyc knowledgebase. The browser can now display information on more diverse topics such as, for example, Princeton University, the Montreal Canadiens, or, if you still prefer, pizza.
• Search: Since the general-knowledge dataset provided by OpenCyc is much larger than the pizza dataset, it no longer made sense to list every object in the left sidebar. A search interface replaced the object listing.
• Asynchronous AJAX: The presentation layer is now asynchronously connected to the view layer.
• New display: I have made lots of changes to the overall user experience, with many more changes to come.
• Separation of data and object attributes: Data attributes (e.g. strings, integers) have been moved out of the graph, into the “Properties” pane in the sidebar. The graph now only displays object properties.

• Dynamic Navigation!: Nodes can now be clicked, to expand them and re-focus the graph. When clicking on a node, its label will first turn orange, to indicate that it is loading. Once the node has loaded, the label will turn green, and any additional nodes will be added to the graph. This gives way to a much more dynamic way to browse the graph. Try, for instance, viewing a node like DomainConcept, and clicking on nodes to explore the dataset from there.

• Many small bugfixes: I am constantly finding and correcting software bugs here-and-there. Last week, there were 2 server outages related to odd problems with both the MySQL and RDF libraries I’m using. In this release, I have found two such bugs and implemented workarounds.
• Node clustering: Nodes are now clustered when they share the same predicate, to simplify the visual display by reducing the number of lines.
  

  Before – No clusters were being drawn.

  

  After – A cluster of 5 elements was detected. The ‘hasTopping ~’ label is now drawn once, instead of 5 separate times.

• MySQL backend integration: The previous version relied on an SQLite database, which was not optimal for anything bigger than a very small semantic database. This new database integration adds robustness and speed. (I will shortly add support for Postgres, as I hear it is much more optimized than MySQL — I just chose MySQL because I know it well.)
• More consistent graph display: The focused node now appears at the centre of the canvas. Superclasses are displayed above. Subclasses and instances are displayed below. Incoming and outgoing nodes, which are not related to class hierarchy are drawn to the left and right, respectively.
  

  Before

  

  After

• Fewer nodes: A lot of the feedback I got was along the lines of, “I don’t get it”. I’ve therefore filtered out many of the predicates that don’t intuitively mean anything to most people (e.g. disjointWith, first, range, domain, onProperty etc.). I’ve also filtered out most of the Restriction objects from the graph. These connections will re-appear in later releases, presented in more functional ways.

So Quo Vadimus?

Where do we go from here? In the early 21st century, it seems anachronistic of us to rely so heavily on these two divergent and equally limited forms of knowledge storage.

Hypertext has been immensely useful in communicating ideas. Wikipedia is the shining example of this paradigm, in which we have been using the web as a glorified book. I’m not trying to diminish this astounding monument to human collaboration. However, one of the key reasons we have computers is to be able to automate tasks we find tedious and navigate information in new and more dynamic ways. What’s the use of having so much knowledge stored digitally and so much computational power in the world, if computers can’t do anything with it except render web pages for humans to decode and interpret?

Metadata tables, on the other hand, allow us to do stuff  with the knowledge they contain, as long as we, as humans, are willing to bend, mash and trim that knowledge to make it fit into a neat grid-like structure. For example, most bureaucracy is an exercise in shaping data to make it fit into forms and tables. Errors are commonplace, and you’d be hard-pressed to find a more de-humanizing chore than filling out any type of application form.

This, to me, is an unacceptable tradeoff.

The Monk

Having now dabbled in careers as both a record store owner, as well as a baking-wiki editor, you now see the big picture, and the contradiction that is the crux of the human/computer dichotomy: knowledge needs to be both  highly expressive and  highly structured.

Somewhere in the middle of the hypertext/metadata battlegrounds, a synthesis is emerging.

Let’s begin by looking at the structure of a grid. For each cell of data, there are 3 elements at play: the row heading, the column heading, and their intersection.

Example Table — The highlighted cell lies at the intersection of the “Benefits” Row (8) and the “Feb. 11″ (C) column.

In cases where the table represents a list of items, the row headings are implied. This was the case with our record catalogue. Here it is again, with implicit row headings added:

Record Catalogue Revisit — The same record catalogue from part 1, with labels to identify each row, and act as the row headings.

If you were to list all of the row, column and cell intersections present in this table, you’d arrive at a list like this one:

• [Item #1] [Title] [Lifes Rich Pageant].
• [Item #1] [Artist] [R.E.M.].
• [Item #1] [Year] [1986].
• [Item #1] [Format] [LP].
• [Item #2] [Title] [Lifes Rich Pageant (25th Anniversary Reissue)].
• [Item #2] [Artist] [R.E.M.].
• [Item #2] [Year] [2011].
• [Item #2] [Format] [2CD].
• [Item #3] [Title] [Atlantic Rhythm and Blues: 1947 - 1974].
• [Item #3] [Artist] [Various Artists].
• [Item #3] [Year] [1991].
• [Item #3] [Format] [Boxset].

Congratulations! You have now mastered the basics of the Semantic Web, or as we can now think of it, the Third Way. By “unfolding” the contents of the table, you lose its structure without compromising the way in which the data is represented.

As with hypertext, you are now dealing with a canvas on which free-form statements can be made. You’re still using human language, so you can be as expressive as your heart demands of you. You can also use hyperlinks — I’ll get to that in a minute. For instance, here are further modifications you could make to your semantic record catalogue:

• [Item #3] [Artist] [Various Artists].
• [Item #3] [Artist] [Big Joe Turner].
• [Item #3] [Artist] [George Freeman].
• [Item #3] [Artist] [Lady John Scott].
• [Item #3] [Artist] [Tiny Grimes].
• (etc.)

Now, instead of using the label “Various Artists”, you can state exactly who appears on the album, and make those names available to search.

You are also unconstrained by the grid in that you may add/remove column headings on a row-by-row basis. Consider these further modifications to your semantic catalogue:

• [Item #1] [Year] [1986].
• [Item #1] [Recording Year] [1986].
• [Item #2] [Year] [2011].
• [Item #2] [Recording Year] [1986].
• [Item #2] [Release Year] [2011].
• [Item #3] [Year] [1991].
• [Item #3] [Release Year] [1991].
• [Item #3] [Spans From] [1947].
• [Item #3] [Spans To] [1974].

The basic form of all statements in the semantic web are composed in the form: [subject] [predicate] [object]. This syntax imposes only a minimal amount of structure to the statements, making them accessible to computing.

Consider we do something a little bit trickier to our record store catalogue than just adding new columns: we give it a bit of intelligence, too.

In the case of the two R.E.M. albums (which are actually two editions of the same album), let’s make a new statement that links these two albums together, like so:

• [Item #2] [Recording Year] [1986].
• [Item #2] [Reissue Of] [Item #1].

Notice that I’ve removed the “Recording Year” entry. By creating a new generalized concept such as “Reissue Of”, I can create logical inference rules that provide specific information (e.g. the recording year of an album) in a more general way (e.g. each reissue). A piece of software known as a reasoner takes these rules and applies them. For instance, I can create a rule which states, “Every reissue of an album was recorded on the same year as the original”.

The reasoner can then use this small bit of intelligence to apply this rule to every reissue:

• [Item #7743] [Recording Year] [1988].
• [Item #7791] [Reissue Of] [Item #7743].
• [Item #7791] [Release Year] [1998].
• [Item #7791] [Recording Year] [1988].

Since the semantic web is all about representing and understanding real-world concepts in highly interconnected and intelligent ways, creating these inference rules (which is akin to programming) doesn’t rely on text markers to understand the data. In other words, if I’m a customer searching a semantic music catalogue for a reissue, I don’t have to look for the word “Reissue” in the album title. I can directly ask to see all albums [x] that meet the condition [x] [Reissue Of] [y].

Many other logical inference rules can be added, to make sure the record catalogue makes sense and doesn’t contain errors. For instance, rules like “An album cannot be released before it is recorded”, or “A reissue cannot be released prior to the original”.

These rules seem overly simple to a human, but to a computer, this is good stuff.

Back to the Bakery

The above describes the “semantic” part of the equation, but where does the web fit in? As a baking-wiki editor, you witnessed first-hand the awesome power of community. Despite their cultural differences, your users were the sous-chefs that kept your virtual bakery fresh and flaky. Or something.

A key element of the semantic web is that it is distributed, exactly like the web as we now know it. This means that when you define the logical rules for your record store catalogue, you have the option of making that logic accessible to others. A music collector elsewhere on the internet might stumble upon your logical rules, and simply link to them in order to apply them to his or her music catalogue. It’s not only the content of the information that can be shared, but the programmatic infrastructure around it, too!

In the case of the baking wiki, this feature could come in immensely handy! For one thing, ingredients in a recipe could be described in terms of certain quantities of a certain food, where both the quantity (e.g. cups) and the food (e.g. chocolate chips) are linked to other places on the web where these concepts are defined more fully.

For instance, perhaps “cups” is linked to a semantic Wikipedia page, where the logic for converting cups to metric is defined. Likewise, “chocolate chips” could be linked to a semantic database that tracks calories, so that now, you can halve recipes and count calories automatically!

The two elements at play here — semantic insight into data, and a distributed format — are what make the semantic web so unique and powerful. As the semantic web gains adoption and popularity, we will start benefiting from the network effect, in that knowledge and intelligence can be catalogued, shared, communicated, presented and utilized in unprecedented and world-changing ways.

With semantic web technologies, we can construct incredibly complex queries, and given the information and logical elements needed to solve that query, have delivered exactly the result we’re looking for. Imagine a world in which a command like this is possible:

“I’d like to bake a cake while listening to music. Create a playlist with songs arranged alphabetically by artist. I only want to hear songs from the 1950′s sung by artists who grew up within 150 km of a city with a population of over 1 million and with at least 2 vowels in its name. Make the playlist exactly as long as it will take me to bake a cake in the exact quantity that would result in the finished product containing a prime number of calories, and making about 10 modest servings.”

This query can be broken down by a computer in exactly the same way a human would break down such a task: look for cities of at least 1 million and only keep those with two vowels in their names. Then cross-reference that list with artists who had songs released in the 1950′s, and who grew up within the specified radius of each matching city. Once you have candidate songs, find a cake recipe, scale it up or down until the calorie count and serving restrictions match the criteria, figure out the cooking time, then construct a playlist accordingly.

(It might also be a good idea to have the semantic helper keep some psychiatrists’ numbers on speed-dial, because that is one crazy-ass request.)

Conclusion

I believe the Semantic Web will save the world because it represents the gateway to massively-collaborative and distributed intelligence. The web, in its current incarnation, has proven to be a singularly powerful tool in the missions of facilitating the cross-cultural sharing of ideas, and pushing education to the farthest reaches of the planet.

I believe the widespread adoption of semantic-based knowledge sharing will have an epochal impact on these efforts and others like them, as we empower computers to better understand our world, and in doing so, help us shine light into previously dark recesses, where injustice and ignorance dwell.

The semantic web has the power to release us from the dehumanizing bondage of metadata bureaucracy. All the while, it allows us to embrace knowledge sharing in the intrinsically-collaborative tradition of the wiki, as we give computers ways of using that knowledge to do everything from diagnose diseases with greater accuracy to helping us plan healthy meals more cost-effectively, with minimal food waste and environmental impact.

The possibilities are virtually limitless, and for the first time, bounded not by an inability to express them, but only by the computational resources at hand.

A more interesting approach to storing knowledge digitally is to use natural language. This is where hypertext comes in. Hypertext is text, in any language, that has been augmented with links to connected documents. These links span over portions of the document, and connect the highlighted text with a relevant document.

This blog is written in hypertext, as is a site like Wikipedia.

In fact, most of the internet is hypertext! The underlying format of web pages is HTML, which is the Hypertext Markup Language. The foundations of the World Wide Web are built around the power of hypertext. Beyond that, the modern web contains many examples of hypermedia, which is a broader category of hyperlinked media (i.e. not just text, but also pictures, sounds, video clips, etc.), as well as full-fledged applications which go even beyond the scope of hypermedia.

But for now, let’s focus on the wiki, as an example of a very straightforward hypertext platform.

The Bakery

Frustrated with your failed venture starting the music store, you’ve decided to put your record catalogue aside and open a bakery. Well, actually, you’ve decided to launch a website sharing recipes and information about baking.

Now that you’re familiar with the ways in which metadata is an imperfect method for storing information, you turn to wiki software to create your recipes website. You’re already a big fan of Wikipedia, and you’d like to do the same thing, but for baking.

So, you set up the software, and, driven by a pure passion for soufflés and strudels, you begin entering content into your wiki: recipes, encyclopedic ingredient descriptions, nutritional data, etc.

In a short time, you notice your traffic increase substantially, as a growing number of users from around the world begin consulting and contributing to your baking wiki. After years of torment, you feel the weight of your failed record store begin to lift off your shoulders.

The feeling is short-lived, however, because you begin hearing a specific type of complaint: some of the recipes use metric measurements (e.g. ml) while others use Imperial measurements (e.g. cups). Even worse, some recipes have been carelessly modified and amended by different users, and contain both types of units side-by-side. Everyone knows this is an insurmountable problem in the baking world, and so your chefs demand action from you.

Your solution is this: you will write a small piece of code that will allow every user to set a preference for the type of unit they’d like to see. Then, your code will dynamically convert text into the correct measurement units as each page is displayed. Sounds perfect, no?

Your code does this by looking for text that matches the pattern “<#> <unit>”, where <#> represents one or more digits, and <unit> can be one of: ml, g, cups, tbsp, tsp, ounces, oz, etc… Your program also maintains a conversion table, so that it can automatically perform the proper arithmetic operations on the number part, and re-label the <unit> part:

Example Conversion to Metric

Example Conversion to Imperial

By and large, this works well, but aside from giving sometimes awkward results (no one measures 1.05 cups of anything!), you notice that there are certain cases where a measurement is detected incorrectly:

Example of Faulty Conversion — The “G” from “Graham” was incorrectly detected as meaning “Gram”, triggering a conversion to “oz”.

Beyond this fairly simple example of regional idiosyncrasy, you notice that there are other places where the international makeup of your wiki’s contributors becomes evident. “What the heck is a Graham cracker?”, asks a British chef, while on the other side of the Atlantic, an American wonders “What on earth is a digestive biscuit?”. In reality, both are talking about the same food under two different names. Due to this kind of regionalism, two wiki pages describing the same concept are created, both containing much of the same encyclopedic information. Worse, twice the effort must be made to maintain two separate pages. You shed a silent tear upon realizing this, as you only ever wanted to break down gingerbread walls, not put up new ones!

Already disheartened, you then start noticing something profoundly more troubling: your wiki software is stupid! I mean fundamentally stupid. It knows a lot about baking, but given a connection between your wiki and a baking robot physically able to carry out any type of baking task, you still couldn’t make strudel one! All of the information is trapped in hypertext form, and although that turns out to be a very expressive format (unlike metadata), the tradeoff is that it’s very difficult for a computer to make sense of the content (unlike metadata).

Even operations like halving/doubling a recipe, which seem algorithmically simple to perform, need to gain quite a deep understanding of language before performing proper recipe adjustments:

Faulty Arithmetic — Here we see how simple arithmetic conversion of recipe excerpts can lead to wrong results. This conversion is naïve, meaning it has no semantic understanding of what it is doing.

You have created an encyclopedia with a lot of information, sure, but that information is of no use to anyone but humans. For instance, you can’t ask a computer to perform a simple series of operations on a recipe (e.g. halve it, count the calories within that, estimate the cost of ingredients, add ingredients to my phone’s shopping list app, etc…). Even though the information you ask for might be entirely within the pages of your wiki, the software has no semantic understanding of the content. It does not understand the concepts that are contained within, it only captures text that users store and revise.

Denouement

At first, you were young and foolish. You thought metadata was going to make you a quick buck, and so you used it to catalogue your music. You never expected that it would be you who would pay the price for using such a non-expressive framework. Once you had licked your battle scars, now older, wiser, and more determined, you turned to hypertext, way on the other end of the expressivity spectrum. Your baking wiki became an international success, except for a few rough edges. Yet here you are, standing at the top of Mount Olympus, feeling… hollow. Sure, you have collected and organized a vast amount of information, but in doing so, you’ve lost the help of your digital servants. Now, your encyclopedic baking wiki requires non-stop curation and maintenance by humans to keep it relevant and accurate. Not only that, software is unable to understand the information, meaning you are using the computer as a glorified book.

“Be careful what you wish for, because you might actually get it,” they always warned you. Atop the mountain, looking out, and feeling empty, you experience a moment of clarity as you understand what that saying means. Calm washes over you. What more is there to do when you’ve climbed the mountain, but sit and meditate?

After failure as a record store owner, and success as a bakery wiki editor, you are finally ready to begin your next life, one of synthesis and a holistic understanding of data. With that, you become the Monk…

Better Living Through Artificial Intelligence

I believe that the Semantic Web has the potential to radically transform our lives, as it ushers in a new age of artificial intelligence that we’ve so far only seen in science fiction movies. As its adoption grows and spreads, the information contained within will touch every aspect of human life, including education, medicine, popular culture, transportation, politics, history… and beyond.

In this series of blog posts, I will attempt to illustrate how it works, and give a glimpse into the bright future of information technology. This first post will focus on one of the deep problems that the Semantic Web solves.

[The Semantic Web will create] a Web in which machine reasoning will be ubiquitous and devastatingly powerful.

— Sir Tim Berners-Lee (1998; The Semantic Web Roadmap)

 

Hyperdata

The Semantic Web proposes a new paradigm for organizing information known as Hyperdata, which will be key to the evolution of information technology. Hyperdata is the intersection of metadata (information about information) and hypermedia (information linked to other information). I’ve written about metadata and hypermedia in previous blog posts.

Metadata is excellent at expressing facts about existing information. Examples include the date that a blog post was published, and tags that describe the contents of a file. Metadata is useful in search contexts: the more information that is available about a data object, the more ways that exist for searching and finding exactly what you’re looking for.

At the same time, metadata has a distinct shortcoming: it is only descriptive in a very narrow sense. An example will illustrate this point.

The Record Store

Suppose you have a small collection of music that you want to sell online. In order to inventory and track your collection, you opt to create a catalogue listing everything you have for sale, and display it online so that visitors to your website can shop for and find exactly what they’re looking for, or browse through lists of what you have available.

As a first step, you create a spreadsheet with some basic columns such as “Title”, “Artist”, “Year” and “Format”. In the vast majority of cases, these four aspects of any given music album are enough to give potential customers a very good idea of what they will be buying. Most shoppers looking for a specific item in your store will be able to find exactly what they want by searching on one or more of these four criteria.

So you create a blank spreadsheet that looks like this:

Perhaps unknowingly, you have chosen a metadata framework for cataloguing your music for sale. Each row lists an item on offer, while the four columns each provide information about those items.

Your first catalogue entry is R.E.M.’s seminal “Lifes Rich Pageant” (sic.). You have an album on vinyl as well as one on CD, and you want to sell both. So, you make the vinyl version the first entry in your catalogue:

So far so good.

Next, you want to add the CD album, the 25th anniversary reissue of “Lifes Rich Pageant”. So, you add a second entry to your catalogue:

Already, you’ve started to become aware of three problems with the spreadsheet you’ve designed:

1. There’s no “Edition” column, so in order to distinguish the two albums, you have to cram more explanatory information into the title column. However, R.E.M. never released an album entitled “Lifes Rich Pageant (25th Anniversary Reissue)”. These are merely two editions of the same album, “Lifes Rich Pageant”. If a customer were to search the website for albums with the exact title “Lifes Rich Pageant”, he would only see one result, the LP. That customer might not have a turntable, might have been very happy to buy the 25th Anniversary Reissue on CD, but unfortunately didn’t find the CD version in his search. The metadata framework and its lack of expressiveness has cost you a sale.
2. The same problem exists in the “Format” column. By writing “2CD”, you meant that the reissue is actually a double CD, even though, strictly speaking, there isn’t a recording format called “2CD”. As you grow your catalogue, how will you indicate a triple CD? How about a concert album that includes both a CD and a DVD of the performance? What about a box set? Further, is it a vinyl box set? A cassette box set? A CD box set?
3. When creating your spreadsheet, you assumed that the concepts of “Recording Year” and “Release Year” were the same, and so you only created a single “Year” column. Again, your spreadsheet has proven to be lossy in the way it catalogues the music, and so you have had to make a choice that “Year” actually means “Release Year”.

Discouraged but undeterred, you add a third entry to your catalogue, a compilation entitled “Atlantic Rhythm and Blues: 1947-1974″:

Again, you notice problems with your spreadsheet:

1. This third album, a collection of Atlantic Records’ greatest hits spanning the years 1947 to 1974, was released in 1991. But imagine a shopper who says: “I’d love to find soulful bluesy music from the 50s or thereabouts”. Although this third album in your catalogue would fit the bill perfectly, it would never turn up in a search for “1950″, since that number appears nowhere in either the Title or the Year columns.
2. There is no artist named “Various Artists” — this is simply a way to express “There is no single artist for this album”. Sadly, the customer who comes to your website, searching for a Big Joe Turner song she heard on the radio, will be disappointed because none of her search queries for “Big Joe Turner”, “Midnight Special”, or “1957″ will yield any results, despite the fact that it describes precisely something that you have and are selling!

(The first problem is not insurmountable. You could write some computer code to make the search engine a little smarter: “Assume any search for a number refers to a year, and include in the search results any item in my inventory that has a range of years (two numbers separated by a dash) which encompasses the sought-after year”.

With this code, a search for “1950″ would successfully retrieve “Atlantic Rhythm and Blues: 1947-1974″.

This code adds a little intelligence to your search engine, but not much, because the search engine has only been instructed to recognize years in a very specific format (two numbers separated by a dash). This logic would not work on a title such as “Glittering Prize 81/92″ (by Simple Minds), since the two numbers are not separated by a dash, and would lead to puzzlement when Rod Stewart’s box set “The Great American Songbook (Vol. 1-5)” appears in the results of a search for Violent Femmes’ album “3″.)

As a result of your non-expressive framework, you lose yet another sale. Disenchanted with metadata, you decide to shut down your music store and start a bakery instead. (I have to say, you give up pretty easily!) Because of metadata’s lack of scope, many customers have been disappointed, and an entrepreneur’s dreams have been crushed.

The Shortcomings of Metadata

To summarize, metadata is excellent at giving information about information, as long as that information all fits neatly into a grid. This is acceptable in the world of computers where objects such as files and blog posts are strictly defined, but human life rarely fits neatly into a grid.

To fix the problems with metadata that are described above, you’d have had to add extra columns every time you encountered an exception (e.g. renaming the “Year” column to “Release Year” and adding an extra “Recording Year” column). This is very easy when your music catalogue contains 3 items, but it doesn’t scale well. As your catalogue grows, more and more exceptions will be encountered; each time a new column is added, or an existing one is renamed, every entry within has to be checked and edited, to make sure the information is accurate and error-free. Try doing this for a music catalogue the size of Amazon’s or GEMM’s!

This is not a purely academic problem, either. Metadata’s lack of expressivity is found in everyday life. Anytime you fill out a form, you are entering information into a grid. If you live your life in a normalized, structured fashion, this is probably fine. If, on the other hand, you have stories to tell, you’ll likely spend some time explaining why your life doesn’t fit into the grid.

As someone who recently relocated to the US, I can testify to this phenomenon first-hand. It is difficult to sign up for something like cable internet service, for instance, when you don’t have a credit history or a Social Security Number. You usually end up leaving those parts of a form blank and/or calling a helpdesk. If you’re lucky enough to get through to a customer sales rep or government official who can help you, they still usually need to ask around to find out how to handle this exception (you) in their system, and a lot of confusion and difficulties may arise as a result.

In the next post, I will examine hypermedia, which proposes a different, much more expressive way of codifying information. It also has non-trivial shortcomings, which is why I’ll conclude by looking at hyperdata, as the intersection of metadata and hypermedia, and describe how it proposes a solution to both problems.

HyperCard, as the name suggests, was based on the idea of stacks of cards, hyperlinked to one another. Think of it as a stack of index cards that you could draw on and annotate with pictures and text. Just like the Memex, you could then use the power of hyperlinks to link cards together so that, for example, clicking on a picture of a dog might bring you to a card with more information and pictures of that dog, with further links to other cards, etc…

I started using HyperCard at a young age, and I remember finding it very frustrating at first. I was used to painting on blank 2D canvases such as the ones in MacPaint. In contrast, when I opened HyperCard for the first times, I found the terminology confusing and foreign. I would get annoyed when, after creating a graphic masterpiece, I would hit a key combination to jump to a new blank card, and see my picture disappear before my eyes.

My world was shaken when I finally got the idea of stacks and cards. It was like all of a sudden, I was able to paint and draw in 3 dimensions rather than 2. At first, I experimented with short stories which would start on the first card, and continue from that card to the next.

The HyperCard 2.0 Home Stack — The home stack was a kind of launch pad into HyperCard. You could start your HyperCard experience through tutorials, by seeing example stacks (such as a phone dialer or address book), or by diving-in to creating your own. There were even customizable tabs, for adding links to your own stacks. Incidentally, this home screen design can be seen in modern products such as the iPhone today. (Image used with kind permission from Robin Silberling)

Shortly after, I discovered that it was possible to make flip-books in HyperCard as well. I would draw a picture on a first card, then make the image on the second card slightly different, and so on. I remember creating countless animations, some of them becoming quite sophisticated, with cut scenes and longer runtimes.

In addition to being able to draw and add text to each card, you could also place a variety of buttons onto each card. In their simplest forms, buttons would skip to another card or play a sound. When I discovered buttons, I started making adventure mini-games. In one game, you started standing outside a house. By clicking on the doorway, you could open the door and step inside the house. Each room had doors leading to other rooms, and so on. All of this was done by drawing pictures of indoor rooms on different cards in a stack, and then using buttons to interlink them.

I experienced another quite profound paradigm shift when I discovered what lay at the heart of buttons: HyperTalk. While Bill Atkinson had written the core of HyperCard (stacks, cards, drawing & painting, etc…), his colleague Dan Winkler had been busy developing a programming language to give stack authors powerful ways of adding extra functionality to their projects. He called this programming language HyperTalk.

The best part about HyperTalk was that it was very intuitive and approachable. Many programming languages tend to look exceedingly cryptic to the uninitiated, but HyperTalk was constructed so that it read mostly in plain English:

on mouseDown
  put "Hard Drive:Documents:Client Contacts" into filePath
  if there is a file filePath then
    open file filePath
    read from file filePath until return
    put it into card field "Client Contacts"
    close file filePath
  end if
end mouseDown

Since HyperCard shipped with a number of demo stacks, I was able to jump in head first, and teach myself HyperTalk very quickly through experimentation and by copying-and-pasting bits of HyperTalk script from other stacks into my own. With every new bit of HyperTalk that I taught myself, I felt like I was pushing my creative boundaries further and further.

Over the next several years, I developed an insatiable thirst for mastering HyperCard. I would rarely do any homework, as generally my bag would get dropped onto the ground with complete disregard as I bee-lined from the front door to my Mac Plus. My experimentation with HyperCard was vast, and the number of stacks I created must have ranged well into the thousands. Among others, I remember creating a Wheel-of-Fortune type game, a trivia game, a maze game, a visual dictionary of wind instruments, a bulletin board system (!!), innumerable Choose-Your-Own-Adventure type games, programs that drew pictures for you, chess puzzles, and the list goes on. And on. And on.

By now, I was fully sold on the merits of HyperCard. Birthday and Christmas gifts would come in the form of new versions of HyperCard or books about advanced programming in HyperTalk. Wanting to evangelize the creative possibilities of this program, I started a computer club at my (primary) school, and (starting in 1995 at age 12) worked for two summers as an instructor in a computer summer camp. There, I met a fellow instructor who was as passionate and savvy with the ‘Card as I was, and we wrote together what I still consider, to this day, my HyperCard Magnum Opus: a battleship program. After the kids had gone home for the day, we’d wire one of the computers to the projector, and we’d take turns programming and reviewing the other person’s work on the projector. By the end, our battleship program became a fierce challenger, with 5 levels of artificial intelligence, each progressively more intelligent than the last. By level 5, the AI would flat-out cheat and look at your board to see your ship positions.

During the day, I taught kids the basics of HyperCard. Some kids chose to make simple flip-books and adventure games, while some of the more advanced learners got into more ambitious projects. I particularly remember working with one boy who was autistic. Stacks and cards were not concepts he was fully able to grasp, but he nevertheless created drawings in HyperCard that showed a lot of talent. The kind of creativity that HyperCard enabled, and which was visible in the kids’ projects constantly surprised me.

From the Memex, to HyperCard, to Wikipedia — the evolution of hypermedia.

By the end of the second summer, I had become a HyperCard guru. By this time, the World Wide Web had come into being in an early form. Not content with simply advocating HyperCard on a local scale, I created a website dedicated to archiving and organizing snippets of HyperTalk code, for the larger community of power users like me. I was the chief contributor.

HyperCard was born in 1987, before the internet and before even colour Macintoshes. By the mid-90′s, HyperCard still didn’t have true internal colour support, and had fallen into disrepair following organizational problems at Apple. In 2000, what little development efforts were being made were ceased, and in 2004, Apple stopped selling HyperCard altogether.

Having learned everything I could about HyperCard, I gradually shifted my focus to other programming languages (like the ones I was using to build the website to promote HyperTalk).

HyperCard was a killer app, and a testament to great software design. The best part about using HyperCard was that it was fun. And yet, it constantly challenged you to explore a little deeper, to try something new. I have not found a single other creative application that was so thoroughly rewarding to use and learn. Sadly, I don’t even remember the last time I used HyperCard. It won’t run on any modern computer, but just in case I get nostalgic someday, I’ve kept my Mac Plus safely stored with a handful of stacks not far away…

He spoke again recently at TED2010, about augmented-reality maps. Reality augmentation is the idea that technologies could come and “enhance” our senses, by adding pertinent information about things around us. Using the pioneering work of Photosynth, he demos a Bing Maps application which juxtaposes information and images of all kinds into a 3D map. Truly jaw-dropping!