Fluidinfo on Blog | Jonas Neubert

Fluidinfo API: Likes & Dislikes

Sun, 24 Apr 2011 00:00:00 +0000

I spent the last two weekends[1] working with the Fluidinfo API (here is why). Using Fluidinfo is pretty easy, I found two weekends enough to discover most of what is possible with Fluidinfo.[2] Here’s my collection of likes and dislikes which accumulated while working with Fluidinfo. In the spirit of a true pessimist, I start with dislikes.

Dislikes

Among the very first things I learned while playing with Fluidinfo: You can delete your root namespace but you can’t re-create it. If you sign up with the username skillshelv.es your root namespace will be skillshelv.es (big surprise) meaning that all your tags will be in the format skillshelv.es/tagname or skillshelv.es/subnamespace/tagname. Because you have the ability to manage sub-namespaces under your root namespace, Fluidinfo exposes the ability to delete namespaces. When I first tried out the Python Library, the first thing I managed to do successfully was to delete my root namespace skillshelv.es. Trying to recreate it results in a permission error. That’s why all Skillshelv.es related tags now live in the alternate namespace skillshelves which I obtained after creating a second account.
There is no results pagination and no way of limiting the result count. The heading of the docs chapter on query results limits sounds promising, but only talks about hard limits imposed by Fluidinfo, not limits set in the query. Try querying for the titles and author names of all O’Reilly books stored in Fluidinfo. You get 2373 book titles and their respective authors, totaling half a megabyte of data. If you only want to show the first ten, you’ll still have to query all of them.
Fluidinfo (sometimes) is slooooooooow. The query mentioned above takes somewhere between 8 seconds and forever to return results. That’s a long time if you know that your user is waiting for a page to display. It’s an even longer time if you are using Google App Engine: GAE has a hard limit of 10 seconds on how long an outgoing HTTP request may wait for a response. On Skillshelv.es I work around long load times by running all Fluidinfo GET queries in Javascript. That way I can display a loading indicator while the Fluidinfo data is loading. This gives the user something to see and shifts the blame to Fluidinfo :-) More generally, though, it means that Fluidinfo shouldn’t be the datastore you query when generating pages. A potential solution might be allowing to limit the result count (see #2) which might reduce processing time on the Fluidinfo side and reduces the amount of data to be transferred.
Object IDs can’t be used in queries. It’s easy to get a list of all tag names that are attached to an object. It’s not so easy getting the values of all those tags. Fluidinfo only provides a clear method to retrieve exactly one tag value associated with a specific object at a time, e.g. the title of a certain O’Reilly book. If you want multiple tag values at the same time, you’ll have to use the /values API endpoint, like in this example which gives you title and price of all O’Reilly books whose title contains “PHP”. Fluidinfo gives you a pretty neat query language to define which objects are included in the response for a query to /values, but querying for an object ID isn’t part of that query language. Therefore, you can’t retrieve multiple tag values for a single object at a time, unless you already know one tag value that uniquely identifies this object. For O’Reilly books the contents of oreilly.com/id seem to be unique and Skillshelv.es uses it equivalently to a primary ID. Update: I mentioned this in the Fluidinfo IRC channel and apparently this is now a ticket and might become a feature in future.
While this Fluidinfo blog post suggests that Fluidinfo’s matches operator for text matching works like MySQL’s LIKE, it doesn’t actually. matches only matches full words. That’s not unsurprising but was a little disappointing when I tried building an auto-complete feature that pulls data directly from Fluidinfo.

Likes

First class user support! When I reported #1 of the dislikes above via Twitter, Fluidinfo’s CEO replied within a few hours acknowledging the issue. Later I figured out that most the action is going on in Fluidinfo’s IRC channel where the folks behind Fluidinfo seem to always be hanging out waiting for people to approach them with their problems. When one of my (valid) queries wouldn’t work one day, the problem was fixed in no time and on top I got a ton of useful advice.
Fluidinfo Explorer. This tool didn’t seem very useful to me at first but then I discovered the context menu and the login button :-) Once you have some data in Fluidinfo and you want to quickly check or change a few entries, nothing beats this tool.
There are ~~a million~~ lots of ways to access Fluidinfo. Libraries are available for Adobe AIR, Clojure, Go, Java, Javascript, Lisp, .NET, Perl, PHP, Python, Ruby and Smalltalk. And Fluidinfo supports JSONP and CORS. It’s really nice to be able to hit the ground running in whatever language you like best. Compare this to some big companies who have tens of APIs, each with libraries available in another random selection of two or three languages.

Footnotes:

[1] Like every good grad student I did not devote the entire weekend but also did some ~~real~~ research work on these weekends.

Fluidinfo API Contest: Design Decisions

Mon, 18 Apr 2011 00:00:00 +0000

I announced yesterday that I am currently working on a submission to the O’Reilly/Fluidinfo API contest. You should check out the other submissions so far:

BookChirpa by Mark McSpadden is a data mashup project showing recent tweets about O’Reilly books enriched with information about these books which is pulled from Fluidinfo. There is an About page with a whole bunch of technical info. Ironically, the top entry on the Bookchirpa library page is “21 Recipes for Mining Twitter”.
David Karapetyan opts for a submission which only adds data to the Fluidinfo data store but doesn’t have a frontend. That’s legit by the contest rules. Here’s a blog post about how he added Amazon Suggestions to each book. 90% of his post are crossed out indicating that the story behind his submission isn’t exactly linear.
Eric Seidel’s submission is really similar to Skillshelv.es. Shockingly similar. Read about it in his blog. By the looks of it he’s been a Fluidinfo aficionado before because his prior project FluidCV is based on it. His submission to the contest is to let people add O’Reilly books to their FluidCV to - hold your breath - show their skills based on the books they own. Disclaimer: I hadn’t seen his project when I started working on Skillshelv.es.
Michael Hawes noticed that the books which Fluidinfo imported into their database are only those English language ones sold in the USA. But O’Reilly also publishes books in other languages in the small part of the earth known as ROTW to Americans (the acronym stands for both Rest of the World and Rim of the World). He embarked on the mission to add links to the foreign language equivalents to all existing Fluidinfo book objects. Like David K.’s entry, Michael’s has no front page, but extensive documentation on his blog.
Rachel Willmer wrote an extension to her eBook price aggregator Luzme.com to send up to date eBook pricing details into the Fluidinfo data store. She writes that she will only sync this data for the duration of the competition but I sure hope she continues to do so after the deadline on Sunday night. It strikes me how open everyone is about what their entry is and how they went about creating it. So I won’t hold back either and write up the details on my submission, predominantly in this blog and maybe later on as part of the Skillshelv.es site. Today, let’s talk about the design decision that had to be made before I started touching my keyboard.

Quick Recap

Here’s the quick technical summary of what my submission will do: Each book will be tagged with one or more skills representing the expertise conveyed in the book. Each book will also be tagged with usernames to represent ownership. Based on this info, users’ skill levels and some other derivative data will be computed.

Hello Fluidinfo API Competition

Sat, 16 Apr 2011 00:00:00 +0000

How does a small startup get the word out about their API? The people at Fluidinfo probably thought that a full page ad in the New York Times is a bad idea. They decided to run a contest instead. If things go well, contests are great because tens or hundreds or thousands of people (“contest participants”) create value according to your specification but you only have to pay a few of them (“lucky winners”). If all goes wrong only a few people create very little value and you have to pay all of them. That’s really all there is to know about contests.