For many needs that I have, I'm not like that. I like to minimize the tools in my belt, stick to them and utilize them. I had been using Bloglines to read blog feeds for many years, until it was shut down/taken over, and I was forced to move. I switched to Google Reader. Now, I'm in a similar boat concerning my bookmarks.

For years, I have been using Google Bookmarks. It's a little-known service, and it is very basic:

• From the Google toolbar, you can access your toolbars in any browser for which you can install that toolbar. You can also add a bookmark that way.
• There's no hierarchy in bookmarks, just "labels", which work like tags. Good enough for me.
• Alternatively, bookmarks can be browsed, searched and added from the Google Bookmarks web page, making your bookmarks accessible to devices and browsers on which you do not have the toolbar.

It's not great, but it works. Until Google in their relentless cleanup momentum decided to drop support for the Google toolbar, as of Firefox 4 I believe. It continued to work for a while, but not anymore.  I can't access my bookmarks from the toolbar anymore, nor can I add new ones.

Unwanting to deal with the problem, I went even more basic by accessing my bookmarks from the webpage, which is a bit painful. Until adding new bookmarks started to fail a few days ago there as well. These are enough signs to get the hell out of the service.

Opportunistic as I am in such unwanted tasks, I was looking for the least painful way to get similar functionality, my requirements to be summarized as follow:

Accessing and adding (private) bookmarks easily from any device, browser or location

Oh, and for free of course. I eventually settled on Delicious, which lets me import all my exported Google Bookmarks in one go, auto tag them and set them to private. That's all I wanted, and it took less time than writing this blog post. Such a relief, as I was thinking the conversion would take me precious hours. 

As a final note, I want to add that due to search and social networking, the importance of bookmarks has decreased for me. Still, I do have a lot of work-related bookmarks that cannot be retrieved or remembered easily, as they are not searchable.

The above meme sums up what I'm currently doing in the JungleDragon area. I've been demonstrating basic specie identification and visualization earlier. That part is fit for production. The back-end of the so-called specie engine isn't yet. I'm hardening it as follow by testing my own photos sets:

1. Identify specie on photo.
2. See if it is found on Wikipedia, if not, investigate the problem, try to fix it, back to step 1.
3. If found, parse it. If a problem occurs, investigate the problem, try to fix it, back to step 1.
4. Success. For this one specie that is.

Each successful identification makes the specie engine better and more reliable. Currently, the fail rate is still quite high, too high to deploy this, so I'm thinking it will be a few more weeks before I release it. Here are some of the more common problems I encounter:

• There simply isn't a page for the specie at Wikipedia. Can't be fixed.
• Varieties in casing and spelling not resolving to the intended Wikipedia page. Problematic, as sometimes Wikipedia has hard redirects for such cases (which I can resolve), yet sometimes it doesn't. This is quite a big problem for which I have yet no solution. For example, "Milky stork" will not work whilst "Milky Stork" does. There is no all-uppercase convention either.
• My parser engine choking on Wikipedia content due to an unexpected format. This problem will shrink over time as I test more specie pages, yet it does make the engine a lot more complex. For example, last week I discovered there is a third way to create a specie taxonomy box. I was prepared for "taxobox", "autotaxobox", but it seems there is also "speciebox". All these templates do the same thing yet are to be parsed differently and there is no consistency at all as for which one is used. 

So the feature is forthcoming within a matter of weeks. It will be awesome, but not perfect upon release. It will always have some degree of problems, which I'm hoping to minimize over time.

Yesterday I explained my progress concerning specie integration in JungleDragon v2. Much to my delight, I received a few very positive comments. I need those to fuel my motivation. This part is hard, complex work with constant setbacks.

Anyway, I also explained I was playing with the idea to automatically integrate Youtube videos with the specie page. Quite a simple, yet powerful way to see a specie in action. My theory was to simply use the Youtube API and to pass it a search string, in my case the name of the specie.

You can see above how this is not going to work. The Impala, other than a specie, is also a car. A Polar bear, other than a specie, is also a cartoon, dance event, and part of the song title of various music tracks. In these two simple test cases, about 90% of the search results have nothing to do with the specie.

So this is a failed experiment. Funny, though: Henriette asked what I was doing and I explained the problem to her. She suggested to use the scientific name as a search string. As this was a 5 second code change, we could instantly see the improvement in relevancy: up to about 80%. Much better, but I'm thinking it's still not good enough. I then had the task of explaining to my beloved one how brilliant her remark was, yet how I was going to ignore it anyway. I'll spare the details. 

How to move forward in this area? I still believe in the power of video to enrich specie information, so here's my plan:

1. Ignore it for now and focus on testing what I got and bringing it live. This feature is non-essential for a first release of the specie identification feature.
2. Once step 1 is stable enough, solve the video challenge as follow: allow users to manually add videos, simply by entering a Youtube or Vimeo URL. Not that hard to implement, much greater relevancy and quality of videos.

What do you think?

Here's a photo uploaded to JungleDragon, of which the specie is identified as a "Polar Bear":

As a result of this identification, an interesting side panel appears. Let's have a closer look at it:

This essentially is a summary of this specie, with the following data visualized:

• Common name of the specie in the header 
• The first 500 characters of the description of the specie, wrapped around the photo of the specie that has the most karma in JungleDragon
• An interesting visualization of the IUCN conservation status and trend of the specie. You should really compliment me on this one as I wasted 6 hours styling it.
• The 2nd, 3rd and 4th image of the specie in JungleDragon, sorted by karma

There's a lot more that JungleDragon knows about this specie. To learn more about it, there's the obvious button "Learn more about this specie". Clicking the photo of the specie will also you bring you to the specie page, which currently looks like this:

So this is the dedicated specie page, which tells everything JungleDragon knows about it. This is only a fragment of the page, in reality it is about 10 times longer.

First, we start with a large header containing a larger version of the most popular image, as well as the full description of the specie.

Below that are a set of dynamic links, the ones with the arrows pointing down. These represent the areas of which JungleDragon has information about the specie. For example, rather than scrolling the entire page, you can instantly jump to "Diet" and from there go back to the top. JungleDragon does not copy all text blocks from Wikipedia, only a select few. 

The left column below the header contains the actual text blocks. These come from Wikipedia, yet they are transformed so that it becomes natural language with various Wikipedia elements removed (such as links, citations, etc). Another interesting element to the text display is that it is mixed with JungleDragon photos, not Wikipedia photos. You'll notice the first text block containing a left-aligned photo. The next text block will have a right-aligned photo. This keeps going until we run out of text blocks or photos. It makes for an attractive format of reading and an interesting way to mix Wikipedia content with JungleDragon content, or so I think.

Moving on the right column, we have a "Distribution and status" block which contains an image of this species' distribution (if available on Wikipedia) and again the IUCN status, richly illustrated.

Below that is the taxonomy block which classifies the specie in the biological "tree". As you can see, the elements in the tree currently are not links. Initially they will not be, even when I bring this live. If I can, I will implement this though. It would allow for tree-like browsing. For example, you could jump to a page of the family "Bears", to see all bears, not just Polar Bears.

So for now this concerns the overview tab of the specie page. There's also a photos tab:

It does what you expect: it shows photos of that specie, and you can sort them by popularity or upload date. 

The third tab "videos" is not implemented yet. It's just an idea I'm having. The idea being that I use the Youtube API to shows videos of this specie based on a search string. I will give this a try to see if I can get it to work reliably.

There's about 10 other things to implement to fully implement the v2 vision of JungleDragon, but this teaser definitely represents the core functionality. Your feedback is highly wanted!

In the last few updates concerning JungleDragon, I mentioned how I'm working on the specie engine, the part that integrates specie information of Wikipedia with JungleDragon photos. None of this is live yet, so you can't see it. Neither was there any development UI to demonstrate, it was just me complaining how tedious it is to get structured data out of Wikipedia. 

That is still true, and my struggles in that area continue, but hereby I do want to share some first UI work of the specie engine. The scenario is simple: you have uploaded a photo and are asked to identify the specie on the photo/ For that there is an "Add specie" button, which brings up this dialog:

Since multiple species can appear on a single photo, you can add more than one specie, yet you add them one by one. As the dialog states, you can search both by common name (i.e. "Polar bear") as well by the latin name (i.e. "Ursus Maritimus").

As you type a specie name, the list will help you using suggestions. These suggestions concern species known to JungleDragon. This means they are used before. I do not have a database with all species. Instead, as you add a specie not known to JungleDragon, it will be a known specie from that point on. 

What makes a valid specie? Here are the current rules:

• There must be an english Wikipedia page for your query, or a redirect to such a page
• That page in particular must be a specie page, meaning:
  • It has the "taxobox" on the right
  • It has to be a specie or a subspecie, meaning it has either the "binomial" or "trinomial" name property. For example, "Bear" is not a specie, but "Brown bear" is.

Ok, given that you entered a valid specie name, one of two things will happen:

• If the specie is known to JungleDragon already, it is instantly associated with a photo.
• If the specie is not known to JungleDragon, yet it is a valid specie, I will parse it from Wikipedia in real-time, which takes a few seconds. A loading indicator will make this clear. From that point on, it is a known specie to JungleDragon.

So, that's how the "Add specie" dialog works. It's how you identify a specie on a photo. Once I know that relationship, I can visualize rich specie information next to such a photo. Here's a very early preview:

Check out the sidebar on the right. This photo of an Impala has been associated with the specie Impala, and as a result, it shows the common name, binomial name, description, and range map. 

Be aware that this is just a simple start. I have a lot more data about the specie and I can also visualize it any way I like. Take note of the concept though. This is where JungleDragon v2 is all about. Instantly learning about what is on the photo. And of course, later on you can click through on the specie name which will show a full page with everything there is to know about it.

Wiki parsing engine updates

I need to reserve some room in this post once again for self-pity. To complain about parsing Wikipedia. The overall complaint is that each time I extend my test set of specie queries, I find new problems, new ways in which Wikipedia pages are structured, that my engine cannot handle yet. It's one step forward, two steps back. Here's two recent situations:

• I've been relying on the taxobox on a specie page to parse the species' taxonomy. Finally I had my engine robust enough to deal with the unlimited ways in which that taxobox can be structured: levels in the taxonomy can be there or not, the amount of levels varies, the spelling of levels varies, the value of a level can be plain text or contain any Wiki markup. Until I discovered yesterday that some specie pages do not use a taxobox, they use an "automatic" taxobox, a complex variations based on specie keys.
• Another problem as a result of testing is this. Say you'd have a photo of an African Elephant. In the "Add specie" dialog the instructions are clear: "Elephant" will not work as it is not a specie, and thereby not specific enough. So you try "African Elephant". You'd expect this to be specific enough, unless you're a zoologist. See, in this case, even "African Elephant" is not enough. It's not a specie, instead "African Bush Elephant" is the correct specie, according to science and according to Wikipedia. But probably not according to you. In these situations, I have therefore implemented a routing mechanism. It's a manual table that I maintain in which I map a source, in this case a commonly failing yet well-intended query, into a valid target specie. So if you'd type "African Elephant", I will map it to "African Bush Elephant" for you. Over time I'm hoping this table will give you a better chance at the result you expect.

Without a doubt, there will be dozens more problems coming my way. But I will persist through them, because I dearfully believe in the concept. No matter what it takes, this will get done, and it will be done right.

I love Google Analytics. It is astonishing to have such an incredibly advanced statistics tool at one's disposal for free. I also believe most owners of small to midsize websites do not get the most out of it, I'm certainly guilty of that. That's why I first want to briefly revisit two earlier posts concerning features you may not expect in Google Analytics:

Measuring your site's speed as seen by your visitors

Change a single line of code in your GA tracking javascript and GA will then sample your site's loading speed. This is a big deal. You can easily see how your site performs based on different bandwidths, locations, browsers, any dimension you like. There's enterprise solutions charging you tons for such a service. In GA it's free, and all you need to do is to add a single line of code.

In-page Analytics

My absolute favorite. We know we have tons of metrics available in GA, but its hard to bring them down to meaningful conclusions. In-page analytics changes that. You will see your page as you designed it, and visually attached you see the click-through data, amongst other metrics. 

This is the first-ever easy way to test your design. Recently I wanted to redesign parts of the navigation of a site, yet I was worried that users would be confused, as some options would be relabeled or removed alltogether. Until I learned that option in question was hardly ever used at all. It is quite powerful to make design decisions based on data, rather than gut feel, personal preference or emotion. This also seriously strengthens your position amongst stakeholders, in case you have to defend a design decision.

Real-time analytics

Ok, so those were the earlier posts. They are game changers, please check them out in detail.

The new one, which actually has been available for several weeks now, is Google Analytics realtime. Where all of GA's former metrics had data with a delay of about one day, realtime analytics show you those metrics as they occur. Perhaps you tried to launch a viral campaign in social media, this way you can check the actual effect as it happens. You can also learn about traffic patterns related to timezones. 

The opening screenshot shows a glimpse of the realtime view, with the currently active visitors, their locations, target pages and more. Truth is to be told though, this really is only interesting in case you have critical mass, meaning a large website with many visitors.

As you may know, I'm currently working on the integration of specie data from Wikipedia into JungleDragon, as part of JungleDragon v2. It is hard, complex, invisible work. But I'm making great progress. There's no UI to demonstrate yet, but allow me to update you on the progress of the back-end.

Finding a specie:

I can now do this:

$specie = $this->ZoologyManager->FindSpecie("Impala");

Which will result in:

• A check if there is a Wikipedia page for this entry
• If there is an entry, yet it is a Wikipedia redirect page, I will resolve the redirect
• Assuming the page exists, a check is made to see if it is a specie page:
  • It has the taxonomy box to the right
  • The page is at the taxonomy level species or subspecies. In practice this means the property "binomial" or "trinomial" name must be present. Anything less specific is no match.
• All checks passed, parse the specie page. This is the most complex and fragile part of the process:
  • Parse the taxonomy information. Implemented in a flexible, forgiving way since multiple taxonomy systems exist and are used, the amount of levels varies between pages and even the naming of properties varies between pages. 
  • Parse the rest of the page, which can contain any Wiki markup. I'm intelligently trying to find selective blocks of information by its headers, next I parse, strip and transform this content in a way I find suitable for JungleDragon storage and display. This is a best effort approach, but it's working reasonably well currently.
• Normalize the parsed elements of both the taxonomy information and the free format text into a specie object that JungleDragon understands.

This is a long explanation of saying that with a single specie search string, I get back a structured object of that specie, containing very rich information. That's quite a powerful API.

Saving a specie

Assuming we found a valid specie, next I can do this:

$this->ZoologyManager->SaveSpecie($specie);

Which will result in:

• A new specie record being created in my database, if it did not exist already. This record contains both the taxonomy information of that specie as well as the normalized text blocks.
• In a seperate table I will update specie synonyms, for example, there can be multiple search strings matching a single specie. "Polar Bear" and "Ursus Maritimus" are the same specie, and I maintain that relation to avoid duplicate specie records. 
• If the specie record contains any specie classifications not yet known to JungleDragon (division, kingdom, class, order, family, genus), it will save it in a seperate table. This table is loosely designed, given that the amount of levels per specie is not consistent in Wikipedia, as well as there not being any guarantee that keys match exactly by string.
• A very cool one: If the Wikipedia specie page has a range map (a graphic showing the distribution of the specie on a map), I will download it, associate it with a specie, and then transfer it to Amazon S3, just like all JungleDragon photos.

Summary

So, progress has been great. The API is powerful and the building blocks are starting to come together. Without a doubt, the actual logic of this back-end will be tweaked many times due to the inconsistency of Wikipedia, but I'm making big steps forward. It's tedious work, but once this is integrated into the UI, you will see why it's worth it.

First, my situation. I'm using MySQL to power JungleDragon. As you know, one can chose the storage engine per table but usually I go for InnoDB, which allows for foreign keys and some other trickery. In this case, I was creating a new table called "specie". It's a bit of an odd table. It has a few regular fields, but it also needs to store very large text blocks that roughly vary in size between 3 and 5K. In characters that is. Since I'm using UTF-8, the byte size storage is likely double, so between 6 and 10K. Each text block is a seperate column and there's 32 of them. As column data type I use MediumText, which allows up to 64KB of data to be stored.

You could question this table design, but for now, let's put that aside. Given that I've selected data types that allow for plenty of room, I did not expect problems when inserting records into this table. Yet I've hit error 139.

Upon further investigation, I've found that this error occurs when the maximum row size limit is exceeded. This limit is 8K per row. That sounds a bit silly given that we can chose column data types that far exceed that number. As you may know, MySQL doesn't store TEXT or BLOB data inside the rows, it stores them seperately.

Well, not really. MySQL stores the first 768 bytes of each TEXT or BLOB column inside the row, even though the actual TEXT far exceeds that size. So in my case, 32 TEXT columns x 768 bytes = 24576 bytes, which is far beyond the 8000 bytes limit. The solution, other than partitioning the table? Switching the storage engine of the table to MyISAM, which allows for a limit of 64KB.

So, the solution is simple, if you can live with the limitations of MyISAM (I can in this case). The reason I am writing this is to point out that several sources on the web are wrong in stating that the 8000 limit is a MySQL limit. In reality, it's a storage engine limit. 

During December we had quite an overexposure of JungleDragon on this blog, so hereby I'll try to bring back some variety by sharing some thoughts on one particular topic: URL shorteners.

What they are for

URL shorteners in my opinion exist for the following reasons:

• To save characters in microblogging, particularly Twitter. The need to save characters there originates from the limited characters available in a cell phone text message.
• In print, where you do not want readers to type long URLs, although even this use is up for debate, since you could perfectly well have nice and short URLs without a specific shortener service. In fact, shortened URLs typically are quite cryptic, not easy to type from print.

So clearly, there is some need and use for shorteners. But I think the use of shorteners should be limited to where they clearly add value, and not be blindly applied everywhere. Therefore, I cringe at the idea that they are increasingly used where there is no real reason for it, as a trendy new default to link to things. Here's some reasons why I hate URL shorteners, or particularly the overuse and misuse of them:

They are insecure

Normal URLs, if designed correctly, provide me with information on where I'm going. This is input for me to decide whether I click a link or not. With a shortened URL, all information of the actual URL is gone, you have no idea where you're going. Sure, I sometimes click on a URL I haven't seen before, but information is taken out, not added.

A second security aspect is analytics. Although most won't care, you're placing potentially valuable traffic information in the hands of a third party, since all traffic will go through the shortening service. Since we increasingly live in an online world, the shortening service gets some pretty interesting customer information across industry sectors and regions. Potentially they sell it to your competitor, who knows?

They solve a symptom, not the problem

URL shorteners are oftend sold as to shorten URLs as well as to avoid ugly URLs. Rather than using a shortener, which effectively moves the problem to the 3rd party, one could also try to design short, yet meaningful URLs at the root to begin with.

This isn't always possible, I know. For example, a SharePoint URL will be long and ugly by default. Yet in allmost all contexts outside of Twitter, for those cases you can create labeled links. As in, click here, to open the team site, where you link here to the long and ugly URL. You can do it in email, in bookmarks, in office documents, on web pages. That covers almost all scenarios.

Potential loss of link value

I am not sure how intelligent search engines are in calculating redirects, but URL shorteners in theory could mean a loss of link value, since you're putting out multiple URLs to the same page, which fragments link juice. Again, this is a theoretic argument, I don't know what the practical effect is.

The dependency

By far my biggest objection to URL shorteners is the dependency you are adding. A dependency you do not control. A URL shortener may go out of business and all your links will break. All your carefully built up links that will be up there for years will no longer work. The URL shortener may change its conditions and start charging you above a redirect treshold, or they may introduce an ad page with a delayed redirect. Who knows, it is not your service. You're placing the destiny of your links at a 3rd party, likely a commercial party. The dependency also means that you have yet another point of failure. If the URL shortener goes down, your online business goes down.

And for what? To have cryptic short URLs that nobody can remember? 

Closing

Wrapping up, I don't really hate URL shorteners for their intended use, I just hate their increased usage in scenarios where they don't belong. What do you think?