Yesterday I explained my progress concerning specie integration in JungleDragon v2. Much to my delight, I received a few very positive comments. I need those to fuel my motivation. This part is hard, complex work with constant setbacks.

Anyway, I also explained I was playing with the idea to automatically integrate Youtube videos with the specie page. Quite a simple, yet powerful way to see a specie in action. My theory was to simply use the Youtube API and to pass it a search string, in my case the name of the specie.

You can see above how this is not going to work. The Impala, other than a specie, is also a car. A Polar bear, other than a specie, is also a cartoon, dance event, and part of the song title of various music tracks. In these two simple test cases, about 90% of the search results have nothing to do with the specie.

So this is a failed experiment. Funny, though: Henriette asked what I was doing and I explained the problem to her. She suggested to use the scientific name as a search string. As this was a 5 second code change, we could instantly see the improvement in relevancy: up to about 80%. Much better, but I'm thinking it's still not good enough. I then had the task of explaining to my beloved one how brilliant her remark was, yet how I was going to ignore it anyway. I'll spare the details. 

How to move forward in this area? I still believe in the power of video to enrich specie information, so here's my plan:

1. Ignore it for now and focus on testing what I got and bringing it live. This feature is non-essential for a first release of the specie identification feature.
2. Once step 1 is stable enough, solve the video challenge as follow: allow users to manually add videos, simply by entering a Youtube or Vimeo URL. Not that hard to implement, much greater relevancy and quality of videos.

What do you think?

Here's a photo uploaded to JungleDragon, of which the specie is identified as a "Polar Bear":

As a result of this identification, an interesting side panel appears. Let's have a closer look at it:

This essentially is a summary of this specie, with the following data visualized:

• Common name of the specie in the header 
• The first 500 characters of the description of the specie, wrapped around the photo of the specie that has the most karma in JungleDragon
• An interesting visualization of the IUCN conservation status and trend of the specie. You should really compliment me on this one as I wasted 6 hours styling it.
• The 2nd, 3rd and 4th image of the specie in JungleDragon, sorted by karma

There's a lot more that JungleDragon knows about this specie. To learn more about it, there's the obvious button "Learn more about this specie". Clicking the photo of the specie will also you bring you to the specie page, which currently looks like this:

So this is the dedicated specie page, which tells everything JungleDragon knows about it. This is only a fragment of the page, in reality it is about 10 times longer.

First, we start with a large header containing a larger version of the most popular image, as well as the full description of the specie.

Below that are a set of dynamic links, the ones with the arrows pointing down. These represent the areas of which JungleDragon has information about the specie. For example, rather than scrolling the entire page, you can instantly jump to "Diet" and from there go back to the top. JungleDragon does not copy all text blocks from Wikipedia, only a select few. 

The left column below the header contains the actual text blocks. These come from Wikipedia, yet they are transformed so that it becomes natural language with various Wikipedia elements removed (such as links, citations, etc). Another interesting element to the text display is that it is mixed with JungleDragon photos, not Wikipedia photos. You'll notice the first text block containing a left-aligned photo. The next text block will have a right-aligned photo. This keeps going until we run out of text blocks or photos. It makes for an attractive format of reading and an interesting way to mix Wikipedia content with JungleDragon content, or so I think.

Moving on the right column, we have a "Distribution and status" block which contains an image of this species' distribution (if available on Wikipedia) and again the IUCN status, richly illustrated.

Below that is the taxonomy block which classifies the specie in the biological "tree". As you can see, the elements in the tree currently are not links. Initially they will not be, even when I bring this live. If I can, I will implement this though. It would allow for tree-like browsing. For example, you could jump to a page of the family "Bears", to see all bears, not just Polar Bears.

So for now this concerns the overview tab of the specie page. There's also a photos tab:

It does what you expect: it shows photos of that specie, and you can sort them by popularity or upload date. 

The third tab "videos" is not implemented yet. It's just an idea I'm having. The idea being that I use the Youtube API to shows videos of this specie based on a search string. I will give this a try to see if I can get it to work reliably.

There's about 10 other things to implement to fully implement the v2 vision of JungleDragon, but this teaser definitely represents the core functionality. Your feedback is highly wanted!

In the last few updates concerning JungleDragon, I mentioned how I'm working on the specie engine, the part that integrates specie information of Wikipedia with JungleDragon photos. None of this is live yet, so you can't see it. Neither was there any development UI to demonstrate, it was just me complaining how tedious it is to get structured data out of Wikipedia. 

That is still true, and my struggles in that area continue, but hereby I do want to share some first UI work of the specie engine. The scenario is simple: you have uploaded a photo and are asked to identify the specie on the photo/ For that there is an "Add specie" button, which brings up this dialog:

Since multiple species can appear on a single photo, you can add more than one specie, yet you add them one by one. As the dialog states, you can search both by common name (i.e. "Polar bear") as well by the latin name (i.e. "Ursus Maritimus").

As you type a specie name, the list will help you using suggestions. These suggestions concern species known to JungleDragon. This means they are used before. I do not have a database with all species. Instead, as you add a specie not known to JungleDragon, it will be a known specie from that point on. 

What makes a valid specie? Here are the current rules:

• There must be an english Wikipedia page for your query, or a redirect to such a page
• That page in particular must be a specie page, meaning:
  • It has the "taxobox" on the right
  • It has to be a specie or a subspecie, meaning it has either the "binomial" or "trinomial" name property. For example, "Bear" is not a specie, but "Brown bear" is.

Ok, given that you entered a valid specie name, one of two things will happen:

• If the specie is known to JungleDragon already, it is instantly associated with a photo.
• If the specie is not known to JungleDragon, yet it is a valid specie, I will parse it from Wikipedia in real-time, which takes a few seconds. A loading indicator will make this clear. From that point on, it is a known specie to JungleDragon.

So, that's how the "Add specie" dialog works. It's how you identify a specie on a photo. Once I know that relationship, I can visualize rich specie information next to such a photo. Here's a very early preview:

Check out the sidebar on the right. This photo of an Impala has been associated with the specie Impala, and as a result, it shows the common name, binomial name, description, and range map. 

Be aware that this is just a simple start. I have a lot more data about the specie and I can also visualize it any way I like. Take note of the concept though. This is where JungleDragon v2 is all about. Instantly learning about what is on the photo. And of course, later on you can click through on the specie name which will show a full page with everything there is to know about it.

Wiki parsing engine updates

I need to reserve some room in this post once again for self-pity. To complain about parsing Wikipedia. The overall complaint is that each time I extend my test set of specie queries, I find new problems, new ways in which Wikipedia pages are structured, that my engine cannot handle yet. It's one step forward, two steps back. Here's two recent situations:

• I've been relying on the taxobox on a specie page to parse the species' taxonomy. Finally I had my engine robust enough to deal with the unlimited ways in which that taxobox can be structured: levels in the taxonomy can be there or not, the amount of levels varies, the spelling of levels varies, the value of a level can be plain text or contain any Wiki markup. Until I discovered yesterday that some specie pages do not use a taxobox, they use an "automatic" taxobox, a complex variations based on specie keys.
• Another problem as a result of testing is this. Say you'd have a photo of an African Elephant. In the "Add specie" dialog the instructions are clear: "Elephant" will not work as it is not a specie, and thereby not specific enough. So you try "African Elephant". You'd expect this to be specific enough, unless you're a zoologist. See, in this case, even "African Elephant" is not enough. It's not a specie, instead "African Bush Elephant" is the correct specie, according to science and according to Wikipedia. But probably not according to you. In these situations, I have therefore implemented a routing mechanism. It's a manual table that I maintain in which I map a source, in this case a commonly failing yet well-intended query, into a valid target specie. So if you'd type "African Elephant", I will map it to "African Bush Elephant" for you. Over time I'm hoping this table will give you a better chance at the result you expect.

Without a doubt, there will be dozens more problems coming my way. But I will persist through them, because I dearfully believe in the concept. No matter what it takes, this will get done, and it will be done right.

I love Google Analytics. It is astonishing to have such an incredibly advanced statistics tool at one's disposal for free. I also believe most owners of small to midsize websites do not get the most out of it, I'm certainly guilty of that. That's why I first want to briefly revisit two earlier posts concerning features you may not expect in Google Analytics:

Measuring your site's speed as seen by your visitors

Change a single line of code in your GA tracking javascript and GA will then sample your site's loading speed. This is a big deal. You can easily see how your site performs based on different bandwidths, locations, browsers, any dimension you like. There's enterprise solutions charging you tons for such a service. In GA it's free, and all you need to do is to add a single line of code.

In-page Analytics

My absolute favorite. We know we have tons of metrics available in GA, but its hard to bring them down to meaningful conclusions. In-page analytics changes that. You will see your page as you designed it, and visually attached you see the click-through data, amongst other metrics. 

This is the first-ever easy way to test your design. Recently I wanted to redesign parts of the navigation of a site, yet I was worried that users would be confused, as some options would be relabeled or removed alltogether. Until I learned that option in question was hardly ever used at all. It is quite powerful to make design decisions based on data, rather than gut feel, personal preference or emotion. This also seriously strengthens your position amongst stakeholders, in case you have to defend a design decision.

Real-time analytics

Ok, so those were the earlier posts. They are game changers, please check them out in detail.

The new one, which actually has been available for several weeks now, is Google Analytics realtime. Where all of GA's former metrics had data with a delay of about one day, realtime analytics show you those metrics as they occur. Perhaps you tried to launch a viral campaign in social media, this way you can check the actual effect as it happens. You can also learn about traffic patterns related to timezones. 

The opening screenshot shows a glimpse of the realtime view, with the currently active visitors, their locations, target pages and more. Truth is to be told though, this really is only interesting in case you have critical mass, meaning a large website with many visitors.

As you may know, I'm currently working on the integration of specie data from Wikipedia into JungleDragon, as part of JungleDragon v2. It is hard, complex, invisible work. But I'm making great progress. There's no UI to demonstrate yet, but allow me to update you on the progress of the back-end.

Finding a specie:

I can now do this:

$specie = $this->ZoologyManager->FindSpecie("Impala");

Which will result in:

• A check if there is a Wikipedia page for this entry
• If there is an entry, yet it is a Wikipedia redirect page, I will resolve the redirect
• Assuming the page exists, a check is made to see if it is a specie page:
  • It has the taxonomy box to the right
  • The page is at the taxonomy level species or subspecies. In practice this means the property "binomial" or "trinomial" name must be present. Anything less specific is no match.
• All checks passed, parse the specie page. This is the most complex and fragile part of the process:
  • Parse the taxonomy information. Implemented in a flexible, forgiving way since multiple taxonomy systems exist and are used, the amount of levels varies between pages and even the naming of properties varies between pages. 
  • Parse the rest of the page, which can contain any Wiki markup. I'm intelligently trying to find selective blocks of information by its headers, next I parse, strip and transform this content in a way I find suitable for JungleDragon storage and display. This is a best effort approach, but it's working reasonably well currently.
• Normalize the parsed elements of both the taxonomy information and the free format text into a specie object that JungleDragon understands.

This is a long explanation of saying that with a single specie search string, I get back a structured object of that specie, containing very rich information. That's quite a powerful API.

Saving a specie

Assuming we found a valid specie, next I can do this:

$this->ZoologyManager->SaveSpecie($specie);

Which will result in:

• A new specie record being created in my database, if it did not exist already. This record contains both the taxonomy information of that specie as well as the normalized text blocks.
• In a seperate table I will update specie synonyms, for example, there can be multiple search strings matching a single specie. "Polar Bear" and "Ursus Maritimus" are the same specie, and I maintain that relation to avoid duplicate specie records. 
• If the specie record contains any specie classifications not yet known to JungleDragon (division, kingdom, class, order, family, genus), it will save it in a seperate table. This table is loosely designed, given that the amount of levels per specie is not consistent in Wikipedia, as well as there not being any guarantee that keys match exactly by string.
• A very cool one: If the Wikipedia specie page has a range map (a graphic showing the distribution of the specie on a map), I will download it, associate it with a specie, and then transfer it to Amazon S3, just like all JungleDragon photos.

Summary

So, progress has been great. The API is powerful and the building blocks are starting to come together. Without a doubt, the actual logic of this back-end will be tweaked many times due to the inconsistency of Wikipedia, but I'm making big steps forward. It's tedious work, but once this is integrated into the UI, you will see why it's worth it.

First, my situation. I'm using MySQL to power JungleDragon. As you know, one can chose the storage engine per table but usually I go for InnoDB, which allows for foreign keys and some other trickery. In this case, I was creating a new table called "specie". It's a bit of an odd table. It has a few regular fields, but it also needs to store very large text blocks that roughly vary in size between 3 and 5K. In characters that is. Since I'm using UTF-8, the byte size storage is likely double, so between 6 and 10K. Each text block is a seperate column and there's 32 of them. As column data type I use MediumText, which allows up to 64KB of data to be stored.

You could question this table design, but for now, let's put that aside. Given that I've selected data types that allow for plenty of room, I did not expect problems when inserting records into this table. Yet I've hit error 139.

Upon further investigation, I've found that this error occurs when the maximum row size limit is exceeded. This limit is 8K per row. That sounds a bit silly given that we can chose column data types that far exceed that number. As you may know, MySQL doesn't store TEXT or BLOB data inside the rows, it stores them seperately.

Well, not really. MySQL stores the first 768 bytes of each TEXT or BLOB column inside the row, even though the actual TEXT far exceeds that size. So in my case, 32 TEXT columns x 768 bytes = 24576 bytes, which is far beyond the 8000 bytes limit. The solution, other than partitioning the table? Switching the storage engine of the table to MyISAM, which allows for a limit of 64KB.

So, the solution is simple, if you can live with the limitations of MyISAM (I can in this case). The reason I am writing this is to point out that several sources on the web are wrong in stating that the 8000 limit is a MySQL limit. In reality, it's a storage engine limit. 

During December we had quite an overexposure of JungleDragon on this blog, so hereby I'll try to bring back some variety by sharing some thoughts on one particular topic: URL shorteners.

What they are for

URL shorteners in my opinion exist for the following reasons:

• To save characters in microblogging, particularly Twitter. The need to save characters there originates from the limited characters available in a cell phone text message.
• In print, where you do not want readers to type long URLs, although even this use is up for debate, since you could perfectly well have nice and short URLs without a specific shortener service. In fact, shortened URLs typically are quite cryptic, not easy to type from print.

So clearly, there is some need and use for shorteners. But I think the use of shorteners should be limited to where they clearly add value, and not be blindly applied everywhere. Therefore, I cringe at the idea that they are increasingly used where there is no real reason for it, as a trendy new default to link to things. Here's some reasons why I hate URL shorteners, or particularly the overuse and misuse of them:

They are insecure

Normal URLs, if designed correctly, provide me with information on where I'm going. This is input for me to decide whether I click a link or not. With a shortened URL, all information of the actual URL is gone, you have no idea where you're going. Sure, I sometimes click on a URL I haven't seen before, but information is taken out, not added.

A second security aspect is analytics. Although most won't care, you're placing potentially valuable traffic information in the hands of a third party, since all traffic will go through the shortening service. Since we increasingly live in an online world, the shortening service gets some pretty interesting customer information across industry sectors and regions. Potentially they sell it to your competitor, who knows?

They solve a symptom, not the problem

URL shorteners are oftend sold as to shorten URLs as well as to avoid ugly URLs. Rather than using a shortener, which effectively moves the problem to the 3rd party, one could also try to design short, yet meaningful URLs at the root to begin with.

This isn't always possible, I know. For example, a SharePoint URL will be long and ugly by default. Yet in allmost all contexts outside of Twitter, for those cases you can create labeled links. As in, click here, to open the team site, where you link here to the long and ugly URL. You can do it in email, in bookmarks, in office documents, on web pages. That covers almost all scenarios.

Potential loss of link value

I am not sure how intelligent search engines are in calculating redirects, but URL shorteners in theory could mean a loss of link value, since you're putting out multiple URLs to the same page, which fragments link juice. Again, this is a theoretic argument, I don't know what the practical effect is.

The dependency

By far my biggest objection to URL shorteners is the dependency you are adding. A dependency you do not control. A URL shortener may go out of business and all your links will break. All your carefully built up links that will be up there for years will no longer work. The URL shortener may change its conditions and start charging you above a redirect treshold, or they may introduce an ad page with a delayed redirect. Who knows, it is not your service. You're placing the destiny of your links at a 3rd party, likely a commercial party. The dependency also means that you have yet another point of failure. If the URL shortener goes down, your online business goes down.

And for what? To have cryptic short URLs that nobody can remember? 

Closing

Wrapping up, I don't really hate URL shorteners for their intended use, I just hate their increased usage in scenarios where they don't belong. What do you think?

This month was the first ever JungleDragon photo contest. As the contest ends today, the winners are announced here. Thanks to all who participated, see you all in 2012!

Today is the last day of this year's JungleDragon marathon, where I dedicated a full month to JungleDragon development. What a month it was. In this post I want to summarize the month, the year and what's ahead. First, let's look at what was achieved this month:

Update 01:

• 3 Bugs solved
• Cleaning up of activity streams, making them more useful
• Simplified and reorganized user profiles
• Better feedback management: one page to check your photo's community feedback

Update 02:

• Improved look of JungleDragon's Youtube channel
• Launch of first-ever JungleDragon photo contest

Update 03:

• All-new personal dropdown menu below your avatar
• Completely redesigned friend feed which includes both an activity and photo stream
• Combined with update 02, a vastly improved personalized experience

Update 04:

• Various usability improvements in fullscreen photo viewer
• Breakthrough launch of HD photo viewing, a unique feature

• 9 Bug fixes, improvements and optimizations
• New karma capability: link love
• Reorganized personal settings menu
• Opened up HD photo viewing for all members

Update 06:

• A day of struggle, yet a breakthrough in JungleDragon's twitter feed which now posts thumbnails of JungleDragon photos, making JungleDragon available on almost every device via Twitter.

Update 07:

• Photo thumbs now also available on JungleDragon's Facebook fan page
• Redesign and cleanup of Facebook fan page
• Improved "unclassified images" overview in JungleDragon, which shows images that are not well described or tagged

Update 08:

• Breakthrough update: added Google sign-in support. Joining that way takes 5 seconds. Followed the next day by an article on how I did that.
• Removed requirement of JungleDragon user names needing to be unique

Update 09:

• Another key milestone: JungleDragon now detects geotagged photos and if detected, displays your photo's location on a map widget next to your photo.
• Improvement of the display of various photo details next to your image, smartly displaying the upload date, capture date, EXIF info and location in a natural sentence.

Update 10:

• Geotagging continues with the launch of an awesome, fullscreen wildlife map that showcases all geotagged photos.

Update 11:

• Auto tagging of geotagged photos
• Advanced map controls added for the fullscreen wildlife map
• Ability to explore a country from a photo or user profile added
• Fullscreen wildlife map now has a search bar for instant location search

Update 12:

• Introduced a really easy way to geotag your photos directly inside JungleDragon

Update 13:

• Fixed two very hard issues concerning the fullscreen wildlife map
• Improved Google sign-in integration

Update 14:

• 4 small improvements and optimizations

Update 15:

• Started work on one of the most important JungleDragon features to-be: specie identification

Update 16:

• One bug fix, one small improvement
• Introduced mass geotagging where you can easily geotag multiple photos inside JungleDragon
• Introduced "My Map", a personal fullscreen wildlife map of all your geotagged photos

Update 17:

• Progress update on the parsing of Wikipedia specie pages, a key element in JungleDragon v2.0

Marathon summary

This marathon has been nothing but positive for me:

• I was highly motivated and productive, leading to a lot of improvements and new features in a short period of time.
• I realized some breakthrough updates: HD photo viewing, a superb geotagging experience, an awesome Twitter feed, Google sign-in support and key improvements in making JungleDragon more personal and usable.
• I learned a lot and had fun
• I've finally addressed items that have been too long on my todo list, things I kept delaying
• With geotagging implemented and specie identification currently in development, I've finally committed to JungleDragon v2. JungleDragon as I envision it, the full concept, is coming together.
• This was the first marathon where JungleDragon was live, and the community interaction was great. I picked up on some requests from the community quickly, and we had fruitful discussions on how specific features should work. A very rewarding way to move forward.

Year summary

I don't want to make this post too long, but hereby a few observations concerning this year of JungleDragon:

• JungleDragon was launched in April, making this the first true year of JungleDragon
• The site has been very well received, almost everybody that encounters it is positive
• JungleDragon is still small in terms of community and content size, but I'm still thrilled that within a few months, a core community was formed and the content base is growing. It's harder than you think to create a working community out of nothing. The site is proven to be viable, yet needs to grow more.
• Despite a few infrastructure incidents, the site has been quite stable technically
• Working with a live site and community is so much more fun than working in isolation, I've learned valuable lessons this way. One of which is to emphasize simplicity and usability, and to focus on what the community really wants. Another is to work more incrementally, rather than the big bang approach I sometimes take.
• JungleDragon has developed an informal tradition to donate to nature for every 1,000 images shared, and I'm planning to do more photo contests in which donations to nature are also made. The contributions are modest, but I think JungleDragon is genuine about not only sharing, but also caring about nature.

What lies ahead

With the marathon over and me getting back to work, development will obviously slow down, but my commitment to push JungleDragon forward remains. Here are the focus areas for next year:

• JungleDragon v2: Blending rich specie information into JungleDragon, a task that likely spans a few months, but a crucial one nevertheless.
• General development: There's always items on my list, and new items being requested by the community. I'll be sure to pick these up on a regular basis. Regular small updates keep the spirit of the site alive and they add up over time.
• Bleeding edge development: JungleDragon does not end at v2. It will never be done. Hidden at the bottom of my todo list are the crazy ideas, things I may or may not do, that may or may not be possible at all. One example would be Mobile, but there's more.
• Marketing: JungleDragon cannot grow if nobody knows it exists, so this area will need constant attention. Obviously my focus will be on low-cost high-impact methods.

I'm proud of JungleDragon and its community, and hereby I wish to thank everyone who contributed to it. Let us take this positive outlook into 2012. 

Note: tomorrow will be my final post of the year, where I will announce the winners of the December photo contest.