First of all, best wishes to everyone for 2009. I hope you live through it in good health and joy. I certainly had an enjoyable time the last few days, but am also glad that things are turning back to normal. Well, one thing that was not enjoyable was some intense tooth ache that kept me up at night, but that's fixed now. In between the holidays I actually had a few days off, so I spent it on progressing project JungleDragon. One particular deliverable was the JungleDragon logo, which I showed you earlier. 

However, I've been working on something far more interesting too. It is something that kept me up at night , something that made me read and study about three books worth of online material and cost me countless hours of prototyping with code. The result is here:

It's a JungleDragon URL. For an image. But...it is no ordinary URL. That image is stored at the Amazon S3 service. And when it gets requested by that URL in a browser, the request is made to  the Amazon S3 service, not my server.

Why is this such a big deal?

Consider that JungleDragon is an image collaboration site, kind of like a mini Flickr or SmugMug. Now imagine for a minute that JungleDragon becomes successfull. One of the biggest challenges that can be foreseen is scaling. There are/will be scaling challenges at the database, web traffic and storage level.

Let us focus on the web traffic and storage scaling issues for now, starting with storage. Images obviously take up a lot of disk space and it is quite easy to accumulate a few TBs of data this way, especially considering that I will store each image in at least 6 different formats. Images will also be the main "capital" of JungleDragon so this means they have to be stored redundantly. And what about backups? How would one go about backing up such huge quantities of data on a regular basis? And what if something fails? Data storage is a key scaling issue for JungleDragon.

What about web traffic? Imagine a gallery page showing thumbs of fourty images on a single page. Including some CSS, Javascript and some generic site images (such as a logo), this single page will make roughly 50 HTTP requests. Yes, scaling web traffic will be an issue too.

By offloading both the image storage and image HTTP requests to Amazon, I have an infinite amount of storage at my disposal, and it will automaticaly be stored redundantly, will be backed up, etc. It will also dramatically reduce HTTP requests on the future JungleDragon server(s), meaning I can take on much more traffic before having to implement scaling solutions there. Of course, the use of Amazon S3 is not free, it's a pay-as-you-go model, but it's looking pretty cheap for now.

All in all, I consider this a MAJOR breakthrough in the development of JungleDragon, as scaling was one my main concerns. This alone does not solve everything, but at least a major part of it.

By the way, implementing the above (offloading storage and http traffic for your files to S3) is super easy. You can be up and running in an hour or so. What is less easy is this:

• Integrating it in a robust way in your back-end
• Lowering your Amazon S3 request load and bill by the smart use of caching
• Avoiding the risk of getting a high Amazon S3 bill when some joker decided to DDOS your image URLs or deeplink to them
• Controlling various Amazon S3 params via code, for example to failover to local storage, controlling the cache age, controlling the URLs to use, controlling the ACL of files, and much more.

RATE THIS CONTENT (OPTIONAL)

	Was this document useful to you?
	Rating:	Awesome  Good  Average  Poor  Useless

CREATE A NEW COMMENT

	Name:
	Email:
	Website:
	Content:	HTML is not allowed. Hyperlinks will automatically be converted.