Making Facebook Platform Apps Scale on the Cheap
You know a web 2.0 application is hitting an upward inflection point when your inbox starts filling up with the likes of “Joe Doe wants to add you as a friend on VagueSter.”
Over the last few months, it’s been Facebook’s turn. The site’s rise in traffic is in no small part due to developers being enabled to deliver their apps using the Facebook API. Some of these apps are getting serious traffic; see the Alexa results for the ex-PayPal guys over at Slide.com, for example. But some words of warning about social applications.
By Surj Patel
As Friendster found out the hard way, exponential network growth is compelling but the hangover can be deadly. In simple terms, your audience can grow linearly but your computational requirements often grow exponentially (social network computation has a lot of what’s known as NP-complete problems). If your app on the Facebook platform succeeds, you don’t then want it to die on the vine for lack of server resources. You need to make it scale and you need to make that happen quickly. And of course, cheaply.
While the guys in Palo Alto are happy that you use Facebook as your platform, they are not going to be happy if, as you succeed, you take them down with you. The techies at Facebook have been careful to ensure that they offload as much processing to you as they can — which means that in order for your app to scale you need to look at buying plenty of machines which, in turn, means lots of money.
The problem with traditional hosting resources is that you need you to pay up front, then they either hammer you with overages for being successful or charge you up the wazoo for unused capacity that just generates heat. What you need is what the world of suits would call “just-in-time resources,” which can be roughly translated into: “I get what I need, when I want it, and I pay only for what I use and no more.”
But of course you’re broke, sleeping under your desk and eating ramen noodles. So what can a poor entrepreneur do?
Welcome to Amazon and S3 and EC2 — processing power (EC2) and storage (S3) on demand. These services let you access computational power and storage only when you need it and, better yet, pay only for what you use. The last time I checked, it was 10 cents an hour for the server, 10 cents for every gigabyte of data written and 18 cents per gigabyte read out – all for a virtual box with 1.7Ghz x86 processor/1.75Gbytes of RAM/250Mbs of bandwidth. Nor are you limited to one usage; use as many as you need or want and can afford. But get ‘em while you can, as a waiting list has appeared in the last few months.
So where could you use such “meta services” when developing an app for Facebook? I talked to Jeff Jolma, the expert Facebook programmer at Snapvine, which has been adding voice services to Facebook and MySpace for the past couple of years. He said they’re great for getting the job done and making sure the job keeps getting done. “You have to be aware of some of the limitations with meta services, but if your app can work within their parameters then it’s a cheap way to scale up and down as needed,” he said. They’re also “definitely worth looking at” for Facebook apps, he said. Other uses for EC2 and S3 include:
- Images – if you have to store or process images beyond Facebooks provision
- Video storage and processing
- Heavy storage - DB replication, video, backups, transfers
- Backup - safe and secure
- Batch processing - large amounts of cycle-intensive processing all at once
- Data mining
- Spike demand handling (the Slashdot antacid)
And many, many more.
Now you may read this and think, “While it’s a great idea, the cost to develop the software to handle this would negate any benefit, right?”
Wrong.
Someone has probably already probably done most of work for you; use Google to find them. There are various meta services out there that can help you as well. Some, such as with WeCeo and RightScale, are selling computation services through a web site/API and will manage everything for you.
If you prefer to roll your own, go directly to the source with EC2 or S3 (for related ideas and support go here for EC2 and S3, respectively).
So go on, give it a try — you really have nothing to lose. Make yourself and the young Mr. Zuckerberg richer. While you’re at it, try to have some fun and please do share what you discover about meta services, be it good or bad. We’d also love to hear about services other than Amazon’s, so please post your findings below.
Thanks to Jeff Jolma at Snapvine for help in researching this article.
All Stories
- What Is The Future Of Enterprise Software
- Enterprise Software’s Future Lies With ISVs
- Sit Up Straight and Listen
- Making Facebook Platform Apps Scale on the Cheap
- Is Enterprise Software Failing The Innovation Test?
- Why Cisco & Microsoft Are Uneasy Frenemies?
- Lowering Barriers to Entry: Open Source and the Enterprise
- Why Google Needs The Desktop (& Adobe)
- Service Providers 2.0: It’s All About Scale
- The Road Ahead for Browsing
GigaOM Network
- Bay Area’s Carbon Tax, the Nation’s First, Rankles Big Oil [Earth2Tech]
- 9 Firefox Extensions to Protect Your Privacy [WebWorkerDaily]
- Vid-Biz: CBS, Akamai, Tube TVs [NewTeeVee]
- What’s Interesting This Morning… [GigaOM]
- The Open Source Cloud [OStatic]
Recent Comments
- [...] John Hsieh In an interview with Vivek Ranadive, CEO of Tibco Software, he articulates the “Is Enterprise Software Failing The Innovation Test? …
- Jeff, This space is getting very squirrly these days. Enterprise Software is languishing in the complexity of enterprises who have had too many years of pl…
- [...] Why Google needs the desktop (& Adobe) [...] …
- hi Nolan, i am computer graduate. your article is very intresting. i want to know more about this topic. i am searching for best profession in comput…
- The battleship Nokia won't be "following in the wake" of the dinky known as the Iphone. Every movement anyone has ever made in the mobile ocean has been cha…
- [...] recent “software-as-service” phenomenon is a particularly interesting example of disruptive change. SaaS was first seen [...] …
Copyright 2001-2008 GigaOmniMedia, Inc. All rights reserved.
Built by Foobuilder.
Powered by WordPress.com.
Marketing consulting by 
ACS.
Comments
I hear that Jeff Jolma character plays a wicked guitar.
WeoCEO is the stable IP, fail-safe monitoring, load-balancing, and auto-scaling solution that we use for our WeoGeo (http://www.WeoGeo.com) mapping marketplace on EC2 and S3. We are almost complete with our private beta period and look forward to a fully open beta shortly.
Feel free to check us out and request an EC2 ISO for your own use.
[...] The site’s rise in traffic is in no small part due to developers being enabled to deliver their apps using the Facebook API. Here is how you can use Amazon Web Services to scale your Facebook Apps. Continue Reading. [...]
You may want to take a look at ElasticDrive, it’s a S3 backed Network storage engine that allows you to use Amazon S3 like a local hard drive. It can be formated to most file systems and configured to act as a software RAID for an instant realtime mirror of content both locally and remotely. http://www.elasticdrive.com
We’ve also create a Amazon EC2 migration tool that allows for OS migrations to and from EC2.
http://www.enomalism.com/features/amazon-ec2-migration/
[...] is discussing an important issue about Facebook applications over at The Future of [...]
Hi- I’m sure you’re getting pinged by quite a few TC readers.
Nevertheless, I have to put my two cents in- as someone who is on the developer board for Facebook literally 12-15 hours a day, I can tell you that developers know about every solution you suggest. 9/10 times, any problems you encounter are on Facebook’s end.
It’s a little patronizing to read stock suggestions when app failure is as exasperating to developers as it is to people who are using apps. Please check before using an obviously influential podium to broadcast incorrect assumptions.
Hi Phil,
Thanks for your comment. From your post I can sense that your at the sharp end of the stick for scaling problems. Sorry to hear that.
People have to start somewhere. This isn’t a technical developer forum but one for entrepreneurs (and yes it is influential), as such I wrote for that audience. Instead of saying “don’t bother writing app’s as its too difficult” i think its better to push things forward by saying ” yeah go ahead and write them and see what we can solve. By the way here are some pointers”.
I am sure there are many problems on Facebook’s end but your going to get growing pains anywhere. Look at the early days of Amazon web services and you’ll see the same “support / care ” issues. You cant change any of that. You have to take what you get until they see value in pouring money into support and being accountable.
It sounds like you have a deeper technical knowledge, as such you know that NP complete problems have no real solution only “strategies” for doing better.
The article is intended as a starting point to inform of potential coping mechanisms for the growth.
You may be interested in this deeper article about API design
http://acmqueue.com/modules.php?name=Content&pa=showpage&pid=488
Also as this is an open forum please do feel free to post your suggestions - as i said in my post - be great to share your findings to help others.
Surj.
Surj,
Sorry if I came off as upset- I should have mentioned that I appreciate your good intentions and that your suggestions are definitely good ones. Scalability is important for developers. I guess I was reacting off of the Tech Crunch article and, unfortunately, you got the remaining vitriol. Forgive me- very immature.
I guess that I’d just like people in tech to realize that app errors aren’t always a dev’s fault and that, as a result, the dev shouldn’t necessarily be judged for those errors when it comes to key funding, advertising, and simple “do I use this app or not” devisions.
But hey, thanks for the article- I’d definitely point this out to someone new to Facebook Apps, and you did a great job providing people a place to talk about it.
Thanks,
Phil
[...] dieses Problem, unter dem dann nicht nur die Anwendungsentwickler, sondern auch Facebook leiden, liegt nach Surj Patel in neuen Diensten wie S3 und EC2 (angeboten von Amazon), in denen man Bandbreite [...]
[...] Making Facebook Platform Apps Scale on the Cheap « The Future of Software (tags: facebook hosting amazon web services) [...]
[...] sich Gedanken über das Wachstum von Apps bei Facebook gemacht und ein paar Ratschläge formuliert: Making Facebook Platform Apps Scale on the Cheap. Letztendlich sind seine Hinweise nicht Facebook-spezifisch, sondern eine generelle Empfehlung, wie [...]
Hi Surj,
Besides the difficult technical hurdles programmers will have to take, perhaps there could be another way of improving the quality. I wonder if it would be possible that the Facebook developers themselves create some sort of “Facebook developers approved” stamp for new apps. I know that there are many applications eveloped, but if the commnity would draw up some quality standards an set up a community testing environment, I feel the quality would definitely go up. And that would benefit all, including developers learning the trick.
Good article,
Alexander
Thanks for covering this space, Surj. We just finished the day participating in Amazon’s “Startup Project” conference for startups and VCs — and the energy and interest around web-scale computing was pretty remarkable.
An example of a RightScale customer with a cool new service based entirely on AWS (EC2 and S3) is Animoto, which combines user photos and music into professional animated slide shows — check out http://www.animoto.com. Their site is snappy, easy to maintain, and scaled right out of the gate. And they’re running both a load-balanced, fault tolerant web front end, and a full master/slave MySQL implementation on top of RightScale and AWS.
You have to look at the root of the problem. The reason these apps fail is because of their so calleded “developers”. Most of these apps are built by individual (mostly young people and students) who don’t have any industry experience in software development. Therefore, the end result is poorly written code which thus results in broken application.
[...] Making Facebook Platform applications scale on the cheap Share This | Sphere | Topic: Featured, Startups | [...]
[...] This is a great article that I found on my good old friend GigaOm site. It’s key to understand that developers need to get a handle on the following. [...]
Rightscale fees are still up there, beyond the budget of students bootstrapping a start-up.
These are two game changing platforms especially when you write abstractions to simplify there use. Services like rightscale I’m sure have they’re place but looking at they’re cost structure its more effective for a developer to write there own management scripts and use available open source tools. EC2 for example is very nicely managed with the EC2 firefox extension.
http://developer.amazonwebservices.com/connect/entry.jspa?entryID=609
[...] Patel is discussing an important issue about Facebook applications over at The Future of [...]
[...] Making Facebook Apps scale on cheap : An interesting writeup By Surj Patel about Scalability issues Facebook itself and the 3rd Party apps on it have. Also discusses EC2 and S3 as an alternative solution to scale in a cost effective way. [...]
[...] Making Facebook Platform Apps Scale on the Cheap « The Future of Software (tags: scalability facebook amazon s3 ec2 aws scaling) [...]
[...] Plattform Ebene 2 (Plug-In APIs): Hier passiert wesentlich mehr als nur ein Datenaustausch, das Nutzungserlebnis auf der Plattform selbst verändert sich. In der Desktop-Welt z.B. wenn ein Nutzer aus freien Stücken eine Plug-In für Firefox installiert. Firefox ist dabei die Plattform, weil sie es Entwicklern ermöglicht Module zu schreiben, die sich bildhaft gesprochen zusammenstecken lassen und so einen tieferen Integrationsgrad als Ebene 1 darstellen. In der Online-Welt - und auch heute kommen wir nicht an unserem neuesten Lieblingsthema vorbei - lautet das Paradebeispiel: Facebook. Allerdings, was mir bis vor kurzem durch die Lappen gegangen war: Die Applikationen werden nicht bei Facebook, sondern wiederum auf der Infrastruktur-Umgebung der Urheber entwickelt und vor allem gehostet. Problem: Wenn diese Applikationen beliebt werden, ergeben sich bei der Skalierug schnell massive Performance-Ausfälle, wie Surj Patel ausführlich schreibt. [...]
[...] Plattform Ebene 2 (Plug-In APIs): Hier passiert wesentlich mehr als nur ein Datenaustausch, das Nutzungserlebnis auf der Plattform selbst verändert sich. In der Desktop-Welt z.B. wenn ein Nutzer aus freien Stücken eine Plug-In für Firefox installiert. Firefox ist dabei die Plattform, weil sie es Entwicklern ermöglicht Module zu schreiben, die sich bildhaft gesprochen zusammenstecken lassen und so einen tieferen Integrationsgrad als Ebene 1 darstellen. In der Online-Welt - und auch heute kommen wir nicht an unserem neuesten Lieblingsthema vorbei - lautet das Paradebeispiel: Facebook. Allerdings, was mir bis vor kurzem durch die Lappen gegangen war: Die Applikationen werden nicht bei Facebook, sondern wiederum auf der Infrastruktur-Umgebung der Urheber entwickelt und vor allem gehostet. Problem: Wenn diese Applikationen beliebt werden, ergeben sich bei der Skalierug schnell massive Performance-Ausfälle, wie Surj Patel ausführlich schreibt. [...]
Surj,
WeoCEO has just come out of private beta. It is free to use for stable IP addressing and auto-recovery for a single managed instance.
Load-balancing and auto-scaling are free until later this year. We expect to be charging for these features in the future.
This blog post (http://blogs.weogeo.com/pbissett/2007/10/18/weoceo-emerging-from-private-beta/) has further details.
thanks,
p
[...] Joyent hopes to make money when people upgrade to their higher-tier services. I don’t know how much they plan to charge, but I am guessing it is going to be a lot cheaper than what an app developer typically has to pay. Surj Patel had recommended using Amazon’s EC2 and S3 services as way to scale in our “Future of Software” series. [...]