Ben Allfree :: Painless Programming

Amazon AWS mainly consists of S3 (infinite storage) and EC2 (infinite computing) in my projects. I'd venture to say that's true for you too even though AWS has other services too.

Amazon's claim is mostly true: they do offer, for all practical purposes, bottomless resources at utility pricing. The pricing isn't actually that great until you get into large capacities, but then it gets very competitive.

Designing an application to use AWS requires - to my way of thinking - a fundamentally different approach to application architecture. Basically, Amazon looked at their own infrastructure efforts and wondered how to sell it. That means your applications need to work according to how Amazon thinks scaling should work. The biggest limitation used to be that EC2 instances lost all their data when the power went out. This is because Amazon used EC2 internally as raw computing power that could be provisioned to any one of a number of specific servers pre-configured to do a job within their infrastructure. So you have to design your application the same way. Is it a bad idea? May be not. But it's certainly not the only or easiest way to design things. As my good friend said, if you are willing to ignore an assumption that has been true for the last 30 years (that data persists when the lights go out), EC2 sounds great. They've worked toward addressing the problem, but I still don't quite trust EC2 like I do other virtualization ISPs.

S3 is basically a big hash table. You give it a key and some data to store and it does it. If you name your keys to look like URL paths, then you can access it through HTTP too. So it's a giant static content bucket. Recently, they introduced edge caching to make S3 even faster. They've also introduced features like key renaming. By golly, one of these days it might behave like another 30-year-old concept: the file system. Until then, it still seems a little quirky and you really need to design with S3 usage in mind.

Now we come to the big question in my mind. I like to question big things, like whether cloud computing and utility computing really address needs you actually have. I can't speak for all businesses, but for you and me, utility computing answers a problem we don't have. I imagine over 90% of the web does not have the type of infrastructure problem AWS solves. Sure, you can invent problems that AWS can solve, but when you really sit down to look at who is using AWS and how, you begin to understand that it's a toy for them. Yes, they get up in front of the audience at the AWS conferences and say how AWS solved a massive infrastructure headache at a fraction of the cost because they didn't need to buy hardware. They espouse the virtues of infinite scalability and how they're ready for traffic spikes. But here on earth, very few of us are affected by scaling problems to an extent that would merit a fundamentally different application architecture.

To put all this in context, I say it from a base of experience. I have designed several apps that leverage AWS successfully. And each time, what I saw was that it was a big effort with a potential payoff down the road.

If you like to plan for success early, AWS may really resonate with you. If you're more in the Agile camp of doing things only after there is a concrete and observable need, then AWS is probably something you can skip until you have so much traffic and cash flow and investors that you can buy your own infrastructure. At that point, you can compare AWS against the big boys like Akamai. In the end, I suspect that if you stay Agile it will always be better to roll your own. See any of my blog posts about frameworks for more thoughts on this.

Overall, as much as I like the idea of AWS, I need to say thumbs down because it just hasn't proven to address a critical need for you. I love the idea and I love thinking about how awesome it would be to have a problem that AWS solves, but so far I have not come across one nor do any of the startups using AWS appear to have a concrete need for it. It's more like they're doing it for the publicity. The central point is that you pay for cool and, in this case, it doesn't pay you back.