Ben Allfree :: Painless Programming

erlang is a programming language that is supposed to be very scalable. It encourages a form of design that is naturally scalable and as such requires some degree of adjustment for many programmers. The whole idea doesn’t sit well with me.

I was talking to a client today who has business plans that revolve around harvesting data from web sites. How do you do it? Here’s my step-by-step process:
1) Crawl for links
2) Fetch link content
3) Index link content
4) Transform link content
If you design your system that way, you end up with something that can be scaled [...]

I help people build their businesses on scalable “cloud” architecture. This is a blossoming field and it’s not always clear how an application should be programmed or architected so it properly leverages all these scalable resources available today. For example, what exactly can be programmed on a cloud? What is a cloud, anyway? If you’re [...]

Well, this looks pretty interesting. Google has released a scalable “app engine” where you upload your web site source code and it runs on their infrastructure.

There is a lot of research going on right now surrounding how to create massive databases. Google, Amazon, and other companies are introducing web service APIs designed for data storage and retrieval.
Storage is pretty much solved now. Amazon S3 covers that. But efficiently querying and retrieving that data is an amazing challenge. I’m working [...]

Ben will be helping an undisclosed client transfer 1 billion files into Amazon’s S3 storage cloud. Further details are not public at this time.