Effortless scalability. I want it. I’ll pay for it.

Planning a smooth scale-out growth of a web application can be a nightmare. You are never certain how popular the application will be. You can’t tell when the users will actually use the service. When your site hits the first page on Digg, you feel elation and terror at the same time. Even worse, it can be very difficult to estimate how your users will use the site, how many videos they will watch, and how many database updates they will trigger.

It is easy to set up a single-server web application, or even to separate the major tiers onto separate servers. But when it comes to scaling out to more servers, particularly more database servers, life gets difficult. You face data concurrency issues, latency challenges, failover needs, and backup requirements. Solving each of these is complex, time-consuming, and often expensive.

None of this is conducive to building the next big Internet startup. Solving the scaling issue is a necessary evil… for now. In my opinion, whoever fixes it will make a heap of money. It’s a commodity problem. Someone ought to fix it.

The cloud computing services (such as AWS and now Google AppEngine) are the first steps towards effortless scalability, but they leave a lot to be desired. With a combination of EC2 and S3, you can build a very credible Linux (or even Windows with emulation) solution that can scale with reasonable ease and cost efficiency. However, you still have to bake your own solution for load balancing, database clustering, and server provisioning. Amazon’s SimpleDB is not a great database alternative (it’s not really a database in my opinion), and has some limitations (eventual consistency, data type limitation) that require a substantial layer of custom code to serve the needs of most applications.

I want a better solution. I suppose the market is heading for it, but just in case, I’m going to whine until I get it. I want effortless scalability.

What is effortless scalability? I build my application on my local machine. Once it’s working, I upload it to the cloud, assign it a URL, and I’m done. I get 100 visitors, I pay a bit. I get 10 million visitors, no problem (although I pay a lot more). Some key requirements:

• No server requisitioning… computing resources are allocated automatically as needed
• Built-in database scaling, without the limitations of SimpleDB
• Automated load balancing, with configurable settings for picky developers
• Automated data storage that looks like a file system to me (think cPanel like you would see in a shared hosting environment)
• Flexible stack support (check out CohesiveFT for an example of how that might work)
• Automatic failover

Amazon Web Services is a good start. A highly scalable relational database in the cloud would be a great next step. Or at least a cloud datastore that more closely approximates the capabilities of a relational database, and abstracts away the complexities from the developer.

Some will argue that one of the things that makes Amazon Web Services so powerful is its very simplicity. Complexity abstracted away from the developer will inevitably accrue to the cloud. That is likely true, although from an allocation of resources standpoint, it makes more sense than requiring developers to solve the same problem over and over.

Solutions such as Media Temple only chip away at the problem, for example offering dedicated MySQL containers as their database scaling solution. How can that support a heavy duty site?

Anyway, someone please turn application scaling into a click of a button. I’ll be happy to pay for it.

[Disclosure: I have a relationship with CohesiveFT mentioned in the post through my role at OCA Ventures, a venture firm that has an investment in the company.]

The blogosphere is replete with references to an interesting post by Michael Mace, “Mobile Applications, RIP“. Michael believes that the age of custom mobile application stacks is dead. Instead, he posits, the mobile web is surging in importance, despite its being inelegant and inferior technologically. He points out that

“A platform that is technically flawed but has a good business model will always beat a platform that is elegant but has a poor business model.”

In principle, I agree with him, but his choice of language and approach to the issue suggest that technology comes first. As he says “A platform (subject) … has a … business model (object).” I don’t mean to sound like a third-grade English teacher, but this is an important distinction. Platforms don’t have business models (or at least they shouldn’t). Business models have platforms.

For the past 14 years I have been managing teams of very capable engineers. I myself do a lot of the software design, and for some projects, the coding. It is very easy to think technology first, because that is often the way things are built. Engineers naturally want to make their software elegant.

But elegant architectures do not necessarily make money. And there is real danger that by starting to plan or build the technology before nailing down the business issues, you’ll build the wrong thing. Of course, given today’s frantic and competitive technology markets, the technology and the strategy usually evolve concurrently. The problem is that the during such concurrent development, the technology is often the most concrete thing you have. It is easy to relegate the business strategy and consumer needs to a secondary role.

The “technology first” attitude is at the heart of more startup failures than I could ever list. Never let the engineers (the crazy people) run the startup (the asylum).

But what can we do to minimize the risk that the technology takes on a greater importance than the business objectives? Here are some suggestions:

• Start every technology meeting with a review of the current business objectives. This will help remind your team of where they’re really trying to head.
• Challenge your team to explain why their technology decisions are the best choice to achieve your business goals. Never let them build the technology “right” unless they can persuasively articulate how that plays into your business needs.
• Use business milestones to motivate your technology team rather than development milestones. Don’t reward them for the release of version 2, but rather for acquiring 3,000 new users on version 2. That will make them think more like a consumer when they’re developing, and will force them to work with marketing to get the job done rather than just passing it off and saying “good luck.”
• Stress the importance of maintainability and flexibility in their software design. They will be tempted to go for scalability and stability instead. Remind them that until we have settled on what we need to build, work on scalability and stability is wasted. Maintainable, flexible code can evolve as we determine our business needs. Once that is set, we can focus on locking down performance.
• Contextual design interviews are key. It’s very easy for engineers to lose track of what customers actually need/want unless they actually see it. It still amazes me sometimes how stupid users can be (like I can be sometimes when using a new product I’m not really paying attention to). Accommodating their (our) ignorance is critical for a good product.

[Fellow engineers, fear not. Some day I’ll write a post about the ignorance of non-engineer managers ignoring the technology realities, and pushing out a product that is not ready, and will never be any good.]

Readers, do you have any other suggestions about how to ensure business objectives remain paramount?

Note: This article derives from a paper I co-authored in 2007, “Understanding Key Success Factors for Social Web Ventures”. It won first place in the Insightory.com competition. You can view the entire paper here. This blog entry is part of a series of entries covering social web ventures.

Metcalfe’s Law says the value of a network equals the square of the number of its users. Larger networks create network externalities, leading to competitive advantage. Users post on YouTube because there are more visitors. And visitors come to YouTube because there are more videos to watch.

Therefore, network size is key to a successful social web venture. More users and content than your competitors means more consumer value, which in turn creates a snowball growth effect.

But when should you start monetizing your traffic? Critics suggest that many startups wait too long to start monetizing. However, until we find revenue models that do not reduce consumer value, the wisest strategy is to hold off monetizing your traffic at least until you reach an inflection point (e.g., second derivative hits zero).

Most revenue models, such as advertising, reduce consumer value. Metacafe, which has long had relatively intrusive ads, launched nearly two years before YouTube, but continues to trail far behind the market leaders.

When a site starts monetizing its traffic, it reduces consumer value, which can give your competition an advantage. Therefore, sites should delay monetizing until they reach the point of diminishing growth… as they come closer to their long-term stable size. This suggests that the larger the potential market for a site, the longer you should wait to monetize.

Once you get big enough (like YouTube), you may have the breathing room to capture some of the user value without reducing your net value proposition below that of the competition.

Heavy.com has done a great job of monetizing traffic without hurting consumer value too much. Of course, the best solution would be to monetize while increasing consumer value. Brickfish.com is doing a fantastic job of that by melding user generated content and revenue generation into one unit [disclaimer - I have performed consulting services for BrickFish in the past].

Do you have any comments, or ideas are for monetizing while enhancing consumer value?

This is my first blog post. Ever. I should have done this years ago. Apparently, the first bloggers started at it in 1994, around the same time I began my first technology startup, a web consultancy. Although I was knee deep in the web, I had never heard of blogging, and was sleeping about four hours a night. No blogging for me at that time.

By 1999 when blogging became more popular, DDS (the unfortunately unappealing acronym for my first startup) had grown so much that I had to split it into two companies with a total of perhaps 60 employees. I was also an executive board member and CTO of a venture backed online mortgage company. In retrospect, I probably would have both enjoyed and benefited from starting a blog at that time. But I was too busy, or at least that is what I told myself.

Which leads us to the question, “How far ahead of the chasm am I?” I am speaking of Geoffrey Moore’s chasm, of course. Am I an enthusiast or a visionary? Or (shudder) am I part of the early majority (the first post-chasm group).

My wife would like to peg me as the earliest of enthusiast adopters, always trying out the newest and uncertain technologies. As evidence, she would present the very nature of Blir, one of my more successful (prior) companies.

Blir was a custom software development and technology strategy consulting firm. For clients such as Albertson’s, Exodus Computing, and Bain & Company, we cut deep into the bleeding edge of technology to find ways to improve performance or create new business opportunities. The technologies we used were typically so new that our clients had very few people to turn to for implementing them. It was, of course, a critical part of the way we won business against the much bigger players in the market.

In response, I would point out that while we used cutting edge technology, our goals were eminently practical. Enthusiasm for the technology itself was far less important than excitement about what it could do for a company. In 1996-97 I had failed with an angel backed e-commerce marketplace. That (hard) lesson taught me to think twice about technology opportunity. Part of Blir’s marketing was that we could leverage our past failures to benefit our clients. The web, after all, was still a very woolly and frightening place.

By now, I can tell far too many stories about how uncertain web technology and startup businesses can be (mine included). Have these experiences changed me? Made me more reluctant to jump into new technologies? For example, I have put off purchasing an iPhone until version 2 comes out (I want 3G support). Also, I’m rarely the one playing with the latest nightly builds of an open source project. I go straight to the most stable release, and stick with it until there’s a reason to change.

As a serial entrepreneur (and sometime venture capitalist), it is my job to keep abreast of the latest technology. I like to think that I’m in the “visionary” bucket of adopters, which is where I think people such as I belong. But are hard knocks pushing me back along the adoption curve? Am I becoming a part of the “early majority” of adopters? My wife would tell you no, but it’s definitely something I’m keeping an eye on.

Readers, where are you on the adoption curve? Where do you think entrepreneurs should be?