Building a server environment that’s scalable and reliable

Yesterday I joined a web cast on http://www.fastcompany.tv about Building a server environment that's scalable and reliable. In this session the architects of Technorati and Friendfeed told about their experiences. They had some interesting quotes about building reliable and scalable applications.
1) Logging is essential to estimate future usages of the system. However, logging in the database is too IO intensive. Log on the web server and aggregate the log files of all web servers in a separate database.
2) Data which are mostly static are not necessary to retrieve from the database. For instance a TOP 10 list of most popular documents. Store these lists locally on the web server. Local file IO is always faster than database retrieval. Build a mechanism to distribute the Top 10 List to all your web server.
3) Every new feature should be able take back. If a new feature does not work, or has a too big impact on the performance of the system, you should be able to remove this feature.
4) Push new code to specific user groups. When you have 700 web servers you are not able to update these web server at the same time.
5) If data is not in the cache, show nothing otherwise the system can't handle the load. If 10.000 users start retrieving this unavailable cached data from the server, your server will die. Build a mechanism to check if cached data is available. If not update your cache, so your application can make use of it.
6) Geographic user performance measurement.
7) During designing of functionality: Keep it Simple. It's already complex enough to manage, deploy and maintain.

The conclusion I had from these web cast:

It is Near Real Time.

It's all about partitioning of data. These quotes did realize me that we still are thinking to traditional in the way we are developing applications. Think in big numbers.