BestCrosswords.com architecture

This document describes the hardware and software architecture of BestCrosswords.com (more info about the site here). Hopefully, it will provide guidance to people wanting to undertake such a project, but are not sure where to start.

WEBSITE STATS

Here are some stats to help you assess the kind of scalability that can be achieved with the architecture on the specified hardware.

As of September 2007:

Monthly visits: 1.4 million
Monthly pageviews: 5.5 million
Members: 224,000
Peak simultaneous visitors: 4,500
Database tables: 30
Database size: 1.6 GB
Average page rendering time: 50 ms
Average load average: 0.33
Average CPU usage: 10%

ARCHITECTURAL OVERVIEW

The site runs on a single machine. An Apache server receives web connections. Its functions include: URL rewriting and sanitizing, SSL and logging. It then proxies connections to a Tomcat appserver. Tomcat is responsible for processing requests. MySQL is used to persist web sessions and business data.

    ,--------,    ,--------,    ,-------,
--->| Apache |--->| Tomcat |--->| MySQL |
    `--------'    `--------'    `-------'

Another machine is used for failover and has the same architecture.

HARDWARE & SOFTWARE

Both main and failover servers have the same configuration:

Hardware: 2.8 GHz Pentium 4 HT with 1 GB of memory
OS: Debian sarge (3.1) using the 2.6 kernel
Web: Apache 2.0.54
Appserver: Tomcat 5.5.17 with a customized JDBC session store
Database: MySQL 4.1.11 using the InnoDB engine
Mail: Exim 4.5


SOFTWARE ARCHITECTURE

I use a multi-tier architecture:

   ,--------,
-->| web UI |
   `--------'
        |
        V
   ,---------,   ,--------,   ,-------------,
   | service |-->| domain |-->| persistence |
   `---------'   `--------'   `-------------'
        ^
        |
   ,-------------,
-->| web service |
   `-------------'

The following technologies are used: Tapestry 4.0 (web UI), Axis 1.3 (web service), Spring 2.0 (web UI, web service, service), Hibernate 3.2 (persistence).

PERSISTENCE LAYER

Hibernate is used to map domain model entities to the database. Transaction support is provided by the InnoDB MySQL storage engine.

I also use Data Access Objects to move data in and out of the database. The bulk of the code the DAOs implement is query logic. Separating that logic from the service layer is essential to make database optimization as easy as possible.

DOMAIN LAYER

The site domain logic is implemented in this layer. The domain model contains about two dozens entities, implemented as POJOs.

SERVICE LAYER

There are about a dozen services in this layer, all implemented using Spring beans. Spring provides transaction demarcation and caching. The services use the DAOs provided by the persistence layer to access and persist the entities.

The service layer has two purposes. First, it is used to implement workflow logic. Workflow logic is business logic that doesn't pertain to the domain model per say, for example, sending a welcome email when a new member signs up. Second, it provides a procedural and remotable API to the layers above.

Data Transfer Objects are used to carry data in and out of this layer. The choice to use DTOs was made at the beginning because it was my original intent to expose that layer through web services. However, I realized that I wouldn't be able to keep the services APIs stable enough for the needs of a public interface so I created a separate web service layer. Given the chance to redo it, I would probably ditch the DTOs and use the domain model entites for data transfers. This would save me a lot of data mapping code and would certainly improve performance.

WEB UI LAYER

The web UI layer contains a couple hundreds Tapestry pages and components.

The components get their data from the service layer. Since that data can oftentimes afford to be a bit out of date, a caching strategy has been devised. I simply stick a caching proxy in front of Spring beans for which I want to cache the results. I then inject the cached bean into the components that can benefit from the caching. This reduces the load on the database and improves page rendering time.

I originally designed the web UI layer to be stateless until the visitor logged in, then it would become stateful. Tapestry makes it easy to do that and since the majority of the visitors would not log in, all was well. However, as web UI layer became more sophisticated, statelessness started to become a hindrance so the web UI is now running statefully from the moment a visitor enters the site. This increased the number of web sessions in the database, so I had to rewrite parts of the Tomcat JDBC session store to accomodate for the extra load.

WEB SERVICE LAYER

This layer gives third-parties a programmatic access to the functionality of the site. It is composed of a thin facade of services adapting and stablizing the APIs provided by the service layer. In order for me to evolve the interface without disrupting established clients, I incorporated a version number to the API. So far, all the changes made have been backward compatible but it reassures me that, if need be, incompatible changes can be made to the API, as long as the version is increased.

FAILOVER

The site was initially hosted on a shared server. Then it was moved to a virtual private server to finally make the transition to dedicated server. Recently, repeated hardware failures and downtimes prompted me to start using DNS failover to provide redundancy to the site.

Before rolling out the solution, I performed an experiment to assess the viability of DNS failover. You can read more about it here. Granted, this solution doesn't provide 100% availability since DNS changes take time to propagate (even with a low TTL), but it's good enough.

The setup is simple, I have two machines, each running its own copy of the site. During normal operations, the DNS A record of the domain name points to the main machine. When a failure occurs, the A record is reconfigured to point to the failover machine. The failover machine is kept in sync with the main server using rsync over SSH for the filesystem data and MySQL master/slave replication over a Virtual Private Network for the database. I host the DNS records at DNSMadeEasy because they offer basic server monitoring and automatic DNS failover. An added bonus of the MySQL master/slave replication is that it simplifies database backups. I simply stop the slave from replicating while I do a database dump. It has no performance impact on the master and the dumps have referential integrity.