A number of issues have come up over the last month or so where people are unable to connect to the 10C API or load a website in under 5 seconds. Others have noted that posting can take a while or retrieved conversation threads contain incomplete sets of data. Considering how this service is still in beta and sees less than 30% the traffic of 10Cv2, these problems are completely unacceptable. Something needs to be done, and done quick.
(Basically) The Current Network Design
One of the problems with the current version of 10C is that the servers are trying too hard to be too many things. At the moment, there are just two of them working together to handle both Website and API traffic. When a request comes in, the first available server responds with the appropriate data. If any data is written to the database, then that data is also copied over to its twin. In times of heavy traffic, clones can be added to the pool, though they will need a few minutes to update their local databases for the changes that have occurred since their images were made. I typically update the clone images once a month.
It's not a very workable plan in the long run, and it's not a great solution when the problem doesn't seem to be the hardware, per se, but the network capacity of the service provider. At the moment, 10Centuries is hosted on a number of virtual private servers at Sakura Internet in Osaka and Tokyo, Japan. I chose to go with this provider because they're local and never charge for network transfer. This means that even if I am sending a terabyte of data every month, it will not cost me a Yen. That said, we get what we pay for. The $1000/year I pay covers a number of servers that are typically always online. The network, however, can be over-provisioned or just downright sluggish for all the other traffic that moves through Sakura's data centres.
It's time to move on.
Next Week's (Basic) Network Design
As of this writing, I'm in the midst of moving 10Centuries from Sakura Internet to Amazon Web Services. Annual operational costs are estimated to grow by 20%, but this will allow the service to grow a little more logically. Audio and image content will be stored in S3 and pushed out via CloudFront for faster service. Web visits will make use of the web servers. API calls will be held by the API servers. Either group can grow or shrink as needs be.
But there's something else that will be possible as a result of this splitting of roles … something that was last seen in 10Cv1 five years ago: a fork and split of the 10C code.
While I could have a single codebase across all servers doing the job they're expected to do, this doesn't really lend itself to speed and efficiency. It won't happen right away but, over the coming months, the code running the API servers will evolve to build a more efficient API, and the web servers will evolve to work as a web server. Oh, and the web server code will be open sourced for anyone who would like to self-host their own 10C-powered site.
The last few weeks have been a little frustrating for people trying to use the service. That sucks, and I want to make it better. With this change, the service can continue to evolve to become a more robust, more responsive service that people can rely on.