The Performance Testing Craft CMS with Blitz article sparked intense discussion among colleagues, especially the observation that Craft CMS demo site we used barely managed to serve 3 successful requests per second without caching. In this article, we collect the most thought-provoking ideas and attempt to address them with, you guessed it, more load tests!
The comments and resulting questions we’re going to address are as follows:
- Most sites will never experience such heavy load. What would the results look like with more reasonable levels of traffic?
- One takeaway from the article is that PHP’s pool settings play a significant role in the server’s throughput. What would the results look like with the pool settings, specifically
max_children, optimised for the server? - The Europa Museum demo site doesn’t use eager-loading or other optimisation strategies. What would the results look like using a performant site?
- The article suggests that concurrent requests that require PHP are much slower to respond to those that don’t. How computationally expensive is it to bootstrap the Craft app, and what would the results look like using a PHP file response versus an early Craft response?
What We’re Doing #
The right thing to do would be to test each of the points above in isolation, but due to time constraints (ours as well as yours!), we’ve come up with a few load tests that we think should cover them all.
We’ll use Locust to spawn at most 10 users, with max_children set to 20, on (a replica of) the PutYourLightsOn homepage, load-testing the following scenarios:
- An HTML file
- A PHP file
- An early Craft response
- A full Craft response
Setup #
Please refer to the original article for a detailed description of the setup.
We’ll use the same web server specs as before:
- 2 CPUs
- 2GB of memory
- Ubuntu 20.04
- San Francisco 3
- PHP 8.1
We’ll then configure PHP-FPM to increase max_children to 20 by running the following command.
sudo nano /etc/php/8.1/fpm/pool.d/www.conf
And updating the value of pm.max_children.
pm = dynamic
pm.max_children = 20
Using a value of 20 is an arbitrary choice, but with 10 users, it ensures that PHP can handle every request thrown at it (since each user can only make one request at a time). You, being a diligent developer, will follow a more scientific approach.
Load Tests #
For each load test, we’ll have Locust spawn 10 users at a rate of 1 per second and allow it to run for 60 seconds.
We’ll use the following locustfile.py file, commenting out the appropriate lines to test each of the 4 scenarios.
from locust import HttpUser, task
class HelloWorldUser(HttpUser):
@task
def hello_world(self):
# The HTML file.
#self.client.get("/home.html")
# The PHP file.
#self.client.get("/home.php")
# The early and full Craft responses.
#self.client.get("/")
HTML File #
This is our control test, as it measures how fast the server can send HTML responses to the client and should be considered the “best-case scenario”. Our home.html file contains the rendered HTML output of the PutYourLightsOn homepage. This test is comparable to the Blitz + Rewrite (warmed) load test, with a max RPS of 528.5 and an average response time of 18 ms.
PHP File #
This test essentially measures the performance of PHP 8.1. There’s a minimal amount of logic, ensuring that PHP has to do at least some work, which is simply a check for the existence of, followed by outputting the contents of, the home.html file.
<?php
$path = './home.html';
if (is_file($path)) {
echo file_get_contents($path);
}
else {
throw new Exception();
}
The results are surprisingly close to the HTML file, with a max RPS of 456.1 and an average response time of 20 ms.
Early Craft Response #
This test should help us get a feel for how much time it takes for Craft to spin up. It is comparable to the Blitz load test. We’ll use a custom module to output the contents of the home.html file and exit as soon as the Craft app initialises.
<?php
use Craft;
use craft\web\Application;
use Exception;
class Module extends \yii\base\Module
{
public function init()
{
parent::init();
Craft::$app->onInit(function() {
$path = './home.html';
if (is_file($path)) {
$response = Craft::$app->getResponse();
$response->content = file_get_contents($path);
$response->send();
exit();
}
else {
throw new Exception();
}
});
}
}
The results are telling – there is a glaring gap between the max RPS of 58.7 and the 456.1 for the PHP File test. The average response time of 170 ms is noticeably higher but not completely unacceptable.
Full Craft Response #
This test fully bootstraps the Craft app and all plugins and then renders the homepage. Although the site is optimised, the server manages to respond to at most 7.7 requests per second with an average response time of 1.3 seconds.
Observations #
With only 10 concurrent users (and therefore at most 10 requests at a time), there were no failures and none of the drama of servers falling over, as in the original article. Plotting all test results on the same scale, however, shows similar order-of-magnitude differences in performance.
The PHP File results are surprisingly similar to those of the HTML File test, meaning that running PHP is not computationally expensive in and of itself. As soon as the Craft app is bootstrapped, average response times jump from 20 ms to 170 ms, meaning that the server can’t respond to nearly as many requests per second.
The Full Craft Response results are better than the previous Stock results, but the average response time of 1.3 seconds with 10 users is sluggish for a production site and means that the server struggles to respond to 8 requests per second.
Running the Full Craft Response load test with only 1 user resulted in an average response time of 310 ms, so it is safe to assume that Craft response times start to suffer when the server is hit with multiple concurrent requests.
We ran one final test to see what the average Full Craft Response time per user would look like, testing with 1, 5, 10 and 20 concurrent users.
Enabling the debug toolbar on the front-end reveals that a request to the homepage requires 63 database queries to be executed, taking 71 ms. In contrast, a fresh installation of Craft without any plugins installed executes 15 database queries.
The number of database queries is not unusual for a homepage that pulls content in from various sections and has a handful of plugins installed, but if you consider that 10 requests result in 630 database queries being executed, it becomes clear why Craft struggles as the number of concurrent requests increases.
Takeaways #
Our main takeaways from the observations above are as follows.
- If you expect your site to receive medium to high levels of traffic, then Craft will struggle without a caching strategy. Our load tests showed that with 20 concurrent users, the site becomes painfully slow, with response times of up to 3 seconds (before images and other resources can begin being fetched). Keep in mind that these results are not representative of real “human” users, who will (hopefully) spend some time looking at pages before moving on to the next one.
- Although it is tempting to assume that your VPS comes optimised for running Craft, you should take the time to familiarise yourself with adjusting, at the very least, PHP settings such as
memory_limit,max_execution_timeandmax_children. Alternatively, using a managed hosting provider such as ArcusTech or Servd will save you the trouble of learning any DevOps. - Eager-loading elements is critical to reducing database queries, which allows Craft to respond to requests quicker and handle more traffic. Both the Blitz plugin and the free Blitz Recommendations plugin will let you know about eager-loading opportunities in your templates and how to fix them.
- Load testing is for everyone! Our experience is that the effort is well worth the insights to be gained about the performance of a site and the opportunities for making it faster and more scalable.