Composr Tutorial: Optimising Performance

Composr is very heavily optimised so that pages load as quickly as possible. This tutorial will provide information on techniques and issues for increasing throughput, so that your site may support more visitors. Some of these techniques are programmer-level in complexity.

If you have need to be able to take particularly high load, professional developers are available for custom tuning.

The tips of most importance are given in bold.

Performance tuning is multi-faceted, broadly fitting into:

• Making all page requests faster
• Making the most common page requests extremely fast (e.g. static caching for guests)
• Improving the number of page requests per unit time (throughput)
• Removing specific bottlenecks
• Stopping requests blocking each other (e.g. database locks)
• Blocking abuse, such as bad bots, DOS attacks, misrouted traffic, and flooders
• Improving page download speeds (e.g. compression and minification)
• Improving front-end rendering time
• Optimising server configuration

There are many angles to come at the problem, and to really do a good job you need to come at all of them, on every layer of the web architecture.

Limiting factors could be:

• CPU (most common)
• Memory
• Disk access speed (quite common if you do not have an SSD and fast I/O channel)
• Networking speed (rare)

Making PHP run faster

Speed can be approximately doubled if an "opcode cache" is installed as a PHP extension. These caches cache compiled PHP code so that PHP does not need to re-compile scripts on every page view. The main solutions for this are:

• Zend OpCache (this is bundled by default in PHP 5.6 and higher, recommended)
• APC (free) (or APCU compiled with APC compatibility)
• wincache (free, Windows only)
• xcache (free)
• eAccelerator (free)
• ionCube PHP Accelerator (free)

Only one opcode cache can be used, and they often need to be manually compiled against the PHP version on the server and then installed as an extension. Such system administration is beyond the scope of this tutorial.

Do not set the persistent cache's memory higher than needed, especially on a fast CGI or module-based PHP installation – for at least some persistent cache backends the memory is pre-allocated, for each PHP process that is running. For APC, this is set via the apc.shm_size setting.

Persistent database connections

For MySQL you can configure Composr to use persistent database connections. This is not recommended on shared hosting because it may annoy the webhost, but if you have a dedicated server it will cut down load times as a new database connection does not need to be established.

These are enabled through the "Installation Options" (http://yourbaseurl/config_editor.php).

Composr caching

Composr provides many forms of cache, to allow the system to run as quickly as is possible.

The Composr caches are:

• language cache: this removes the need for Composr to parse the .ini language files on each load
• template cache: this removes the need for Composr to parse the .tpl template files on each load
• Comcode page cache: this removes the need for Composr to parse the .txt Comcode pages on each load
• Comcode cache: this removes the need for Composr to parse Comcode whenever it is used
• block cache: this removes the need for many blocks to be fully executed whenever they are viewed – they are cached against the parameters they are called up with using a per-block tailored scheme
• theme image cache: this removes the need for Composr to search for theme image files whenever they are referenced by code (a code could translate to perhaps 10 different URLs, due to the flexibility of the system)
• values caches: this isn't a specific cache, but caching of values such as member post counts removes the need for Composr to recalculate them on-the-fly
• persistent cache: this is a very special cache that is explained in the next section
• advanced admin cache: this can be turned on in the Admin Zone configuration to let admins have cached pages on their computer that are immediately (without server communication) interstitially displayed (for roughly 1 second) while the server generates the up-to-date page
• fast spider cache: this can be turned on from the Installation Options editor to feed static pages to bots, to stop bots hurting site performance
• self learning cache: this can be turned on from the Installation Options editor and allows pages to learn what resources they need, for efficient bulk loading of essentials while avoiding loading full resource sets upfront

General tips

• Use PHP 7, as PHP 7 introduced massive speed improvements.
• Don't have addons installed that you don't need. Some may have general performance requirements, such as catalogues (adds custom fields checking to all content types) – while others may just have general over-head in terms of sitemap generation, language string presence, moniker registration, and hook execution.
• Enable the disable_smart_decaching option in the installation options. You can make it intelligent, by only allowing smart decaching if the FTP log was modified recently.
• You can mark out parts of a complex Comcode layout as having quick_cache by moving the content into individual Comcode pages then using the main_include_module block to insert it back, with the quick_cache parameter turned on for that block.
• The defer-loading option block parameter is also useful, although it puts a requirement on JavaScript.
• If you are using a particularly visually complex block (e.g. deep pop-out menus) then use the quick_cache block parameter on it if possible.
• The main_news and main_forum_news blocks have an optimise parameter which will simplify down the Comcode of anything displayed such as it will be stored statically. This usually will have no impact, but may increase performance if a lot of Comcode tags were used. It does have an impact if dynamic elements are used within posts, such as Comcode that check's a user's usergroup or displays the current date/time.
• Avoid making very long Comcode pages if they are just going to be static for all users. Ideally for usability pages should not be very long, but there are cases where you may choose for them to be (e.g. large glossaries). You can put {$,page hint: no_smart_conversion} into a Comcode page to prevent the WYSIWYG editor converting it to Comcode and otherwise use the Comcode html tag to include pure HTML content.
• If you get a lot of 404 errors, it is best to make a static 404 page instead of using Composr's which can be relatively intensive. You activate a custom 404 page by putting a reference to it in the .htaccess file (our recommended.htaccess file does this for Composr's). Or for IIS, there is sample code in the web.config file.
• If you are optimising your server infrastructure to be able to handle large numbers of users don't forget to change the value of the "Maximum users" config option to something much higher!
• Even though Composr supports various database vendors it is optimised for MySQL. You should also run the latest stable version so that you benefit from all their performance optimisations.
• Even though Composr supports various third party forums it is optimised for Conversr.
• Servers may have incorrectly firewalled DNS resolution, in which case the Composr "Spammer checking level" setting must be set to "Never" to avoid a 28 second timeout happening on each anti-spammer check.
• If your website doesn't need to be able to run without a wider Internet connection (i.e. isn't an Intranet), then you could blank out any of the unmodified JavaScript libraries Composr includes (e.g. jQuery) and instead include references to a JavaScript CDN via direct includes in the HTML_HEAD.tpl template.
• Comcode tabs can take a page-link rather than normal Comcode contents, for the tab to open up via AJAX (don't use this on the default tab though, as that won't work).
• If you make very heavy use of large and complex tooltips then you may want to consider loading these via AJAX. This will require some fairly advanced coding skills, but it may be worth doing. Look at the COMCODE_MEMBER.tpl template for an example of AJAX tooltip code.
• Disabling URL monikers can help with performance; however this will also impact SEO if you think your URL paths are key to that.
• Block unwanted bots you see in your logs using robots.txt, block URLs that you don't care to be indexed, and set a Crawl-delay value in there.
• Automatically kill slow searches (see "Auto-kill slow searches" in the Search tutorial).
• Enable the "Gzipped output" option. Unless you're using Cloudflare's optimisation features, in which case disable it so Cloudflare can do that instead.
• Sometimes you may be flooded with random traffic. For example, the chinese "great firewall" has been observed to randomly reroute traffic to avoid it reaching certain sites. If this happens, you may want to use a .htaccess file (Apache-only) to spot undesired request patterns and terminate those requests.
• Don't allow thousands of posts in forum topics, as navigating within becomes very inefficient for the database.
• Make sure PHP was not compiled with --disable-ctype.
• You can add {$,Quick Cache} into Comcode pages to enable the equivalent of quick block caching for that page.
• You can edit the sessions table to be of type HEAP, which will have it stored in memory rather than disk (it has no need to be saved to disk permanently).
• Avoid very long forum topics – split them up if they get too long. This is because jumping to the latest post has to search back through all posts in the topic to work out pagination positioning. Additionally just going deep into the topic uses a lot of resources for the same kind of reason.
• If you have large numbers of topics then consider deleting topics that have little value.
• Deny guest access to pages that are necessarily computational intensive – to stop bots consuming too many resources.
• If you have custom .php scripts that are called a lot, consider whether they could just be static files – or if the output can be statically cached and a transparent webserver-level redirect can be used to bypass PHP. PHP has infinitely more overhead than a static file, and is often a bottleneck due to limited quantities of FastCGI workers.

Server configuration tips

The fastest way to configure PHP is usually as FPM (FastCGI), on top of a threaded web server (e.g. Apache's mpm_event):

• While the traditional Apache mod_php is fast and simple, it cannot run on top of mpm_event because it is not thread-safe, so it doesn't scale up well – you'll find you need more Apache processes just to deal with static requests, but each comes with the overhead of mod_php.
• suPHP and CGI are likely too slow for you.
• FPM will give you a suEXEC-like capability without actually needing to configure suEXEC.
• FPM has the capability of scaling up and down the number of PHP FastCGI processes dynamically, within the bounds you configure.
• Be aware that this requires a baseline of RAM for every website on the server that is running under a different username because the FastCGI processes are user-specific.
• The more RAM you have, the quicker parallel PHP requests can be handled as more FastCGI instances can be configured to remain resident in RAM.
• Don't have an Apache or FastCGI (FPM) configuration that allows more Apache/PHP processes than you have memory for when a server is under high load – it is much better to have saturated CPU than to have your server start page swapping.
• If you are using Composr features where AJAX is involved, expect parallel PHP requests even when only a single user is on the website.
• You'll likely want to carefully configure your FastCGI settings for each website, based on available RAM and anticipated load.
• If you are configuring your websites via a hosting control panel such as ISPConfig, make sure the default www site isn't reserving FastCGI instances when it's not even being used.
• Configure your SSL options appropriately; basically there is a tradeoff between perfect privacy ("Perfect Forward Secrecy") and speed.
• Set up fail2ban so that you aren't constantly having to log failed login attempts by hackers to SSH etc (this can be a big I/O performance hit).
• You may want to use only systemd's journald, rather than rsyslogd; on some Linux system rsyslogd is configured to mirror entries written to journald, which can be very intensive due to the frequent reading in and out. journald is higher performance as it is an efficient binary format with a cursor feature.
• Consider whether you want journaling or not – it has a significant I/O overhead.
• If you use Dropbox for server backup, and have other users actively saving files into the account, only keep it started for a short time after your backup script finishes – as it will do a lot of I/O updating its SQLite databases whenever any file in Dropbox is changed.

Your server should be using none, or very little, swapspace. If swapspace is being used it implies that the server doesn't think the free RAM is enough and it will be actively moving stuff into and out of that swap space as your server serves web requests – a real performance killer. You can adjust the threshold when a Linux server will start using swapspace.

Persistent cache

The persistent cache is a cache that aims to store regularly-accessed data in-memory between requests, so that it does not actually need to be loaded from the database or recalculated on each page load. This cache removes about 30% of the page load time, but most importantly, takes load away from the database, allowing the database to become less of a limiting factor in high throughput situations.

The cache is implemented to work with any one of:

• APC (or another opcode cache mentioned in the intro paragraph), which provides in-memory storage features as well as an opcode cache and is associated with core PHP development (hence why we recommend it as the opcode cache to use)
• memcache ('memcache' is the PHP extension for the 'memcached' server), which provides a heavyweight solution to memory sharing – it is not recommended that this be used for typical websites, as memcached requires additional configuration
  • or memcached, which works via the other PHP memcached server extension
• Wincache – a PHP accelerator developed by Microsoft, optimised for Windows
• XCache – another PHP accelerator
• eAccelerator – an old PHP accelerator that is no longer developed, so using this is not recommended
• disk cache – while this does increase disk access, it still provides a performance boost over not having a persistent cache

Composr will not use a persistent cache by default but it may be enabled from the Installation Options editor.

Composr does not cache processed content in memory that has no special featured status, as this would only trade reliance on CPU for reliance on memory in a non-productive fashion.

Be aware that some persistent cache backends may be tied into a PHP process, rather than held system wide. This is the case for APC. Therefore if you are using a CGI architecture the full contents of the cache will be dumped each time a web request finishes (defeating the entire purpose); for fast CGI it's not as bad, as processes serve multiple requests, although it does mean there is a lot of duplication between each fast CGI process.

Aggressive caching for bots

If you want to serve cached pages to bots, put a line like this into your _config.php file:
All user agents identified as bots or spiders, or listed in the text/bots.txt file, will be served out of a cache. HTTP caching is also properly implemented.
The cache lifetime in this example would be 3 hours, but you can change it to whatever you require.
The cache files are saved under the caches/persistent directory.

If you want any Guest user to be cached like this, set:

'keep' parameters

This is not recommended, but if you really need to squeeze performance, you can disable the 'keep' parameters:

Disk activity

If you have a hard disk that is slow, for whatever reason, you can put these settings into _config.php to reduce access significantly:
They all have effects, so be careful! There are reasons these settings aren't the defaults.

Rate limiting

A site can be impacted by a flood of requests from a machine, and it is wise to block this.

If you're on a shared host, requests floods can even get you suspended, or your whole site automatically rate limited.

To enable add this to _config.php:

This produces soft errors with the correct HTTP header. The errors happen early, before Composr boots up.

Note that anyone behind a shared proxy server will share an IP address. If this is common you'll need to raise limits to compensate, or not enable this feature.

CloudLinux

CloudLinux is popular with webhosts. It is a modified version of Red Hat Linux that can limit the performance of individual accounts, based on the lve tools.

CloudLinux tracks maximum average CPU, calculated for the worst 5 second time period your site performance has in any particular hour (by default). This is calculated across all CPU cores, so it tends to trip on request floods (rather than any single slow request), with the request processing spanning multiple cores.

CloudLinux also tracks average CPU, but this is less of an issue because the average tends to be much less than the maximum, but it is the maximum that hosts will typically look at.

Webhosts may automatically produce warnings based on the CloudLinux maximums getting hit, or even automatic suspensions. At best, sites will automatically have their CPU kept to the configured maximum.

cPanel users will find a resource usage graph is made available. Note that the 100% line on this graph represents 100% of the configured limit, not 100% of a CPU core or of combined CPU. The average and maximum lines are scaled proportional to this configured limit also. This will confuse you if you aren't aware of it, because everywhere else lve is configured and monitored in terms of percentage of total capacity.

Database backups

You may have some kind of script that does database backups. Follows is some advice for that script…

1. Call mysqldump with the --skip-lock-tables --quick --lock-tables=false parameters. In MySQL 5.7+ there is a mysqlpump command, which is an alternative to mysqldump. It is more efficient and doesn't need tuning as much.
2. Do not backup the contents of certain volatile/large/non-important tables. For mysqldump use multiple --ignore-table=database.table parameters (you cannot omit the database name in this syntax). For mysqlpump use a single --exclude-tables=table1,table2,table3 style parameter.
   The tables:
   
   • cache
   • cached_comcode_pages
   • captchas
   • cron_caching_requests
   • post_tokens
   • messages_to_render
   • ip_country
   • sessions
   • stats
   • temp_block_permissions
   • url_title_cache
   • urls_checked
   • digestives_tin
   (put the table prefix in front of these names of course)
3. Run the backup at a time when nobody is accessing the site.
4. Avoid backing up test databases, there's no need to stress your machine load to back up what does not need to be backed up.
5. Consider a professional backup solution. If you have a large site, any amount of load peaking and row locking could cause major issues, so consider investing in a MySQL/disk backup solution that is smarter.

PHP time limit

If you are really constrained for resources, and worry some requests may take too long in some cases, you can lower the PHP time limit from Composr's default 60 seconds.
Note that Composr does raise the limit on its own when really needed.


Note that PHP's time limit does not count time taken doing database queries and other external tasks.

Image dimensions

Composr won't hard-code image dimensions into templates, but some may use the IMG_WIDTH/IMG_HEIGHT symbols to auto-detect them. This has a very small performance impact – you may wish to hard-code it once your theme is complete.

E-mails

Sending e-mails can be an intensive task because Composr has to work out what CSS to include for each e-mail, put templating together, and connect through and send to the e-mail server.
Turning on the "E-mail queue" option will avoid this happening immediately during user actions, deferring it. It will make forum posting, for example, faster.

Server

A faster and more dedicated server will make Composr run faster. This may seem obvious, but in the efforts of optimisation, is easily forgotten.

CPU speed will be the most limiting factor for most websites: so this is the first thing that should be considered. Our analysis of web servers targeted to enthusiasts and small businesses show that servers commonly only have around 10% of the performance of a good desktop machine – look at the CPU family carefully, don't just count the Ghz! Also go with an SSD if possible.

Searches

MySQL fulltext search can be a resource hog if your server is not configured properly and you have a large amount of content.

To make it run efficiently, the MySQL key_buffer_size setting (think of it as a general index buffer, it's not just for keys) must be high enough to contain all the indexes involved in searching:

• the fulltext index on the translate table
• indexes on the fields that join into that table
• other indexes involved in the query (e.g. for sorting or additional constraints)

It can be a matter of trial and error to find the right setting, but it may be something like 500M.
If the key buffer size is not large enough then indexing will work via disk, and for fulltext searches or joins, that can be very slow. In particular, if a user searches for common words, the index portion relating to those words may be large and require a large amount of traversal – you really don't want this to be running off of disk.
If you notice searches for random phrases are sometimes fast, and sometimes slow, it's likely indicating the key buffer has filled up too far and is pushing critical indexes out.

You can test cache coverage via priming the key buffer via the MySQL console. This example would be for searches on forum posts, and a key buffer size of 500MB:

You'll get a result like:
You don't want Key_blocks_unused to be 0 like in this example, as it illustrates the key buffer is too small. So tune your settings until you have them right.

Once you get it right, fulltext searches on large databases should complete in a small number of seconds, rather than tens of seconds. The first search may be slow, but subsequent ones should not be.

It also never hurts to optimise (via OPTIMIZE TABLE or myisamchk -r *.MYI) your MySQL tables. This helps MySQL know how to better conduct queries in general, and re-structures data in a more efficient way.

If getting Composr search to work well does not seem feasible, there is a simple non-bundled addon for using Google to do your site searches. Of course this would not have any sense of permissions and be limited to Guest content, but that's fine for most use cases.

The Cloud

If you have outgrown normal shared hosting / a single VPS or server, you can do a cloud deployment of Composr, and there are a few options:

1. Composr runs perfectly on Rackspace Cloud Sites hosting, where you can set up quite large instances (we have been told that behind-the-scenes there is database replication, to share database load).
2. You can run on Amazon EC2 Instances (or pretty much any cloud service that supports custom images, or PHP). There is a Composr Bitnami image for easy installation on Amazon's infrastructure. Bitnami also provide their own Amazon-based hosting service. If you want to use multiple instances you will need to set up your own MySQL replication and file-synching, but Composr does have support for it (see "Server farms"). There are some solutions to auto-scale out cloud instances, although many companies handle it manually.
3. You can set up your own cloud system on your own servers, or cheap dedicated servers. You could look at using something like OpenStack to achieve this, and you could use HHVM for great performance. This is similar to Facebook's infrastructure and a very challenging option, but perhaps the best from a huge-scale cost perspective.
4. You can use Google App Engine, to get automated scaling, with a tradeoff that it is a bit harder to initially set up. This is the most elegant and maintainable option and we generally prefer it over '1' and '2' if you are seriously investing in a maintainable infrastructure. We discuss this in the separate Google App Engine tutorial. At the time of writing Google App Engine PHP support is unstable and won't run Composr properly – this is outside our control, and we have been discussing it with Google directly.

Amazon EC2

Composr does quite a lot of disk space when checking for installed files, etc (much more than a traditional hard-coded custom application would need to do).
Because of this we recommend using 'magnetic' rather than 'SSD' storage, because SSD storage on Amazon's infrastructure has extremely very low IOPS allowances (presumably so that their internal bandwidth is used for very-fast burst reads/writes from/to fast SSD rather than more regular reads/writes).

Other optimisations

There are many possible config options, or set_value commands to tune performance described in the Code Book.

Also see the non-bundled performance_compile addon.

Ensure configuration is set

If a configuration page has never been saved, default config values from that page will be calculated on the fly, which is a little slower.
Go through and save them all, or run this Commandr command:

Huge databases

If you have really large databases then two issues come into play:

1. Composr will start doing sensible changes to site behaviour to stop things grinding to a halt
2. You might start worrying about databases being too large for a single database server and need to implement 'sharding'

You may also want to switch tables to InnoDB instead of MyISAM. If you do this then run this command in Commandr too:
This works around that InnoDB can't do quick row counts in tables, so implements an approximation for certain table counts.

Composr adaptations

Composr has been tested up to a million of the following:

• Comment topic posts for a single resource
• Ratings for a single resource
• Trackbacks for a single resource
• Forum/topic trackers (if you do this though things will get horribly slow – imagine the number of e-mails sent out)
• Authors
• Members
• Newsletter subscribers
• Point transactions
• Friends to a single member
• Friends of a single member
• Banners
• Comcode pages
• Calendar events
• Subscribers to a single calendar event
• Catalogues (but only a few hundred should contain actual entries – the rest must be empty)
• Catalogue categories
• Catalogue entries in a single catalogue category
• Shopping orders
• Chatrooms (only a few can be public though)
• Chat messages in a single chatroom
• Download categories
• Downloads in a single download category
• Polls
• Votes in a single poll
• Forums under a single forum
• Forum topics under a single forum
• Forum posts in a single topic
• Clubs (but not usergroups in general)
• Galleries under a single gallery
• Images under a single gallery
• Videos under a single gallery (unvalidated, to test validation queue)
• Quizzes
• Hack attempts
• Logged hits
• News
• Blogs
• Support tickets
• Wiki+ pages
• Wiki+ posts

(where we have tested the million resources 'under a single' something this is to place additional pressure on the testing process)

If there is a lot of data then Composr will do a number of things to workaround the problem:

1. Choose-to-select lists will either become non-active or be restricted just to a selection of the most recent entries (instead the user can follow in-situ edit links to get to edit something).
2. A very small number of features, like A-Z indexes, will become non-functional.
3. Pagination features will become more obvious.
4. In some cases, subcategories may not be shown. For example, if there are hundreds of personal galleries, those galleries will need to be accessed via member profiles rather than gallery browsing. This is because pagination is not usually implemented for subcategory browsing.
5. The sitemap might not show subtrees of content if the subtree would be huge.
6. Some Composr requests will average become very slightly slower (more database queries) as optimised algorithms that load all content from database tables at once have to be replaced with ones that do multiple queries instead.
7. Entry/Category counts for subtrees will only show the number of immediate entries rather than the recursive number
8. Birthdays or users-online won't show (for example)
9. The IS_IN_GROUP symbol and if_in_group Comcode tags will no longer fully consider clubs, only regular usergroups
10. Usergroup selection lists won't include clubs except sometimes the ones you're in
11. With very large numbers of catalogue entries, only in-database (indexed) sorting methods will work, so you can't have the full range of normal ordering control
12. Selectcode will not work thoroughly when using category tree filters if there are more than 1000 subcategories

All normal foreground processes are designed to be fast even with huge amounts of data, but some admin screens or backend processes may take a long time to complete if this is necessarily the case (for example, CSV export). Composr has been programmed (wherever possible) to not use excessive memory even if a task will take a long time to complete, and to not time-out. Composr implements a Composr task queue to be able to perform expensive tasks in a managed way.

There is a risk that people could perform a DDOS attack. For example, someone might submit huge numbers of blog items, and then override default RSS query settings to download them all, from lots of computers simultaneously. Composr cannot protect against this (we don't put in limits that would break expected behaviour for cases when people explicitly ask for complex requests, and if we did it would just shift the hackers focus to a different target), but if you have this much exposure that hackers would attempt this you should be budgeting for a proper network security team to detect and prevent such attacks.

Be aware of these reasonable limits (unless you have dedicated programming resources to work around them):

1. Don't create more than 60 Custom Profile Fields, as MySQL will run out of index room and things may get slow!
2. Composr will stop you putting more than 300 children under a single Wiki+ page. You shouldn't want to though!
3. Composr will stop you putting more than 300 posts under a single Wiki+ page. You shouldn't want to though!
4. Don't create more than about 1000 zones (anything after the 50th shouldn't contain any modules either). Use customised page monikers to build a 'directory structure' instead.
5. LDAP support won't run smoothly with 1000's of LDAP users in scope (without changes anyway).
6. Just generally don't do anything unusually silly, like make hundreds of usergroups available for selection when members join.

Sharding

If you have so much data (100's of GB, millions of records) that you can't house it in a single database server then you have a good kind of problem because clearly you are being incredibly successful.
It's at this point that serious programming or database administration will need to happen to adapt Composr to your needs. MySQL does have support for 'sharding' that can happen transparently to Composr, where you could use multiple hard-disks together to serve a single database. However this is not the commodity hardware approach many people prefer.
An alternative is to implement a No-SQL database driver into Composr. There is nothing stopping this happening so long as SQL is mapped to it. We have no out-of-the-box solution, but we do have full SQL parsing support in Composr for the intentionally-limited SQL base used by Composr (in the XML database driver) so have a lot of the technology needed to build the necessary translation layer. Practically speaking though this is a serious job, and at this point you are so huge you should be having a full-time team dedicated to performance work.

Server farms (custom)

Composr has special support for running on server farms (or custom cloud platforms):

• Composr file changes are architected so that any changes call up a synchronisation function which can be used to distribute filesystem changes across servers. As Composr requires no usage of FTP once installed, it presents the ideal managed solution for automated mirroring, once the initial network and synchronisation hook are created
• the Composr database system supports replication

In order to implement file change synchronisation, you need to create a simple PHP file in data_custom/sync_script.php that defines these functions:

In order to implement replication, just change the db_site_host and db_forums_host values using http://yourbaseurl/config_editor.php (or editing _config.php by hand in a text editor) so that they contain a comma-separated list of host names. The first host in the list must be the master server. It is assumed that each server has equivalent credentials and database naming. Due to the master server being a bottleneck, it will never be picked as a read-access server in the randomisation process, unless there is only one slave.
It is advised to not set replication for the Composr sessions table, as this is highly volatile. Instead you should remove any display of 'users online' from your templates because if you're at the point of replication there will be too many to list anyway (and Composr will have realised this and stopped showing it consistently in many cases, to stop performance issues).

Round-Robin DNS could be used to choose a frontend server from the farm randomly, or some other form of load balancing such as one based on a reverse proxy server.

Reverse proxying notes

If you are using a reverse proxy then it's important PHP sees the correct IP addresses of end-users, not your proxy server.
A solution is to set PHP's auto_prepend_file option to a script like the following:
This assumes that the server is only accessible from your reverse proxy. If not, you need to white-list the $_SERVER['REMOTE_ADDR'] values that may trust the $_SERVER['HTTP_X_FORWARDED_FOR'] setting.

Some web server configurations don't appear to support auto_prepend_file, in which case you can include by referencing it in _config.php:

Geographic distribution of servers

Some people with very ambitious goals want to have multiple servers dotted around the world. This requires some custom programming but the developers have studied the scenario and can role out solutions to clients if required. Our solution would involve priority-based geo-DNS resolution coupled with a reverse proxy server to also cover fail-over scenarios. The custom programming to implement this is in setting up the complex network infrastructure as well as implementing ID randomisation instead of key auto-incrementing to avoid conflicts (across distances replication can not be set to be instant).

Content Delivery Networks

There are a number of approaches you can take if you want to serve media from third party servers. Doing this reduces latency, and your own hosting costs, as well as some other potential side benefits such as automatic transcoding.

Theme files

You can have randomised content delivery networks used for your theme images and CSS files.

Change the Content Delivery Network option to something like:

As you can see, it is a comma-separated list of domain names. These domain names must be serving your files from a directory structure that mirrors that of your normal domain precisely.

Composr will randomise what CDN's it uses, so parallelisation can work more efficiently (this is the only reason for the comma-separation). Web browsers have a limit on how many parallel web requests may come from a single server, and this works around that.
DNS may be used to route the CDN servers based on geographic location.

There are many commercial CDN services available that can be configured in via this option. Many of them automatically will transfer over any file you reference on their service (via URL equivalencies) so that you don't need to worry about copying files over yourself.

If you use a service like CloudFlare it can automatically provide CDN functionality similar to the above, although from a single server only. CloudFlare works a little differently as it automatically rewrites your pages as they route through its systems. The CDN options above directly affect Composr's output.

Upload syndication

This section is mainly applicable to programmers, although specific service implementations may be available within addons

Composr contains 2 kinds of hooks for sending off uploads to third-party servers:

1. upload_syndication – transfers attachments and gallery files to other services, useful both for syndication and remote hosting; supports a basic UI for determining whether syndication happens
2. cdn_transfer – transparently transfers uploads directly to other services at an early stage; has no UI

S3 storage

If you receive more uploads than you can handle through normal disk space, you could use S3 storage. This will also give your users a marginal improvement in download speeds and provide better value storage (on cloud hosting traditional disk space may be more limited).

Composr has no direct inbuilt S3 support, but this is intentional, because you do not need it. Assuming you have a Linux dedicated server or cloud instance, you may use the 'FUSE' system to make Composr upload directory/directories use S3.

First you need to mount S3 storage to a directory, such as uploads/attachments, using a filesystem driver:
http://tecadmin.net/mount-s3-bucket-centosrhel-ubuntu-using-s3fs/
You'll want to mount a subdirectory of your S3 storage rather than a root. I recommend you mirror the same basic filesystem structure as Composr, so that you are mapping subdirectories with equivalence.
You'll need to rename uploads/attachments to something else to do this, and then move the old contents of that directory back into the S3 version of it.

Now immediately you are using S3 for storage, however URLs are still coming through your own server, which will work, but not efficiently.
Let's say we want normal image attachments to route through S3. We would edit the MEDIA_IMAGE_WEBSAFE.tpl template, changing:
to:
You'll need to disable the Composr "Reveal usage figures" option, as the usage figures can't be maintained with requests not routing through Composr.

Alternatively, you could do a similar thing using rewrite rules in the .htaccess/web.config file, but this would add additional latency.

Facebook's HHVM

HHVM is not officially supported, as it is fast evolving and sometimes has bugs. It should work though. Support can be provided individually upon commercial demand. As PHP 7 improved performance a lot over the previous official PHP releases, HHVM is not as interesting as it once was, as HHVM is a very heavyweight solution.

Facebook released an alternative implementation of PHP that works as a JIT virtual machine instead of an interpreter. It more than doubles performance, and is set to improve further as it is developed.
This is not suitable for most people for the following reasons:

• It requires server admin access
• The HHVM server is hard to run in parallel to normal web servers
• It only runs on 64 bit systems running common varieties of Linux (Fedora/CentOS and Debian/Ubuntu are all the ones known to work at the time of writing)
• You need to be comfortable compiling HHVM yourself (well actually there are packages, but this is relatively new)
• It generally requires some advanced skills

However, it is very suitable for those who need to serve a high capacity and have the skills and staffing to manage all the above.

The hphp_buildkit addon (bundled by default) provides a config and initialisation scripts for HHVM.
To invoke (we are assuming that HHVM has been properly installed and is in the path):

• Open a shell to the Composr directory
• Type sh hphp.sh (this is in the hphp_buildkit addon)

In order to support HHVM (which imposes some language restrictions) we made some changes to Composr, and updated our coding standards. There's no guarantee that any individual non-bundled addon will meet the coding standards, however.

We expect Facebook will improve HHVM performance will increase over time. They are working on improvements to the PHP language, which Composr could benefit from with further development. Type hinting is particularly interesting, as it has great performance potential and mirrors our code quality efforts that have already made Composr type-strict. We have preliminary support for this already, but it is not consistently maintained to be production-ready.

We have tried in the past to support:

• Roadsend (commercial software)
• Phalanger/PeachPie (Microsoft-sponsored)
• Quercus (Caucho-sponsored)
• Project Zero (IBM-sponsored)

But after quite a lot of work we have concluded these projects are too buggy to be realistic, not well maintained, or don't actually offer any performance improvement. Never-the-less, making Composr work on these platforms in the past has led to improvements in the Composr code quality, as these PHP replacements have a tendency to float different kinds of subtle coding errors to the surface.
There are also newer PHP alternatives such as HippyVM, or approaches that limit the PHP language significantly, but nowadays HHVM is a leading the others by a long way.

mod_pagespeed

Google's mod_pagespeed adds some nice micro optimisations. Composr doesn't do these itself because they would hurt CPU performance and modularity, but applying them in a fast Apache module is useful. To be honest, this will not really result in any particularly noticeable performance gain, but if you've got a good budget for trying lots of small things at once, every little bit helps and we thought it worth mentioning and documenting against.

mod_pagespeed is not supplied with Apache, but can be installed on most Linux servers. However you would need server access or for the webhost to do this for you.

Interesting optimisations include:

• https://www.modpagespeed.com/doc/filter-trim-urls (this breaks modularity – but that is fine for primary output streams)
• https://www.modpagespeed.com/doc/filter-quote-remove and https://www.modpagespeed.com/doc/filter-attribute-elide (Composr is XHTML-compliant, this breaks that – but that is fine for primary output streams)
• https://www.modpagespeed.com/doc/filter-whitespace-collapse (this will save some bytes, although not nearly as many as you think if you already have gzip encoding on)
• https://www.modpagespeed.com/doc/filter-image-optimize (mod_pagespeed may do a better job that your manual optimisation, and its support of WebP is very interesting)
• https://www.modpagespeed.com/doc/filter-js-defer (this will improve performance initial rendering time, but at the expensive of delaying interaction time – this may be a fairly good idea for very non-interactive sites, but a terrible idea for very-interactive sites – use with great care)

The work on HTTP/2 is also interesting (previously Google SPDY), and SDCH is extremely interesting. At this time of writing it is too early to really think about these, but they should provide automatic optimisations to Composr.

The CloudFlare service provides some of the same optimisations that mod_pagespeed does, as well as a CDN. However personally I'd opt to use mod_pagespeed and a custom CDN, as it is a more straight-forward configuration that doesn't require proxying your traffic (which itself will impact performance). That said, CloudFlare does provide nice anti-spam and traffic filtering, so if you have a big spam/DOS problem, it can be a very useful tool. CloudFlare's preloading feature also looks interesting, but it is not likely to result in noticeable speedup for most sites.

Preloaded links

Web browsers are starting to support link preloading. If you have a situation where you can predict what links people will click, you can use a standard HTML approach to define preloading for those links.

Profiling

A developer can run a profiler to work out bottlenecks in code, and decide optimisations.

xdebug

For hard-core programming, xdebug would be used. It collects the most detailed profiling information and there are many tools for analysing the data.

tideways-xhprof

Facebook developed xhprof, a fast profiler that can be used on live servers. It collects less detailed data than xdebug. xhprof was discontinued, but tideways-xhprof is a fork that is well maintained.

To use on a live server you can install the tideways-xhprof extension, then add this PHP code to your _config.php file:
The collected xhprof data can be viewed in Facebook's standard xhprof_html tool.
The collected .context files can be viewed in any text editor, and are designed to give a little context to the environment status when the profiling happened.

Commercial dashboards

Newrelic and Tideways both provide high-quality performance monitoring dashboards, for a fee. To a limited extent Google Analytics can also identify performance issues.

Composr's inbuilt profiler

Composr's built-in profiler is enabled via a hidden option (described in the Code Book, and comments in sources/profiler.php).

MySQL

MySQL has a "slow query log" feature which is useful for debugging database performance issues. Additionally the MySQL query SHOW FULL PROCESSLIST is very useful for showing current activity when you notice the database being slow.

Composr's page stats

If you have the stats addon you can also see what percentage of clock time each page is running for using this query:

Filesystem

If you create a data_custom/debug_fs.log file then all disk activity will be written to it if you load the site with keep_debug_fs=1.

Calculating hosting requirements

This section contains a simple methodology with made up figures…

Let's say it's a 0.5 seconds load time per page, and that is for 1 core of a 2 core 2Ghz machine. Let's say that at peak, there are 25 users, loading a new page every 20 seconds.
We only really consider full page hits, as that's where serious processing lies. Things like image downloads have minimal CPU impact.

Max page hits per second on said machine:
1 / 0.5 seconds * 2 cores = 4 hits per second.

Peak load:
25 users producing hits / 20 seconds between hits = 1.25 hits per second

So, in this sample we can take more hits than at peak. But of course you need to use real numbers.

It's quite a simplistic model, as things often burst, which means things queue up a bit, but also even out over time. Also if there's locking, such as a write operation locking a database table, things can queue, but that doesn't necessarily mean there'll be a high CPU cost – it may just mean traffic is rearranged while locked requests wait a bit longer.

If you are planning on using Amazon instances, you can resize them after-the-fact, but it's rather complex:
Change the instance type - Amazon Elastic Compute Cloud
You effectively have to take it down, then get Amazon to shift over your hard disk image to where they need to put it for a larger instance.

Debugging CRON performance

If you find the scheduler is running slowly, you can find out which individual hook is the cause by trying each one individually, like:
http://yourbaseurl/data/cron_bridge.php?limit_hook=calendar
Find the hooks that run by looking in sources/hooks/systems/cron.

Diagnosing server slow-downs

If you are administrating a server you could come across situations where the server 'grinds to a halt', or spits out depleted resources messages. This isn't a Composr problem, but just like any desktop computer can take on too much work, so can a server.

In these kinds of situations you need to identify what server resource is depleted. Typically it is either:

1. Disk I/O
2. Memory
3. CPU
4. Network I/O (available bandwidth)

A good system administrator will:

1. Stay on top of performance metrics, to know what needs optimising or where to spend more on hardware.
2. Develop experience isolating the cause of slow-downs, pointing to programmers where they need to do profiling and optimisation.
3. Configure a server so that excess activity results in appropriate error messages, rather than a crashed server (for example, by configuring the Apache settings to limit the number of processes and threads to what the server can actually handle).

Potential bottlenecks

Here's a list of metrics you may want to be considering…

• Memory:
  • Free non-swap memory
  • Assigned swap memory
  • Swapping rate
  • MySQL memory usage specifically
  • Combined PHP memory usage specifically
• Processes:
  • Process count
  • Process count of PHP CLI processes specifically
  • Process count of PHP web processes specifically
• Queues:
  • Queued MySQL queries
  • Queued web requests
• CPU (†):
  • Uptime
  • CPU utilisation
• Local I/O:
  • Disk I/O load
  • I/O wait time
  • I/O queue length
  • Performance:
    • I/O read latency
    • I/O write latency
    • I/O read throughput
    • I/O write throughput
• Network I/O:
  • Network I/O load
  • Inbound bandwidth utilisation
  • Outbound bandwidth utilisation
  • Packet loss percentage
  • Inbound ping time
  • Outbound ping time
  • Inbound network pipe speed
  • Outbound network pipe speed

† I/O load will also raise these metrics, which can be confusing.

Diagnostic tools

There are are various commands on Linux that are useful during diagnosis…


Additionally, the Apache web server has mod_status which can be configured to show what web requests are coming in, and how long they last for. This tells you a lot more than the Apache access log will, although looks at the Apache access log is still important to find actual timestamps of requests (i.e. to gauge throughput) and to make sure you're not overlooking request volume from individual hosts (which may use KeepAlive and therefore keep on the same Apache slots).

Some of the sample commands above are configured to keep showing results in a loop. Note that you will miss things doing this. For example, from experience if swapping is happening you may not actually see it in the vmstat output.

There is an 'art' to finding the cause of a performance slow-down. Often just one depleted resource will have a knock-on effect on others. For example, if I/O is saturated, memory may become depleted as Apache processes back up.

Tips

Some further tips:

• If you are on a VPS or shared hosting ask the webhost whether there is too much load on the host machine at some points during the day. For a VPS, consider whether you have dedicated cores, have virtual cores in a small shared pool (worst, depending on your 'neighbours'), or whether you have virtual cores in a large pool (very efficient and adaptive, so long as the host's cores aren't all full). Note that a VPS cannot truly know what load the host machine is under, so will not be optimal in how it manages system load – for example, low-priority tasks such as flushing disk write caches may happen when the VM 'thinks' the disk is under low load, when the real disk may actually be under high load.
• If you are on a VPS then don't assume that I/O will be as a regular machine's HDD or SSD – there's a good chance the host is using some kind of storage array with its own latency, in addition to VM overhead. I/O is always queued on any architecture so you'll get queue buildup, on both the VPS and the host, exacerbated by the latency. From my testing it seems Linux will flush disk writes early (to log files, for example) if it thinks the disk is free, which can actually lock things up badly on a VPS.
• Don't forget that Cron scripts may have a performance impact, especially because by default Opcode caches are not enabled for them.
• Be wary that web requests may be slowed down by things external to that request – for example if requests have been backlogged, resulting in saturated processing – or if intensive background tasks (such as Cron) are running in parallel.
• If you are seeing swap space used even when you have plenty of free RAM and a low swappiness setting, you can do swapoff -a; swapon -a to force it to clear.
• Check your robots.txt file actually can be called by URL (it's easy to accidentally set the wrong file permissions on it and not notice, for example).

Improving Apache access logs

Staring at an Apache access log can be frustrating, especially if you have a lot of static file requests, or your server's traffic is split across different logs. You also can't graph a raw log.

It may therefore be useful to use a custom script to visualise a log. Here is one I wrote for the compo.sr web server (which runs ISPConfig):

The script outputs a tab-separated format that can be directly copy & pasted into Excel. Excel can then be used to sort and graph the results very easily.

This script will need customising for your particular architecture.

Benchmarking

Slow-downs may be a result of something inherent in your architecture that you didn't expect, and not directly related to your load.

Here are a few useful Linux benchmarking tools:

• sysbench – examine your CPU performance
• dd – while not actually a benchmarking tool, it can help you find your disk read and write performance (you can find examples online)
• bonnie++ – examine detailed disk I/O characteristics
• siege – gauge your web throughput

See also

• Moving sites
• Installing on Google App Engine
• Webhosting for Composr
• https://www.woorank.com
• https://compo.sr/track…g_view_page.php?tag_id=16

Feedback

Have a suggestion? Report an issue on the tracker.