Code Book, part 3 (Miscellany)

« Return to Code Book table of contents

Composr core performance

The Composr core is extremely optimised.

Most of the Composr page generation time can be put down to detecting the context the user is operating in, retrieving all components from their storage locations, structuring, processing whatever aspects of the output are dynamic, and outputting. This typically all-together takes a few hundred milliseconds. With opcode caching and persistent caching, it can be under 100ms, and potentially further with a platform like HipHop-PHP (which compiles code, and keeps code even more permanently resident between requests than an opcode cache).

The very core Composr responsibilities are as follows…

Input:

• Set correct character sets, for unexpected inputs (fast default case: ASCII request)
• Input sanitisation / checking / gating to defaults
• Input state management
• File/HTTP I/O stubs

Output:

• Find character set (fast default case: hard-coded character set optimisation)
• CSS and JavaScript inclusion/merging
• Breadcrumbs (disableable)
• Comcode-page display (fast default case: Comcode cache)
• Date/time display
• Time period display
• Output framing
• Tempcode assembly
• Tempcode rendering
• Tempcode symbol processing
• Tempcode output escaping
• Theme images (fast default case: persistent cache)
• Output state management
• Primitive type displaying
• Locales
• Basic template bindings

Internal dispatch:

• Finding linked scripts (internal caching)
• Error generation
• General link generation
• URL-moniker linking
• Page-link decoding
• Keep-parameter propagation
• Page-identification and dispatch
• Page searching (fast default case)
• Match-keys
• Hook searching
• Block and module calling
• Block processing
• Metadata querying
• Addon state detection (fast default case: persistent cache)

Configuration:

• Read base config and provide defaults
• Software versioning
• Config options API (fast default case: self-learning cache)
• Config values API

Caches:

• Find if a decaching request is active
• Static caching for Guests and bots (the ultimate optimisation)
• Persistent cache
• Self-learning cache
• Block caching

Context detection:

• Getting base URLs, based on SSL and zone context (internal caching)
• Detect safe mode (fast default case: hard-coded setting)
• Find current script context
• Mobile detection
• Rendering option detection [e.g. wide mode]
• Cookie detection (may be disabled)
• JavaScript detection (may be disabled)
• Browser capability/flag detection
• IP banning (disableable)
• Detection of user language (fast default case if internationalisation disabled)
• Self-linking/context-identification
• Detect of timezone (fast default case if internationalisation disabled)
• Server/HTTP-environment querying
• Users-online limit (disableable)

Users/Members:

• Find the current member
• Session restoration (fast default case: persistent cache)
• Session clean up (only runs randomly)
• Cookie logins
• Alternative login types
• Display names
• Custom profile fields
• Forum driver object initialisation and driver binding and driver
• Detecting and actioning logins

Processing:

• Memory management
• Intelligent code inclusion, overrides (fast default case: persistent caching of current code profiles)
• Database API (delayed connection instantiation; structural management code not loaded by default)
• Database API low-level driver
• Error handlers (only called exceptionally)
• Comcode (fast default case: simple conversions)
• Language requiring (fast default case: self-learning cache)
• Lookup language strings, support parameters, pluralisation differences, vowel/consonant differences
• Lookup translations, including linking to Comcode parsing
• Translation database storage
• Stats logging (disableable)
• utf-8 support
• Sorting
• Primitive types data conversions
• IP address detection and processing

We try and keep the core code size to an absolute minimum, due to memory requirements and PHP parsing time. Sometimes we have a little more code loaded up than needed, but only for things that are common even if not 100% routine. It is reasonable to assume a performance-conscience user will have an opcode cache, which removes the issue of code quantity.

Anything not marked with a specific bypass optimisation (in brackets) will still have been heavily optimised: the Composr core is extremely optimal.

Note that while we have a lot of user-specific and security features in the above list, but these can be bypassed when not needed via the static cache.

Various hidden optimisation flags are described later in the Code Book, and also in the Optimising Performance.

Philosophy

Engineering standards and trade-off

When coding for Composr, you should be careful to make sure all our design goals are maintained. Sometimes trade-off s have to be made between design goals, sometimes an option can remove the need for a trade-off, and sometimes an option would just bloat and confuse the product. Good balance and thoughtfulness is required before rushing to add the latest cool ideas.

We have established the following design goals (in no particular order):

• Security – Composr should be completely secure and un-hackable, but also be able to track offenders and be easily restorable if the worst happens
• Performance – Composr should be as efficient as possible, with both CPU and memory usage; things like caches and optimised code, and sensible feature design, can achieve this. Tip: use xdebug for profiling, along with WinCacheGrind.
• Great design – Composr should look excellent on every page
• General quality – Composr should not contain things such as spelling errors
• Flexibility / generality – Composr should be able to work in many different kinds of website. Features should be usable in different ways for different people. It should not have a restricted layout (hence Comcode, modules, blocks, and templates)
• International – Composr should work well across the world, across language and culture (as long as someone puts the effort in to do a translation)
• Compatibility – Composr should work well with common versions of required technologies
• Modability – Composr should be modular, and overridable; easy to change without tearing the things to pieces, preferably without even editing things (just adding)
• Usability – Composr should be very easy to use. The best user-experience and interaction-design methods should be used.
• Power / features – Composr should be as powerful as any competing product, but that should not be an excuse for implementing crap that bloats and over-complicates the product; we all want loads of features, but we don't want to confuse the users or tie ourselves up so tight that future development is very difficult
• Standards compliance – An accessible web, based on open standards, which is stable and not tied to any specific vendor, is desirable for all; by sticking to standards, you are ensuring Composr works on a wide range of technologies, and for a wide audience
• Documentation – Remember that the best documentation is when the interface describes itself, but the written documentation must be excellent.
• Consistency – The whole system should have common ways of doing things. Everything should seem consistent to the user, and be so behind the scenes. Consistency is piece of mind, consistency is ease of use, consistency is simplicity, consistency is less code to test and fix. I can't stress how important it is to be consistent.
• Independence – Composr should not have unnecessary dependencies, such as MySQL.
• Stability / Error-trapping / Input-sanitisation – Composr should catch all errors, not let things like PHP notices slip through. This means that programmers have to get used to a new level of strictness, we are much less tolerant of errors slipping through because for complex software like Composr we can't afford weak foundations.
• Scalability – Composr must work on sites both small and huge. For example, it's wrong to put all downloads in a list on any single screen – there could be 1000's.
• Portability – People should be able to move their websites between servers without much difficulty.
• Simplicity – Features only needed by a minority should be "off-by-default"

Why not more object-orientation?

Most PHP CMSs have gone in the direction of having very sophisticated object systems, with dependency injection, design patterns and namespaces.

Composr however doesn't use this kind of stuff, and breaks what some people would consider best practices.

This is entirely intentional and we want to explain why, because there are a lot of people out there with strong opinions on how to design a web system and we realise our approach is not the conventional one that a lot of people are pushing. We want to explain why we think we have built a far more productive environment in Composr. This will seem like a bit of a wild rant, but really we've gone into detail to hopefully shine a light on the Composr design decisions.

It's a matter of good engineering to consider all factors when coming up with a design. Here are some design considerations, in rough order of decreasing importance:

1. Ease and efficiency of coding for the framework
2. Performance of live code
3. Ease of understanding and maintaining the framework
4. Ability to automatically test
5. Strict separation of concerns to stop developers interfacing incorrectly with functionality
6. Ability to create complex abstractions and contextual shifts when dealing with special scenarios – for example, test runs and simulations

'1' is so important. We feel that with OOP-heavy frameworks, where you have to worry about relatively complex initialisations of every part of the system you want to touch, and how you are going to plumb all the different components together, it really makes the code take a lot longer to write due to the extra verbosity. In fact, 66% of code can easily be taken up just by extra plumbing, and this really hurts when you're trying to get ROI on your time investment. Yes, extra plumbing is not that hard, but it is time consuming to do and maintain and to read through.

Regarding '2', method calls are expensive in PHP, and variable storage uses a lot more memory than you think, as does the memory simply to load up PHP code. So, there really is a high cost in all the plumbing you might want to do – 66% more code could certainly mean 40% more memory usage, and 30% slower CPU performance. Users increasingly expect sites to respond extremely fast, yet CPU speeds have not really increased in a decade. On top of all this, frameworks also mean you need to load a lot more code than you need, due to having to load up structures that are more complete and complex than you strictly need to serve the simple requests you are normally serving. Most OOP-heavy framework users respond by slapping on a full-page cache, but this is very limiting because then you really block the social interactivity you really want from a system like Composr.

Regarding '3', OOP-heavy architectures try to abstract away details to make things simpler, but in the process of defining custom interfaces for all modes of interaction, and plumbing to connect all the objects together, you end up with a huge amount of new interface complexity to both understand and maintain. We think this really is a zero-sum game. There's a really important design principle in the field of agile programming called 'You don't need it' (YDNI), and OOP-heavy architectures often go against this by trying to plan out for all access scenarios upfront and abstract them perfectly – it's much better that we don't waste resources (both programmer time, and system resources) until we need to make a particular access pattern work, and then we can optimise it for how we need it. Apart from cost/schedule, YDNI is also important because you can't predict future requirements well – trying to anticipate too much by putting in too much structure between components actually works against you rather than for you, giving you into patterns you actually would not have wanted. In other words, too much structure costs a lot to make and can tie you down as much as it can guide you.

'4' is definitely something OOP-heavy coding helps with. However, you can achieve equivalent things in simpler designs also.

'5' is another win for OOP-heavy systems, but it does kind of assume you are employing code monkeys, and that is never going to work however much you try and mollycoddle them.

'6' is something OOP-heavy systems do very well, but read on.

As you can see, simpler designs solve the most important concerns best, and then remaining concerns are well served by OOP-heavy designs. However, our Composr design is not actually a naive simple design at all – it's actually very carefully structured, and has a high degree of strictness/discipline in many areas. Composr is definitely not anti-OOP by any means, it is only anti-over-engineering – we actually rely on the strengths of OOP in many key areas of the system. Here's how our approach works to achieve the commonly-cited OOP-heavy advantages (flexibility, maintainability, clarity, and stability) without slowing development, complicating code, or hurting performance:

• We can achieve the equivalence to dependency injection as our output and execution context states support a stack-based switching mechanism, allowing us to create sandboxes very effectively for those few cases where it is useful. The critical difference is we achieve it in far less code, with better performance, in a way that is easier to understand, and where the burden for it is only felt when you actually need this advanced functionality, rather every time you use any component of the system.
• We can achieve inheritance via our code overrides system, which also provides the ability for users to override components without having to get down and dirty with coding. Different versions of code can be placed and tested as overrides, and new functionality can be placed using overrides. This does not suffer from the same problems as copy and paste coding, as overrides are designed to work on the function level, via some clever code we have in place, and we can even make line-by-line changes should we need to, or extend individual functions.
• We do have unit tests to test critical functionality. Admittedly these are more prone to break if we start redesigning stuff without confirming the interfaces used by the unit tests are still appropriate, but the difference is really not that much.
• We keep a very tight rein on global variables that we define. There is a formal naming scheme, and we have an automated test that scans the codebase and verifies that we don't start accessing any of them all over the place. Generally a global belongs in the file it is defined in, and from this point of view the source code files are like classes – the globals in them are their private properties. In essence, rather than enforcing this through verbose code plumbing and memory consumption, we automatically enforce it through automated testing for compliance to our coding standards. If a global is allowed for multiple access in different places (usually for performance), then we have a rule that it has to be given a formal annotation to what it does.
• We have the ability (via APIs and simple glue objects) to 'reflect' on content types within Composr in a really abstract way, even though the content types aren't defined by "fat objects". This provides the same power that rigid conformability allows, but without hurting performance or actually requiring rigid conformability.

The Composr design is actually based on how PHP is supposed to work, rather than Java-envy. Critics often think of PHP as inelegant, and it is true there are a lot of consistencies in the language and that some old legacy PHP features (that we don't use) were very poorly designed – but PHP is also widely popular because it is cost effective and easy to use, and this has been proven with top sites like Facebook using it. PHP is built on the concept that web requests should be simple and fast – they should get straight to the point and get what is needed done, and this is why PHP runs each request in its own 'sandbox' rather than as a thread on an application server. Web requests start up, do what they need to do, then are killed off. We therefore design Composr around the idea that these web requests are simple and fast, rather than trying to build a castle in the sky for a process that will terminate anyway as soon as its finished sending a result back. PHP environment access is based around 'super globals', just like Composr's access to key interfaces is – like the database connections. However, we improve super globals by adding the stack based context switching functionality described earlier.

So… hopefully you can now see past the anal arguments some people make that everything should be designed a certain way – an expensive, complex, way, that performs really poorly in the name of a theoretical but unrealisable maintainability. Don't believe any methodology's hype without doing a good analysis first – maintainability really comes down to the efforts the team does in keeping a codebase tidy and consistent, not much else. As good engineers we also know we want a framework that provides the most efficient possible coding method, in addition to the robustness, flexibility, and maintainability that we need. Knuckling down and over-engineering everything is going to make coding progress very slow/expensive and probably going to make code less maintainable rather than more maintainable – "simple by default" is a better plan if implemented with care and maintained with agility.

Rant over .

This is a good time to mention we have two frameworks derived from Composr called Conposr and Conposr++. These are built for standalone web apps and have a different set of design principles behind them.

Troubleshooting

There are a number of issues developers face when programming for Composr. Our high standards and development mode means issues are raised to the surface that less skilled programmers may not easily notice when working with other systems.

XHTML

Composr is coded against XHTML5, which is the XHTML version of HTML5. In other words, it is a stricter (in a good way) version of HTML5.

The extra strictness of XHTML is now largely historic, as no websites (including Composr) truly force the browser to enable the full XHTML-strictness, while browser XHTML modes aren't cut-down like they once were.
Never-the-less, it encourages good coding discipline – clean, intuitive code.

Composr notes:

• Ideally do not use the .innerHTML JavaScript property. Instead you will should use the Composr  set_inner_html and get_inner_html functions that Composr supplies. Our functions have been carefully tuned over the years to work on all browsers to handle things like embedded JavaScript code in a consistent way.

MySQL errors

Composr is not just built for MySQL. It is designed to work with any kind of database software. It is all to common for developers to assume specifics of the MySQL platform are standard SQL and can be used freely. MySQL in fact is very non-standard in certain ways. Composr enables "MySQL strict mode" to force MySQL to (correctly) complain when mistakes (such as non-specified field values) are made.

It is advisable for you to:

• write SQL conservatively: avoid SQL syntax/functionality that Composr does not already use somewhere because that syntax/functionality probably is vendor-specific
• avoid being sloppy and getting MySQL to perform automatic type conversions and to automatically guess missing values during insert queries.

See the Using PostgreSQL with Composr tutorial for a practical discussion for PostgreSQL as a specific example.

Blank pages

If you find you are getting a blank page it is likely because you either have exceeded the PHP memory limit, or a PHP parse error has happened.

You can often find out what the error was by putting &keep_show_parse_errors=1 into the URL. This will make Composr run slower and it might introduce some weirdness, but it often will also reveal the errors. Additionally, enabling the PHP display_errors and display_startup_errors in your main php.ini is a good idea (for a development server only).

Sometimes this advice will not help: if there is an "@" (error suppression operator) somewhere above the function call chain for where a fatal error happened then this will make fatal errors result in a silent blank screen. Composr cannot detect them. You should use "@" sparingly anyway.

Query limits

If you get errors about query limits consider optimising your code to use less queries. If you cannot do this, you can set:
…in your code.

You can also put &keep_no_query_limit=1 into the URL to have query limits temporarily disabled. Often when caching is disabled or empty the query limit can be exceeded.

Debugging, and stack traces

IDEs

Some developers like to use IDEs with breakpoint support, watches, code-stepping, etc. PhpStorm is generally considered the best IDE, and is our official IDE; NetBeans is good too. The core Composr staff generally work with simpler editors most of the time.

Simple but effective

When not regularly using IDEs, the core Composr staff usually debug via just putting temporary bits of code in, like:

You don't need all that code all the time, but if you don't use "exit" then you do need "ob_end_flush". This is because Composr uses an "output buffer" for its Tempcode system, and by default any output you do will get written into an arbitrary point in the Tempcode tree – often never to be seen at all!

In fact, rather than the ugly code above that just dumps information into the output stream, you can do something a lot fancier:

This should show the message in a nice position in the Composr output.

Of course, make sure you don't leave any temporary debugging code around when you finish!

Dev-mode

If development mode is running then the inspect function will be available in PHP. You can run commands like:
…and Composr will output the contents of these variables very neatly for you, and give a stack trace.

If output has already started then Composr will give you your details in an info box at the bottom of the displayed screen; otherwise you'll get a text dump.

This is very useful when stepping through code. Often bugs are because you are using the wrong data types (e.g. Tempcode objects being treated as strings). Temporarily placing inspect calls in your code and moving them along as you analyse exactly what is happening is a good way to test exactly what is happening for the occasions when the automatic stack dumps are not available or do not reveal the real cause of the problem.

If you have an IDE with a debugger then this is better, but we find most developers do not have one, and also PHP debuggers tend to be quite unreliable. This old fashioned manual technique of manually tracing through code works well if you are quick at making changes and refreshing the browser window.

Stack traces

Composr will show a "stack trace" if it receives an error it treats fatally (a fatal error is one that terminates normal execution – because execution cannot finish). These stack traces are amazingly good at helping you find bugs, so if you get one have a close look at it.

A stack trace indicates the function call chain from the call of the script handling your current URL (at the bottom) to where Composr failed (near the top). In other words, as you look through the trace you are looking back through each function call all the way back to the top level script that the URL is calling. It doesn't show what the code has finished doing, only what it was doing when it failed and the function call chain that led to that action. At any level you can see the file the call was on, the line number, the function called from that line, and the arguments passed to that function (although large arguments like big arrays and Tempcode are omitted).

Lateral reasoning usually allows spotting of the source of a bug in a few seconds, although sometimes it can be more difficult. Often the bug is caused by unexpected parameters, and you can see them reflected in the stack trace and laterally work out why those parameters were passed and thus work out how to solve the problem.

One confusing thing about stack traces is that the error chain is included within the trace. With time you'll learn to read past this and see where the error was actually triggered. Typically anything above the fatal_exit or composr_error_handler function calls relates to the error handling process itself.

eval

If code is passing through eval (common with code overrides) then a stack trace can be difficult to analyse because you'll have a rough line number but no filename.

One technique is to turn on safe mode, because if the error still happens in safe mode you may also get a real line number.

Here's a helpful script to search a codebase for a function call happening on a particular line, to help narrow down a cause:

Tips

This section contains a number of general programming tips. Much of this applies to all programming, not just Composr programming. However there are also a number of tips for Composr that will help you on a day-to-day basis.

Setting default parameters

Sometimes on a complex project you will want a module to have a default Filtercode string, or a default virtual root, or some other default parameter different to what Composr hard-codes.

The best way to achieve this is to code up a startup hook. Using PHP code you can detect non-presence of a URL variable, and then set your own default directly into $_GET according to your own context.

For example, if you wanted a module as seen in one zone to have one virtual root, and the same module as seen in another zone, to have another – you could code up a startup hook that sets the default root accordingly.

Deployment

Be very careful deploying these three files:

1. _config.php
2. .htaccess
3. php.ini

_config.php is obvious, you don't want to overwrite the live server environment settings with development ones.
Sometimes it is a good idea to code up _config.php to detect what server it is on, then share that file across all platforms. This is mainly a personal choice.

.htaccess can be very server-specific, so be aware these pitfalls:

1. If a live server is CGI rather than an Apache module, over-assumptions can cause a 500 Internal Server Error
2. Any assumptions about Apache modules being there which may not be can cause a 500 Internal Server Error
3. If a RewriteBase line is there, it may be wrong for the server
4. The server could have its own file, with its own special settings (it's quite common to use the file to tune live settings)

php.ini is similar to .htaccess in that it can encode very server-specific settings.

Tidying up ugly Comcode or running conversions

Often Composr users will write some horrible Comcode, that is impossible to reasonably extend without first cleaning it up.
Additionally, you may want to run some conversions, such as seeing what HTML some Comcode produces.

The data/comcode_convert.php can help you in these areas. It has options for converting between HTML/semihtml/Comcode, and has an optional reindent tool and HTML cleaner built-in.

Regular expressions

Have a good understanding of regular expressions and an eagerness to make use of them. You can often find uses for them, especially when searching code or preparing data-sets. They can save a lot of time manually reformatting things.

Webhosting

Never trust webhosts to provide decent hosting. Expect lots of bugs, unexpected limitations, performance bottlenecks, old software, and missing PHP extensions.
That's life, so try and minimise your dependencies and environment assumptions as much as possible – write simple, compatible, code.

git

It is in your interest to build up a decent understanding of git, including:

• The very basics: Cloning, Adding files, Committing, Pushing, Pulling
• Creating branches
• Merging branches
• Reverting uncommitted code back to the last commit (git checkout -- <path>)
• SSH keys
• Resolving conflicts
• Creating diffs of uncommitted code
• Creating diffs between branches

Git is a useful tool for maintaining history on a project, deploying across servers bypassing issues of slow Internet connections (far better than FTPing), and testing experimental changes – even if you are working alone on a project.
It will save you a lot of time, and give you more security and stability in your work.

Writing new code

Many programmers struggle through things due to:

1. failing to split up tasks, getting overwhelmed
2. not writing clean code and confusing themselves and others (not splitting things up [abstracting] properly, copy & pasting too much, poor variable naming, poor code structure, poor code layout)
3. patching over problems via workarounds that "seem to work", rather than properly understanding what's going on

The solution is to be logical, calm, and rigorous. Also, over time your ability to build up a mental model of how things fit together in large code-bases like Composr, will improve.

Here is a good workflow for writing high-quality code for Composr:

1. Work out generally what needs doing.
2. Create a new feature branch in git, if you are doing dangerous/experimental changes or are coding for core Composr (as opposed to a client website).
3. Lay out the basic structure of how the code will come together (files, classes, functions).
4. Document the solution using pseudo-code comments throughout the structure. Put a "TODO" comment under each of the pseudo-code comments.
5. Fill in the TODOs with real code. Make sure you are writing quality code and matching appropriate coding standards.
6. Run the ocProducts Code Quality checker on the code. Specifically: recalculate function signatures, then check the files you've added/changed. Fix any errors found, it should be possible to have zero reported errors with the default settings (all of Composr passes).
7. Write unit tests to prove to yourself it works well, if appropriate. This makes most sense if your code implements algorithms or APIs. User-interfacing testing is a lot harder, more subtle, and the tests more fragile, so frankly this is usually left to human testers.
8. Test manually / debug.
9. Add new files to git, and Commit/push (actually you may do this a few times throughout the work, depending on task length).
10. Come back the next day and check over your work. A fresh look often reveals some things you missed before. Also fix any issues that you already realised in the interim (often you will think of things later).
11. If you created a branch, do whatever is appropriate for it. That might be merging it into the master repository, submitting a merge request on GitLab, or asking a senior developer to review then merge it.

Here are some general tips:

• It's sometimes a good idea to program algorithms in complete isolation. Separate the complexities of integration from the complexities of implementation. This also leads to better quality code because you will have been forced to abstract the problem already.
• If working with other remote developers, Cloud9 IDE is absolutely excellent. You can watch each other code and see the results of running your code.It is very useful for teaching programming technique, team debugging, and pair programming.

Writing unit tests

Unit tests allow you to initially verify code works according to your expectations, but also provides enormous long term value. If you have to change code structure around later, or need to test on different environments, a library of unit tests greatly reduce the time-cost of re-testing your work.

Composr's git repository contains the testing_platform addon, which is based on SimpleTest. It is located under the _tests directory.

Writing new tests is really easy, just write a _tests/tests/unit_tests/whatever.php file. Use the SimpleTest assert system to verify test results match expectations.
If you are coding for core Composr, make sure the new test file is added to the sources_custom/hooks/systems/addon_registry/testing_platform.php file (so that the testing_platform addon will bundle your test).

You can browse/run tests via http://yourbaseurl/_tests.

Never install the testing platform on a live site, or at least if you do block public access to it via the web server configuration.

Debugging

When programming bear in mind that it is human to make lots of mistakes. Don't blame yourself for errors. The key is having a rigorous practices to identify and deal with them.

If you are stuck with a problem you haven't been able to quickly resolve, consider these logical steps for resolving it:

1. Output data involved in calculations so you can confirm you are calculating on the data you think you are.
2. Isolate the problem with a simpler test case that is safely and trivially repeatable. For example, if the problem is happening when submitting a form with a unique codename and an upload, make a test script that simulates the same environment but without you having to fill in the form.
3. Step through the code either using an IDE debugger (PHPStorm & xdebug), or simply by putting temporary var_dump and exit calls in the code.
4. At some point you'll narrow in on the problem, as you'll have inspected yourself at a point in the code where things aren't going as expected. Usually it is then relatively easy to solve the problem as you will have found a clue in the process, such as a non-true assumption you had previously made.

Here are some general tips:

• Always think of quick ways to prove/disprove theories, and check facts, like variables match the values you expect them to. If you narrow in on facts, you make solving a problem much easier.
• Get fully comfortable with the powerful debug tools in a browser such as Google Chrome, such as: the DOM inspector, setting JavaScript breakpoints, setting to pause on JavaScript exceptions, typing into the console, watching the console for errors, and analysing network requests and responses.
• Get comfortable with unified diff files. They are very useful for many purposes.
• Have a mind-set of automatic tasks to increase your own efficiency. Write temporary scripts to get things done faster or to reveal data to prove/disprove hypotheses about the nature of a problem, or to build confidence that you've properly fixed things.
• Whenever something confuses you, try and understand your way through it, even if you've already found another solution. Confusion is an opportunity for learning, a revelation of a gap in your knowledge – make the best of it so that you get a strong understanding of all technologies. With time, things will get easier and easier, but only if you constantly work at expanding your understanding. Work to build up a unified mental model of how everything works and fits together across all of computer technology.
• If you want to see how a function is operating in random different calls (i.e. not always just the first call), add some temporary code like if (rand(0,10)==1) exit($some_variable); and refresh the page a few times. This is a useful technique for many cases where you need to sample data.
• PHP often will not tell you about syntax/parse errors in your code. If you are stuck with the code randomly exiting, try using php -l <filepath> to see if PHP's lint tool will tell you about a parse error. Use of the ocProducts Code Quality Checker tool is even more useful, returning all kinds of potential problems.

If you are debugging a standalone PHP script (i.e. outside Composr), make sure you have errors turned on:
Having full reporting of all PHP notices, and display of them on screen, helps greatly in finding causes of issues.

Debugging on live servers

Composr keeps a number of logs, which can be very helpful for debugging things. Try and build up a model of what happened via careful analysis of logs, like Sherlock Holmes.
Useful Composr logs include:

• The error log (data_custom/errorlog.php)
• The logged page activity, available via 'Investigate user' on an IP address / member ID / username (useful also for finding all a member's recent IP addresses)
• The eCommerce log, if enabled (enable it via creating an empty writable data_custom/ecommerce.log file)
• The mail log

Other very useful logs:

• The web server's own access log; look carefully in it and you are likely to see things like HTTP response codes (e.g. 500 means Internal Server Error)
• Any errorlog files the web server may leave behind (native PHP error logs, which may or may not be enabled)

Correlate times and IP addresses to get a model of what happened.
Also the Compsor 'SU' feature (see the Testing access and privileges tutorial, Access control and privileges section) can be very useful for testing under different users without having to know their passwords.

If it is acceptable to debug code on a live server, putting exit and var_dump commands to only run for your own IP address can be useful (if (get_ip_address()=='...') { ... }). This is not recommended for a high-traffic site though, you're likely to make an occasional mistake that'll get spotted. So for this case, you'll just have to clone the database back to a development server and test like that.

Logging into live sites

It is common to need to log in as an administrator on a live site and not know the admin password (only knowing the hosting access).
If you add a line to _config.php like the following:
It will automatically log you in under the first administrator account. You can find your IP address from common websites.

Note that if you're using localhost, you'll probably be using ipv6, so you'll need to use the ipv6 local address, ::1.

Form fields

To save time and auto-populate form fields during testing, if you have the Web developers toolbar extension for Firefox, you can right click, then choose: Web Developer → Forms → Populate form fields.

Advanced deployment and customisation

Using git to deploy

Deploying to a live server using git is a great idea. However, be aware some caveats…

1. Disable dev-mode in the config file using:
   

   Code (PHP)

   $SITE_INFO['dev_mode'] = '0';
    
   git automatically enables dev mode, which intentionally interferes with things to help you develop quality code – such as adding extra checks, or disabling cookies so that you don't rely on them. This is great for development, where the developer is in control of everything, but not correct for production.
2. Use a decent .gitignore so you don't get a lot of noise when running git status. Our default one is good but may need tuning.
3. Either keep the config file out of git, or put code in there to differentiate between machines via checking $_SERVER['SERVER_ADDR'] ($_SERVER['LOCAL_ADDR'] on a Windows server).
4. Make heavy use of git branches so that you don't get stuck having to perfect all code before you can update anything onto the server.

You can consider a half-way house, by having a full checkout on the server, then separately symlinking specific directories into the main public_html. This is useful if you only want certain directories in git, such as a docs zone (docs in this example being managed offline, and pushed live using git).

You can also consider multiple checkouts on the server, such as a testing site, and a live site. This allows you to test things with greater care.

Hidden features inside Composr

Composr contains a number of hidden 'values', 'keep' parameters, and empty file flags. These allow activation of special functionality that isn't considered important/mainstream enough to warrant user-interface space within Composr.

Empty files

The presence of the following empty files in the root directory have a special meaning to Composr:

• install_ok – don't complain if install.php is left present (DO NOT use this unless your install is not connected to the Internet or if you definitely have an install_locked file)
• install_locked – whether to lock the installer (prevent it running)

Other special files

If you put a file called closed.html then this will be shown to people rather than the normal closed-site screen, regardless of whether the site is currently closed or not.

Hidden 'values'

Values are like hidden configuration options. They are either hidden because they are managed by code (perhaps used for keeping track of something), or because they are obscure. To set a value, open up Commandr (Admin Zone > Tools > Commandr – or clicking the symbol at the bottom-left of any page) and type :set_value('<name>','1'); (replace '1' if appropriate, but usually we do use '1' to enable). In normal PHP code, you can use the same set_value function, and also the get_value function. You can automatically add a new hidden value just by setting it for the first time.

We have the following which are either unlikely to be useful, or potentially unstable or unpolished:

• Integrations
  • force_admin_auth – set this to '1' if you want any super-administrator to have to have some kind of authorisation-based login by redirecting them to http://yourbaseurl/admin_login/ which presumably you have configured to be a redirect-through that catches logins (Apache only)
  • no_password_hashing – set this to '1' if you want passwords to be stored as plain-text in the database (not recommended)
  • xhtml_strict – set this to '1' to do some magic to make Composr's output XHTML strict. This is slow and not really advisable as it will degrade the user experience if JavaScript is not available (new-window links would break)
  • webdav_root – set this to the subdirectory the WebDAV addon should run from. On Windows it needs to run from the root, i.e. '/'. However, to do this you would need to map onto a separate domain name. If you change this setting, you'll need to change the redirect in your .htaccess file
  • allow_member_mail_relay – set to '1' if the form_to_email.php script should allow e-mailing to arbitrary members via the allow_member_mail_relay parameter. This potentially allows people to spam your members, so you may want to add some kind of extra security here at a web-server level (i.e. restricting access to the script).
  • windows_auth_is_enabled – set to '1' to enable Integrated Windows Kerberos Authentication for automatic desktop client logins
  • postgres_fulltext_language – set to a PostgreSQL full-text configuration name for defining how to tokenise text (only set if you don't want 'english'). Existing indices would need hand-editing to the new value too.
• Unsupported functionality
  • unofficial_ecommerce – set to '1' if your forum driver contains usergroup manipulation functions (we don't support this, but wanted to be able to allow those willing to extend forum drivers to use our full eCommerce framework)
• Workarounds for difficult server configurations
  • disable_iconv – set this to '1' if your iconv extension causes PHP to crash, or if it is very slow
  • disable_mbstring – set this to '1' if your mbstring extension causes PHP to crash
  • prefer_curl – set this to '1' if the server requires you to not use network sockets directly from PHP and instead use the CuRL extension (although if you need this it is likely you actually need to just configure the proxy server options instead)
  • real_memory_available_mb – set this to the number of MB GD should consider available for image generation, if PHP is lying about its memory limit
  • http_faux_loopback – a regular expression of URLs that may be accessed via direct filesystem reading rather than HTTP loopback. Be very careful with this, as it could seriously harm security. It can improve performance however. Some poor webhosts may also require it (we've only seen one though). It does support PHP scripts, which are given special treatment, via passing through the php-cgi interpreter (file uploads and POST requests not supported). Set via a command such as:

• • base64_emails – set this to '1' if you want base64 e-mails to be used. These are more robust (PHP's wordwrapping, which we use to make non-base-64 e-mails compliant, can be buggy) but some servers do not like them or consider them very slightly more spammy
  • cloudflare_workaround – set this to '1' if you need to fix the IP address of requests marked as coming from CloudFlare. It reduces security.
  • disable_openssl – set this to '1' if you are on Windows and PHP<5.3.4, which has a bug in openssl_random_pseudo_bytes. It is much better to upgrade PHP though, for better security.
  • disable_ssl_for__<domain> – set this to '1' to disable SSL certificate checking for the given domain name. This is not a great idea, but may be necessary if a required service has a broken certificate yet it only supports HTTPS. Note that this will not work on a Mac by default because Apple's build of cURL (using https://opensource.app…curl/lib/vtls/darwinssl.c) will use Apple SecureTransport instead of OpenSSL, which doesn't support disabling of verification
  • manualproc_mail – set this to '1' to debug problems with the PHP mail command (the PHP mail command won't relay error information from the sendmail process, this will replace it with something that does); it only works on servers that allow PHP process control
  • disable_cookie_checks – set this to '1' if you are somehow proxying traffic into your web server with an incorrect hostname header, so cookie checks must be skipped
  • disable_sitemap – set this to '1' if you need to disable Sitemap menus for any reason (perhaps it is too slow to generate or there is some kind of infinite loop bug that is not yet fixed). Set this to '2' to greatly reduce the depth of the Sitemap (this is probably what you want, as you don't want your admin menus disappearing).
  • http2_post_fix – set to 1 to delay an AJAX POST request while a warmup request runs; used to fix issue with HTTP/2 connections where write-back is broken

• Used internally by Composr
  • cdn – this is used for storage of autodetected CDNs, prior to the config option for them being properly filled in
  • brand_base_url – the base URL to the brand site (by default this is https://compo.sr but may be changed)
  • rebrand_name – if Composr is called something else as far as the site staff are concerned, set the name here
  • rebrand_base_url – to change where branding URLs go to (e.g. the stub that is used when linking to documentation), set it here
  • company_name – if Composr is being rebranded to be 'made' by another company, set the name here (this has no effect of copyright of course, but ocProducts allows this)
• Developer testing
  • notification_safety_testing – set this to '1' to enable extra checks to ensure notifications or welcome e-mails don't get misdirected/over-sent
  • memory_tracking – set this to the number of megabytes peak memory usage after which you want the execution URL to be silently logged to the error log (useful for debugging memory usage on a live site). The value must be an integer, with no trailing 'M'. This does not impose a memory limit, which is 64MB by default in Composr, but may be turned off within certain carefully-chosen execution contexts (usually admin tasks which are necessarily memory consuming). Requires PHP 5.2+.
  • enable_profiler – set this to '1' if you want to enable the Composr profiler, which will write select performance information to data_custom/profiling.(ID).log. For your security you should ensure these files are web-accessible (we will do this by default if we can via the data_custom/.htaccess, but you should take care). The web server needs permissions to save files into data_custom/ (this is possible by default on modern suEXEC environments). Set to '2' if you want to read Linux performance data, which will then be included in the logging (takes a bit more work by the profiler, assumes Linux and sufficient permissions).
  • monitor_slow_urls – set this to a number of seconds to log any requests taking at least that many seconds seconds to the error log
  • permission_log_success_too – set this to '1' if you want the permission log file (data_custom/permissioncheckslog.php) to show both successful and unsuccessful permission checks. Usually it only shows fails.
• For system-integrators
  • agency_email_address – set this to a secondary e-mail address where you would like all error e-mails to go (the primary one being the staff address). Only set this if you are an agency and want to receive error e-mails in addition/instead of your client
• For back-end developers
  • textmate – set this to '1' if you are developing on a local machine and use the Apple Mac TextMate editor. It will cause TextMate editing links to come up in stack traces
  • allow_php_in_templates – set this to '1' to enable the undocumented Tempcode 'PHP' directive, and its short-hand syntax (written like normal PHP long-tags). This allows you to write PHP code inside templates, but please aware there are strong risks associated with this. It means that themes you install may contain PHP code, or people submitting Comcode content may put in PHP code which you could accidentally unwittingly activate when validating content (as it would be validated against your own security credentials). Note this option is not designed to work from Comcode, because Comcode will add HTML formatting around your PHP code, corrupting it (from a security point of view you need to be aware though, as that could be avoided with some care by a hacker)
  • git_autopull – set this to '1' if git changes should automatically be pulled every minute (less frequently if the CRON interval is longer). Assumes a suEXEC-like server and a git deployment and working shell_exec
• WYSIWYG editor
  • commercial_spellchecker – set this to '1' if you want to enable the WYSIWYG editor's spellcheck.net spellchecker
  • wysiwyg_normal_font_sizes – set this to '1' if you want to have normal font sizes rather than relative font sizes (relative sizes allow context-adaptive markup)
  • wysiwyg_spellchecker_default – set this to '1' if you want to allow the browser's spellcheck to be available by default (otherwise a button must be pressed)
• Low-level tuning
  • no_forum_moderate – set this to '1' to hide the staff checklist mention of moderating the forum
  • show_menu_items_as_url – set this to '1' if you want the menu editor to show URLs instead of page-links (they will then be converted back)
  • kid_gloves_html – set this to '0' if you want to completely disable the kid-gloves HTML escaping mode, i.e. avoid the need to make declarations in your overrides. You may want to use this if you override various files but aren't really writing custom code and don't really understand how to make the declarations
  • whitelisted_blocks – set this to a comma-separated list of block names to whitelist any user to use those blocks independently. This is useful for AJAX-loaded blocks on sites where users may leave pages open, letting their sessions or block authorisations expire.
  • no_individual_gallery_view – set this to '1' if all images/videos in a flow-mode gallery must be viewed from the flow-mode view (never individually)
  • root_cat__images – you can set this to the codename of a gallery, which will be the root for gallery selection on the gallery addon's Add Image form (root is not shown as an option). Alternatively, it can be an Selectcode string.
  • root_cat__videos – you can set this to the codename of a gallery, which will be the root for gallery selection on the gallery addon's Add Video form (root is not shown as an option). Alternatively, it can be an Selectcode string.
  • levels_to_expand__<ajax_tree-hook> – set some ajax_tree hook to auto-expand by a certain number of levels
  • no_confirm_url_spec_cats – set this to '1' if you don't want to have to confirm the category for a new catalogue entry when the category was specified in a URL parameter already
  • no_spec_cat__<catalogue-name> – set this to '1' if you don't want to have to confirm the category for a new catalogue entry when the category was specified in a URL parameter already
  • comment_forum__galleries / comment_forum__images / comment_forum__videos / comment_forum__downloads / comment_forum__calendar / comment_forum__news / comment_forum__polls – override the comment forum for a particular content type. Make sure you refresh the page with the comment form in your browser after changing this, as the settings also get saved into the page HTML
  • comment_forum__catalogues__<catalogue-name> – define a comment forum for a specific catalogue
  • allow_no_lang_selection – set this to '1' if you want members to be able to leave their language selection as 'Unset' (otherwise normally it'll copy the auto-detected language into their profile or they can change it, but it never stays as unset). This option is useful on multi-site-networks, with each site having a different language
  • no_base_check – set this to '1' if you do not want Composr to empty caches if the base URL gets changed (this is if the base URL is not defined, and someone accesses from something different)
  • pagination_when_not_needed – set this to '1' if you want the pagination wrapper template to load even if there is no pagination (useful for collecting result metadata via Tempcode)
  • catalogue_seo_source_map__<catalogue-name> – a comma-separated list of field IDs to use for SEO keyword generation (if not set, uses all fields). If all keywords in the field should be used, put an exclamation mark after the field ID
  • disable_password_change_notifications_for_staff – set this to '1' if you want staff password edits to not result in notifications to users
  • filtercode_protected_fields – a comma-separated list of fields Filtercode cannot work on (above the ones hard-coded as forbidden)
  • memory_limit – set this to a memory limit to override Composr's built-in one (e.g. "128M"). This will make memory bugs on your server more dangerous, so use at your own risk (a memory usage problem can take down a whole server) – although memory bugs are unlikely given that Composr users report if they ever see this error
  • canonical_keep_params – set this to a comma-separated list of keep_ parameters that should be allowed to propagate within search engines
  • no_tech_login_messages – set to '1' if you don't want technical explanations attaching when a user is forced to login
  • cc_sort_date__<catalogue-name> – set this to '1' if you want catalogue category children to be sorted by date rather than title
  • resize_rep_images – set this to '0' if you do not want uploaded rep-images to be sized down automatically to the standard thumbnail size
  • default_event_type – set this to the ID of the default event type to select for new events
  • edit_with_my_comcode_perms – set this to '1' if content you edit should display with your Comcode permissions. Only enable this if you trust yourself to spot XSS attacks.
  • site_location – a real-world address for where your website is, sometimes used in metadata.
  • google_news_urls – set this to '1' if news URLs should contain a Google-News-friendly entry ID.
  • enable_delayed_inserts – set this to '1' to enable MySQL delayed inserts. Delayed inserts can cause a very small performance improvement if log calculations happen quite regularly; however they only work on MyISAM tables and are deprecated as of MySQL 5.6
  • really_want_highlighting – set this to '1' if you want to force HTML to be converted to Comcode when searches happen, so keyword highlighting works better.
  • search_days__<search-hook> – set the default day search period for a search hook (e.g. 60, or blank for no limit). This may be set for reasons of user experience, or for performance – searching without a day constraint is actually faster.
  • no_cannot_access_url_messages – set this to '1' to turn off attached messages about broken URLs that Composr tries to process
  • force_spell_checker – set to 'pspell' or 'enchant' to force that particular spell-checker, rather than auto-detecting
  • avoid_register_shutdown_function – set this to '1' if register_shutdown_function is not reliable on this system (may appear to be unreliable on some Windows servers, for example)
  • skip_fulltext_sqlserver – set this to '1' if you are using Microsoft SQL Server and want to disable full-text search support. You may do this if you don't like the limitations of SQL Server full-text search (per-table searching, not per-field searching).
  • mail_theme – set this to the name of a theme to force that theme to be used for all outbound e-mails. This is useful because otherwise the theme of the user triggering the e-mail to be sent will be used.
  • force_memory_sort__<catalogue-name> – set this to '1' to force an in-memory sort on a particular catalogue (this is slower, but it will sort more accurately by resolving language string fallback and Comcode fully before sorting)
  • search_do_days_fallback – set this to '0' if you don't want to re-search without a recency filter if no results were found
  • sitemap_orphans_to_pages – set this to '0' if you don't want orphaned pages to be put under the default Pages page grouping of the Sitemap
  • time_sensitive_rankings__<catalogue_name> – set this to a number for catalogue rating sorting to only consider this number of records. This is useful for a upvote/downvote system that is responsive to change in voting pattern. Requires the database to support window functions (MySQL 8+ does, Maria DB 10.2+ does).
  • keep_gallery_gps – set this to '1' if you want galleries (and related to feature) to not strip GPS metadata from images (which is done by default for privacy reasons)
• Performance tuning (positive) [not official options as will break normal expectations of how the system will behave / complicate things]
  • slow_php_dns – set this to '1' if the server has very slow DNS resolution on the web server's main network interface, and therefore to do DNS resolution via the command line instead
  • lots_of_data_in_* – set this to '1', with '*' replaced with a database table name, if you want the Selectcode mechanism to work with recursive DB lookups rather than one huge flat lookup
  • disable_cat_cat_perms – set this to '1' if you want to disable catalogue category permissions
  • catalogue_limit_cat_field_load__<cataloguename> – set this to '1' if you want that catalogue to not have all fields loaded on category views, an optimisation that reduces templating flexibility
  • no_catalogue_field_assembly – set this to '1' if you want catalogue entries to be 100% hand-templated; removing this will improve performance but remove pre-assembly
  • no_catalogue_field_assembly_fieldmaps – see above, but only if fieldmap assembly not required
  • no_catalogue_field_assembly_grid – see above, but only if grid assembly not required
  • no_catalogue_field_assembly_tabular – see above, but only if tabular assembly not required
  • assume_modules_correct – set to '1' if module auto-installation/auto-upgrade should not happen; this will improve performance
  • no_priority_redirects – set to '1' if redirects should only be considered if a page cannot be found via other means
  • no_tags – set to '1' if you don't want tags to be generated (small performance tweak)
  • no_awards_in_titles – set this to '1' if you don't want awards to be displayed on content screens (small performance improvement)
  • disable_user_online_counting – set this to '1' if you don't want to count online users (small performance improvement)
  • no_banner_count_updates – set this to '1' if you don't want to maintain banner view counts (small performance improvement)
  • pinpoint_submitban_check – set this to '1' if you only want submitter ban checks to occur in the CMS zone (small performance improvement)
  • no_block_timeout – set this to '1' if blocks should never timeout (stops stuttered performance if a site is not used regularly)
  • disable_sitemap_for__* – disable sitemap generation for a particular page (e.g. if there's too much data in it and the server can't cope, and the data isn't so important anyway). Replace '*' with the page name
  • no_view_counts – set this to '1' if you don't want view counts to be maintained (reduces database writes). You will probably want to disable the normal "Reveal usage figures" option too, to hide all the zero counts
  • avoid_normal_topic_read_history – set this to '1' if you don't want topic-read history to be maintained for things other than private topics
  • no_member_tracking – set this to '1' if you want to disable the member tracking feature where it's supported (it shows who is viewing a forum topic, for example)
  • min_lastvisit_frequency – set this to the number of seconds waited between the last-visit time being updated (performance tweak, minimises database writes)
  • no_flood_control – set this to '1' if you will not have flood control
  • implicit_usergroup_sync – set this to '1' to enable synching of implicit usergroups (see the Advanced Composr member system tutorial for more information)
  • shortened_tempcode – set this to '1' to not store parameter reference origins in Tempcode, which makes Tempcode use less memory/storage/CPU
  • save_jpegs_as_png – set this to '1' to allow Composr to detect when JPEG files would be smaller, and save JPEG data into PNG file extensions. It is a bit of a hack, and goes against standards, but works well
  • maximum_result_count_point – the number of search results to count up to (defaults to '1000'); a higher figure means that for multi-condition searches extra computation will be required to see which result rows match the search conditions (single-condition search counting is usually fast because an index gives an instant answer)
  • search_days__<search-hook> – (see "Low-level tuning")
  • js_keep_params – handle keep variable propagation via JavaScript. Used in conjunction with $SITE_INFO['no_keep_params'] = '1';
  • slow_counts – set this to '1' if you have switched to InnoDB, as table row counts are slow and this switch enables an approximation instead in certain places. Do not use with other databases than MySQL or PostgreSQL.
  • innodb – set this to '1' if you have switched to MySQL's InnoDB
  • no_post_rating – set this to '1' if you don't want to allow post ratings, which saves one query per displayed post
• Performance tuning (negative)
  • flush_cache_on_action – set this to '1' to automatically flush the static cache when actions are performed
  • allow_admin_in_other_zones – set this to '1' if you want to allow admin and CMS modules to run in other zones (small performance cost)
  • memory_limit_simulate_hard – set this to '1' if you do not want the memory limit to be disableable; useful for testing compatibility with hosts that have low limits
  • cron_block_caching_careful_contexts – set this to '1' if you want Cron generated block-caching to use the requested theme and language to fill the cache; this requires loopback HTTP requests
• Easter eggs
  • stupidity_mode – set to leet or bork, for a fun April Fools-style Comcode joke (clear the Comcode cache after you're done though)
• Experimental functionality
  • list_unsubscribe_target – set to a URL for a List-Unsubscribe header to be attached to e-mails with that URL, e.g. mailto:you@example.com?subject=Unsubscribe&body=Please%20unsubscribe%20me
  • access_protocol_redirect – set this to '1' if you want to redirect if the wrong access protocol (e.g. HTTP instead of HTTPS) is used to access the site. Note that this option won't apply when pages are served from the static cache, so it is still a good idea to configure the web server to do this directly. Sample code is in the Security tutorial (Migrating HTTP websites).
  • captcha_member_days – set this to a number if new members should continue to have CAPTCHA for this many days
  • captcha_member_posts – set this to a number if new members should continue to have CAPTCHA for this many forum posts
  • reorganise_uploads – set this to '2' for various upload directories to have a hierarchical structure of subdirectories to organise your files based on category; set this to '1' for a flat structure; set this to '0' to disable it completely; to retroactively move things run the "Organise uploads" cleanup tool that will then appear.
  • urls_simplifier – set this to '1' to have simplified (not-RFC-compliant) URLs shown in URL inputs on edit forms. This makes them a lot easier to use, and we convert them back to compliant URLs on the server. It is a safe process where it is not possible to cause any corruption by double encoding or double decoding.
  • download_associated_media – set this to '1' to add an option on posting forms to have any referenced media automatically downloading locally, to protect against the risk of future broken links
  • highlight_extracted_file_data – set this to '1' if downloads searches should show highlighted data from extracted keywords, if there's no match on the body text of the download

Hidden 'keep' parameters

'keep' parameters are placed inside URLs in order to give Composr some extra information when loading a page. Their URL-presence is automatically relayed/preserved in all Composr links within the page. To enable a 'keep' parameter, simply append &<name>=1 to the URL (replace '1' if appropriate, but usually we do use '1' to enable). If there are no parameters in the URL and URL Schemes are not enabled, use a '?' instead of a '&'.

The 'keep' parameters available are:

• keep_cache – set to '1' to temporarily enable caching or '0' to temporarily disable. You can also use cache so it works on a per-page basis, with the exception of this not affecting the template and language caching.
• keep_no_dev_mode – set to '1' to disable development mode (which only runs if you're working out of a subversion repository anyway)
• keep_hide_dev_mode_message – set to '1' if you don't want the "Dev-mode is on" message to show if dev-mode is on (maybe you're taking screenshots?)
• keep_no_ext_check – set to '1' to force the webstandards checker to not check dependency files
• keep_force_htaccess – set to '1' to force the Conversr login to go via htauth in precedence over other login methods (e.g. login cookies, sessions)
• keep_textonly – set to '1' if the 'text only' stylesheet is to be used
• keep_simplified_donext – set to '1' to temporarily act as if the simplified do-next option is turned on
• keep_mobile – set to '1' to pretend you're using a PDA
• keep_has_js – set to '1' to force Composr to think you have JavaScript (useful if the has-JS cookie isn't saving yet you need to use a JS-based interface)
• keep_ldap_debug – set to '1' to forcefully show LDAP errors, when suppressing them might normally be wise due to over-sensitivity
• keep_show_query – set to '1' if you want the SQL query used by a search to be echoed out
• keep_just_show_query – set to '1' if you want Composr to echo out the query as above, but then to exit before running it
• keep_currency – set to an ISO currency code to tell Composr which currency you use
• keep_country – set to an ISO country code to tell Composr which country you're in
• keep_region – similar to above, but supports whatever regions are configured on the website
• keep_cat_display_type – set to a number representing the catalogue category display-type to try out (0=entry-tables, 1=lists, 2=matrix)
• keep_id_order – use this in conjunction with the admin_cns_groups module to force a usergroup reordering based on the named DB field of your choice
• keep_backup_alien – set this to '1' if the backup module should only be backing up non-Composr files (i.e. do a backup of the website, but not the software)
• keep_module_dangerous – set this to '1' to allow uninstallation of modules that are locked
• keep_preserve_ids – set this to '1' when doing an CMS-merge import, to preserve IDs in the import (and not import duplicated IDs); this is not supported officially
• keep_theme – set this to a theme name, to temporarily try out a different theme
• keep_safe_mode – set this to '1' so that where possible, any customised files will be ignored; the default theme will be used
• keep_no_output_streaming – set this to '1' to disable output streaming (i.e. enable full pre-processing)
• keep_fatalistic – set this to '1' if terminal warning errors should be handled as terminal fatal errors (which include stack traces); useful for debugging the origin of an error message
• keep_firephp – set this to '1' to dump permissions checks that occurred to FirePHP. It only works for authenticated administrators, but if the administrator is using keep_su to masquerade as another user it will work (which is particularly useful for debugging permissions problems).
• keep_firephp_queries – set this to '1' to dump queries that occurred to FirePHP. It only works for authenticated administrators, but if the administrator is using keep_su to masquerade as another user it will work.
• keep_show_parse_errors – set this to '1' if you think you have a corrupt PHP file but you can't tell which as your PHP display errors option isn't on (and hence just get blank screens). This will allow Composr to generate stack traces for most corrupt files it tries to include. Note another cause of blank screens can be the PHP memory limit being exceeded.
• keep_send_immediately – set this to '1' if you want to debug newsletter code, and not have newsletters sent in the background
• keep_su_online – set this to '1' if you want the user keep_su is used with to also show as online, and for their last online/submit statuses to be updated
• keep_su_strict – set this to '1' if to strictly masquerade as another user rather than to allow a few exceptions like the display of stack traces
• keep_no_minify – set this to '1' if you want to use a JavaScript debugger and hence do not want the JS to be minified (unreadable)
• keep_show_loading – set this to '1' if you want comments inside the HTML source showing memory usage taken up as major server-side dependency files load
• keep_show_loading_code – set this to '1' if you want comments inside the HTML source showing memory usage taken up as server-side PHP scripts load
• keep_avoid_memory_limit – do not put on a memory limit (admins only); using this will also add in the option to remove paginations, for those with the remove_page_split privilege
• keep_memory_over_speed – use less memory for Tempcode
• keep_no_query_limit – do not place a query limit during development mode
• keep_on_same_msn – set this to '0' if you are doing a Composr site merge import and want to force Conversr content to import even if the importer thinks that you are on the same MSN as the other site
• keep_theme_test – set this to '1' if you are testing lots of themes. It will cause installed theme addons to have their Comcode pages extracted with a special prefix, it will remove that prefix on addon packaging, and page detection will give preference to the prefix for the particular theme you are currently using (so use in conjunction with keep_theme).
• keep_debug_notifications – set this to '1' if you are debugging the notifications system and therefore need them to be sent before the page output finishes
• keep_debug_tasks – set this to '1' if you are debugging the tasks system and therefore need them to be run immediately (i.e. without going into the task queue)
• keep_devtest – used during development mode to test URL building is working correctly, of no use manually
• keep_debug_has_cookies – useful for developers to tell Composr cookies are supported when JavaScript may be off
• keep_memory_limit_test – set to a number of megabytes under 32 to reduce the memory limit to this value; useful for performance debugging
• keep_realtime_test – used for testing the realtime feature's display (it injects random data)
• keep_wide – remove panels
• keep_wide_high – remove panels, as well as the header and footer
• keep_rating_test – set this to '1' if you are testing rating and don't want to be locked out for having already rated
• keep_all_cpfs – set this to '1' if you want to edit bundled CPFs that aren't currently marked as editable
• keep_oembed_cache – set this to '0' to disable the oembed URL cache temporarily, when debugging oembed endpoints
• keep_urlmonikers – set to '0' to disable URL monikers
• keep_no_url_scheme – set this to '1' to temporarily disable URL Schemes. This is useful for working out page-links/match-keys
• keep_force_open – set this to '1' if you want to ignore a closed.html file and get to the website
• keep_no_frames – set this to '1' to disable optional iframes / AJAX frame-simulation
• keep_failover – set this to '1' or '0' to force failover mode to on or off
• keep_has_cron – set this to '1' or '0' to override UI adaptations to CRON being on or not (e.g. to enable scheduling options as if CRON was on, even if they won't run)
• keep_smart_decaching – set this to '1' for smart decaching to be enabled, even if disabled in _config.php (it is on by default, so you may want to use this flag if you disabled it but want to ease your development workflow)
• keep_infinite_scroll – set this to '0' or '1' to temporarily change the infinite scrolling setting
• keep_test_version – set to a dotted version number to test the version block in the Admin Zone dashboard as if you're running a different version. Likely needs combining with keep_cache_blocks=0
• keep_no_notifications – set this to '1' to suppress sending of notifications (only works if logged in as an administrator)
• keep_quick_hybrid – set this to '1' in a quick installer to let the installer read code files from disk while installing if they are there; this allows hot-fixing/hot-debugging of installation bugs
• keep_show_timings – set this to '1' for the installer to spit out some timings information (for performance debugging) and turn off auto-continue
• keep_translate_sort – set this to 'recent' to make the content translation interface show recent strings first; useful during debugging
• keep_debug_fs – set this to '1', while in the installer, or logged in as an admin, for disk I/O to be slowed and file system case sensitivity to be checked. Or, to a higher number of milliseconds if you want to introduce fake I/O latency. It is for developer filesystem testing.
• keep_manual_import_config – set this to '1' in the import module to disable automatic detection of the import target's database settings from its filesystem config file (if you don't have it, for example)
• keep_timer_display – set this to '1' if you want quiz timers to show in the quiz module, even to users with the bypass_quiz_timer privilege

The following 'keep' parameters have interface triggers, but are also handy for manual activation:

• keep_markers – set this to '1' if you want template start/end markers to be output in the page HTML source when viewed in Composr
• keep_webstandards_check – set this to '1' to perform advanced output checking (for staff only), giving errors if technology guidelines are broken (such as XHTML5, CSS, and WCAG). If you enable this flag you will be informed of any accessibility problems on your website, but it severely effects performance as well as your administration experience.
• keep_su – set this to a username or member ID when logged in as admin, to pretend to be that member. May choose the guest member ('Guest' on Conversr by default).
• keep_theme_seed – set this to kiddie or random or a 6-character HTML colour code, to get dynamic themegen going for your page view (this is very slow, but fun!); you must be staff, with a confirmed session, for this to work
• keep_theme_dark – used in conjunction with keep_theme_seed; set to '1' if it is a dark seed
• keep_theme_source – used in conjunction with keep_theme_seed; set to the name of the theme being used as the source
• keep_theme_algorithm – used in conjunction with keep_theme_seed; set this to 'equations' or 'hsv' to set the theme algorithm
• keep_print – set this to '1' to display a 'printer friendly' version of the page
• keep_lang – set to a two-letter ISO language code to tell Composr which language to use
• keep_session – this isn't useful for manual editing, but it is used to note session-IDs when [session-]cookies are not working
• keep_timezone – this is the timezone code (tz style) to view in
• keep_catalogue_<catalogue-name>_root – set to a category ID for the current viewed content type to trick the breadcrumbs into showing a 'virtual root'; this can also be activated by browsing to the category you'd like to be virtual root (as staff) and then clicking on the final link in the breadcrumb chain (the link representing the current category)
• keep_wiki_root – as above, but used to propagate Wiki+ virtual roots
• keep_gallery_root – as above, but used to propagate gallery virtual roots
• keep_download_root – as above, but used to propagate download virtual roots
• keep_forum_root – as above, but used to propagate forum virtual roots
• keep_page_root – as above, but used to propagate Comcode page virtual roots

The following 'keep' parameters have interface triggers and are not useful for manual use:

• keep_full_structure – whether to avoid merging nodes together in the Sitemap, used by the Permissions tree editor so that permissions on non-accessible nodes are settable

The following aren't 'keep' parameters, but are still useful and otherwise behave in the same way:

• auth – set to '1' to make Composr request HTTP authentication. This is useful for RSS URLs, if you want the feeds to be generated as authenticated.
• wide – set to '1' if you don't want to show side panels
• wide_high – set to '1' if you don't show to show panels or the TOP/BOTTOM
• start – set to the start number in the result browsing
• max – set to the maximum number of results to show per-page
• page/type/id – standard URL routing parameters
• kfsXX – this isn't useful for manual editing but people ask what it's for; it's to preserve the browse positions in forumviews, such that when returning, the browse positions are remembered

Extra _config.php settings

Setting,
in _config.php will disable e-mails. This is useful on a development server running with live user accounts.

If you use the config_editor.php script to edit the base configuration then you'll find lots of other settings to play with.

Providing default parameters to forms

Many forms in Composr accept GET parameters for specifying default field values. This includes most 'add' forms in the system. Generally the GET parameter is exactly the same as the actual field name in the HTML, so it's easy to work it out.

We also have some special cases coded up directly. These are often referenced between different parts of the Composr code. The details below are divided by module, then type, then parameter name.

• topics
  • new_post
    • quote
    • intended_solely_for
• warnings
  • add
    • member_id
    • post_id
• admin_aggregate_types
  • add
    • aggregate_type
  • sync
  • type
• admin_ecommerce_logs
• add
  • group_id
  • trigger
  • type_code
• admin_cns_forums
  • add
    • parent_forum
    • forum_grouping_id
• admin_cns_merge_members
  • browse
    • from
    • to
• cms_authors
  • _add
    • author
• cms_banners
  • add
    • b_type
• cms_blogs
  • add
    • cat
    • validated
• cms_booking
  • add_booking
    • bookable_id
    • day
    • month
    • year
• cms_catalogues
  • add_entry
    • catalogue_name
    • category_id / category_id_suggest
    • field_*
    • validated
  • add_category
    • catalogue_name
    • parent_id
  • add_catalogue
    • id
• cms_calendar
  • add
    • date
    • e_type
    • validated
• cms_downloads
  • add
    • cat
    • validated
  • add_category
    • parent_id
• cms_galleries
  • import
    • member_id
    • cat
  • add
    • cat
    • validated
  • add_other
    • cat
    • validated
  • add_category
    • parent_id
• cms_news
  • add
    • cat
• cms_quiz
  • add
    • validated

All default parameters are flagged as 'non-canonical' to search engines, so that the search engine knows that variations in value are not indicative of entirely separate web pages.

Log files

These log files exist by default in data_custom:

• errorlog.php – Run-time error logging (using the PHP error logging system) [shown in Composr Error log page]

The follow log files can be created in data_custom, and will then be appended to as Composr runs:

• permissioncheckslog.php – Permission checks (useful for debugging why accessing is failing) [shown in Composr Error log page]
• ecommerce.log – raw eCommerce transaction data (useful for debugging IPN)
• queries.log – queries running (debugging)
• big_query_screens.log – slow queries running (debugging performance)
• profiling*.log – profiling [if enabled] (debugging performance)
• resource_fs.log – resource-fs operations (debugging and audit)

If you create these files it is your responsibility to make sure users cannot download their contents. We supply a default Apache data_custom/.htaccess file for this, but this requires Apache and .htaccess files to be enabled; we also supply some rules in web.config for IIS.

Overriding files

If you change an original Composr file for a client project you must save it into the _custom equivalent of its original directory. E.g. if you change site/pages/modules/calendar.php you need to save into site/pages/modules_custom/calendar.php. Otherwise if the client ever upgrades their website chances are all the customisations you've made would be lost when a newer version of the original file is placed.

This said, It's best to avoid completely overriding default files if possible, so that any bug fixes from upgrades are more likely to work. Composr provides some techniques so you can override just stuff you want to change rather than a whole file.
This is explained in the The Composr programming framework tutorial (there are two techniques: replacing/supplementing, and programmatic alternation).

If you are considering overriding a file just so that you can add some new functions to it used by one of your new modules (for example, adding a new form_input_something function), consider adding to a new file (or existing file in your module's addon) instead. There is no rule that says similar functions have to go together, and it makes no sense unless they are all core functions – in fact it increases average memory usage to have excess functions defined in core files (each required-up function uses quite a lot of memory).

Hints for developing websites for other people

Composr is a great platform for web developers. A web developer can build all kinds of sites using Composr – from simple new media sites, to social networks, to seemingly-bespoke sites such as property directories.

However, if you're making a website for somebody else it is is important to bear in mind they probably do not want to learn Composr like you have.

Here are some assorted hints:

• You might want to use the debranding feature, Admin Zone > Style > Debranding
• You might want to give the user a non-admin username (e.g. a super-moderator user instead), and then set Admin Zone permissions so that many unnecessary pages are hidden. Composr is then smart enough to automatically simplify its interface to compensate.
• When deploying make sure to get your canonical URL right. Ask the client whether they want 'www.' in their URL or not. Set up a .htaccess file (https://css-tricks.com…pets/htaccess/www-no-www/) that redirects the non-canonical URL to the correct one. This will remove a lot of headaches related to duplicate cookies, bad SEO, confusion, and Composr error messages.
• Always, always, test the e-mails a site gives out. It is very embarrassing if you accidentally customise your MAIL.tpl template in a custom theme and then find e-mails are being sent out of the Admin Zone using the default theme and Composr logo. Always save your MAIL.tpl in the default theme, and make sure that the default theme's 'logo/-logo' and 'logo/standalone-logo' theme images correctly reflect your design. And, test it! If you made complex changes, test in different e-mail programs/webmail-apps.
• Develop using a staging/developer website, and deploy to live/customer-demo after you've finished all changes. This primarily involves uploading changed files. Composr has a great tool for transferring data between Composr sites using XML too, if you find you are needing to sync data.

Considerations for design and themeing:

• If there are separate frontend and backend developers, the frontend developer can do an effective mockup by coding a simple mini-module/mini-block that uses a static array of data passed directly into templates. This way the built CSS and markup is "Composr ready" (particularly things like the 'IMG' symbol can be used from the start). This requires the frontend developer to have basic PHP and Tempcode experience.
• In particular, do not make theme image changes using Composr's theme editor, as these are really painful to sync. Make use of Composr's automatic theme image detection by simply saving new theme image files into appropriate directory locations; note you will need to clear the caches if you are overriding a default image.
• If you are adding new functionality and write new templates, it is best to save these into the default theme's templates_custom/javascript_custom/xml_custom/text_custom/css_custom directories, so that your functionality works on all themes. If you are overriding default theme templates/CSS, it is then that you save them into your site's theme. A similar rule applies for custom images.

Take particular care with Comcode pages:

• If you have to hand code a Comcode page with HTML then include the following hint in it: {$,page hint: no_smart_conversion}. This will stop the WYSIWYG editor trying to convert the HTML back into Comcode, which can mess up hand-crafted markup and CSS (normally WYSIWYG only guarantees to preserve style/structure created within itself).
• If for some reason you don't want the WYSIWYG editor to run for a page (e.g. if it has a Google ad on, that can cause the editor to malfunction) then include the following hint in it: {$,page hint: no_wysiwyg}
• Remember when uploading to a live site that you do not want to overwrite customised Comcode pages, or other files that may have since been changed. Be careful what you upload, and keep backups.

It's overkill to apply the full range of Composr coding standards if you're writing PHP code for use on just one website. You shouldn't get into bad habits by skipping any of the conventions but some of our standards are particularly time consuming to meet, and in particular the following time savers can be applied:

• It is better to have "hackerish" overrides (using the code rewrite facility) than to override whole files. This is because it makes upgrades easy, which is far more important than code maintenance for a one-off Composr project.
• You can almost always assume the site runs one theme and thus save your new templates into the new theme. One caveat is the 'CMS' zone will need to run your theme, and also bear in mind that any files used in outgoing e-mails must be available from the default theme.
• You don't need to write everything using language strings.
• You don't need full MVC. For example, you could code in add/edit functionality straight into modules rather than via a sources file (model).
• You can make more assumptions than you ordinarily would be able to. For example, you might be able to assume cURL is installed. It is sensible to document your assumptions though, to make sure they continue to be met, or so someone breaking them has an idea that they need to act to change your code.
• For simple dynamic functionality you can use mini-modules and mini-blocks instead of full blown object-orientated modules and blocks.
• You do not need to write PHPdoc comments unless you think other developer's would benefit from them.
• On some sites you may be able to assume JavaScript is enabled, or that accessibility isn't an issue (e.g. on a video site it's unlikely to need to cater for blind users – unless you plan to support audio description). Generally we would advise to be very cautious here, it usually does not take much longer to support full accessibility, and is worth doing routinely. But it is worth us pointing this out because some sites may require particular complex JavaScript interfaces and it might be unviable to support JavaScript being disabled.

Bear in mind that some sites need maintenance or have multiple people working on them, so you should bear this in mind when you decide the level of engineered design and code documentation to implement for your custom functionality. It is a trade-off between initial development cost, and long-term time savings.

Scripts and tools

There are a large number of scripts and tools for helping in particular situations, outside the normal content management processes of running a Composr website.

Many of these are designed just for programmers, to speed up development. But some are particularly useful just for basic configuration management and DevOps.

In this section we document all the scripts and tools that may be generally useful to you as a developer, with a few categories of exceptions:

1. Commandr commands (go to Commandr and type commands to see a list)
2. Website cleanup tools (go to Admin Zone > Tools > Website cleanup tools)
3. Tools within the upgrader (upgrader.php)
4. Scripts with code listings provided in the Professional upgrading tutorial
5. IP address lookup (when you do an Investigate User in Composr you get various links to do investigations on the IP address like geographic lookup)

This section is not designed as primary documentation for the tools. Those tools not designed for programmer use may be described in various individual tutorials. The tools for programmers are typically undocumented, but a programmer can analyse the code to see how they work (and make improvements as needed).

Website management

DevOps

Theme development

Development

Core development

Documentation development

Release management

API services

Composr version naming

In order to have a consistent upgrade system, Composr versions are named according to a rigid scheme.

For written version names we use the "pretty" naming scheme. This is best shown by example…

The following are examples of valid "pretty" names:

• 3
• 3.1
• 3.1.4
• 4 alpha1
• 4 beta1
• 4 RC1

The following are examples of invalid "pretty" names:

• 3.0 [reason: leading ".0" components]
• 3.0.0 or 3.1.0 [reason: leading ".0" components]
• 4alpha1 [reason: missing space]
• 4 alpha 1 [reason: extra space before "1"]
• 4Alpha1 [reason: alpha/beta must always be lower case]
• 4 RC 1 [reason: extra space before "1"]
• (anything with dashes in)
• (anything with 'final' or 'gold' in)

Rules:

• Versions should never be skipped. E.g. There can't be a 3.2 if there is no 3.1.
• All versions have to have formally named news posts, in the 'New releases' news category. Our upgrade scripts do this automatically.
• Dots are not decimal points. They split major/minor/patch levels. You can have numbers in any dot section exceeding 9 and going to 10.
• alpha/beta/RC is technically attached to a patch-level release, not a major/minor version. i.e. any patch release can have alpha/beta/RC.

Inside our code we always use the "dotted" scheme, which is the same except that the space is replaced by a dot.

See the notes at the top of sources/version2.php for a full run-down of our ways of expressing versions.

Conclusion

We're glad you've made it to the end of our Code Book. Thanks for reading this far, and we hope you find Composr an effective and enjoyable environment to code for.

Feedback

Have a suggestion? Report an issue on the tracker.