Composr Supplementary: Professional upgrading

Written by Chris Graham (ocProducts)
This tutorial provides a professional process for updating customer website(s). Steps here may also be useful for professional website "tune ups".

It is only intended for programmers who are experienced in the use of Composr.

The tutorial is currently written for updating ocPortal sites to Composr v10 sites.

Table of contents

  1. Composr Supplementary: Professional upgrading
    1. Getting organised
    2. Process
    3. See also

Getting organised

If you are updating multiple websites then create a spreadsheet with the following columns:
  • Website URL
  • Prospective upgrade date
  • Cost quote
  • Website owner name
  • Website owner e-mail
  • Support ticket / CRM URL
  • Contacted owner?
  • Have access to website?
  • Cost agreed with owner?
  • Any concerns

This will keep you organised and prevent you from losing your sanity trying to remember all these details.

Process

This is the process repeated for each website being upgraded. The steps are not set in stone, but rather designed to be interpreted as guidelines by an experienced developer.
We assume you are running your own local AMP-style stack, have git installed, and have a Composr git checkout under <webroot>/Composr
<webroot> refers to your webroot, wherever it is. On my machine it is ~/www.

Step Description Extra code / notes
1 Transfer these instructions to a separate TODO list so you can work through them Copy to clipboard · Personalise paths
Downloading, assessing, reconfiguring
2 Check for any special circumstances If a multi-site-network or a third-party forum is involved, or other third-party integrations, extra consideration for compatibility will be needed
3 Make sure the website owner is aware that an upgrade cost doesn't include training for functionality changes Customers need to be aware of costs, if you don't protect yourself you could have a customer expecting you to include days of free training for what you're only invoicing as a few hours of work. Customers also need to be aware that things may change, including features being removed (link them to any announcement about this). Consider also asking for a list of all the design or code changes made on the site so that you can cost it up / plain it; anything non-listed might be charged extra for.
4 Download all the website files to your local machine, under <webroot>/<codename> It's much more efficient to work locally. Leave this running in the background while you work on other steps / tasks.
5 Check the remote server meets the Composr minimum requirements If the requirements are not met you'll need to circle back to the website owner; they may have intentionally held back software on their hosting and have the capability to update it, or they may need to do a server upgrade or host switch.
Composr's minimum requirements provide a lot of info, if you can't access that it's very easy to just get PHP's standard phpinfo:

Code (PHP)

<?php

phpinfo();
 
Additionally you can upload and run the first two steps of Composr's installer and see if it complains about anything.
6 Confirm the remote site is running a version of ocPortal/Composr that you can upgrade from If not you'll need to circle back and inform them of the cost of jumping through other intermediate ocPortal/Composr version(s).
† 7 Create <webroot>/<codename>/_temp and put a .htaccess file in it

Code

# < Apache 2.4
<IfModule !mod_authz_core.c>
   Order Allow,Deny
   Deny from all
</IfModule>

# >= Apache 2.4
<IfModule mod_authz_core.c>
   Require all denied
</IfModule>
† 8 Download the website database to your local machine, put it in <webroot>/<codename>/_temp/initial.sql If you can't access phpMyAdmin, then use a script:

Code (PHP)

<?php

$filename = 'sql-backup-' . date('Y-m-d') . '.sql';

$h = 'Content-Disposition: attachment;
        filename="' . $filename . '"';
header($h);

$db_site = 'TODO';
$db_site_host = 'localhost';
$db_site_user = 'TODO';
$db_site_password = 'TODO';

if ($db_site == 'TODO') {
    // LEGACY
    if (is_file('_config.php')) {
        require('_config.php');
    } else {
        require('info.php');
    }
    $db_site = $SITE_INFO['db_site'];
    $db_site_host = $SITE_INFO['db_site_host'];
    $db_site_user = $SITE_INFO['db_site_user'];
    $db_site_password = $SITE_INFO['db_site_password'];
}

$sql_cmd = 'mysqldump';
$sql_cmd .= ' -u' . escapeshellarg($db_site_user);
$sql_cmd .= ' -p' . escapeshellarg($db_site_password);
$sql_cmd .= ' -h' . escapeshellarg($db_site_host);
$sql_cmd .= ' ' . escapeshellarg($db_site);
$sql_cmd .= ' 2>&1 > ' . $filename;
echo '/*';
echo shell_exec($sql_cmd);
echo '*/';

@ob_end_clean();
readfile($filename);

unlink($filename);

if (!empty($_GET['self_destruct'])) {
    unlink('sqldump.php');
}
 
9 Replace data/upgrader2.php from the latest patch release of the version being upgraded from We've had some legacy bugs in this crucial upgrade file.
10 If your server is not suEXEC-style, set recursive world write permissions

Code (Bash)

cd <webroot>/<codename>
sudo chmod -R 777 .
 
† 11 Search and replace the old table prefix in the SQL dump if you're upgrading an ocPortal site Usually ocPortal sites used ocp_ or ocf_, and we want to use cms_ so that ocPortal legacy doesn't need to be remembered in the future.
† 12 If you're updating from ocPortal then replace DEFAULT CHARSET=latin1 with DEFAULT CHARSET=utf8mb4 in the SQL dump If you are having to fudge a server running an old version of MySQL, use utf8 instead and be ready to tell the website owner they cannot use emojis or install any new addons (as that will try and use utf8mb4 for new tables).
† 13 (Following directly from the above) Change utf8mb4 to utf8 for just the following tables: cms_addons, cms_chat_rooms, cms_f_saved_warnings, cms_f_emoticons, cms_gsp, cms_import_parts_done, cms_msp, cms_newsletter_subscribe, cms_sessions, cms_theme_images, cms_tickets, cms_url_title_cache, and cms_wordfilter If you are having to fudge a server running an old version of MySQL, use utf8 instead and be ready to tell the website owner they cannot use emojis.
14 If you're updating from ocPortal then find all non-default files containing this regexp [^\x00-\x7F] and re-save them as utf-8 The regexp finds non-ASCII files, so you don't have to re-save every single file
† 15 Import the SQL dump into your local MySQL installation, in a database named client_<codename>, using the command line mysql client

Code (Bash)

cd <webroot>/<codename>
mysql client_<codename> < _temp/initial.sql
 
16 If you're updating from ocPortal then add some settings to _config.php/info.php:

Code (PHP)

$SITE_INFO['self_learning_cache'] = '1';
$SITE_INFO['session_cookie'] = 'cms_session__<whatever>';
 
17 Reconfigure _config.php/info.php to have local details on your development hostname

Code (PHP)

$host = gethostname();
switch ($host) {
    case 'TODO': // dev machine
        unset($SITE_INFO['db_forums']);
        unset($SITE_INFO['db_forums_host']);
        unset($SITE_INFO['db_forums_user']);
        unset($SITE_INFO['db_forums_password']);
        unset($SITE_INFO['cns_table_prefix']);
        $SITE_INFO['base_url'] = 'http://localhost/<codename>';
        $SITE_INFO['db_site'] = 'client_<codename>';
        $SITE_INFO['db_site_host'] = 'localhost';
        $SITE_INFO['db_site_user'] = 'root';
        $SITE_INFO['db_site_password'] = '';
        $SITE_INFO['debug_mode'] = '0';
        $SITE_INFO['dev_mode'] = '0';
        $SITE_INFO['no_email_output'] = '1';

        // These must not exist on any live site,
        // they allow temporary unauthenticated
        // local admin access to make upgrading
        // easier a third-party developer
        $SITE_INFO['admin_password'] = '';
        $SITE_INFO['backdoor_ip'] = '127.0.0.1';
        break;
}
 
18 Change mysql to mysqli in _config.php/info.php if upgrading a really old site
† 19 Save a new SQL dump into the _temp folder

Code (Bash)

cd <webroot>/<codename>
mysqldump client_<codename> > _temp/stage1.sql
 
20 Set up a local git repository to reduce the chance of losing work

Code (Bash)

cd <webroot>/<codename>
git init
cp <webroot>/composr/.gitignore .
 
edit .gitignore to remove the second half of the file

Code (Bash)

git add .
git commit -a -m "Initial commit"
 
Main upgrade
† 21 Log into the website's upgrader on your local machine
† 22 Empty the caches
23 Do the file transfer using the omni-upgrader package if upgrading from ocPortal, or a custom upgrader build otherwise Upgrading from ocPortal v8/v9 to Composr v10 - Composr
24 Do the integrity checker scan and delete files from the prior version
† 25 Do the database upgrade
26 If upgrading from ocPortal delete the info.php file
Cleanup and quarantine
27 Save a new SQL dump into the _temp folder

Code (Bash)

cd <webroot>/<codename>
mysqldump client_<codename> > _temp/stage2.sql
 
28 Delete old page revisions, old config files, old backups, old .latest_in_ocp_edit files, etc
29 Temporarily move any custom code and themes into a parallel directory structure under _old_code
† 30 If upgrading from ocPortal switch to non-multi-lang-content using Commandr (unless the user really does want multi-lang-content)

Code

:
require_code('database_multi_lang_conv');
disable_content_translation();
† 31 If jumping across versions run through the "Correct MySQL schema issues (advanced)" cleanup tool (Admin Zone > Tools > Website cleanup tools) to find and resolve any unexpected issues that might have developed Don't just trust the SQL given, check it really is correct to repair the problems found.
32 Run some cleanup tools Go to Admin Zone > Tools > Website cleanup tools, run "Sync for lost disk content"; if you're really enthusiastic also "Find orphaned uploads" and "Broken URLs"
33 Delete some empty directories that may have been left over (Composr only tracks alien files, not directories) This will find empty directories, only delete with care:

Code (Bash)

find . -type d -empty
 
On a Mac it is more tricky, as .DS_Store files may be littered around. You can use Thomas Tempelmann | Find Empty Folders for macOS or run this command first:

Code (Bash)

find . -name .DS_Store -exec rm {} \;
 
(The upgrader does delete empty directories caused by alien file removal.)
34 Consider replacing all the tar files in addons/imports so you're sure they're fresh If you download a custom upgrade package you can find all the TAR files in it (the from version number in the URL is irrelevant)
35 Do a git commit

Code (Bash)

cd <webroot>/<codename>
git add .
git commit -a -m "Finished cleanup and quarantine"
 
Rebuild custom code and design
36 Build a new theme with the correct seed.
37 Use a diff tool to find all the changes made in the old theme (you put them in _old_code). You can usually diff-compare against the .editfrom files, otherwise you'll need to go back and compare with the originals from the version you are upgrading from.
There is a themechanges.sh script which may be useful, but realistically you probably just want to do it yourself on a file-by-file basis.
38 Fix for specific known theme changes. Buttons and CSS classes have changed significantly since old versions of ocPortal, these may need reworking if the icons are directly used anywhere.
39 Use a diff tool to find all the changes made in code files you put in _old_code. You'll need to compare overrides against the originals from the version you are upgrading from. Custom code and overridden code will need to be assessed to make sure it still is valid. By the time you're done you should be able to erase the _old_code directory.
40 Run CQC. Run the Code Quality Checker on any custom code. Do this by copying data_custom/functions.bin from a Composr GitLab pull, and then run the CQC out of that GitLab pull while reconfiguring the scan folder to the <webroot>/<codename> directory.
41 Users may be used to some different configuration if upgrading from ocPortal, maybe set simplified_attachments_ui to 0. US users may also like yeehaw to be set to 1.
42 Do final local testing, particular some embedded links may need reconfiguring if upgrading from ocPortal, as the Composr display defaults are different Go through testing page by page, and also at least get a feel for content (such as news articles) working well
43 Review errorlog.php then clear it out
44 Do a git commit

Code (Bash)

cd <webroot>/<codename>
git add .
git commit -a -m "Finished code rebuild"
 
Get back in sync
 Defer this section if you are going to provide a client demo before the final upgrade
45 Close the remote site. If the site being upgraded has database content changing regularly (perhaps member activity), you need to update your local database so you have those changes. Skip this and the next step if you do not.
46 Get your database back in sync Do one of 3 things:
  1. Identify that nothing needs synching. If you are using MyISAM then Commandr can tell when tables were recently changed with ls /database.
  2. Take a new database backup and use a diff against your very first database backup, then manually copy those changes through. This is only a good idea if you expect a tiny number of simple changes. Of course in most cases you can ignore changes to logging tables.
  3. Install MySQL utilities and Connector/Python, put your _temp/stage1.sql file into client_<codename>_old then run something like:

    Code (Bash)

    mysqldbcompare \
     --server1=remote_user@remote_host \
     --server2=root@localhost \
     remote_database:client_<codename>_old \
     --run-all-tests \
     --difftype sql \
     --disable-binary-logging
     
    …to get SQL to do an update. Then carefully filter it to what you really want, then run it.
  4. Take a new database backup, repeat steps from this tutorial marked † to re-upgrade that database. It should not take so long given the file upgrading you've already done will still apply. You'll need to re-open the site of course, given you closed the live site just before backing it up.
Options 3 or 4 are the most likely you will want to use.
47 If the site being upgraded has files changing regularly (perhaps edited Comcode pages), download those changed files and extract locally This is a script to identify changed remote files and zip them:

Code (PHP)

<?php /*

 Composr
 Copyright (c) ocProducts, 2004-2017

 See text/EN/licence.txt for full licencing information.

 Meta Script:
   Find files modified within the last days days
*/

header('Content-type: text/plain');

if (!isset($_GET['days'])) {
    exit('Must give a days parameter');
}

$days = intval($_GET['days']);
$cutoff = time() - 60 * 60 * 24 * $days;

if (isset($_GET['path'])) {
    chdir($_GET['path']);
}

$files = array();

$out = do_dir('.');
sort($out);
foreach ($out as $file) {
    if (filemtime($file) > $cutoff) {
        $files[] = $file;
    }
}

if (!empty($_GET['zip'])) {
    if (count($files) == 0) {
        exit('No files');
    }

    header('Content-type: application/octet-stream');
    $h = 'Content-Disposition: attachment;
        filename="new_files.zip"';
    header($h);

    $cmd = 'zip new_files.zip';
    foreach ($files as $file) {
        $cmd .= ' ' . escapeshellarg($file);
    }
    shell_exec($cmd);

    @ob_end_clean();
    readfile('new_files.zip');

    unlink('new_files.zip');

    exit();
}

foreach ($files as $file) {
    echo $file . chr(10);
}

echo 'Done; set &zip=1 to download a zip file';

if (!empty($_GET['self_destruct']) {
    unlink('find_new_files.php');
}

function do_dir($dir)
{
    $out = array();
    $_dir = ($dir == '') ? '.' : $dir;
    $dh = opendir($_dir);
    if ($dh) {
        while (($file = readdir($dh)) !== false) {
            if ($file[0] == '.') {
                continue;
            }

            if (is_file($_dir . '/' . $file)) {
                $out[] = $_dir . '/' . $file;
            } elseif (is_dir($_dir . '/' . $file)) {
                $_under = $dir .
                    (($dir != '') ? '/' : '') .
                    $file;
                $_out = do_dir($_under);
                $out = array_merge($out, $_out);
            }
        }
    }
    return $out;
}
 
Uploading
48 Put a copy of bigdump.php into the _temp folder and configure it for the live database http://www.ozerov.de/bigdump/
49 Upload a zip of the files into a _new folder, being careful to not include the .git folder in the zip If some upload directories are too big, omit them and resolve this by moving back the old ones from the live server.
For a further level of professionalism you may choose to deploy using your git repository rather than a zip file, if so push git up to a shared repository (e.g. GitLab or Bitbucket) and do a pull from the server, rather than uploading and extracting a zip.
50 Extract the zip live on the server (control panel file managers can do this, or a simple custom PHP script)

Code (PHP)

<?php /*

 Composr
 Copyright (c) ocProducts, 2004-2017

 See text/EN/licence.txt for full licencing information.

*/

if (!isset($_GET['file'])) {
    exit('Need \'file\' parameter (filename to unzip)');
}

$filename = $_GET['file'];

error_reporting(E_ALL);
ini_set('display_errors', '1');

$zip = new ZipArchive;
$res = $zip->open(dirname(__FILE__) . '/' . $filename);
if ($res === true) {
    $zip->extractTo('.');
    $zip->close();
    echo 'done';
} else {
    echo 'failed ';
    switch ($res) {
        case ZIPARCHIVE::ER_EXISTS:
            echo 'ER_EXISTS';
            break;

        case ZIPARCHIVE::ER_INCONS:
            echo 'ER_INCONS';
            break;

        case ZIPARCHIVE::ER_INVAL:
            echo 'ER_INVAL';
            break;

        case ZIPARCHIVE::ER_MEMORY:
            echo 'ER_MEMORY';
            break;

        case ZIPARCHIVE::ER_NOENT:
            echo 'ER_NOENT';
            break;

        case ZIPARCHIVE::ER_NOZIP:
            echo 'ER_NOZIP';
            break;

        case ZIPARCHIVE::ER_OPEN:
            echo 'ER_OPEN';
            break;

        case ZIPARCHIVE::ER_READ:
            echo 'ER_READ';
            break;

        case ZIPARCHIVE::ER_SEEK:
            echo 'ER_SEEK';
            break;
    }
}
 
51 Run bigdump
52 Delete the _temp folder and the zip file
53 Amend _config.php live to remove your temporary details, and possibly also add a backdoor_ip line

Code (PHP)

$SITE_INFO['backdoor_ip'] = 'TODO';
 

Smarter configuration management

If you want to have the exact same configuration file running across multiple machines, you can code in varying details using a switch statement…

Code (PHP)

switch (gethostname()) {
        ...
}
 
54 Set file permissions on live if necessary
55 Get the client to test out of the _new folder, if appropriate (in which case you'll need to temporarily set the base_url setting in _config.php)
56 Move the old files on server into an _old folder
57 Move files down from _new folder, changing base_url setting in _config.php back to normal if we changed it in step 55
58 Final live testing
Customer interfacing
59 Contact the website owner with whatever you need to say, including details about old files and database tables
Your site has now been updated, the changes are live.

I have left a copy of your old site files under the ""_old"" folder.
Additionally, your old database tables are still in the database using the ""ocp_"" table prefix. The new database tables use the ""cms_"" table prefix.

I have granted my IP address administrative access so I can quickly run administrative tasks should I need to. Please let me know if you would like this to be removed.
60 Charge the website owner for the work


See also


Feedback

Please rate this tutorial:

Have a suggestion? Report an issue on the tracker.