History: TRIM
Source of version: 146
Copy to clipboard
__TRIM__ is an acronym for the __Tiki Remote Instance Manager__.
TRIM is installed on a master server. TRIM then connects to various servers and permits you to manage various instances of Tiki. You can install, upgrade, backup, check the file integrity and do various other things. Some actions can be run unattended on a cron job.%%%
For more details on its development status see [http://dev.tiki.org/TRIM].
{REMARKSBOX(type="information" title="Superceded feature" highlight="y" close="n" store_cookie="n" )}The __"Tiki Remote Instance Manager (TRIM)"__ has gone into maintenance mode, and the code was forked and revamped to become its replacement, now known as the __"((Manager|Tiki Manager))__".{REMARKSBOX}
!! What
* A combination of shell and PHP scripts to install, update, backup, restore and monitor (check security of) a large number of Tiki installations (instances).
* TRIM facilitates/automates the operations you would normally do by logging into each server.
* A great tool if you want to setup a ((dev:WikiFarm))
* It is __not__ a script to update your Linux or MySQL installation (but it may one day do some server checks to see if your Apache, PHP, MySQL, GD, etc. versions are OK).
* Originally developed for Tiki, it was designed so it could be ported/used for just about any PHP / MySQL application which has SVN as a source. If you want to port to a different app, let's work together!
* See also ((TRIM use cases))
!! How it works
TRIM is installed on a __master__ server. TRIM then connects by SSH (and more recently via FTP) to various servers. Via a wizard (you answer the questions interactively) TRIM permits you to manage various __instances__ of Tiki. You can install, upgrade, backup, check the file integrity and do various other things. Some actions can be run unattended on a cron job.
!! Limitations
* FTP support has been added, however, you can run into memory limitations (outside of TRIM's control)
!! About supported versions
* In theory, TRIM works for all versions of Tiki, and on all versions of GNU/Linux. In practice the most testing happens for:
** [http://wikisuite.org/ClearOS|ClearOS]
** Latest stable versions of Tiki 15.x, 18.x and trunk. Ex.: we may find a bug on 15.1, and fix it for 15.2, so you should always be on latest versions of each supported branch.
!! Alternatives
((MultiTiki)) is an alternative to TRIM.
* ((MultiTiki)): all files (php/tpl/etc) are shared amongst several domain names (but you can have .css and .tpl which override for your domain) but each domain has a distinct database.
* TRIM: each domain is totally independent. Distinct database __and__ distinct files __and__ can be on different hosts.
As of 2017-01, TRIM can not be used to manage a Multi-Tiki, but this could be developed (please do get in touch if you have a use case for this)
!!! Similarities
# Both TRIM and MultiTiki are tools to facilitate mangement of a large number of Tiki instances,
!!! Advantages of TRIM vs MultiTiki
# With TRIM, each instance can be on totally different servers or on various directories on the same server, so you can easily scale to as few or as many servers as you want.
# Once an instance is in a MultiTiki, it's a little tricky to extract it. For example, it's easier to provide a backup of just one instance, using TRIM.
# If you have multiple domains and need to provide distinct access for each (emails, FTP accounts, access to logs, etc.), it can be easier to have distinct server space.
# With MultiTiki, if you have a security issue, all your Tikis are potentially compromised because they share the same files. (even for the Tikis that have not enabled the unsafe feature)
# With TRIM, you can pick & choose which Tikis to upgrade. With MultiTiki, all the instances get the upgrade. Even if they don't need the bug fixes or enhancements, they are still at risk of being affected by a ((dev:regression)). So all instances need to be checked after each upgrade.
** __This takes a lot of time__. And then, you can introduce issues for projects that were quite happy with the previous version. They didn't need new stuff, but inherited some problems.
# With TRIM, you can modify the templates/*.tpl files and still use the merging features of svn to take advantage of the enhancements to the main code base (There could be conflicts to manually resolve during upgrades though). Using multiTiki, you must put all changes in a separate directory and can't easily benefit from improvements to the main code base.
# TRIM handles offsite backups
!!! Advantages of MultiTiki vs TRIM
# Uses less disk space
# Can upgrade hundreds of Tikis in one operation (whereas TRIM connects one by one to upgrade)
# MultiTiki is part of the main Tiki code base whereas TRIM is maintained separately.
# --TRIM requires SSH access-- TRIM can now work via FTP, whereas maybe MultiTiki can be run just with FTP. Depending on your server, you may have a lot of permissions to change by FTP though.
# --TRIM requires more things on the server (sqlite, SVN, etc)-- not anymore, you can use FTP mode and heavy work is done on TRIM master, and result is pushed to the TRIM instance.
# --MultiTiki has a web-based interface-- TRIM Web Administration was added
!! What if I want to share data between instances?
Sharing data between Tiki instance is not specific to TRIM. If you want to share logins, groups and preferences throughout many Tiki sites, you can use ((InterTiki)).
!! License
LGPL, like the rest of Tiki
!! Documentation
Here is a brief explanation of the basic commands that can be used. All commands follow a wizard pattern.
!!! make instance
* -+make instance+- adds an instance to be managed by TRIM, which can mean either:
** create a fresh Tiki instance (including a database)
** or detect an existing Tiki installed via Subversion (SVN) and "adopt" it.
*** You may need to ((dev:Convert a site installed via FTP to now use SVN|convert your Tiki instance from FTP to SVN)).
When running make instance, TRIM attempts to perform various operations from this point:
# Share an RSA key to connect to the remote host in the future
# Make a few verifications on remote host
# Detect the application installed remotely and the version
# If nothing is installed, propose to install something
You will be prompted to enter a pass phrase. If you enter one, you will be prompted every time you use TRIM. Unattended backups and security checks will not be possible.
You have three options for the location of your instance
* ftp: another server via FTP access
* ssh: another server via SSH access
* local: same server
The installation process will ask to create a database and user for Tiki. To do this, root access on the MySQL database (or user with similar privileges) is required. Otherwise, database settings can be entered manually (The database needs to have been created before in cPanel, webmin or similar).
!!! make editinstance
* -+make editinstance+- permits to modify an instance which is managed by TRIM. Added on 2017-11-16
!!! make blank
* Like make instance but it doesn't actually add a Tiki. It just creates and instance than can be restored to or cloned to. Alternatively, you do make instance but cancel when TRIM offers you to install Tiki.
!!! make check
This is somewhat equivalent to a ((Security Admin|#Check_your_files|secdb)) check, except that it's safer because it's made from a trusted host and will take your custom modifications on host into consideration. Upon first run, the check will ask where it should fetch the hashes from.
* source : obtain the hashes from the official repository. After setting up the expected hashes, a check will be performed.
* current : use the files currently online as the source of hashes. If this is a recent install, this may be safe. However, if the host was already compromised, hacks will be undetected.
* skip : Move on for now
When files are altered, added or removed, TRIM will prompt with various solutions.
* For modified files, it will allow to view the changes that were made and propose to accept or reject them. Rejecting changes will fetch a copy of the file from the repository.
* For added files, it will be possible view the file and then accept or reject it. Rejecting the file will remove it from the remote host.
* Removed files will allow to accept or reject the removal. Rejection will fetch a copy from the repository.
!!!! make verify
A synonym of [#make_check|make check]
!!!! make watch
Set-up a cron job on the TRIM master to perform the Hash check ([#make_check|make check]) automatically every day. The script will prompt for a contact email address to notify with the log in the case of a suspicious file change (which could be an intrusion or someone who legitimately changed a file on the server) and the time at which the script should run.
!!! make update
* Does a dry-run first, and aborts on any conflicts
* Updates to latest code in that branch (or trunk) using -+svn up+- and thus merging any changes
* And all operations should be done after updating the code
** Updates the file hashes accordingly. The hash verification/update may prompt with some files containing conflicts if modifications were made on the instance.
** Performs the database update.
** Runs setup and Composer,
** Clears cache
** Rebuilds search index
During the update process, the instance is disabled using a .htaccess file (previous one is preserved), making the site unavailable until the update is completed.
!!!! To setup automated updates
run:
{CODE()}
php scripts/setupupdate.php
{CODE}
Do not pick the same time as automatic backup!
!!! make upgrade
Similar to update. Requests for the branch to switch to over the update. You will have several choices of branches. Please note that you should __NOT__ downgrade as Tiki doesn't support a downgrade database script. An upgrade is a one-way street! You should make a backup before you upgrade so you can return to this version if issues arise.
If you choose to do a manual upgrade (with svn switch), run a [#make_detect|make detect] after so TRIM knows about it.
!!! make convert
Converts an instance from CVS BRANCH-1-9 to SVN branches/2.0.
* All script customizations will be lost
* Database, img/wiki and img/wiki_up are preserved
!!! make fix
Run setup.sh on the remote host using automated parameters. It should work in most cases. If the command proposed my setup.sh without parameters or super user rights are required, you should connect to the remote host manually using `make access`.
!!! make cli
* -+make cli+- permits to run any ((Console)) command. Added on 2017-11-16
!!! make access
Opens a shell to the remote host. Because TRIM manages all your connections, using this command simply avoids needing to remember passwords.
!!! make backup
What can go wrong and how should this script protect me?
* My host disappears
* A vandal messes up my files
* User error (oups, I erased some files)
TRIM performs a complete backup of the Tiki instance. The backup includes all files (using rsync for efficient bandwidth management) on remote host (including user files and files stored out of the web root) and a dump of the database. Each backup is archived in the TRIM_ROOT/backup/archive folder as a .tar.bz2 file. The backup file contains:
* manifest.txt: Indicates where folders were located on the remote host
* database_dump.sql: Self-explanatory
* ~np~[a-f0-9]{32}~/np~: Folders named using a hash. Content of the folders on remote host. The manifest.txt file lists these hashes.
Archives are cleaned up after each run. The rules are the following:
* Backups made on the first of any month are kept forever.
* Backups made on Sunday are kept for 31 days.
* Daily backups are kept for a week.
On your first backup, you may get a harmless error message
{CODE()}
rm: cannot remove `/home/trim/xyz/backup/1/manifest.txt': No such file or directory
{CODE}
!!!! To setup automated backups
run:
{CODE()}
php scripts/setupbackup.php
{CODE}
Do not pick the same time as automatic update!
!!! make restore
You can restore on a blank install. Ref: [#make_blank|make blank]
If you have data files that are not stored in the database, you should use the ((File Storage|#Ideal_scenario|ideal scenario for data file storage and relative paths)).
!!! make delete
Delete the instance via the command line (you could also do via the web interface). This does __NOT__ delete your Tiki. It just deletes the instance connection to it.
!!! make enablewww
Activate the ''TRIM Web Administration''. The script asks a few questions and changes some permissions.
!!!!- Rewrite rules are necessary for TRIM Web Administration
* On a standard install of Ubuntu, when trying to use WebTrim. Presumably because Rewrite Rules are not OK
{CODE()}
Not Found
The requested URL /webtrim/view/1 was not found on this server.
Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.1 with Suhosin-Patch Server at localhost Port 80
{CODE}
---
Search the Web to find instructions on how to activate Rewrite rules
http://www.google.com/search?q=+enable+mod_rewrite+module
!!! make viewdb
View the SQLite database. For debug purposes.
!!! make profile
* Apply ((profiles))
!!! make report
* Generate reports, and send reports to a Tiki instance using ((Data Channels)). More info at: https://profiles.tiki.org/TRIM_Report_Receiver
!!! make copysshkey
* Copy the SSH key
!!! make detect
Detect Tiki branch or tag, and PHP version. For debugging purposes. Also useful if you manually proceeded to ''svn switch'' and TRIM needs to update its internal database about a Tiki instance.
!!! make clean
This is to delete all state (sqlite3 database), backup, cache, and log files. Useful for development.
!!! make clone
Makes another identical copy of Tiki. This is basically a combination of ''make backup'' and ''make restore'' in one operation. You will be asked:
* Select the source instance:
* Select the destination instance -> __Be very careful as the data here will be wiped and replaced by the data from the source instance__ This destination instance must exist with a database. It can't be blank.
{img type="fileId" fileId="1354" thumb="mouseover"}
If you have data files that are not stored in the database, you should use the ((File Storage|#Ideal_scenario|ideal scenario for data file storage and relative paths)).
2018-05-02: An issue was noticed with make clone of branch 12.x to 15.x So workaround is to upgrade to 15.x.
!!! make clone and upgrade
Like above but with an extra upgrade operation
{CODE(caption="To manually add an upgrade to the clone operation")}
[root@trim]# php scripts/clone.php upgrade 1 3 branches/15.x
Creating snapshot of: mern.suite.wiki
Downloading files locally...
Obtaining database dump...
Creating archive...
...
...
{CODE}
2018-05-02: An issue was noticed with make clone of branch 12.x to 15.x So workaround is to upgrade to 15.x.
!!!! Automating make clone and upgrade
{CODE(caption="An example of a trim-clone.sh file (which you can put on a cron job) to clone and test upgrade to a few other versions")}
#!/bin/bash
PHP="/usr/bin/php -d memory_limit=256M"
cd /root/trim || exit 1
$PHP /root/trim/scripts/clone.php clone 1 2 >> /var/log/trim-clone.log 2>&1
$PHP /root/trim/scripts/clone.php upgrade 1 3 tags/15.4 >> /var/log/trim-cloneupgrade-15x.log 2>&1
$PHP /root/trim/scripts/clone.php upgrade 1 4 tags/16.2 >> /var/log/trim-cloneupgrade-16x.log 2>&1
$PHP /root/trim/scripts/clone.php upgrade 1 5 trunk >> /var/log/trim-cloneupgrade-trunk.log 2>&1
exit 0
{CODE}
__Explanation: __
{CODE()}$PHP /root/trim/scripts/clone.php upgrade 1 3 branches/15.x >> /var/log/trim-cloneupgrade-15x.log 2>&1
{CODE}
This clones instance #1 to #3, and upgrades to 15.x and sends all info to log file
!! Requirements
* Tiki ((Check)) can be used to dynamically check the requirements on your server
* __PHP 5.6__ (or newer) with SQLite extension (on master server) (uses SQLite 2.1 format)
** --As of 2016-04-15, PHP 5.3 seems to be sufficient for all operations (but of course, you need PHP 5.5 to install Tiki 13.x on remote server)-- On PHP 5.5 you will get a parse error: -+PHP Parse error: syntax error, unexpected '.', expecting ',' or ';' in /var/local/trim/src/rclib.php on line 30+- because of the DIRECTORY_SEPARATOR constant usage
* PHP Command Line Interface (PHP-CLI) on both master and remote servers
* Shell (SSH) access on master and remote (remote servers with only FTP will work but with fewer features and slower management)
* Both master and instances can be shared hosting as long as you have SSH access. It is recommended that the master be installed on a very safe machine because it has SSH access keys to all your instances.
* Rewrite rules (on master server) are necessary for TRIM Web Administration
!!! Shell commands/scripts
* make
* subversion (svn) (1.10, 2.x and 3.x sites use svn)
* php5-cli (This is the command line version of PHP, which is not always installed)
* sqlite (This is the database where TRIM will store data about the various instances)
* php5-sqlite (extension)
* php5-pspell (extension)
* php5 (for webtrim)
* cron (optional, to automate unattended tasks)
* rsync (for backups)
* sendmail (to send you email notifications of changes to your files)
* bzip2 (for backups)
!!!! To restore backups on TRIM (for testing)
* ssh
* phpmysql
!!!! ext/ssh2
^
__ext/ssh2__ is not mandatory, but highly recommended to speed up TRIM. It will also avoid many weird error messages and it is much stabler. Without it, it ''should'' work but some things may not.
To install it under Debian or Ubuntu:
{CODE()}
# sudo apt-get install libssh2-php
{CODE}
If that didn't work:
{CODE()}
# sudo apt-get install php5-dev libssh2-1-dev php-pear
{CODE}
{CODE(caption="see http://pecl.php.net/ssh2 for latest version number")}
# sudo pear install pecl/ssh2-0.11.0
{CODE}
If you get questions, just press "Enter" to accept the defaults.
{CODE(caption="add to the end or uncomment ''extension=ssh2.so'' in php.ini")}
# sudo nano /etc/php5/cli/php.ini
{CODE}
{CODE(caption="to test if it works, look for ssh2")}
# php -m
{CODE}
Related
http://kevin.vanzonneveld.net/techblog/article/make_ssh_connections_with_php/
^
!!! Linux
To install all of these under Debian or Ubuntu
{CODE()}
sudo apt-get install make subversion php5-cli sqlite php5-sqlite php5-pspell php5 cron rsync bzip2
{CODE}
If you don't have postfix or sendmail installed already, then you can install at least sendmail. But beware that if you have postfix installed and configured for a custom setup in your server, and you run the command to install sendmail, you will have your postfix automatically removed from your server before sendmail is installed :
{CODE()}
sudo apt-get install sendmail
{CODE}
!!! Windows
There are no current plans for TRIM (the master) to work natively on Windows (but you are welcome to join the community and make this happen!) but it works on Cygwin. So you should use something one of options below. The remote host can be a Windows instance however as TRIM supports FTP. Related: ((Platform independence))
!!!! TRIM on Cygwin
* ((TRIM on Cygwin))
!!!! Ubuntu on Windows
# Download and install Ubuntu on Windows solution
# Follow instructions for Ubuntu (apt-get install...)
!!!!! Options
* [http://www.my-guides.net/en/content/view/112/33/|How to install Ubuntu Linux on Windows using VirtualBox]
* [http://www.andlinux.org/|andLinux] is a complete Ubuntu Linux system running seamlessly in Windows 2000 based systems (2000, XP, 2003, Vista; 32-bit versions only).
* [http://sourceforge.net/projects/portableubuntu/|PortableUbuntu]
!!! Mac
On a Mac, you'll need something like [http://www.macports.org/|MacPorts]. The [http://www.macports.org/ports.php|package names may vary a bit].
{CODE()}
sudo port install make
sudo port install php5-sqlite3
...
{CODE}
!! How to download and use
You may first want to use Tiki ((Check)) to see if you server has all the requirements.
!!!# Get TRIM via SVN (to the current directory)
On a [http://wikisuite.org/ClearOS|ClearOS] server, the recommended location is -+/var/local/trim+-.
For other servers, this should be in a non-web accessible directory.
{CODE()}
svn co https://svn.code.sf.net/p/tikiwiki/code/trim/ .
{CODE}
If you got TRIM before 2012-12-12, you need to:
{CODE(caption="relocate TRIM" colors="shell")}svn switch --relocate https://tikiwiki.svn.sourceforge.net/svnroot/tikiwiki/trim https://svn.code.sf.net/p/tikiwiki/code/trim/ .
{CODE}
{CODE(caption="update to latest version of TRIM" colors="shell")}
svn up
{CODE}
!!!# Make an instance
The first time TRIM is ran, it will complete the setup (e.g. run ((Composer)), set keys, etc.)
{CODE()}
make instance
{CODE}
(or just type ''make'' and you will get choices available like ''make check'', ''make update'', etc)
__If TRIM is not detecting php5-cli (It's important to have at least php-cli version 5.6)__
To test:
{CODE()}
$ php -v
{CODE}
This should give you the PHP version of your default command line version. Ex.:
{CODE()}
PHP 4.4.9 (cli) (built: Sep 17 2008 12:02:18)
Copyright (c) 1997-2008 The PHP Group
Zend Engine v1.3.0, Copyright (c) 1998-2004 Zend Technologies
with Zend Extension Manager v1.2.2, Copyright (c) 2003-2007, by Zend Technologies
with Zend Optimizer v3.3.3, Copyright (c) 1998-2007, by Zend Technologies
{CODE}
If you have more PHP versions available but the default one is lesser than PHP 5.6:
In this case you should manually edit the file -+Makefile+-
Replace:
{CODE()}
PHP = php -d memory_limit=256M
{CODE}
with something like:
{CODE()}
PHP = php56 -d memory_limit=256M
{CODE}
or:
{CODE()}
PHP = /usr/local/php5/bin/php -d memory_limit=256M
{CODE}
The 'which php' command does provide you with the info. If not, ask your host.
||Question|note
Instance name | Just a pretty name for the Tiki instance
Contact email | Email of person in charge of this Tiki instance
Web root | full path, no trailing slash (/)
Web URL | full URL, with http
Working directory | You can use /tmp or create/use a directory on your remote server. Full path, no trailing slash (/)
SSH host name | domain to connect by SSH
SSH port | if you are not using the usual port 22
SSH user | username to connect by SSH||
When __make instance__ reports:
Branch tags/1.10.0 was found. If this is not correct, enter the one to use :
type: __branches/1.10__
!! Example
{CODE(caption=example)}
[avantech]$ make instance
/usr/local/php5/bin/php -d memory_limit=256M scripts/addinstance.php
Default editor used (vim). You can change the EDITOR environment variable.
Default diff used (diff). You can change the DIFF environment variable.
Instance name : RTO test
Contact email : <hidden>
Web root : /home/rto/public_html
Web URL : http://rto.ourwiki.net
Working directory : /home/rto/temptrim
Instance information saved.
SSH host name : rto.ourwiki.net
SSH user : rto
The authenticity of host 'rto.ourwiki.net (<snip>)' can't be established.
RSA key fingerprint is <snip>.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'rto.ourwiki.net,<snip>' (RSA) to the list of known hosts.
rto@rto.ourwiki.net's password:
Testing connection...
Were you asked for a password? [yes|no] no
warning: locate: warning: database /var/lib/slocate/slocate.db' is more than 8 days old
Branch tags/1.10.0 was found. If this is not correct, enter the one to use : branches/1.10
{CODE}
!! To restore backups on TRIM machines
We want to test that the backups are indeed working so we need a fully functioning php-mysql environment. By installing phpMyAdmin, we should get this. We also want SSH because TRIM connects via SSH to restore these backups, the same as it would connect anywhere.
To install necessary software under Debian or Ubuntu:
{CODE(caption="Install what is needed to run php-mysql apps and for TRIM to connect")}
# sudo apt-get install phpmyadmin ssh
{CODE}
After the install, visit http://localhost/phpmyadmin/
!! An example of creating an instance (where to restore to)
{CODE()}
trim@trim:~/avantech-trim$ make instance
php5 -d memory_limit=256M scripts/addinstance.php
Default editor used (vim). You can change the EDITOR environment variable.
Default diff used (diff). You can change the DIFF environment variable.
Connection type [ssh|ftp] : ssh
Host name : localhost
User : trim
Instance name : [localhost]
Contact email :
Web root : [/home/trim/public_html] /var/www/restoredfromtrim/
Web URL : [http://localhost] http://localhost/restoredfromtrim
Working directory : [/home/trim/trim_temp] /tmp
Instance information saved.
The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is <snip>.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
trim@localhost's password:
Testing connection...
After successfully entering your password, were you asked for a password again? [yes|no] no
Detecting remote configuration.
No applications were found on remote host.
Which one do you want to install? (none to skip)
[0] tikiwiki
>>>
No instance to install.
trim@trim:~/avantech-trim$
{CODE}
-=Related=-
* ((System Configuration))
* ((Multitiki))
* http://wikisuite.org/How-to-install-Tiki-with-TRIM
* http://wikisuite.org/TRIM-and-Syncthing-for-automated-offsite-backups
-=Alias=-
* (alias(Tiki Remote Instance Manager))
* (alias(TikiWiki Remote Instance Manager))