TikiWiki
Remote
Instance
Manager
What
- A combination of shell and PHP scripts to install, update, backup and monitor (check security) a large number of TikiWiki installations (instances).
- TRIM facilitates/automates the operations you would normally do by logging into each server.
- A great tool if you want to setup a WikiFarm
- It is not a script to update your Linux or MySQL installation (but it may one day do some server checks to see if your Apache, PHP, Mysql, gd, etc versions are OK).
Table of contents
Current state
State as of May 19th, 2008:
- It is not for production servers. Use at your own risk.
- It is somewhat usable in "ideal" conditions (ideal PHP configuration, certain tools must be installed on the server), but not yet robust to deal with various server configurations.
- It is not yet fully documented. It is a wizard-like app, so if you are familiar with shell scripts and installing/upgrading TikiWiki, you should be OK.
- However, it was demonstrated to work on different hosts, so somewhat usable.
- In active development. Make sure to update regularly and report issues / improve the documentation.
What soon/maybe
- It is designed to be modular and could be ported to other Web applications in the future.
- At first, it will require SSH access but it is designed so it could eventually work for FTP as well.
How it works
TRIM is installed on a
master server. TRIM then connects by SSH to various servers and, via a wizard (you answer the questions interactively), permits you manage various
instances of TikiWiki. You can install, upgrade, backup, check the file integrity and do various other things.
If you want to share logins, groups and preferences throughout many TikiWiki sites, you can use
InterTiki.
Alternatives
If you have a large number of Tiki installations, you could use
MultiTiki where all files (php/tpl/etc) are shared amongst several domains (but you can have .css and .tpl which override for your domain) but each domain has a distinct DB. TRIM is different where each domain is totally independent. Distinct database
and distinct files.
Advantages of TRIM over MultiTiki
- With TRIM, each instance can be on totally different servers or on various directories on the same server, so you can easily scale to as few or as many servers as you want.
- Once an instance is in a MultiTiki, it's a little tricky to extract it. For example, it's easier to provide a backup of just one instance, using TRIM.
- If you have multiple domains and need to provide distinct access for each (emails, ftp accounts, access to logs, etc., it can be easier to have distinct server space.
- With MultiTiki, if you have a security issue, all your Tikis are potentially compromised. (even for the Tikis that have not enabled the unsafe feature)
- With TRIM, you can pick & choose which Tikis to upgrade. With MultiTiki, all the instances get the upgrade. Even if they don't need the bug fixes or enhancements, they are still at risk of being affected by a regression. So all instances need to be checked after each upgrade.
- With TRIM, you can modify the templates/*.tpl files and still use the merging features of cvs/svn to take advantage of the enhancements to the main code base (There could be conflicts to manually resolve during upgrades though). Using multiTiki, you must put all changes in a separate directory and can't easily benefit from improvements to the main code base.
- TRIM handles offsite backups
Advantages of MultiTiki over TRIM
- Uses less disk space
- Can upgrade hundreds of TikiWikis in one operation
- MultiTiki has been developed and refined for years and is used in production, whereas TRIM is new and still in heavy development
- TRIM requires SSH access (for now anyway), whereas maybe MultiTiki can be run just with FTP. Depending on your server, you may have a lot of permissions to change by FTP though.
- TRIM requires more things on the server (sqlite, svn, etc)
- MultiTiki has a web-based interface
Who is behind this?
- Louis-Philippe Huberdeau
- Marc Laporte
- Louis-Martin Richard (tester)
- You? (We are looking for Beta-testers!!)
Requirements
- PHP 5.x with SQLite extension (on master server) (uses SQLite 2.1 format)
- PHP 4.x should be OK for remote servers.
- PHP Command Line Interface (PHP-CLI) on both master and remote servers
- Shell (SSH) access on master and remote (eventually, it should work on remote servers with only FTP)
- Both master and instances can be shared hosting as long as you have SSH access. It is recommended that the master be installed on a very safe machine because it has SSH access keys to all your instances.
- Rewrite rules (on master server) are necessary for TRIM Web Administration
Shell commands/scripts
- make
- subversion (svn) (1.10 sites use svn)
- cvs (assuming you'll be maintaining some 1.9x sites via CVS)
- php5-cli
- sqlite
- php5-sqlite (extension)
- php5 (for webtrim)
- cron (optional)
- rsync (for backups)
ext/ssh2 is not mandatory, but highly recommended to speed up TRIM.
To install it under ubuntu:
- sudo apt-get install php5-dev libssh2-1-dev
- Get and extract http://pecl.php.net/get/ssh2-0.10.tgz
- In the directory, modify ssh2.c to remove the precompiler condition around line 481 (only keep the code in the ELSE clause) and run:
- phpize
- autoconf
- ./configure --enable-shared --prefix=/usr
- make
- sudo make install
- Edit /etc/php5/cli/php5.ini to load extension ssh2.so
How to download and use
1- Get TRIM via SVN
svn co https://tikiwiki.svn.sourceforge.net/svnroot/tikiwiki/trim trim
2- Move to trim directory
cd trim
3- Make an instance
make instance
(or just type
make and you will get choices like
make check,
make update, etc)
| Question | note
|
| Instance name | Just a pretty name for the TikiWiki instance
|
| Contact email | Email of person in charge of this TikiWiki instance
|
| Web root | full path, no trailing slash (/)
|
| Web URL | full URL, with http
|
| Working directory | You can use /tmp or create/use a directory on your remote server. Full path, no trailing slash (/)
|
| SSH host name | domain to connect by SSH
|
| SSH user | username to connect by SSH |
When
make instance reports:
Branch tags/1.10.0 was found. If this is not correct, enter the one to use :
type:
branches/1.10
Example
example
[avantech]$ make instance
/usr/local/php5/bin/php -d memory_limit=256M scripts/addinstance.php
Default editor used (vim). You can change the EDITOR environment variable.
Default diff used (diff). You can change the DIFF environment variable.
Instance name : RTO test
Contact email : <hidden>
Web root : /home/rto/public_html
Web URL : http://rto.ourwiki.net
Working directory : /home/rto/temptrim
Instance information saved.
SSH host name : rto.ourwiki.net
SSH user : rto
The authenticity of host 'rto.ourwiki.net (<snip>)' can't be established.
RSA key fingerprint is <snip>.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'rto.ourwiki.net,<snip>' (RSA) to the list of known hosts.
rto@rto.ourwiki.net's password:
Testing connection...
Were you asked for a password? [yes|no] no
warning: locate: warning: database /var/lib/slocate/slocate.db' is more than 8 days old
Branch tags/1.10.0 was found. If this is not correct, enter the one to use : branches/1.10
License
LGPL, like the rest of TikiWiki
Documentation
Here is a brief explanation of the basic commands that can be used. All commands follow a wizard pattern.
make instance
Adds an instance to be managed by TRIM. Trim attempts to perform various operations from this point:
- Share an RSA key to connect to the remote host in the future
- Make a few verifications on remote host
- Detect the application installed remotely and the version
- If nothing is installed, propose to install something
You will be prompted to enter a pass phrase. If you enter one, you will be prompted every time you use TRIM. Unattended backups and security checks will not be possible.
The installation process will ask to create a database and user for tikiwiki. To do this, root access on the MySQL database (or user with similar privileges) is required. Otherwise, database settings can be entered manually.
make check
This is somewhat equivalent to a
secdb check, except that it's safer because it's made from a trusted host and will take your custom modifications on host into consideration. Upon first run, the check will ask where it should fetch the hashes from.
- source : obtain the hashes from the official repository. After setting up the expected hashes, a check will be performed.
- current : use the files currently online as the source of hashes. If this is a recent install, this may be safe. However, if the host was already compromised, hacks will be undetected.
- skip : Move on for now
When files are altered, added or removed, TRIM will prompt with various solutions.
- For modified files, it will allow to view the changes that were made and propose to accept or reject them. Rejecting changes will fetch a copy of the file from the repository.
- For added files, it will be possible view the file and then accept or reject it. Rejecting the file will remove it from the remote host.
- Removed files will allow to accept or reject the removal. Rejection will fetch a copy from the repository.
make update
Allows to update a CVS/SVN instance (cvs up/svn up) and update the file hashes accordingly. It also performs the database update. The hash verification/update may prompt with some files containing conflicts if modifications were made on the instance.
During the update process, the instance is disabled using a .htaccess file (previous one is preserved), making the site unavailable until the update is completed.
make upgrade
Similar to update. Requests for the branch to switch to over the update.
make convert
Converts and instance from CVS BRANCH-1-9 to SVN branches/2.0.
- All script customizations will be lost
- Database, img/wiki and img/wiki_up are preserved
make fix
Run setup.sh on the remote host using automated parameters. It should work in most cases. If the command proposed my setup.sh without parameters or super user rights are required, you should connect to the remote host manually using `make access`.
make access
Opens a shell to the remote host. Because TRIM manages all your connections, using this command simply avoids to remember passwords.
make watch
Set-up a cron job on the TRIM master to perform the Hash check automatically every day. The script will prompt for a contact email address to notify with the log in the case of an intrusion and the time at which the script should run.
make backup
Performs a complete backup of the Tikiwiki instance. The backup includes all files on remote host (including user files and files stored out of the web root) and a dump of the database. Each backup is archived in the TRIM_ROOT/backup/archive folder as a .tar.bz2 file. The backup file contains:
- manifest.txt : Indicates where folders were located on the remote host
- database_dump.sql : Self-explanatory
- a-f0-9{32} : Folders named using a hash. Content of the folders on remote host. The manifest.txt file lists these hashes.
Archives are cleaned up after each run. The rules are the following:
- Backups made on the first of any month are kept forever.
- Backups made on Sunday are kept for 31 days.
- Daily backups are kept for a week.
At this time, the backup script must be called manually. However, it would not be complicated to make an automated version to run on a cron job.
make enablewww
Activate the
TRIM Web Administration. The script asks a few questions and changes some permissions.
Rewrite rules are necessary for TRIM Web Administration
- On a standard install of Ubuntu, when trying to use WebTrim. Presumably because Rewrite Rules are not OK
Not Found
The requested URL /webtrim/view/1 was not found on this server.
Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.1 with Suhosin-Patch Server at localhost Port 80
Apache must load the mod_rewrite module and allow .htaccess files.
Configuration
LoadModule rewrite_module /usr/lib/apache2/modules/mod_rewrite.so
AccessFileName .htaccess
On Ubuntu (different per distribution)
sudo nano /etc/apache2/sites-enabled/000-default
change to "AllowOverride All" in <Directory /var/www/>
make viewdb
For debug purposes.
make detect
For debug purposes.
To do
Known Bugs
-
If the SVN version on TRIM remote instance is lower than TRIM master, installs from TRIM will not be updatable. When you try to use SVN on the instance, you'll get "svn: This client is too old to work with working copy ''; please get a newer Subversion client"
- templates_c (Smarty cache folder) is problematic
- On first backup, I get this error message "rm: cannot remove `/home/user/trim/trim/backup/1/manifest.txt': No such file or directory" (note: this has no side effects, warning only.)
- Perform database setup... -> use stars for 2 password fields
On update:
1- Need space between profile name and URL
2- 2 error messages
-
Subcommand 'info' doesn't accept option '--xml'
- ssh: profiles.tikiwiki.org: Name or service not known — THis is a network error... no can do
user@ubuntu804desktop:~/trim/trim$ make update
php5 -d memory_limit=256M scripts/update.php
Default editor used (vim). You can change the EDITOR environment variable.
Default diff used (diff). You can change the DIFF environment variable.
Note: Only CVS and SVN instances can be updated.
Which instances do you want to update?
[0] profiles.tikiwiki.orghttp://profiles.tikiwiki.org lphuberdeau@tikiwiki.org
>>> 0
Subcommand 'info' doesn't accept option '--xml'
Type 'svn help info' for usage.
ssh: profiles.tikiwiki.org: Name or service not known
ERROR 1050 (42S01) at line 27: Table 'tiki_events' already exists
ERROR 1060 (42S21) at line 38: Duplicate column name 'file'
<snip>
ERROR 1062 (23000) at line 1781: Duplicate entry 'tiki_p_view_actionlog_owngroups' for key 1
Deleted files were found on remote host :
[0] ./installer/tiki-installer.php
What do you want to do about it?
(R)estore version in repository
(D)elete hash to accept file removal
(S)kip
>>> D 0
-- ./installer/tiki-installer.php
Roadmap
Phase 1
Disable site during upgrade (via .htaccess) Done
Update DB (1.x to 1.y sql script) Done
Handle both 1.9 and 1.10
Cron job for automatic check and report on file integrity (what about cache files?) Done
Something to indicate TRIM is working (sometimes, it is busy for many minutes and we are just left to wait, not knowing if it's stalled) ... may not be possible
Ok. Then, please indicate (in general) what TRIM is doing. Something like "TRIM will now transfer all files from SourceForge to TRIM (master) and then upload them to your instance. As long as you are not getting an error message, TRIM is hard at work! This operation could take 10-20 minutes or even a few hours, depending on server performance and network speed." Done
run fixperms.sh Done
Be more robust to various paths for php5 (ex.: using "which php" to find it) Done (using variables in Makefile so modification only required once)
Phase 2
For TRIM to handle backups Done
Make instance should suggest some values (ask for SSH host name and SSH user first). If you just type "Enter", it picks this value. These defaults should be editable somewhere in a config file. It also serves as a good example for the proper format. The default format which works for Fantastico hosting:
- Web root : /home/$sshuser/public_html
- Web URL : http://$sshhost
- Working directory : /home/$sshuser/trim_working (TRIM should create with appropriate rights)
Done
Admin interface to edit and delete instances (maybe a ssl protected web interface?) or here
To create database, database user and permissions from this user when installing Tiki. Could be in tiki-install.php as well
- TRIM will need to be indicated the database login with sufficient rights to do this
- Use this new info to populate db/local.php
- Maybe pick profile from TRIM? (equivalent to tiki-install.php?)
Done
Nicer error message (from .htaccess when site is being updated) Done
Backup
- Cron job
- Option to backup things outside of TikiWiki (ex.: mail directory, access logs, etc)
- To be able to handle symlinks
- /home/user/www is a symlink of /home/user/public_html and if I indicate /home/user/www as the web root of the instance, TRIM backup will not backup files) bug or feature enhancement request?
When suspicious files are detected, in addition to deleting them, it should be possible to send them to a non Web accessible "quarantine" for further analysis or even optionally send to the securitysquad.
bundled _htaccess
Rename the bundled _htaccess to .htaccess for instances running on Apache
- Could enhancements to _htaccess be merged to .htaccess? (ex.: rename .htaccess to _htaccess just before svn/cvs up)
major upgrades
Easier to manage in shell interface
- document so people know to type things like C 0-52
- "Next page" when too much info in one screen (sometimes hundreds of lines of scrollback), but maybe this is a bad idea because it'll prevent unattended operations. hmmm
- Add a note before entering a passphrase in addinstance.php : "If you enter a passphrase, you will need to enter it every time you run TRIM, and thus, automatic, unattended operations (like backups, file integrity checks, etc.) will not be possible."
- If mysqladmin fails (ex.: bad login, not mysql but postgres, etc), it should offer to re-enter value or to abort database creation. In this last case, the system should indicate something like : "TRIM was unable to create your database with the information you provided. You should create it via your admin panel (cPanel, etc) and provide the information when you point your browser to tiki-install.php"
Indicate which warnings and error messages not to worry about (could be a link to a wiki page)
- time stamp 2008-06-02 22:33:48 is 3938 s in the future
- errors on setup.sh (when insufficient perms)
- "warning: locate: warning: database /var/lib/slocate/slocate.db' is more than 8 days old"
- ssh: profiles.tikiwiki.org: Name or service not known
Phase 3
- Once everything is nice and stable with SSH, add FTP support.
- In addition to check and backup, having an automatic update script option (useful to always keep test sites like 110.tikiwiki.org up to date. Unattended updates won't be able to deal with SVN conflicts.
- Possibly automated running of testing feature.
- Check used/available disk space. (% of used space) (Tiki misbehaves when disk is full and often we forget to check for that)
- Report if less than x Megs left or x% (email warning)
- Check on master and instances.
- Add a check for domain name expiry
- This is useful because the person hosting the TikiWiki may not be listed in the DNS registries and won't be informed of DNS changes (expiry, domain theft, etc)
- Some Tikis can have several domain names (domain parking)
- Will this be easy provided the anti-spammer protections?
- Warn by email (both to TRIM manager and instance contact)
- 30 days before expiry
- Ideally, on any change to the entry (for domain theft)
- See script by Jean-François Pelletier, started at Codefest
Phase 4
- Uptime/performance/content testing
- Some of this, like spell checking and link testing, may be better done as a Tiki feature instead, so as to be able to access all content (web crawler will have problem with perms), and be usable without TRIM, but be "triggerable" via TRIM.
- Check domain for presence of Google Analytics code
- Check to see if Tiki is running properly (check presence of keyword like http://host-tracker.com/ and/or changes like http://www.changedetection.com/log/org/americana/index_log.html#changelist)
- Check for the presence of a customized 404 apache error page (we often to forget to set this up)
- Check response time
- Check for broken links
- Check spelling
- Check missing images
- Check for missing plugins (especially for non-bundled plugins which cause error when migrating text from site to another)
- Check for standards compliance
Phase 5
- Backup
- Result could be sent to a UsbTiki format so Tiki admins could have backup in their pocket at all times.
- Save to Amazon S3 and similar online storage services
- Backup of the database should handle multitiki like doc/devtools/backupdb.sh
- Investigate if scripts like automysqlbackup, rsync-incr or rsnapshot have any concepts, features or code that would be useful for us.
- Investigate if we should make additional use of .htaccess in various directories (only useful for Apache)
- Profiles
- Investigate if/how TRIM could help maintaining: OpenSourceCMS type demo to test/develop and show off profiles
- Investigate any synergy with the profile manager to apply database change to one, many or all instances.
- Turn off unsafe, buggy or unused features
- Add watch all wiki pages or watch all trackers to multiple Tikis
- Use existing TikiWiki functions for mass operations. Ex.: add a TikiWiki user on dozens of TikiWikis, etc (In this case, it's important to use existing TikiWiki functions rather than alter the database)
Phase 6
- Test/document on Cygwin so users could manage remote TikiWikis via their Windows PC or Mac. Or a VMware appliance
- Initially built for Linux-Apache-MySQL-PHP (LAMP). Later, we will adjust TRIM to work with different setups.
- Investigate a TRIM-like application that works directly on the remote instance. A small php script which fetches the latest version of Tiki and installs it. (Could be impossible because of limited file permissions that php scripts, with Apache rights, that have limited permissions). mods have this issue.
- import mysql data a bit like BigDump
- Move an instance from one server to another (maybe via a restore of the backup?)
Wiki
,
and
.
Last Modified Comments