Search tracker summary

TRIM

Cut down on your workload!!
Print

TikiWiki Remote Instance Manager

What

  • A combination of shell and PHP scripts to install, update, backup and monitor (check security) a large number of TikiWiki installations (instances).
  • TRIM facilitates/automates the operations you would normally do by logging into each server.
  • A great tool if you want to setup a WikiFarm
  • It is not a script to update your Linux or MySQL installation (but it may one day do some server checks to see if your Apache, PHP, Mysql, gd, etc versions are OK).

Table of contents



Current state

State as of May 19th, 2008:
  • It is not for production servers. Use at your own risk.
  • It is somewhat usable in "ideal" conditions (ideal PHP configuration, certain tools must be installed on the server), but not yet robust to deal with various server configurations.
  • It is not yet fully documented. It is a wizard-like app, so if you are familiar with shell scripts and installing/upgrading TikiWiki, you should be OK.
  • However, it was demonstrated to work on different hosts, so somewhat usable.
  • In active development. Make sure to update regularly and report issues / improve the documentation.


What soon/maybe

  • It is designed to be modular and could be ported to other Web applications in the future.
  • At first, it will require SSH access but it is designed so it could eventually work for FTP as well.

How it works

TRIM is installed on a master server. TRIM then connects by SSH to various servers and, via a wizard (you answer the questions interactively), permits you manage various instances of TikiWiki. You can install, upgrade, backup, check the file integrity and do various other things.

If you want to share logins, groups and preferences throughout many TikiWiki sites, you can use InterTiki.

Alternatives

If you have a large number of Tiki installations, you could use MultiTiki where all files (php/tpl/etc) are shared amongst several domains (but you can have .css and .tpl which override for your domain) but each domain has a distinct DB. TRIM is different where each domain is totally independent. Distinct database and distinct files.

Advantages of TRIM over MultiTiki

  1. With TRIM, each instance can be on totally different servers or on various directories on the same server, so you can easily scale to as few or as many servers as you want.
  2. Once an instance is in a MultiTiki, it's a little tricky to extract it. For example, it's easier to provide a backup of just one instance, using TRIM.
  3. If you have multiple domains and need to provide distinct access for each (emails, ftp accounts, access to logs, etc., it can be easier to have distinct server space.
  4. With MultiTiki, if you have a security issue, all your Tikis are potentially compromised. (even for the Tikis that have not enabled the unsafe feature)
  5. With TRIM, you can pick & choose which Tikis to upgrade. With MultiTiki, all the instances get the upgrade. Even if they don't need the bug fixes or enhancements, they are still at risk of being affected by a regression. So all instances need to be checked after each upgrade.
  6. With TRIM, you can modify the templates/*.tpl files and still use the merging features of cvs/svn to take advantage of the enhancements to the main code base (There could be conflicts to manually resolve during upgrades though). Using multiTiki, you must put all changes in a separate directory and can't easily benefit from improvements to the main code base.
  7. TRIM handles offsite backups

Advantages of MultiTiki over TRIM

  1. Uses less disk space
  2. Can upgrade hundreds of TikiWikis in one operation
  3. MultiTiki has been developed and refined for years and is used in production, whereas TRIM is new and still in heavy development
  4. TRIM requires SSH access (for now anyway), whereas maybe MultiTiki can be run just with FTP. Depending on your server, you may have a lot of permissions to change by FTP though.
  5. TRIM requires more things on the server (sqlite, svn, etc)
  6. MultiTiki has a web-based interface


Who is behind this?

  • Louis-Philippe Huberdeau
  • Marc Laporte
  • Louis-Martin Richard (tester)
  • You? (We are looking for Beta-testers!!)


Requirements

  • PHP 5.x with SQLite extension (on master server) (uses SQLite 2.1 format)
  • PHP 4.x should be OK for remote servers.
  • PHP Command Line Interface (PHP-CLI) on both master and remote servers
  • Shell (SSH) access on master and remote (eventually, it should work on remote servers with only FTP)
  • Both master and instances can be shared hosting as long as you have SSH access. It is recommended that the master be installed on a very safe machine because it has SSH access keys to all your instances.
  • Rewrite rules (on master server) are necessary for TRIM Web Administration

Shell commands/scripts

  • make
  • subversion (svn) (1.10 sites use svn)
  • cvs (assuming you'll be maintaining some 1.9x sites via CVS)
  • php5-cli
  • sqlite
  • php5-sqlite (extension)
  • php5 (for webtrim)
  • cron (optional)
  • rsync (for backups)


ext/ssh2 is not mandatory, but highly recommended to speed up TRIM.

To install it under ubuntu:
  1. sudo apt-get install php5-dev libssh2-1-dev
  2. Get and extract http://pecl.php.net/get/ssh2-0.10.tgz (external link)
  3. In the directory, modify ssh2.c to remove the precompiler condition around line 481 (only keep the code in the ELSE clause) and run:
    1. phpize
    2. autoconf
    3. ./configure --enable-shared --prefix=/usr
    4. make
    5. sudo make install
  4. Edit /etc/php5/cli/php5.ini to load extension ssh2.so

How to download and use

1- Get TRIM via SVN

svn co https://tikiwiki.svn.sourceforge.net/svnroot/tikiwiki/trim trim


2- Move to trim directory

cd trim


3- Make an instance

make instance

(or just type make and you will get choices like make check, make update, etc)

Questionnote
Instance name Just a pretty name for the TikiWiki instance
Contact email Email of person in charge of this TikiWiki instance
Web root full path, no trailing slash (/)
Web URL full URL, with http
Working directory You can use /tmp or create/use a directory on your remote server. Full path, no trailing slash (/)
SSH host name domain to connect by SSH
SSH user username to connect by SSH


When make instance reports:
Branch tags/1.10.0 was found. If this is not correct, enter the one to use :
type: branches/1.10


Example


example
[avantech]$ make instance
/usr/local/php5/bin/php -d memory_limit=256M scripts/addinstance.php
Default editor used (vim). You can change the EDITOR environment variable.
Default diff used (diff). You can change the DIFF environment variable.
Instance name : RTO test
Contact email : <hidden>
Web root : /home/rto/public_html
Web URL : http://rto.ourwiki.net
Working directory : /home/rto/temptrim
Instance information saved.
SSH host name : rto.ourwiki.net
SSH user : rto
The authenticity of host 'rto.ourwiki.net (<snip>)' can't be established.
RSA key fingerprint is <snip>.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'rto.ourwiki.net,<snip>' (RSA) to the list of known hosts.
rto@rto.ourwiki.net's password:
Testing connection...
Were you asked for a password? [yes|no] no
warning: locate: warning: database /var/lib/slocate/slocate.db' is more than 8 days old
Branch tags/1.10.0 was found. If this is not correct, enter the one to use : branches/1.10


License

LGPL, like the rest of TikiWiki


Documentation

Here is a brief explanation of the basic commands that can be used. All commands follow a wizard pattern.

make instance

Adds an instance to be managed by TRIM. Trim attempts to perform various operations from this point:
  1. Share an RSA key to connect to the remote host in the future
  2. Make a few verifications on remote host
  3. Detect the application installed remotely and the version
  4. If nothing is installed, propose to install something

You will be prompted to enter a pass phrase. If you enter one, you will be prompted every time you use TRIM. Unattended backups and security checks will not be possible.

The installation process will ask to create a database and user for tikiwiki. To do this, root access on the MySQL database (or user with similar privileges) is required. Otherwise, database settings can be entered manually.

make check

This is somewhat equivalent to a secdb (external link) check, except that it's safer because it's made from a trusted host and will take your custom modifications on host into consideration. Upon first run, the check will ask where it should fetch the hashes from.

  • source : obtain the hashes from the official repository. After setting up the expected hashes, a check will be performed.
  • current : use the files currently online as the source of hashes. If this is a recent install, this may be safe. However, if the host was already compromised, hacks will be undetected.
  • skip : Move on for now

When files are altered, added or removed, TRIM will prompt with various solutions.
  • For modified files, it will allow to view the changes that were made and propose to accept or reject them. Rejecting changes will fetch a copy of the file from the repository.
  • For added files, it will be possible view the file and then accept or reject it. Rejecting the file will remove it from the remote host.
  • Removed files will allow to accept or reject the removal. Rejection will fetch a copy from the repository.

make update

Allows to update a CVS/SVN instance (cvs up/svn up) and update the file hashes accordingly. It also performs the database update. The hash verification/update may prompt with some files containing conflicts if modifications were made on the instance.

During the update process, the instance is disabled using a .htaccess file (previous one is preserved), making the site unavailable until the update is completed.

make upgrade

Similar to update. Requests for the branch to switch to over the update.

make convert

Converts and instance from CVS BRANCH-1-9 to SVN branches/2.0.

  • All script customizations will be lost
  • Database, img/wiki and img/wiki_up are preserved

make fix

Run setup.sh on the remote host using automated parameters. It should work in most cases. If the command proposed my setup.sh without parameters or super user rights are required, you should connect to the remote host manually using `make access`.

make access

Opens a shell to the remote host. Because TRIM manages all your connections, using this command simply avoids to remember passwords.

make watch

Set-up a cron job on the TRIM master to perform the Hash check automatically every day. The script will prompt for a contact email address to notify with the log in the case of an intrusion and the time at which the script should run.

make backup

Performs a complete backup of the Tikiwiki instance. The backup includes all files on remote host (including user files and files stored out of the web root) and a dump of the database. Each backup is archived in the TRIM_ROOT/backup/archive folder as a .tar.bz2 file. The backup file contains:
  • manifest.txt : Indicates where folders were located on the remote host
  • database_dump.sql : Self-explanatory
  • a-f0-9{32} : Folders named using a hash. Content of the folders on remote host. The manifest.txt file lists these hashes.

Archives are cleaned up after each run. The rules are the following:
  • Backups made on the first of any month are kept forever.
  • Backups made on Sunday are kept for 31 days.
  • Daily backups are kept for a week.

At this time, the backup script must be called manually. However, it would not be complicated to make an automated version to run on a cron job.

make enablewww

Activate the TRIM Web Administration. The script asks a few questions and changes some permissions.

Rewrite rules are necessary for TRIM Web Administration

  • On a standard install of Ubuntu, when trying to use WebTrim. Presumably because Rewrite Rules are not OK

Not Found

The requested URL /webtrim/view/1 was not found on this server.
Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.1 with Suhosin-Patch Server at localhost Port 80



Apache must load the mod_rewrite module and allow .htaccess files.
Configuration
LoadModule rewrite_module /usr/lib/apache2/modules/mod_rewrite.so
AccessFileName .htaccess



On Ubuntu (different per distribution)
sudo nano /etc/apache2/sites-enabled/000-default

change to "AllowOverride All" in <Directory /var/www/>

make viewdb

For debug purposes.

make detect

For debug purposes.


To do

Known Bugs

  • If the SVN version on TRIM remote instance is lower than TRIM master, installs from TRIM will not be updatable. When you try to use SVN on the instance, you'll get "svn: This client is too old to work with working copy ''; please get a newer Subversion client"
  • templates_c (Smarty cache folder) is problematic
  • On first backup, I get this error message "rm: cannot remove `/home/user/trim/trim/backup/1/manifest.txt': No such file or directory" (note: this has no side effects, warning only.)
  • Perform database setup... -> use stars for 2 password fields


On update:
1- Need space between profile name and URL
2- 2 error messages
    • Subcommand 'info' doesn't accept option '--xml'
    • ssh: profiles.tikiwiki.org: Name or service not known — THis is a network error... no can do

user@ubuntu804desktop:~/trim/trim$ make update
php5 -d memory_limit=256M scripts/update.php
Default editor used (vim). You can change the EDITOR environment variable.
Default diff used (diff). You can change the DIFF environment variable.
Note: Only CVS and SVN instances can be updated.

Which instances do you want to update?
[0] profiles.tikiwiki.orghttp://profiles.tikiwiki.org  lphuberdeau@tikiwiki.org
>>> 0
Subcommand 'info' doesn't accept option '--xml'
Type 'svn help info' for usage.
ssh: profiles.tikiwiki.org: Name or service not known
ERROR 1050 (42S01) at line 27: Table 'tiki_events' already exists
ERROR 1060 (42S21) at line 38: Duplicate column name 'file'
<snip>
ERROR 1062 (23000) at line 1781: Duplicate entry 'tiki_p_view_actionlog_owngroups' for key 1
Deleted files were found on remote host :
	[0] ./installer/tiki-installer.php


	What do you want to do about it? 
	(R)estore version in repository
	(D)elete hash to accept file removal
	(S)kip
>>> D 0
-- ./installer/tiki-installer.php




Roadmap

Phase 1
  • Disable site during upgrade (via .htaccess) Done
  • Update DB (1.x to 1.y sql script) Done
  • Handle both 1.9 and 1.10
  • Cron job for automatic check and report on file integrity (what about cache files?) Done
  • Something to indicate TRIM is working (sometimes, it is busy for many minutes and we are just left to wait, not knowing if it's stalled) ... may not be possible
    • Ok. Then, please indicate (in general) what TRIM is doing. Something like "TRIM will now transfer all files from SourceForge to TRIM (master) and then upload them to your instance. As long as you are not getting an error message, TRIM is hard at work! This operation could take 10-20 minutes or even a few hours, depending on server performance and network speed." Done
  • run fixperms.sh Done
  • Be more robust to various paths for php5 (ex.: using "which php" to find it) Done (using variables in Makefile so modification only required once)

Phase 2
  • For TRIM to handle backups Done
  • Make instance should suggest some values (ask for SSH host name and SSH user first). If you just type "Enter", it picks this value. These defaults should be editable somewhere in a config file. It also serves as a good example for the proper format. The default format which works for Fantastico hosting:
    • Web root : /home/$sshuser/public_html
    • Web URL : http://$sshhost (external link)
    • Working directory : /home/$sshuser/trim_working (TRIM should create with appropriate rights) Done
  • Admin interface to edit and delete instances (maybe a ssl protected web interface (external link)?) or here (external link)
  • To create database, database user and permissions from this user when installing Tiki. Could be in tiki-install.php as well
    • TRIM will need to be indicated the database login with sufficient rights to do this
    • Use this new info to populate db/local.php
    • Maybe pick profile from TRIM? (equivalent to tiki-install.php?) Done
  • Nicer error message (from .htaccess when site is being updated) Done
  • Backup
    • Cron job
    • Option to backup things outside of TikiWiki (ex.: mail directory, access logs, etc)
    • To be able to handle symlinks
      • /home/user/www is a symlink of /home/user/public_html and if I indicate /home/user/www as the web root of the instance, TRIM backup will not backup files) bug or feature enhancement request?
  • When suspicious files are detected, in addition to deleting them, it should be possible to send them to a non Web accessible "quarantine" for further analysis or even optionally send to the securitysquad.
  • bundled _htaccess
    • Rename the bundled _htaccess to .htaccess for instances running on Apache
    • Could enhancements to _htaccess be merged to .htaccess? (ex.: rename .htaccess to _htaccess just before svn/cvs up)
  • major upgrades
  • Easier to manage in shell interface
    • document so people know to type things like C 0-52
    • "Next page" when too much info in one screen (sometimes hundreds of lines of scrollback), but maybe this is a bad idea because it'll prevent unattended operations. hmmm
    • Add a note before entering a passphrase in addinstance.php : "If you enter a passphrase, you will need to enter it every time you run TRIM, and thus, automatic, unattended operations (like backups, file integrity checks, etc.) will not be possible."
    • If mysqladmin fails (ex.: bad login, not mysql but postgres, etc), it should offer to re-enter value or to abort database creation. In this last case, the system should indicate something like : "TRIM was unable to create your database with the information you provided. You should create it via your admin panel (cPanel, etc) and provide the information when you point your browser to tiki-install.php"
  • Indicate which warnings and error messages not to worry about (could be a link to a wiki page)
    • time stamp 2008-06-02 22:33:48 is 3938 s in the future
    • errors on setup.sh (when insufficient perms)
    • "warning: locate: warning: database /var/lib/slocate/slocate.db' is more than 8 days old"
    • ssh: profiles.tikiwiki.org: Name or service not known


Phase 3
  • Once everything is nice and stable with SSH, add FTP support.
  • In addition to check and backup, having an automatic update script option (useful to always keep test sites like 110.tikiwiki.org up to date. Unattended updates won't be able to deal with SVN conflicts.
    • Possibly automated running of testing feature.
  • Check used/available disk space. (% of used space) (Tiki misbehaves when disk is full and often we forget to check for that)
    • Report if less than x Megs left or x% (email warning)
    • Check on master and instances.
  • Add a check for domain name expiry
    • This is useful because the person hosting the TikiWiki may not be listed in the DNS registries and won't be informed of DNS changes (expiry, domain theft, etc)
    • Some Tikis can have several domain names (domain parking)
    • Will this be easy provided the anti-spammer protections?
    • Warn by email (both to TRIM manager and instance contact)
      • 30 days before expiry
      • Ideally, on any change to the entry (for domain theft)
    • See script by Jean-François Pelletier, started at Codefest (external link)

Phase 4
  • Uptime/performance/content testing
    • Some of this, like spell checking and link testing, may be better done as a Tiki feature instead, so as to be able to access all content (web crawler will have problem with perms), and be usable without TRIM, but be "triggerable" via TRIM.
      • Check domain for presence of Google Analytics code
      • Check to see if Tiki is running properly (check presence of keyword like http://host-tracker.com/ (external link) and/or changes like http://www.changedetection.com/log/org/americana/index_log.html#changelist) (external link)
      • Check for the presence of a customized 404 apache error page (we often to forget to set this up)
      • Check response time
      • Check for broken links
      • Check spelling
      • Check missing images
      • Check for missing plugins (especially for non-bundled plugins which cause error when migrating text from site to another)
      • Check for standards compliance

Phase 5
  • Backup
    • Result could be sent to a UsbTiki format so Tiki admins could have backup in their pocket at all times.
    • Save to Amazon S3 (external link) and similar online storage services
    • Backup of the database should handle multitiki like doc/devtools/backupdb.sh
    • Investigate if scripts like automysqlbackup (external link), rsync-incr (external link) or rsnapshot (external link) have any concepts, features or code that would be useful for us.
    • Investigate if we should make additional use of .htaccess in various directories (only useful for Apache)
  • Profiles
    • Investigate if/how TRIM could help maintaining: OpenSourceCMS type demo to test/develop and show off profiles
    • Investigate any synergy with the profile manager to apply database change to one, many or all instances.
      • Turn off unsafe, buggy or unused features
      • Add watch all wiki pages or watch all trackers to multiple Tikis
      • Use existing TikiWiki functions for mass operations. Ex.: add a TikiWiki user on dozens of TikiWikis, etc (In this case, it's important to use existing TikiWiki functions rather than alter the database)

Phase 6
  • Test/document on Cygwin (external link) so users could manage remote TikiWikis via their Windows PC or Mac. Or a VMware appliance (external link)
  • Initially built for Linux-Apache-MySQL-PHP (LAMP). Later, we will adjust TRIM to work with different setups.
  • Investigate a TRIM-like application that works directly on the remote instance. A small php script which fetches the latest version of Tiki and installs it. (Could be impossible because of limited file permissions that php scripts, with Apache rights, that have limited permissions). mods have this issue.
  • import mysql data (external link) a bit like BigDump (external link)
  • Move an instance from one server to another (maybe via a restore of the backup?)




Contributors to this page: marclaporte13263 points  , lawiz17 points  and lphuberdeau404 points  .
Page last modified on Sunday 24 August, 2008 20:26:05 UTC by marclaporte13263 points .

To register

To have an account at this site, please register at Tikiwiki.org (external link), and then use that user name and password to log in here.

This site gets user information from Tikiwiki.org with the InterTiki feature.

keywords

The following is a list of tikiwiki keywords that should serve as hubs for navigation within the tiki development and should correspond to documentation keywords

Each feature in Tiki has a wiki page which regroups all the bugs, requests for enhancements, etc. It is somewhat a form of wiki-based project management. You can also express your interest in a feature by adding it to your profile. You can also try out the Dynamic filter.

Accessibility (WAI – 508)
Action log (Tiki 2.0)
Administration
Ajax (Tiki 2.0)
Articles & Submissions
Backup
Banner
Blog
Bookmark
Browser Compatibility
Cache
Calendar
Category
Charts
Chat
Comment
Communication Center
Compression (gzip)
Contact Address book
Contact us
Content template
Contribution (Tiki 2.0)
Cookie
Copyright
Custom Home (and Group Home Page)
Database independence
Database MySQL
Date and Time
Debugger Console
Directory (of hyperlinks)
Documentation link from Tiki to doc.tikiwiki.org (Help System)
Drawing
Dynamic Content
Dynamic Variable
Ephemeride
External Authentication
FAQ
Featured links
File Gallery
Forum
Friendship Network (Community)
Game
Gmap Google maps
Group
Help System
Hotword
HTML Page
i18n (Multilingual, l10n, Babelfish)
Image Gallery
Install
Integrator
Interaction
Inter-User Messages
InterTiki
JS Calendar
Karma
Live Support
Lost edit protection
Mail-in
Map with Mapserver
Menu
Meta Tag
Mini-Calendar
Missing features
Mobile Tiki and Voice Tiki
Mods
Module
Multimedia (Tiki 2.0)
MultiTiki
MyTiki
Newsletter
Newsreader
Notepad
OS independence (Non-Linux, Windows/IIS, Mac, BSD)
PDF
Permission
Poll
Profile Manager
Quicktags
Quiz
Rating
Registration
RSS
Score
Search Engine Friendly
Search
Security
Shoutbox
Site Identity
Slideshow
Smarty Template
Smiley
Spam protection (Anti-bot CATPCHA)
Spellcheck
Spreadsheet
Stats
Structure
Survey
System log
Tags (Tiki 2.0)
Task
Tell a Friend + Social Bookmarking (Tiki 2.0)
TikiTests (Tiki 2.0)
Theme
Trackers
Trackers (Mirror)
User Administration
User Files
User Menu
Watch
WebHelp
Webmail
Wiki 3D
Wiki Plugin extends basic syntax
Wiki Syntax text area, parser, etc
Wiki History, page rename, etc
Workflow (Galaxia Workflow engine)
WYSIWYCA
WYSIWYG (Tiki 2.0)
XMLRPC

Search Wiki PageName

Exact match

Online users

20 online users
RSS feed Wiki RSS feed Blogs RSS feed Calendars