Colplot

Introduction

Whether you instruct collectl to generate plot formatted output at the time of data collection or afterwards by playing back a raw file and converting it to a plottable one, you now have to decide how to generate your actual plots and there are actually several choices:

load the file into a tool that is capable of reading space-delimieted files, such as Excel, and use its native plotting capabilities to generate the plots
load the file into an rrd database (being sure to tell collectl to set the delimiter to a colon and the timestamp to UTC) and use rrdtool's plotting capabilities
use a plotting tool such as gnuplot or ploticus to manually generate your plots or perhaps write some scripts to automate running them
use colplot!

At its core, colplot is nothing more than a script that has intimate knowledge of collectl plot file and directory format. Based on the types of plots being requested and some basic formatting options, it will build the necessary gnuplot commands and automatically run them for you.

Colplot can be run either through a web-based interface or a command line. As this introduction is not intended to be a User's Guide, see the colplot man page to identify which command line switches map to which web-based options. Also note that to produce plots on your terminal in command mode you must be running on a system that supports X terminals or windows. No, that is not a typo. Colplot does run on PCs. Also note if you direct colplot to write your plots to a file X is not required.

It should also be noted that colplot should be able to run with any browser on any operating system as long as perl and gnuplot are also installed. However it has only been testing on linux and windows. Furthermore, installing the RPM has only been tested with the Apache web server and so there will probably be additional work required to get it installed, configured and running with other servers.

Using

When you browse to a server running colplot you will see the following:

By design colplot will generate its plot(s) against all files in the selected directory that match the selected timeframe and optional filename match strings. In the example below, which uses the sample plot files distributed with colplot, you can see the results of selecting only files with client in their name and choosing to plot only CPU and Network data:

When generating plots for multiple systems, colplot will actually format the output as if in a spreadsheet where each cell is a plot and the rows/columns are organized by system names and plot types,. You can change the organization of the cells with the Display drop-down box. In the following example, 2 systems have been selected by choosing system names that match client or server, setting the plot widths to something narrower, setting the x-axis time scale to 5 minutes and removing the legends for a more compact display:

To learn more about colplot read the man page, click on the Help button at the top of the display and don't forget to read the FAQ. You should also click on Help next to the Plot By Name(s) field to see a list of all the available plots. Don't forget to experiment with the sample files in colplot's default directory, noting you can later change the default location in colplot.conf.

Real-Time Display
One nifty feature of colplot is its ability to display graphics in real-time, which is controlled by 2 fields at the top of the display:

By checking Enable Refresh Every, colplot will redisplay all its plots at the associated interval specified by the number of seconds.
By checking the radio button to the right of the number of minutes on the next line, colplot will dynamically set the x-axis to those number of minutes ending at the current time.

The trick to make this all work is to have collectl write data in uncompressed plot format and continuously flush its buffers so the latest sample is always on disk (collectl -P -oz -F0 etc...). Now, when colplot displays its plots it will display the last n-minutes of data at the specified refresh interval. While you can set a pretty small interval such as 5 or 10 seconds, the redisplay of the graphics can be visually annoying and there is more processing involved. Intervals between 15 and 60 seconds tend to be easier on the eyes and CPU.

TIPS

having multiple systems write to the same nfs mounted directory, allows you to monitor them at the same time
if you use narrow plots without legends, you can fit 5 or more per row resulting in a very readable dense display

Installation

If you're installing on a linux system running Apache there shouldn't be anything special to do as colplot will do the right thing to allow someone to browse to http://systemname/colplot. There are a number of pieces of installation magic that makes all this work, some or all of which you may have to manually do if running in a different environment:

To allow one to simply specify colplot after the node name, an additional apache stanza was created that defines /var/www/html/colplot as a valid directory capable of excecuting cgi scripts (that's why you can specify colplot with no extension as part of the URL - it's treated as a directory name). The RPM also creates that directory.
The real magic comes from creating a soft link in that directory named index.cgi which points to colplot.pl. Now when the user specifies colplot in the URL the web-server's behavior is to treat it as a directory name and try to run the script in that directory with the name index.cgi.
Soft links are also placed in that directory, one pointing to colplot-help.html and a second pointing to the colplot FAQ so those buttons on the GUI will work

I did the following to make colplot run on IIS on Vista (after installing perl and gnuplot), noting I wasn't really interested in trying to get as fancy as I did with Apache. To run it with the following settings you will need to specify a URL of http://systemname/colplot.pl:

create a directory called \colplot and put all colplot related files into it which are:
- colplot.pl
- colplot.conf
- colplotlib.ph
- colplotlib.defs
- colplot-help.html
- FAQ-colplot.html
- plotfiles\*
create a new website in IIS. I called mine colplot and used port 80, which means I had to stop the default web site before I could start it. I could have probably put everything into the default directory but prefer to keep things separate.
edit colplot.conf to specify the path to gnuplot as well as set PlotDir to C:\colplot\plotfiles
If you want to be able to generate PDF files you're also going to have to install ghostscript and put its full path name into colplot.conf

It is not an objective to document how to get colplot running in other environments, especially since I don't even have access to them. Therefore it will have to become the responsibility of the community to send me instructions, which I'll be happy to post along with YOUR email addresses so you can support your instructions if something goes wrong.

updated Feb 22, 2011