vmstat is a very useful tool, especially when you are doing performance tuning of an application like MySQL. The only problem with the output from vmstat is that it is rather non-visual. A simple solution is to import the output in a spreadsheet like Openoffice Calc and create a graph, but it is long and painful. Creating many such graphs is not fun at all.

Fortunately, in the Linux world, there are tools to generate graph nearly automatically. Here is my own set of tools to creates graphs from vmstat output. You need the gnuplot program and awk, which is on all Linux version.

The output of vmstat looks like this:

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 164 48604 45136 4871464 0 0 94 111 4 7 1 0 98 0 0
0 2 164 48476 45188 4871552 0 0 35 15 1062 133 1 0 99 0 0
0 6 164 44960 46540 4870400 0 0 3159 497 4149 5618 2 1 78 19 0
2 1 164 64420 46116 4843140 0 0 8651 1794 4815 7719 10 3 73 15 0
2 11 164 45180 46452 4855156 0 0 2325 3827 5846 10320 13 3 72 11 0
1 3 164 45528 45652 4852376 0 0 8652 1824 4857 6831 7 5 67 22 0
0 3 164 59024 45764 4834444 0 0 2394 4118 4768 6544 12 3 70 14 0
...


We first need a script to get rid of the titles and to create an X axis number. I wrote a small script, filtervmstat, which look like this:

 $ more ~/bin/filtervmstat

cat $1 | grep -v memory | grep -v buff | perl -p -e "s|  | |g" -| perl -p -e "s|  | |g" - | perl -p -e "s|  | |g" - | awk 'BEGIN {x=1}; {print x++ " " $1 " " $2 " " $3 " " $4 " " $5 " " $6 " " $7 " " $8 " " $9 " " $10 " " $11 " " $12 " " $13 " " $14 " " $15 " " $16;}'

I am not a awk expert, maybe the column enumeration could be simplified. This will give use the following output:

1 1 0 164 48604 45136 4871464 0 0 94 111 4 7 1 0 98 0
2 0 2 164 48476 45188 4871552 0 0 35 15 1062 133 1 0 99 0
3 0 6 164 44960 46540 4870400 0 0 3159 497 4149 5618 2 1 78 19
4 2 1 164 64420 46116 4843140 0 0 8651 1794 4815 7719 10 3 73 15
5 2 11 164 45180 46452 4855156 0 0 2325 3827 5846 10320 13 3 72 11
6 1 3 164 45528 45652 4852376 0 0 8652 1824 4857 6831 7 5 67 22
7 0 3 164 59024 45764 4834444 0 0 2394 4118 4768 6544 12 3 70 14
8 1 12 164 46284 45792 4842704 0 0 5980 2804 6378 9984 10 6 68 16
9 1 3 164 58760 45568 4823504 0 0 5203 4712 6470 9687 15 4 62 19


Next, we need a gnuplot template in order to plot column 14, 15 and 16 which are respectively, user, system and idle cpu. My cpugraph.plt template is here:

#!/usr/bin/gnuplot -persist
#
#
# G N U P L O T
# Version 4.0 patchlevel 0
# last modified Thu Apr 15 14:44:22 CEST 2004
# System: Linux 2.6.22
#
# Copyright (C) 1986 - 1993, 1998, 2004
# Thomas Williams, Colin Kelley and many others
#
# This is gnuplot version 4.0. Please refer to the documentation
# for command syntax changes. The old syntax will be accepted
# throughout the 4.0 series, but all save files use the new syntax.
#
# Type `help` to access the on-line reference manual.
# The gnuplot FAQ is available from
# http://www.gnuplot.info/faq/
#
# Send comments and requests for help to
#
# Send bugs, suggestions and mods to
#
#
# set terminal x11
# set output
unset clip points
set clip one
unset clip two
set bar 1.000000
set border 31 lt -1 lw 1.000
set xdata
set ydata
set zdata
set x2data
set y2data
set timefmt x "%d/%m/%y,%H:%M"
set timefmt y "%d/%m/%y,%H:%M"
set timefmt z "%d/%m/%y,%H:%M"
set timefmt x2 "%d/%m/%y,%H:%M"
set timefmt y2 "%d/%m/%y,%H:%M"
set timefmt cb "%d/%m/%y,%H:%M"
set boxwidth
set style fill empty border
set dummy x,y
set format x "% g"
set format y "% g"
set format x2 "% g"
set format y2 "% g"
set format z "% g"
set format cb "% g"
set angles radians
unset grid
set key title ""
set key right top Right noreverse enhanced box linetype -2 linewidth 1.000 samplen 4 spacing 1 width 0 height 0 autotitles
unset label
unset arrow
unset style line
unset style arrow
unset logscale
set offsets 0, 0, 0, 0
set pointsize 1
set encoding default
unset polar
unset parametric
unset decimalsign
set view 60, 30, 1, 1
set samples 100, 100
set isosamples 10, 10
set surface
unset contour
set clabel '%8.3g'
set mapping cartesian
set datafile separator whitespace
unset hidden3d
set cntrparam order 4
set cntrparam linear
set cntrparam levels auto 5
set cntrparam points 5
set size ratio 0 1,1
set origin 0,0
set style data points
set style function lines
set xzeroaxis lt -2 lw 1.000
set yzeroaxis lt -2 lw 1.000
set x2zeroaxis lt -2 lw 1.000
set y2zeroaxis lt -2 lw 1.000
set tics in
set ticslevel 0.5
set ticscale 1 0.5
set mxtics default
set mytics default
set mztics default
set mx2tics default
set my2tics default
set mcbtics default
set xtics border mirror norotate autofreq
set ytics border mirror norotate autofreq
set ztics border nomirror norotate autofreq
set nox2tics
set noy2tics
set cbtics border mirror norotate autofreq
set title "" 0.000000,0.000000 font ""
set timestamp "" bottom norotate 0.000000,0.000000 ""
set rrange [ * : * ] noreverse nowriteback # (currently [0.00000:10.0000] )
set trange [ * : * ] noreverse nowriteback # (currently [-5.00000:5.00000] )
set urange [ * : * ] noreverse nowriteback # (currently [-5.00000:5.00000] )
set vrange [ * : * ] noreverse nowriteback # (currently [-5.00000:5.00000] )
set xlabel "Sample" 0.000000,0.000000 font ""
set x2label "" 0.000000,0.000000 font ""
set xrange [ * : * ] noreverse nowriteback # (currently [-10.0000:10.0000] )
set x2range [ * : * ] noreverse nowriteback # (currently [-10.0000:10.0000] )
set ylabel "CPU" 0.000000,0.000000 font ""
set y2label "" 0.000000,0.000000 font ""
set yrange [ * : * ] noreverse nowriteback # (currently [-10.0000:10.0000] )
set y2range [ * : * ] noreverse nowriteback # (currently [-10.0000:10.0000] )
set zlabel "" 0.000000,0.000000 font ""
set zrange [ * : * ] noreverse nowriteback # (currently [-10.0000:10.0000] )
set cblabel "" 0.000000,0.000000 font ""
set cbrange [ * : * ] noreverse nowriteback # (currently [-10.0000:10.0000] )
set zero 1e-08
set lmargin -1
set bmargin -1
set rmargin -1
set tmargin -1
set locale "C"
set pm3d scansautomatic flush begin noftriangles nohidden3d implicit corners2color mean
unset pm3d
set palette positive nops_allcF maxcolors 0 gamma 1.5 color model RGB
set palette rgbformulae 7, 5, 15
set colorbox default
set colorbox vertical origin 0.9,0.2 size 0.1,0.63 bdefault
set loadpath
set fontpath
set fit noerrorvariables
plot "/tmp/cpudata.dat" using 1:16 with lines title "Idle", "/tmp/cpudata.dat" using 1:15 with lines title "System", "/tmp/cpudata.dat" using 1:14 with lines
title "User"
# EOF


It is a very simple template that produce the graph on screen in a X windows with the data coming from /tmp/cpudata.dat. I wrote another small script just to ease the process:


$ more ~/bin/cpuvmstat
HOME=/home/yves
cat $1 | filtervmstat > /tmp/cpudata.dat
if [ "a$2" == "apng" ] ; then
$HOME/bin/cpugraphpng.plt
exit
else
$HOME/bin/cpugraph.plt
fi
rm -f /tmp/cpudata.dat


The cpugraphpng.plt is the same template as before except the output has been change to a png file. So, calling this script with a vmstat output as parameter give the following result on you screen:

The graph is created is less then one second, a very desirable feature for an impatient guy like me…

If you want to save the graph as a png file, create another template as cpugraphpng.plt (copy cpugraph.plt) and change the first two lines to this:

set terminal png
set output "/tmp/cpugraph.png"

Yves

Today, I am in Boston for training, I am shadowing Ronald Bradford, another MySQL consultant. It is fun and I learn a lot although it is not always easy to follow someone else mind. But we needed to evaluate the performance and the stability of a SAN. Do do so, we have used the bonnie++ tool. The server has 32 GB of memory and the filesystem was mounted under /data3. The command line used was the following:

mkdir /data3/testbonnie; nohup bonnie++ -d /data3/testbonnie -s 65000 -n 10 -r 32000 -x 5 -u root -g root &

After more six hours… the results were the following:

cat nohup.out | grep proddb | bon_csv2txt

Version @version@ ——Sequential Output—— –Sequential Input- –Random-
-Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
proddb06 65000M 71639 84 67884 13 54084 7 60194 60 126326 7 572.7 0
proddb06 65000M 79194 92 73790 14 53934 7 60226 60 125639 7 566.9 0
proddb06 65000M 78795 94 65855 12 53893 7 60292 60 126335 7 576.1 1
proddb06 65000M 80131 94 66034 12 53338 7 60308 60 125753 7 587.5 1
proddb06 65000M 79590 95 67575 13 54467 7 60745 60 126381 7 589.7 0
——Sequential Create—— ——–Random Create——–
-Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
proddb06 10 9133 99 +++++ +++ +++++ +++ 9375 99 +++++ +++ +++++ +++
proddb06 10 9085 99 +++++ +++ +++++ +++ 9330 100 +++++ +++ +++++ +++
proddb06 10 9002 99 +++++ +++ +++++ +++ 9386 99 +++++ +++ +++++ +++
proddb06 10 9165 100 +++++ +++ +++++ +++ 9436 99 +++++ +++ +++++ +++
proddb06 10 9140 99 +++++ +++ +++++ +++ 9398 99 +++++ +++ +++++ +++

Which is typical of a medium range San using Raid5. It is worth notice that the first time we ran bonnie++, we actually frose the server. Bonnie++ stressed the disk I/O to a point that showed a problem with the fiber channel adapter driver. Good to have catch this problem before going into production.

Hi,

for the whole week I have been in training in Cupertino.  Basically, what is covered is HA/Cluster/Performance tuning.  Nothing very special this week,  I just continue to discover MySQL from the inside and I’m pleased with what I see.  Tonight, we have a dinner with our VP, Ulf Sandberg.  Since such a large proportion of MySQL employees work remotely,  such events are important make ties between people.

I started looking at the MySQL proxy and although some may argue that features it allows should be in the server, it is so easy to do interesting things.   Of course, I will have to learn LUA and… of course Jan had chosen to use LUA 5.1 and my brand new Ubuntu Feisty Laptop has LUA 5.0.  Anyway, I don’t really need to compile it right now, there is a binary version available.

I almost done with the configuration of my Lenovo T61, yesterday I got my bluetooth working.  What I actually had to do was “modprobe hci_usd”, “modprobe ibm_acpi” and then press “Fn-F5″.  With this my bluetooth starts working but… I lost my wifi and I had to reboot in order to get it back.  Not very fun but now, at least, I can transfer my picture from my bluetooth to my laptop.  I need to learn more about bluetooth.  That’s all for now.

Hi everyone,
this is my first blog entry and my maiden language is french, so, it is not as easy as it should be. Also, I will never pretend being a good writer so, be warned. I joined MySQL earlier in July as a senior consultant and I have created this blog at the suggestion of Ronald Bradford, a senior consultant on the same team as I am. I am just back from vacation and I currently flying to Cupertino CA for a performance tuning and high availability course. I am rather new to MySQL so I hope to learn a lot, especially from the performance tuning part. What I really like from MySQL is the scalability it allows, I don’t know any other generally available database that allows a similar scaling.

I spend my last week of vacation doing camping with my wife and three daughters at the Oka national park camping near Montreal. It’s a very nice camping in a mature forest maid of large pines and oak trees. I have to say, there are some mosquitoes but not too much, I’m still alive. The fun part with camping is the low level of stress and slow life rhythm but you have to accept a relatively low comfort and a smelly camp fire smoke perfume. I was wondering why camping was so popular. Of course there are what I just mention, the low stress life and the contact with the nature. But I believe there is something more when I saw at night, at every lot in the camping, people burning wood in a camp fire. Is it because, one million years ago, our ancestors where making fires to heat them and protect them against wild animal? Maybe there are traces of this in our genes, there is something fascinating about a camp fire. For the rest, sleeping outside on a rather… hard ground, maybe we just want to prove to ourselves that we are able to survive with limited resources. Anyway, I’m a computer science specialist, not an anthropologist.
That’s all for my first time.