Tag Archives: piwik

piwik archive.sh out of memory

Posted on by
Reply

if you happen to run into memory issues piwik/core/DataTable.php with the archive.sh script of piwik you might go to free up memory on the server, like i did. memory problems with the archive script are known.

since php5/cli/php.ini shows memory_limit=-1 which means no mem limit (which i thought is a good value), i didn´t care too much about this. but at the end it turns out, a little add (-d memory_limit=1024M) to the archive shell script worked things out for me:


// snip---
echo "Archiving period = $period for idsite = $idsite..."
 CMD="$PHP_BIN -d memory_limit=1024M -q $PIWIK_PATH -- module=API&method=VisitsSummary.getVisits&idSite=$idsite&period=$period&date=last52&format=xml&token_auth=$TOKEN_AUTH"
 $CMD
// snap---

this made finally happen to bypass the archiving process for month/year and the server now has lower memory footprint because i tuned stuff down as much as i could before that :)

so, if you want to override an ini setting for the script you´re calling, the -d flag is the one to help.

piwik in version 0.5.2 + update script

Posted on by
Reply

while waiting for an almost ten gigabytes large piwik database to be updated, i’m just writing down that piwik got a new release yesterday.

upgrading piwik is easy. i pushed all steps to upgrade it on a linux (debian) server in a small shell script you may find useful.

simple update script for piwik

#!/bin/bash
# auto-update piwik
# @author bigbabou <bigbabou@web.de>
#

# where my current piwik resides
installdir=/var/www/

# where the ugly stuff should be made
workingdir=/home/bigbabou/

# used for copying the new piwik source
piwikdir=/home/bigbabou/piwik/*
logfile=updatepiwik.log

# where the new piwik can be downloaded
piwikurl=http://piwik.org/latest.zip

cd ${workingdir}

# check if update is necessary
wget -N -o${logfile} ${piwikurl}

# overwrite silently
unzip -q -o latest.zip

# tidy up old piwik dir
cd ${installdir}

rm -rf core js lang libs misc plugins tests themes tmp
rm index.php piwik.js piwik.php robots.txt README

# safe global.ini before updating
chown root config/global.ini.backup_by_script
chmod 770 config/global.ini.backup_by_script

echo 'copying new files'
cp -r ${piwikdir} .
chown -R www-data *
chgrp -R www-data *
ls --color=auto -lA ${installdir}

# make nightly precaching exectuable
chmod 770 ${installdir}'misc/cron/archive.sh'

# clean up stuff
cd ${workingdir}
rm latest.zip
rm How\ to\ install\ Piwik.html
rm -rf piwik

echo 'piwik update successful. have a nice day!'

## update db on shell
php ${installdir}index.php

echo ' - update script end -'
exit 0

two lines could be added for convenience

chmod 777 ${installdir}tmp/cache
chmod 777 ${installdir}tmp/templates_c

piwik shows General error: 126

Posted on by
Reply

last week there was everything working fine. no changes made. today, stuff is not running fine. piwik in version 0.4.1 does not show useful content anymore. if you ever get to see all your widgets showing the General error 126″ you can start over with mysqlrepair.

General error: 126 Incorrect key file for table ‘/tmp/#sql_6d3c23.MYI’; try to repair it

SQLSTATE[HY000]

Yes sir, we’re going to repair the database totally. To do so, we fire up

mysqlrepair -B piwik -force -p

which will show a lot of OK rows. option -A is for all databases, so if your mysql server is in production and you’ve got a bunch of dbs on it hot, you propably consider using -B option to point to the single piwik database. I had to retry this with different options but now the things looks okay.

so today’s view is fixed, but since the problem existed for around three days, two days of data are corrupted. so i open config/global.ini.php and set the debugsters to 1.

[Debug]
; the archiving process will always be triggered, even if the archive has already been computed
always_archive_data_period = 1;
always_archive_data_day = 1;

; if set to 1, all the SQL queries will be recorded by the profiler
enable_sql_profiler = 1;

after having reprocessed the complete archive, which does take quite a while, there is still a report of an incorrect key file. But now with nice SQL profiling., d’oh.

<?xml version="1.0" encoding="utf-8" ?>
<result>
<error message="SQLSTATE[HY000]: General error: 126 Incorrect key file for table '/tmp/#sql_694e23.MYI'; try to repair it" />
</result>

solution

the problem was caused by a /tmp partition defined too small (16mb on a guest). now as i raised the tmpfs size the system works normal again. in vserver landscapes you can search a while before finding the corresponding fstab in /etc/vservers/yourvserver/fstab on the host. All this can be found in the vserver faq section.

Thanks to @schommr for the tip.

Best of it all, your data won’t be gone if you have a problem like this. It just can’t be processed to show correctly. My concerns about several days loss of data were flushed away when piwik’s archive.sh ran through in about twenty minutes. go piwik!

piwik statistics and analytics

Posted on by
Reply

some of you may have noticed the Piwik link in my blog´s menu bar. well, I am using Piwik as open source website analysis tool for more projects now and it works just fine. this is why I just wanted to throw another blog post about it into the web. and here it is! Piwik is cool.

Other posts describe the pros and con of Piwik far better than I´d do in the next few minutes, that´s why I´ll tap it short:

  • MySQL & PHP – everybody should have that nowadays
  • easy installation – piece of cake if you can handle a ftp client
  • js based tracking – kind of customizable
  • excellent data export possibilities
  • it´s open source
  • you gonna own your own collected data

Speeding up things

For websites with some traffic, you´ll need quite more than the suggested minimum 128mb of memory for php. To have the Piwik interface sound and responding, you may want to precollect the tracking data with a cron job.

Generate a new user for Piwik ( piwik/index.php?module=UsersManager&action=index ) and copy the authorization token that you´ll get to paste it into the archive.sh file found under misc/cron/ in your Piwik folder.

crontab -u www-data -e

Append this line

5 0 * * * /path/to/piwik/misc/cron/archive.sh > /dev/null

Your box will respond way faster after having processed this. This is why I can check smoothly that it stais the way it used to be: my blog is being majorly visited with uptodate browsers and merely around 10% with IE or other funny stuff. Hoorray.