[ZendTo] Re: rrd fixes

ROGERS Richard Richard.Rogers at staffs.ac.uk
Fri Sep 23 10:43:50 BST 2011


Yes,  I agree, that's why I'm so baffled... I can't see that there's anything wrong with the commands that ZendTo is executing (or the ones I'm executing manually), yet I'm ending up with no valid data in the RRD.

All very strange - maybe some rrdtool bug... (I'll check for any updates/patches).

I will keep digging and post back if I find out anything useful. (Phil, is yours working correctly now?)

Thanks for your input

Regards

Richard

From: zendto-bounces at zend.to [mailto:zendto-bounces at zend.to] On Behalf Of Jules
Sent: 23 September 2011 10:13
To: ZendTo Users
Subject: [ZendTo] Re: rrd fixes



On 23/09/2011 09:55, ROGERS Richard wrote:
Hi Jules -

I stuck in an extra print statement to get the creation command; I now have

# /usr/bin/php /opt/zendto/sbin/rrdInit.php /opt/zendto/config/preferences.php
Initial creation of RRD file (start date of 2011-09-16)
/usr/bin/rrdtool create /var/zendto/rrd/zendto.rrd --start 20110916 --step 86400 DS:dropoff_count:GAUGE:90000:0:U DS:total_files:GAUGE:90000:0:U DS:total_kb:GAUGE:90000:0:U DS:files_per_dropoff:COMPUTE:total_files,dropoff_count,/ RRA:AVERAGE:0.5:1:365 RRA:AVERAGE:0.5:30:12017
/usr/bin/rrdtool update /var/zendto/rrd/zendto.rrd 1316214000:1:1:11.9

18
/usr/bin/rrdtool update /var/zendto/rrd/zendto.rrd 1316732400:1:1:7.0
That proves to me that it is doing the right thing. You have got 1 drop-off of 11.9KB (or is it MB?) from about 6.5 days ago and 1 drop-off of 7KB from about 9 or 10 hours ago.
Look in your list of all drop-offs (login as an admin) and you should see that in the table.

Is that anywhere close to the truth? If so, the ZendTo RRD code is doing the right thing.

Jules.


The odd thing about this is that if I run the rrdtool create command manually I get a much larger RRD (maybe my print statement is wrong - just crudely pasted in from the line that does the system command - PHP is not a language I speak fluently!)

-rw-r--r-- 1 root root 398200 Sep 23 09:08 zendto.rrd.manual1
-rw-r--r-- 1 root root  17496 Sep 16 14:24 zendto.rrd.old1

Adding data manually fails now though (not clear at all what is different from yesterday!) - no errors, just NaNs in the data sequence.

Anyway, reverting to a saved copy of an old RRD, I seem only to be able to get valid data to register if I increase the timestamp to something well in the future (like 1316905200).

Some issue with intervals??

Thanks and regards

Richard

From: zendto-bounces at zend.to<mailto:zendto-bounces at zend.to> [mailto:zendto-bounces at zend.to] On Behalf Of Jules
Sent: 23 September 2011 09:25
To: ZendTo Users
Subject: [ZendTo] Re: rrd fixes

Can you send me the output of rrdInit.php? I want to see all the rrdtool commands it is doing.

On 23/09/2011 09:02, ROGERS Richard wrote:
Aaagh, back to all NaN... I'll dig a bit and see if I can get anything useful.

Regards

Richard

From: zendto-bounces at zend.to<mailto:zendto-bounces at zend.to> [mailto:zendto-bounces at zend.to] On Behalf Of Jules
Sent: 22 September 2011 15:27
To: ZendTo Users
Subject: [ZendTo] Re: rrd fixes

Yes, give it a few data points to work on.

With my own production site, all the stats now match the counts in the ZendTo database (not looking at the RRD database). So the graphs say what they should say.

Jules.

On 22/09/2011 14:54, ROGERS Richard wrote:
Hmmm... initial data population still seems to have an oddity somewhere - perhaps because I had only one unexpired dropoff in the database - it still gave all "NaN" in the RRD. However a manual update with a forged time seemed to insert a proper data line. I have done a new upload, and will recreate the database now and then check the overnight RRD update in the morning.

Thanks and regards

Richard

From: zendto-bounces at zend.to<mailto:zendto-bounces at zend.to> [mailto:zendto-bounces at zend.to] On Behalf Of Jules
Sent: 22 September 2011 14:19
To: ZendTo Users
Subject: [ZendTo] Re: rrd fixes

Right. Now I've got it to generate correct data, and correct graphs.
It really does match what the "All Dropoffs" says it should say.

So please try the attached versions. You must delete your zendto.rrd file first. It will re-create all the data, so you won't lose any old stats. Then run rrdInit.php and then rrdUpdate.php, then check the graphs on your site.

Please let me know if you think this works. If it does, I'll start on a release.

Thanks for your patience,
Jules.

On 21/09/2011 09:48, ROGERS Richard wrote:
I'm sorry to say I think there are still problems with this. I don't think the data is getting into the RRD (which is why the graphs are flat!). My experience was as follows:

# /usr/bin/php /opt/zendto/sbin/rrdInit.php /opt/zendto/config/preferences.php
Initial creation of RRD file (start date of 2011-09-07)
16
/usr/bin/rrdtool update /var/zendto/rrd/zendto.rrd 1315436400:1:1:2926562.0

17
/usr/bin/rrdtool update /var/zendto/rrd/zendto.rrd 1316214000:1:1:11.9

Which looks OK, although on closer inspection I think the dates may be suspect. What's actually shown in the Outbox is:

Claim ID                                                Sender                 Recipient             Size                        Created
daWQaXD7S2bMGMfK                 ROGERS Richard, Staffordshire University staff (xxxxxx at staffs.ac.uk<mailto:xxxxxx at staffs.ac.uk>)    RMR <richard at xxxxxxxxx.uk><mailto:richard at xxxxxxxxx.uk>
                                                                                                                                11.9 KB                 16 Sep 2011 02:14:53 PM
GJYBiNngywQJjXiR                          ROGERS Richard, Staffordshire University staff (xxxxxx at staffs.ac.uk<mailto:xxxxxx at staffs.ac.uk>)    RMR <richard at xxxxxxxxx.uk><mailto:richard at xxxxxxxxx.uk>
                                                                                                                                2.8 GB                   07 Sep 2011 07:42:47 PM

Now an rrdtool dump (excerpt) shows:

                        <!-- 2011-08-31 01:00:00 BST / 1314748800 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>
                        <!-- 2011-09-01 01:00:00 BST / 1314835200 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>
                        <!-- 2011-09-02 01:00:00 BST / 1314921600 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>
                        <!-- 2011-09-03 01:00:00 BST / 1315008000 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>
                        <!-- 2011-09-04 01:00:00 BST / 1315094400 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>
                        <!-- 2011-09-05 01:00:00 BST / 1315180800 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>
                        <!-- 2011-09-06 01:00:00 BST / 1315267200 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>
                        <!-- 2011-09-07 01:00:00 BST / 1315353600 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>
                        <!-- 2011-09-08 01:00:00 BST / 1315440000 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>
                        <!-- 2011-09-09 01:00:00 BST / 1315526400 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>
                        <!-- 2011-09-10 01:00:00 BST / 1315612800 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>
                        <!-- 2011-09-11 01:00:00 BST / 1315699200 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>
                        <!-- 2011-09-12 01:00:00 BST / 1315785600 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>
                        <!-- 2011-09-13 01:00:00 BST / 1315872000 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>
                        <!-- 2011-09-14 01:00:00 BST / 1315958400 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>
                        <!-- 2011-09-15 01:00:00 BST / 1316044800 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>
                        <!-- 2011-09-16 01:00:00 BST / 1316131200 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>
                </database>

It ends after 16/9 (because the last dropoff was on the 16th? - the time stamp on the file is "now" and the time of last update at the top is "<lastupdate>1316214000</lastupdate> <!-- 2011-09-17 00:00:00 BST -->")

Hope this helps in further diagnosis!

ZendTo 4.05 on CentOS VM.

Regards

Richard

From: zendto-bounces at zend.to<mailto:zendto-bounces at zend.to> [mailto:zendto-bounces at zend.to] On Behalf Of --[ UxBoD ]--
Sent: 21 September 2011 08:24
To: ZendTo Users
Subject: [ZendTo] Re: rrd fixes

Hmm, graphs are all empty :(
--
Thanks, Phil
________________________________
From: "Jules" <Jules at zend.to><mailto:Jules at zend.to>
To: "ZendTo Users" <zendto at zend.to><mailto:zendto at zend.to>
Sent: Tuesday, 20 September, 2011 9:35:05 PM
Subject: [ZendTo] Re: rrd fixes

To save you poor busy folks a load of patching, I might just release a new version of ZendTo tomorrow. It's the first in a month or so, and it will iron out these niggles for you.

There won't be any new preferences.php settings or database changes, though there are a few *.tpl template changes. Should be a painless upgrade (he says, cursing it forever :-)

Cheers,
Jules.

On 20/09/2011 21:14, Jules wrote:
I've found and fixed the problems you've all been having with rrdInit.php and rrdUpdate.php.

Attached are new copies of
    /opt/zendto/sbin/rrdInit.php
    /opt/zendto/sbin/rrdUpdate.php
    /opt/zendto/lib/MySQL.php
    /opt/zendto/lib/SQLite.php
You will probably need to delete your RRD data file (/var/zendto/rrd/zendto.rrd by default) first, and then copy in the 4 files. Then run
    /opt/zendto/sbin/rrdInit.php /opt/zendto/config/preferences.php
    /opt/zendto/sbin/rrdUpdate.php /opt/zendto/config/preferences.php

The rrdInit script creates the data file if necessary and updates it with all the latest stats.
The rrdUpdate script just creates all the graphs.

Sorry it's taken me so long to work out the solution to this!

Jules








_______________________________________________

ZendTo mailing list

ZendTo at zend.to<mailto:ZendTo at zend.to>

http://mailman.ecs.soton.ac.uk/mailman/listinfo/zendto


Jules



--

Julian Field MEng CITP CEng

www.Zend.To<http://www.Zend.To>



Follow me at twitter.com/JulesFM

PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654



'Teach a man to reason, and he will think for a lifetime.' - Phil Plait

'All programs have a desire to be useful' - Tron, 1982

_______________________________________________
ZendTo mailing list
ZendTo at zend.to<mailto:ZendTo at zend.to>
http://mailman.ecs.soton.ac.uk/mailman/listinfo/zendto








_______________________________________________

ZendTo mailing list

ZendTo at zend.to<mailto:ZendTo at zend.to>

http://mailman.ecs.soton.ac.uk/mailman/listinfo/zendto






Jules



--

Julian Field MEng CITP CEng

www.Zend.To<http://www.Zend.To>



Follow me at twitter.com/JulesFM

PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654



'It's okay to live without all the answers' - Charlie Eppes, 2011

'All programs have a desire to be useful' - Tron, 1982

'That is the land of lost content,

 I see it shining plain,

 The happy highways where I went,

 And cannot come again.' - A.E. Houseman






_______________________________________________

ZendTo mailing list

ZendTo at zend.to<mailto:ZendTo at zend.to>

http://mailman.ecs.soton.ac.uk/mailman/listinfo/zendto





Jules



--

Julian Field MEng CITP CEng

www.Zend.To<http://www.Zend.To>



Follow me at twitter.com/JulesFM

PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654



'It's okay to live without all the answers' - Charlie Eppes, 2011

'All programs have a desire to be useful' - Tron, 1982

'That is the land of lost content,

 I see it shining plain,

 The happy highways where I went,

 And cannot come again.' - A.E. Houseman





_______________________________________________

ZendTo mailing list

ZendTo at zend.to<mailto:ZendTo at zend.to>

http://mailman.ecs.soton.ac.uk/mailman/listinfo/zendto




Jules



--

Julian Field MEng CITP CEng

www.Zend.To<http://www.Zend.To>



Follow me at twitter.com/JulesFM

PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654



'It's okay to live without all the answers' - Charlie Eppes, 2011

'All programs have a desire to be useful' - Tron, 1982

'That is the land of lost content,

 I see it shining plain,

 The happy highways where I went,

 And cannot come again.' - A.E. Houseman




_______________________________________________

ZendTo mailing list

ZendTo at zend.to<mailto:ZendTo at zend.to>

http://mailman.ecs.soton.ac.uk/mailman/listinfo/zendto



Jules



--

Julian Field MEng CITP CEng

www.Zend.To<http://www.Zend.To>



Follow me at twitter.com/JulesFM

PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654



'It's okay to live without all the answers' - Charlie Eppes, 2011

'All programs have a desire to be useful' - Tron, 1982

'That is the land of lost content,

 I see it shining plain,

 The happy highways where I went,

 And cannot come again.' - A.E. Houseman
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ecs.soton.ac.uk/pipermail/zendto/attachments/20110923/f265c3c7/attachment-0001.html 


More information about the ZendTo mailing list