perfsonar-user - Re: [perfsonar-user] Looks like memory corruption in RRD-MA
Subject: perfSONAR User Q&A and Other Discussion
List archive
- From: Roman Lapacz <>
- To: Jon K Hellan <>
- Cc:
- Subject: Re: [perfsonar-user] Looks like memory corruption in RRD-MA
- Date: Wed, 16 May 2007 16:10:48 +0200
Hi Jon,
I took your metadata config file and just changed paths (they are very similar to yours) and data source strings (see attachment). I run the service and I was able to receive correct response (using your request) without any error message (there was NaN values because I had wrong startTime and endTime parameters). So everything was fine for me.
Roman
Jon K Hellan wrote:
I'm attaching the request, the response, and a minimal configuration file to reproduce the error. I was able to reduce the configuration from both directions for 35 interfaces to the "in" direction for 14 interfaces. This time, the truncated string is '/lib/nav/cricket-data/router-interfaces/tromso-gw.uninett.no/pos1_2.rrd'
(needs '/var' in front).
The log agrees:
2007-05-16 14:16:16,896 ERROR - RRDStorageManager.rrdFetch: Cannot retrieve data from rrdjtool library: RRDFETCH failed: opening '/lib/nav/cricket-data/router-interfaces/tromso-gw.uninett.no/pos1_2.rrd': No such file or directory
Regards
Jon Kåre
Jon K Hellan wrote:
Hi
I'm seeing behaviour in RRD-MA which looks a lot like memory corruption.
This is the perfsonar-bundle-2.0-RC1 bundle.
The error in the log is
ERROR - RRDStorageManager.rrdFetch: Cannot retrieve data from rrdjtool library: RRDFETCH failed: opening 's0_0.rrd': No such file or directory
In one sense, this is correct. There is indeed no such file.
The string s0_0.rrd only occurs in my config file as the end of a file name:
<nmwg:parameter name="file">/var/lib/nav/cricket-data/router-interfaces/trd-gw.uninett.no/pos0_0.rrd</nmwg:parameter>
I'm fetching utilization data for all interfaces in the config file. First in the 'in' direction, then in the 'out' direction. The query in the 'in' direction completes successfully.
I've traced the rrd calls with gdb, and the strange string is passed to rrdOpen after a number of reasonable rrd file names have been successfully processed.
Any tips on how to debug this?
Regards
Jon Kåre
<?xml version="1.0" encoding="UTF-8"?> <nmwg:store type="MAStore" xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/" xmlns:nmtm="http://ggf.org/ns/nmwg/time/2.0/" xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/" xmlns:perfsonar="http://ggf.org/ns/nmwg/tools/org/perfsonar/1.0/" xmlns:netutil="http://ggf.org/ns/nmwg/characteristic/utilization/2.0/" xmlns="http://ggf.org/ns/nmwg/base/2.0/"> <!-- Router interface --> <nmwg:metadata id="meta174633"> <netutil:subject id="subj174633"> <nmwgt:interface> <nmwgt:hostName>tromso-gw.uninett.no</nmwgt:hostName> <nmwgt:ifAddress type="ipv4">128.39.46.57</nmwgt:ifAddress> <nmwgt:ifIndex>2</nmwgt:ifIndex> <nmwgt:ifName>POS1/2</nmwgt:ifName> <nmwgt:ifDescription>ifInOctets</nmwgt:ifDescription> <nmwgt:capacity>2488000000</nmwgt:capacity> <nmwgt:direction>in</nmwgt:direction> <nmwgt:authRealm>UNINETT-uninett</nmwgt:authRealm> </nmwgt:interface> </netutil:subject> <nmwg:parameters> <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter> </nmwg:parameters> </nmwg:metadata> <nmwg:data id="data174633" metadataIdRef="meta174633"> <nmwg:key> <nmwg:parameters> <nmwg:parameter name="file">/home/romradz/tmp/perfsonar-test/lib/nav/cricket-data/router-interfaces/tromso-gw.uninett.no/test.rrd</nmwg:parameter> <nmwg:parameter name="dataSource">bytes</nmwg:parameter> <nmwg:parameter name="valueUnits">octets</nmwg:parameter> </nmwg:parameters> </nmwg:key> </nmwg:data> <!-- Router interface --> <nmwg:metadata id="meta174651"> <netutil:subject id="subj174651"> <nmwgt:interface> <nmwgt:hostName>tromso-gw.uninett.no</nmwgt:hostName> <nmwgt:ifAddress type="ipv4">128.39.47.98</nmwgt:ifAddress> <nmwgt:ifIndex>1</nmwgt:ifIndex> <nmwgt:ifName>POS1/1</nmwgt:ifName> <nmwgt:ifDescription>ifInOctets</nmwgt:ifDescription> <nmwgt:capacity>2488000000</nmwgt:capacity> <nmwgt:direction>in</nmwgt:direction> <nmwgt:authRealm>UNINETT-uninett</nmwgt:authRealm> </nmwgt:interface> </netutil:subject> <nmwg:parameters> <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter> </nmwg:parameters> </nmwg:metadata> <nmwg:data id="data174651" metadataIdRef="meta174651"> <nmwg:key> <nmwg:parameters> <nmwg:parameter name="file">/home/romradz/tmp/perfsonar-test/lib/nav/cricket-data/router-interfaces/tromso-gw.uninett.no/test.rrd</nmwg:parameter> <nmwg:parameter name="dataSource">bytes</nmwg:parameter> <nmwg:parameter name="valueUnits">octets</nmwg:parameter> </nmwg:parameters> </nmwg:key> </nmwg:data> <!-- Router interface --> <nmwg:metadata id="meta176829"> <netutil:subject id="subj176829"> <nmwgt:interface> <nmwgt:hostName>stolav-gw.uninett.no</nmwgt:hostName> <nmwgt:ifAddress type="ipv4">128.39.46.250</nmwgt:ifAddress> <nmwgt:ifIndex>9</nmwgt:ifIndex> <nmwgt:ifName>POS2/0</nmwgt:ifName> <nmwgt:ifDescription>ifInOctets</nmwgt:ifDescription> <nmwgt:capacity>2488000000</nmwgt:capacity> <nmwgt:direction>in</nmwgt:direction> <nmwgt:authRealm>UNINETT-uninett</nmwgt:authRealm> </nmwgt:interface> </netutil:subject> <nmwg:parameters> <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter> </nmwg:parameters> </nmwg:metadata> <nmwg:data id="data176829" metadataIdRef="meta176829"> <nmwg:key> <nmwg:parameters> <nmwg:parameter name="file">/home/romradz/tmp/perfsonar-test/lib/nav/cricket-data/router-interfaces/stolav-gw.uninett.no/test.rrd</nmwg:parameter> <nmwg:parameter name="dataSource">bytes</nmwg:parameter> <nmwg:parameter name="valueUnits">octets</nmwg:parameter> </nmwg:parameters> </nmwg:key> </nmwg:data> <!-- Router interface --> <nmwg:metadata id="meta176835"> <netutil:subject id="subj176835"> <nmwgt:interface> <nmwgt:hostName>stolav-gw.uninett.no</nmwgt:hostName> <nmwgt:ifAddress type="ipv4">128.39.46.18</nmwgt:ifAddress> <nmwgt:ifIndex>11</nmwgt:ifIndex> <nmwgt:ifName>POS4/0</nmwgt:ifName> <nmwgt:ifDescription>ifInOctets</nmwgt:ifDescription> <nmwgt:capacity>2488000000</nmwgt:capacity> <nmwgt:direction>in</nmwgt:direction> <nmwgt:authRealm>UNINETT-uninett</nmwgt:authRealm> </nmwgt:interface> </netutil:subject> <nmwg:parameters> <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter> </nmwg:parameters> </nmwg:metadata> <nmwg:data id="data176835" metadataIdRef="meta176835"> <nmwg:key> <nmwg:parameters> <nmwg:parameter name="file">/home/romradz/tmp/perfsonar-test/lib/nav/cricket-data/router-interfaces/stolav-gw.uninett.no/test.rrd</nmwg:parameter> <nmwg:parameter name="dataSource">bytes</nmwg:parameter> <nmwg:parameter name="valueUnits">octets</nmwg:parameter> </nmwg:parameters> </nmwg:key> </nmwg:data> <!-- Router interface --> <nmwg:metadata id="meta176841"> <netutil:subject id="subj176841"> <nmwgt:interface> <nmwgt:hostName>stolav-gw.uninett.no</nmwgt:hostName> <nmwgt:ifAddress type="ipv4">193.10.68.62</nmwgt:ifAddress> <nmwgt:ifIndex>13</nmwgt:ifIndex> <nmwgt:ifName>POS6/0</nmwgt:ifName> <nmwgt:ifDescription>ifInOctets</nmwgt:ifDescription> <nmwgt:capacity>2488000000</nmwgt:capacity> <nmwgt:direction>in</nmwgt:direction> <nmwgt:authRealm>UNINETT-uninett</nmwgt:authRealm> </nmwgt:interface> </netutil:subject> <nmwg:parameters> <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter> </nmwg:parameters> </nmwg:metadata> <nmwg:data id="data176841" metadataIdRef="meta176841"> <nmwg:key> <nmwg:parameters> <nmwg:parameter name="file">/home/romradz/tmp/perfsonar-test/lib/nav/cricket-data/router-interfaces/stolav-gw.uninett.no/test.rrd</nmwg:parameter> <nmwg:parameter name="dataSource">bytes</nmwg:parameter> <nmwg:parameter name="valueUnits">octets</nmwg:parameter> </nmwg:parameters> </nmwg:key> </nmwg:data> <!-- Router interface --> <nmwg:metadata id="meta176853"> <netutil:subject id="subj176853"> <nmwgt:interface> <nmwgt:hostName>stolav-gw.uninett.no</nmwgt:hostName> <nmwgt:ifAddress type="ipv4">128.39.47.109</nmwgt:ifAddress> <nmwgt:ifIndex>12</nmwgt:ifIndex> <nmwgt:ifName>POS5/0</nmwgt:ifName> <nmwgt:ifDescription>ifInOctets</nmwgt:ifDescription> <nmwgt:capacity>2488000000</nmwgt:capacity> <nmwgt:direction>in</nmwgt:direction> <nmwgt:authRealm>UNINETT-uninett</nmwgt:authRealm> </nmwgt:interface> </netutil:subject> <nmwg:parameters> <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter> </nmwg:parameters> </nmwg:metadata> <nmwg:data id="data176853" metadataIdRef="meta176853"> <nmwg:key> <nmwg:parameters> <nmwg:parameter name="file">/home/romradz/tmp/perfsonar-test/lib/nav/cricket-data/router-interfaces/stolav-gw.uninett.no/test.rrd</nmwg:parameter> <nmwg:parameter name="dataSource">bytes</nmwg:parameter> <nmwg:parameter name="valueUnits">octets</nmwg:parameter> </nmwg:parameters> </nmwg:key> </nmwg:data> <!-- Router interface --> <nmwg:metadata id="meta176859"> <netutil:subject id="subj176859"> <nmwgt:interface> <nmwgt:hostName>stolav-gw.uninett.no</nmwgt:hostName> <nmwgt:ifAddress type="ipv4">193.10.68.2</nmwgt:ifAddress> <nmwgt:ifIndex>10</nmwgt:ifIndex> <nmwgt:ifName>POS3/0</nmwgt:ifName> <nmwgt:ifDescription>ifInOctets</nmwgt:ifDescription> <nmwgt:capacity>2488000000</nmwgt:capacity> <nmwgt:direction>in</nmwgt:direction> <nmwgt:authRealm>UNINETT-uninett</nmwgt:authRealm> </nmwgt:interface> </netutil:subject> <nmwg:parameters> <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter> </nmwg:parameters> </nmwg:metadata> <nmwg:data id="data176859" metadataIdRef="meta176859"> <nmwg:key> <nmwg:parameters> <nmwg:parameter name="file">/home/romradz/tmp/perfsonar-test/lib/nav/cricket-data/router-interfaces/stolav-gw.uninett.no/test.rrd</nmwg:parameter> <nmwg:parameter name="dataSource">bytes</nmwg:parameter> <nmwg:parameter name="valueUnits">octets</nmwg:parameter> </nmwg:parameters> </nmwg:key> </nmwg:data> <!-- Router interface --> <nmwg:metadata id="meta176859"> <netutil:subject id="subj176859"> <nmwgt:interface> <nmwgt:hostName>stolav-gw.uninett.no</nmwgt:hostName> <nmwgt:ifAddress type="ipv4">193.10.68.54</nmwgt:ifAddress> <nmwgt:ifIndex>10</nmwgt:ifIndex> <nmwgt:ifName>POS3/0</nmwgt:ifName> <nmwgt:ifDescription>ifInOctets</nmwgt:ifDescription> <nmwgt:capacity>2488000000</nmwgt:capacity> <nmwgt:direction>in</nmwgt:direction> <nmwgt:authRealm>UNINETT-uninett</nmwgt:authRealm> </nmwgt:interface> </netutil:subject> <nmwg:parameters> <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter> </nmwg:parameters> </nmwg:metadata> <nmwg:data id="data176859" metadataIdRef="meta176859"> <nmwg:key> <nmwg:parameters> <nmwg:parameter name="file">/home/romradz/tmp/perfsonar-test/lib/nav/cricket-data/router-interfaces/stolav-gw.uninett.no/test.rrd</nmwg:parameter> <nmwg:parameter name="dataSource">bytes</nmwg:parameter> <nmwg:parameter name="valueUnits">octets</nmwg:parameter> </nmwg:parameters> </nmwg:key> </nmwg:data> <!-- Router interface --> <nmwg:metadata id="meta176859"> <netutil:subject id="subj176859"> <nmwgt:interface> <nmwgt:hostName>stolav-gw.uninett.no</nmwgt:hostName> <nmwgt:ifAddress type="ipv4">193.10.68.7</nmwgt:ifAddress> <nmwgt:ifIndex>10</nmwgt:ifIndex> <nmwgt:ifName>POS3/0</nmwgt:ifName> <nmwgt:ifDescription>ifInOctets</nmwgt:ifDescription> <nmwgt:capacity>2488000000</nmwgt:capacity> <nmwgt:direction>in</nmwgt:direction> <nmwgt:authRealm>UNINETT-uninett</nmwgt:authRealm> </nmwgt:interface> </netutil:subject> <nmwg:parameters> <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter> </nmwg:parameters> </nmwg:metadata> <nmwg:data id="data176859" metadataIdRef="meta176859"> <nmwg:key> <nmwg:parameters> <nmwg:parameter name="file">/home/romradz/tmp/perfsonar-test/lib/nav/cricket-data/router-interfaces/stolav-gw.uninett.no/test.rrd</nmwg:parameter> <nmwg:parameter name="dataSource">bytes</nmwg:parameter> <nmwg:parameter name="valueUnits">octets</nmwg:parameter> </nmwg:parameters> </nmwg:key> </nmwg:data> <!-- Router interface --> <nmwg:metadata id="meta180111"> <netutil:subject id="subj180111"> <nmwgt:interface> <nmwgt:hostName>tromso-gw3.uninett.no</nmwgt:hostName> <nmwgt:ifAddress type="ipv4">128.39.46.58</nmwgt:ifAddress> <nmwgt:ifIndex>2</nmwgt:ifIndex> <nmwgt:ifName>POS1/2</nmwgt:ifName> <nmwgt:ifDescription>ifInOctets</nmwgt:ifDescription> <nmwgt:capacity>2488000000</nmwgt:capacity> <nmwgt:direction>in</nmwgt:direction> <nmwgt:authRealm>UNINETT-uninett</nmwgt:authRealm> </nmwgt:interface> </netutil:subject> <nmwg:parameters> <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter> </nmwg:parameters> </nmwg:metadata> <nmwg:data id="data180111" metadataIdRef="meta180111"> <nmwg:key> <nmwg:parameters> <nmwg:parameter name="file">/home/romradz/tmp/perfsonar-test/lib/nav/cricket-data/router-interfaces/tromso-gw3.uninett.no/test.rrd</nmwg:parameter> <nmwg:parameter name="dataSource">bytes</nmwg:parameter> <nmwg:parameter name="valueUnits">octets</nmwg:parameter> </nmwg:parameters> </nmwg:key> </nmwg:data> <!-- Router interface --> <nmwg:metadata id="meta183687"> <netutil:subject id="subj183687"> <nmwgt:interface> <nmwgt:hostName>kristiansand-gw.uninett.no</nmwgt:hostName> <nmwgt:ifAddress type="ipv4">128.39.47.110</nmwgt:ifAddress> <nmwgt:ifIndex>1</nmwgt:ifIndex> <nmwgt:ifName>POS1/1</nmwgt:ifName> <nmwgt:ifDescription>ifInOctets</nmwgt:ifDescription> <nmwgt:capacity>2488000000</nmwgt:capacity> <nmwgt:direction>in</nmwgt:direction> <nmwgt:authRealm>UNINETT-uninett</nmwgt:authRealm> </nmwgt:interface> </netutil:subject> <nmwg:parameters> <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter> </nmwg:parameters> </nmwg:metadata> <nmwg:data id="data183687" metadataIdRef="meta183687"> <nmwg:key> <nmwg:parameters> <nmwg:parameter name="file">/home/romradz/tmp/perfsonar-test/lib/nav/cricket-data/router-interfaces/kristiansand-gw.uninett.no/test.rrd</nmwg:parameter> <nmwg:parameter name="dataSource">bytes</nmwg:parameter> <nmwg:parameter name="valueUnits">octets</nmwg:parameter> </nmwg:parameters> </nmwg:key> </nmwg:data> <!-- Router interface --> <nmwg:metadata id="meta183699"> <netutil:subject id="subj183699"> <nmwgt:interface> <nmwgt:hostName>kristiansand-gw.uninett.no</nmwgt:hostName> <nmwgt:ifAddress type="ipv4">128.39.47.113</nmwgt:ifAddress> <nmwgt:ifIndex>2</nmwgt:ifIndex> <nmwgt:ifName>POS1/2</nmwgt:ifName> <nmwgt:ifDescription>ifInOctets</nmwgt:ifDescription> <nmwgt:capacity>2488000000</nmwgt:capacity> <nmwgt:direction>in</nmwgt:direction> <nmwgt:authRealm>UNINETT-uninett</nmwgt:authRealm> </nmwgt:interface> </netutil:subject> <nmwg:parameters> <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter> </nmwg:parameters> </nmwg:metadata> <nmwg:data id="data183699" metadataIdRef="meta183699"> <nmwg:key> <nmwg:parameters> <nmwg:parameter name="file">/home/romradz/tmp/perfsonar-test/lib/nav/cricket-data/router-interfaces/kristiansand-gw.uninett.no/test.rrd</nmwg:parameter> <nmwg:parameter name="dataSource">bytes</nmwg:parameter> <nmwg:parameter name="valueUnits">octets</nmwg:parameter> </nmwg:parameters> </nmwg:key> </nmwg:data> <!-- Router interface --> <nmwg:metadata id="meta187077"> <netutil:subject id="subj187077"> <nmwgt:interface> <nmwgt:hostName>stavanger-gw.uninett.no</nmwgt:hostName> <nmwgt:ifAddress type="ipv4">128.39.47.114</nmwgt:ifAddress> <nmwgt:ifIndex>2</nmwgt:ifIndex> <nmwgt:ifName>POS1/2</nmwgt:ifName> <nmwgt:ifDescription>ifInOctets</nmwgt:ifDescription> <nmwgt:capacity>2488000000</nmwgt:capacity> <nmwgt:direction>in</nmwgt:direction> <nmwgt:authRealm>UNINETT-uninett</nmwgt:authRealm> </nmwgt:interface> </netutil:subject> <nmwg:parameters> <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter> </nmwg:parameters> </nmwg:metadata> <nmwg:data id="data187077" metadataIdRef="meta187077"> <nmwg:key> <nmwg:parameters> <nmwg:parameter name="file">/home/romradz/tmp/perfsonar-test/lib/nav/cricket-data/router-interfaces/stavanger-gw.uninett.no/test.rrd</nmwg:parameter> <nmwg:parameter name="dataSource">bytes</nmwg:parameter> <nmwg:parameter name="valueUnits">octets</nmwg:parameter> </nmwg:parameters> </nmwg:key> </nmwg:data> <!-- Router interface --> <nmwg:metadata id="meta187101"> <netutil:subject id="subj187101"> <nmwgt:interface> <nmwgt:hostName>stavanger-gw.uninett.no</nmwgt:hostName> <nmwgt:ifAddress type="ipv4">128.39.47.118</nmwgt:ifAddress> <nmwgt:ifIndex>1</nmwgt:ifIndex> <nmwgt:ifName>POS1/1</nmwgt:ifName> <nmwgt:ifDescription>ifInOctets</nmwgt:ifDescription> <nmwgt:capacity>2488000000</nmwgt:capacity> <nmwgt:direction>in</nmwgt:direction> <nmwgt:authRealm>UNINETT-uninett</nmwgt:authRealm> </nmwgt:interface> </netutil:subject> <nmwg:parameters> <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter> </nmwg:parameters> </nmwg:metadata> <nmwg:data id="data187101" metadataIdRef="meta187101"> <nmwg:key> <nmwg:parameters> <nmwg:parameter name="file">/home/romradz/tmp/perfsonar-test/lib/nav/cricket-data/router-interfaces/stavanger-gw.uninett.no/test.rrd</nmwg:parameter> <nmwg:parameter name="dataSource">bytes</nmwg:parameter> <nmwg:parameter name="valueUnits">octets</nmwg:parameter> </nmwg:parameters> </nmwg:key> </nmwg:data> </nmwg:store>
- Looks like memory corruption in RRD-MA, Jon K Hellan, 05/15/2007
- Re: [perfsonar-user] Looks like memory corruption in RRD-MA, Jon K Hellan, 05/16/2007
- Re: [perfsonar-user] Looks like memory corruption in RRD-MA, Roman Lapacz, 05/16/2007
- Re: [perfsonar-user] Looks like memory corruption in RRD-MA, Roman Lapacz, 05/16/2007
- Re: [perfsonar-user] Looks like memory corruption in RRD-MA, Roman Lapacz, 05/16/2007
- Re: [perfsonar-user] Looks like memory corruption in RRD-MA, Jon K Hellan, 05/16/2007
Archive powered by MHonArc 2.6.16.