Subject: Grouper Users - Open Discussion List
RE: [grouper-users] old records not removed from grouper_change_log_entry table
- From: "Hyzer, Chris" <>
- To: Ben Beecher <>, "" <>
- Subject: RE: [grouper-users] old records not removed from grouper_change_log_entry table
- Date: Wed, 30 Nov 2016 23:42:59 +0000
- Accept-language: en-US
- Authentication-results: spf=none (sender IP is ) ;
- Spamdiagnosticmetadata: NSPM
- Spamdiagnosticoutput: 1:99
Run this query: select status, started_time, ended_time, millis, job_message from grouper_loader_log where JOB_NAME = 'MAINTENANCE_cleanLogs';
Are you getting successes?
I think what can happen is the database can choke on the delete query since there are so many records. Can you remove the records with a loop of SQL? If we need to change the logic we can delete records a chunk at a time… though I haven’t seen this problem before in Grouper (I have seen it in other systems though)
On Behalf Of Ben Beecher
For the past two weeks we've been dealing with disk space issues in our Grouper database. On 11/10 the data tablespace grew very large, apparently because we had assigned view privileges to a large number of users. We have about 41700 reference groups and we had assigned the view privilege to each group for about 130000 users. This produced a large number of transactions and the grouper_change_log_entry table grew very large. On 11/17 there were 148 million records in that table and it was growing by 16 million records per day. Our data tablespace grew from about 30GB to 171GB. We truncated the grouper_change_log_entry table and we added these lines to the grouper-loader.properties file and restarted the loader:
We removed the view privileges for those 130000 users to the reference groups and that created several million new transactions. We expected the table to shrink but it still has 131 million records. It is currently growing by about 4133 records per day. The old records are not being removed. Our data tablespace has grown to 179GB.
Why are the old records not being removed from the grouper_change_log_entry table?
Should we truncate the table again?
The created_on field should contain milliseconds since epoch but those values are much larger than expected (E+15 instead of E+12):
sql> select (sysdate - to_date('01-Jan-1970', 'dd-Mon-yyyy')) * 86400000 from dual;
1.4804E+12 (millis since epoch for today)
sql> select created_on,TO_DATE('1970-01-01','YYYY-MM-DD') + created_on / 86400000 from grouper_change_log_entry where sequence_number=166996954;
1.4794E+15 25-OCT-14 (millis since epoch for the oldest record in that table)
sql> select created_on,TO_DATE('1970-01-01','YYYY-MM-DD') + created_on / 86400000 from grouper_change_log_entry where sequence_number=297672536;
1.4804E+15 23-AUG-46 (millis since epoch for the newest record in that table)
I set log level to DEBUG and I did not see any error messages in the log regarding the grouper_change_log_entry table.
- [grouper-users] old records not removed from grouper_change_log_entry table, Ben Beecher, 11/29/2016
- RE: [grouper-users] old records not removed from grouper_change_log_entry table, Hyzer, Chris, 11/30/2016
Archive powered by MHonArc 2.6.19.