Grouper Users - Open Discussion List

[Scott Koranda <>]

> How many groups are loaded?  Millions?

This morning I see this:

SQL> SELECT COUNT(*) FROM GROUPER_GROUPS;
     62221

SQL> SELECT COUNT(*) FROM GROUPER_CHANGE_LOG_ENTRY_TEMP;
  26791591

SQL> SELECT COUNT(*) FROM GROUPER_CHANGE_LOG_ENTRY_TEMP WHERE STRING04 NOT 
LIKE 'etc:attribute:loaderMetadata:%';
         0

SQL> SELECT COUNT(*) FROM GROUPER_CHANGE_LOG_ENTRY;
   6856091

There were no entries in either the change log or the change log temp
before patching.

I do not understand how so many entries were put into
GROUPER_CHANGE_LOG_ENTRY_TEMP by the new functionality.

I also drastically underestimated that amount of time it will take for
this deployment to process 27 million rows in the change log. It is
clear it will not be able to do it in a reasonable time (less than a few
weeks) so I am going to TRUNCATE the tables now instead.

> > Maybe I am not, however, understanding how those 27 million rows in
> > GROUPER_CHANGE_LOG_TEMP come about. Are they only created when the
> > "real" loader jobs run? 
> 
> Hmmmm.  I deleted two of the new attributes, and change the incorrect value 
> in a list of groups with two groups.  I ran the loader.  I got two change 
> log entries per loaded group.  One for 
> etc:attribute:loaderMetadata:grouperLoaderMetadataLastSummary and one for 
> etc:attribute:loaderMetadata:grouperLoaderMetadataLastFullMillisSince1970
> I believe you already have assignments on loaded groups for 
> grouperLoaderMetadataLoaded and grouperLoaderMetadataGroupId.
> 
> Not saying this is better, but before the patch, you could create those 
> attributes, and run a GSH script to assign them to loaded groups, maybe a 
> sleeping a second after a dozen assignments.  Up to you.

I will consider it as an option. Thanks.

Another issue with these patches: every time a loader job runs it
generates thousands of log entries at the WARN level with a message like

2018-02-16 13:04:55,826: [DefaultQuartzScheduler_Worker-9] WARN
GrouperLoaderType.updateLoaderMetadataForGroupsNoLongerInLoader(2081) -
- Non-fatal error removing metadata on group:
  my:group:name:here

This is filling up the Grouper logs. 

Is this expected? 

Thanks,

Scott K