Grouper Users - Open Discussion List

[Shilen Patel <>]

Just to make sure, you do have the latest grouper.base.properties with this config:

 

grouper.attribute.namesOfAttributeDefNamesToIgnoreAuditsChangeLogPit.elConfig = $$grouper.attribute.rootStem$$:attestation:attestationCalculatedDaysLeft,$$grouper.attribute.rootStem$$:attestation:attestationLastEmailedDate,$$grouper.attribute.rootStem$$:loaderMetadata:grouperLoaderMetadataLastFullMillisSince1970,$$grouper.attribute.rootStem$$:loaderMetadata:grouperLoaderMetadataLastIncrementalMillisSince1970,$$grouper.attribute.rootStem$$:loaderMetadata:grouperLoaderMetadataLastSummary

 

(making sure it includes all the loaderMetadata attributes)

 

And you haven’t overwritten this config in grouper.properties.  Just making sure.

 

Thanks!

 

- Shilen

 

 

On 2/16/18, 1:16 PM, " on behalf of Scott Koranda" < on behalf of > wrote:

 

    > How many groups are loaded?  Millions?

   

    This morning I see this:

   

    SQL> SELECT COUNT(*) FROM GROUPER_GROUPS;

         62221

   

    SQL> SELECT COUNT(*) FROM GROUPER_CHANGE_LOG_ENTRY_TEMP;

      26791591

   

    SQL> SELECT COUNT(*) FROM GROUPER_CHANGE_LOG_ENTRY_TEMP WHERE STRING04 NOT LIKE 'etc:attribute:loaderMetadata:%';

             0

   

    SQL> SELECT COUNT(*) FROM GROUPER_CHANGE_LOG_ENTRY;

       6856091

   

    There were no entries in either the change log or the change log temp

    before patching.

   

    I do not understand how so many entries were put into

    GROUPER_CHANGE_LOG_ENTRY_TEMP by the new functionality.

   

    I also drastically underestimated that amount of time it will take for

    this deployment to process 27 million rows in the change log. It is

    clear it will not be able to do it in a reasonable time (less than a few

    weeks) so I am going to TRUNCATE the tables now instead.

   

    > > Maybe I am not, however, understanding how those 27 million rows in

    > > GROUPER_CHANGE_LOG_TEMP come about. Are they only created when the

    > > "real" loader jobs run?

    >

    > Hmmmm.  I deleted two of the new attributes, and change the incorrect value in a list of groups with two groups.  I ran the loader.  I got two change log entries per loaded group.  One for etc:attribute:loaderMetadata:grouperLoaderMetadataLastSummary and one for etc:attribute:loaderMetadata:grouperLoaderMetadataLastFullMillisSince1970

    > I believe you already have assignments on loaded groups for grouperLoaderMetadataLoaded and grouperLoaderMetadataGroupId.

    >

    > Not saying this is better, but before the patch, you could create those attributes, and run a GSH script to assign them to loaded groups, maybe a sleeping a second after a dozen assignments.  Up to you.

   

    I will consider it as an option. Thanks.

   

    Another issue with these patches: every time a loader job runs it

    generates thousands of log entries at the WARN level with a message like

   

    2018-02-16 13:04:55,826: [DefaultQuartzScheduler_Worker-9] WARN

    GrouperLoaderType.updateLoaderMetadataForGroupsNoLongerInLoader(2081) -

    - Non-fatal error removing metadata on group:

      my:group:name:here

   

    This is filling up the Grouper logs.

    

    Is this expected?

    

    Thanks,

   

    Scott K