Grouper Users - Open Discussion List

["Hyzer, Chris" <>]

Ok, the restriction in the change log is for the value assignments, not the 
attribute assignments.  So these are a one time thing.

> Since all of those rows have to be processed and moved from
> GROUPER_CHANGE_LOG_ENTRY_TEMP to GROUPER_CHANGE_LOG, and then the
> various change log consumers have to process each of those rows, any
> "real" changes in group memberships are not going to be processed for
> quite some time.

Do you know how long it would take?  Is it just the change log temp to change 
log or is it individual processors?
Maybe we need to switch to a batched SQL paradigm for the change log temp to 
change log processing...

About your recipe

- make sure all "real" work is processed
- stop the loader

right

- delete loader jobs

I don't know why you need to delete the jobs

- patch
- start the loader

run all the loader jobs

- wait for the change log temp to stop growing

wait for jobs to finish?

- stop the loader
- truncate GROUPER_CHANGE_LOG_ENTRY_TEMP
- truncate GROUPER_CHANGE_LOG_ENTRY

Run the PIT sync in GSH: new 
edu.internet2.middleware.grouper.misc.SyncPITTables().syncAllPITTables()

- re-create the loader jobs

If you aren't deleting them you don't need to recreate them

- start the loader


-----Original Message-----
From: Scott Koranda 
[mailto:]
 
Sent: Thursday, February 15, 2018 12:16 PM
To: Hyzer, Chris 
<>
Cc: grouper-users 
<>
Subject: Re: [grouper-users] StaleObjectStateException thrown by loader jobs 
after patch

> There are no new rows for this patch, but there were a few attribute
> assignments added to loaded groups from a previous patch.  There are
> no audits or point in time associated with the attributes which change
> each time the loader runs for the job in this patch.  We have worked
> hard to make sure this is as lightweight as possible and Grouper
> assumes they are there.  Added some metadata on groups is not on my
> radar as something that needs to be optional for performance reasons.
> Ok?  :)

I think there is an issue, however, that will impact performance right
after the patching.

After patching and starting up the loader I see a couple of million new
rows added to GROUPER_CHANGE_LOG_ENTRY_TEMP. Those rows include things
like 'etc:attribute:loaderMetadata:grouperLoaderMetadataLoaded' and
'etc:attribute:loaderMetadata:grouperLoaderMetadataLastFullMillisSince1970'.

Since all of those rows have to be processed and moved from
GROUPER_CHANGE_LOG_ENTRY_TEMP to GROUPER_CHANGE_LOG, and then the
various change log consumers have to process each of those rows, any
"real" changes in group memberships are not going to be processed for
quite some time.

My plan to avoid this is:

- make sure all "real" work is processed
- stop the loader
- delete loader jobs
- patch
- start the loader
- wait for the change log temp to stop growing
- stop the loader
- truncate GROUPER_CHANGE_LOG_ENTRY_TEMP
- truncate GROUPER_CHANGE_LOG_ENTRY
- re-create the loader jobs
- start the loader

Will that work to avoid the extra millions of rows in the change log?

Thanks,

Scott K