Grouper Users - Open Discussion List

["Hyzer, Chris" <>]

How many groups are loaded?  Millions?

> If I do not delete the loader jobs then when the loader runs it will run
> the loader jobs. The loader jobs may create "real" changes that end up
> in GROUPER_CHANGE_LOG_TEMP. I want to be able to really TRUNCATE
> GROUPER_CHANGE_LOG_TEMP since that is a fast operation, rather than
> trying to delete by row. So I do not want any "real" changes in the
> GROUPER_CHANGE_LOG_TEMP.

Gotcha.  Didn't realize they run every minute.

> Maybe I am not, however, understanding how those 27 million rows in
> GROUPER_CHANGE_LOG_TEMP come about. Are they only created when the
> "real" loader jobs run? 

Hmmmm.  I deleted two of the new attributes, and change the incorrect value 
in a list of groups with two groups.  I ran the loader.  I got two change log 
entries per loaded group.  One for 
etc:attribute:loaderMetadata:grouperLoaderMetadataLastSummary and one for 
etc:attribute:loaderMetadata:grouperLoaderMetadataLastFullMillisSince1970
I believe you already have assignments on loaded groups for 
grouperLoaderMetadataLoaded and grouperLoaderMetadataGroupId.

Not saying this is better, but before the patch, you could create those 
attributes, and run a GSH script to assign them to loaded groups, maybe a 
sleeping a second after a dozen assignments.  Up to you.

> > Run the PIT sync in GSH: new
> > edu.internet2.middleware.grouper.misc.SyncPITTables().syncAllPITTables()
> 
> Why is that necessary?

Gets the PIT data up to date and non corrupt.  If you need more info ask 
Shilen (and maybe we can update the PIT wiki, im looking and not seeing an 
explanation).

Thanks
Chris


-----Original Message-----
From: Scott Koranda 
[mailto:]
 
Sent: Thursday, February 15, 2018 4:26 PM
To: Hyzer, Chris 
<>
Cc: grouper-users 
<>
Subject: Re: [grouper-users] StaleObjectStateException thrown by loader jobs 
after patch

> Ok, the restriction in the change log is for the value assignments,
> not the attribute assignments.  So these are a one time thing.

It makes sense it is one time, but it has a significant impact on how
long the outage has to be for Grouper to incorporate these patches.

> > Since all of those rows have to be processed and moved from
> > GROUPER_CHANGE_LOG_ENTRY_TEMP to GROUPER_CHANGE_LOG, and then the
> > various change log consumers have to process each of those rows, any
> > "real" changes in group memberships are not going to be processed
> > for quite some time.
> 
> Do you know how long it would take?  

After patching to 95 and starting the loader there were 27 million rows
added to GROUPER_CHANGE_LOG_ENTRY_TEMP.

I estimate it will take this deployment about 8 hours to grind through
that.

> Is it just the change log temp to
> change log or is it individual processors?  Maybe we need to switch to
> a batched SQL paradigm for the change log temp to change log
> processing...

The 8 hours is primarily the change log temp to the change log. The
change log consumers fire on the changes once they get to the change log
but they just ignore these changes.

> About your recipe
> 
> - make sure all "real" work is processed - stop the loader
> 
> right
> 
> - delete loader jobs
> 
> I don't know why you need to delete the jobs

If I do not delete the loader jobs then when the loader runs it will run
the loader jobs. The loader jobs may create "real" changes that end up
in GROUPER_CHANGE_LOG_TEMP. I want to be able to really TRUNCATE
GROUPER_CHANGE_LOG_TEMP since that is a fast operation, rather than
trying to delete by row. So I do not want any "real" changes in the
GROUPER_CHANGE_LOG_TEMP.

Maybe I am not, however, understanding how those 27 million rows in
GROUPER_CHANGE_LOG_TEMP come about. Are they only created when the
"real" loader jobs run? 

If so, then I may not be able to just TRUNCATE, and will instead have to
delete by row. But that could also take real time.

This deployment has loader jobs that run every minute. Changes in group
memberships are expected to show up in the downstream provisioned
services within 5 minutes or so. 

I am trying to find a way to patch but not have an 8 hour outage.

> - patch - start the loader
> 
> run all the loader jobs
>
> 
> - wait for the change log temp to stop growing
> 
> wait for jobs to finish?
> 
> - stop the loader - truncate GROUPER_CHANGE_LOG_ENTRY_TEMP - truncate
> GROUPER_CHANGE_LOG_ENTRY
> 
> Run the PIT sync in GSH: new
> edu.internet2.middleware.grouper.misc.SyncPITTables().syncAllPITTables()

Why is that necessary?

Thanks,

Scott K

> - re-create the loader jobs
> 
> If you aren't deleting them you don't need to recreate them
> 
> - start the loader
> 
> 
> -----Original Message----- From: Scott Koranda
> [mailto:]
>  Sent: Thursday, February 15, 2018 12:16 PM
> To: Hyzer, Chris 
> <>
>  Cc: grouper-users
> <>
>  Subject: Re: [grouper-users]
> StaleObjectStateException thrown by loader jobs after patch
> 
> > There are no new rows for this patch, but there were a few attribute
> > assignments added to loaded groups from a previous patch.  There are
> > no audits or point in time associated with the attributes which
> > change each time the loader runs for the job in this patch.  We have
> > worked hard to make sure this is as lightweight as possible and
> > Grouper assumes they are there.  Added some metadata on groups is
> > not on my radar as something that needs to be optional for
> > performance reasons.  Ok?  :)
> 
> I think there is an issue, however, that will impact performance right
> after the patching.
> 
> After patching and starting up the loader I see a couple of million
> new rows added to GROUPER_CHANGE_LOG_ENTRY_TEMP. Those rows include
> things like 'etc:attribute:loaderMetadata:grouperLoaderMetadataLoaded'
> and
> 'etc:attribute:loaderMetadata:grouperLoaderMetadataLastFullMillisSince1970'.
> 
> Since all of those rows have to be processed and moved from
> GROUPER_CHANGE_LOG_ENTRY_TEMP to GROUPER_CHANGE_LOG, and then the
> various change log consumers have to process each of those rows, any
> "real" changes in group memberships are not going to be processed for
> quite some time.
> 
> My plan to avoid this is:
> 
> - make sure all "real" work is processed - stop the loader - delete
> loader jobs - patch - start the loader - wait for the change log temp
> to stop growing - stop the loader - truncate
> GROUPER_CHANGE_LOG_ENTRY_TEMP - truncate GROUPER_CHANGE_LOG_ENTRY -
> re-create the loader jobs - start the loader
> 
> Will that work to avoid the extra millions of rows in the change log?
> 
> Thanks,
> 
> Scott K