Grouper Users - Open Discussion List

["Bee-Lindgren, Bert" <>]

> If you're not interested in that [flattened changes], you can actually disable it.  

> That would improve performance and you can still get immediate membership changes.

Just fyi...

PSPNG provisions based on the flattened changes, so you'd be stuck with only full-syncs if change-flattening were disabled.

--Bert

---

From: <> on behalf of Shilen Patel <>
Sent: Tuesday, February 14, 2017 11:18 AM
To: Michael R Gettes
Cc: grouper-users
Subject: Re: [grouper-users] changelogTemp to changelog main processing

 

Maybe.  This was created back when the loader was single threaded and PSP
was the main provisioner.  Both of which were often the bottlenecks.  Now
with the loader being multi-threaded and PSPNG being much faster, the
change log increasingly becomes an issue when you're loading a lot of data.

The records are processed one at a time and in a single thread mainly to
make sure that flattened membership and privilege changes are represented
correctly in the change log.  E.g. If you're added to a group in two
different ways, there's only one flattened membership add in the change
log.  And it also ensures that changes are committed sequentially and in
the right order.

Also, from what I've seen in the past, the most expensive part of the
process is determining the flattened changes.  If you're not interested in
that, you can actually disable it.  That would improve performance and you
can still get immediate membership changes.

We can take another look to see what improvements may be possible.  In the
past, I think there were other thoughts floated around like using hooks to
speed up provisioning.  In any case, do you have more info on what
specifically you're seeing?

Thanks!

- Shilen



On 2/13/17, 3:14 PM, "Michael R Gettes" <> wrote:

>From what I can tell, moving data from the temp changelog to the main
>changelog is handled by
>
>select from temp, insert to main, delete from temp
>
>loop - one record at a time.
>
>Can this be handled more efficiently?  This seems to be the slowest part
>of processing with Grouper.  If there are good reasons for it being the
>way it is - education is appreciated.
>
>Thanks!
>
>/mrg
>