Skip to Content.
Sympa Menu

grouper-users - Re: [grouper-users] changelogTemp to changelog main processing

Subject: Grouper Users - Open Discussion List

List archive

Re: [grouper-users] changelogTemp to changelog main processing


Chronological Thread 
  • From: "Bee-Lindgren, Bert" <>
  • To: Shilen Patel <>, Michael R Gettes <>
  • Cc: grouper-users <>
  • Subject: Re: [grouper-users] changelogTemp to changelog main processing
  • Date: Thu, 16 Feb 2017 11:07:50 +0000
  • Accept-language: en-US
  • Authentication-results: spf=none (sender IP is ) ;
  • Ironport-phdr: 9a23:w22kthDn8EcD2r341Bd+UyQJP3N1i/DPJgcQr6AfoPdwSP37o82wAkXT6L1XgUPTWs2DsrQf2reQ6/yrAzVIyK3CmUhKSIZLWR4BhJdetC0bK+nBN3fGKuX3ZTcxBsVIWQwt1Xi6NU9IBJS2PAWK8TW94jEIBxrwKxd+KPjrFY7OlcS30P2594HObwlSijewZbN/IA+qoQjeucUan5ZuIbstxxXUpXdFZ/5Yzn5yK1KJmBb86Maw/Jp9/ClVpvks6c1OX7jkcqohVbBXAygoPG4z5M3wqBnMVhCP6WcGUmUXiRVHHQ7I5wznU5jrsyv6su192DSGPcDzULs5Vyiu47ttRRT1kyoMKSI3/3/LhcxxlKJboQyupxpjw47PfYqZMONycr7Bcd8GQGZMWMFeWjBfDo64b4YDFvYNMOVFpIX5p1YCsR6yCA+xD+3t1zBInGf7060k3eo8DQHI0g8uH9wMv3TVsNr6O7wfXvqpwKXU1zjPc+9a1Sr75YPVch4hu/aMXbdofMbf1EQgCR3KgkyWp4zkOTOay+sNs3KF4OpmTe6klnMqpB9rojizyMsnl47EiZkLxVDe6SV12p06KsOmR0JiZd6kEZ1QuDyVN4tsXMwiRXpluCE7yr0ao5K7czIFyIg5yBHDa/yLaZaH7Q/+VOuIJzpzmXFreKqnihqs60es1vDwW8yp3FpXsyZIl9fMu38R2xDP78WHROVy80il1DuK0g3e6O9JLV0smabFK5Mt2KM8m5gdvEjZAyP7mET7gLWLekgq5OSl7fnsbK/8qZ+GLYB0jxnzMqQwlcy7BuQ1KhAAUnSc9+ig2rDv5EP3TrJEg/EvlanWq47VKd4cpq6kHw9ayYEj6wu5Dzi7ytgYhWMHLFVZeB2Zk4fpJ1DOIPf+DfulhFSsjStrx/TBPr3mAZXBNGTMkLDkfbpl6k5czhQ8zcxH6p5KEL4NPO//V0rsuNDFExM0NgK0zun7BNlh044eV3yAD6CdPa7XtFKF5OwiL/SUaI8Qojn9Kvwl5/D0jX8+nF8QZbKm0oUNaHCkBPhpOFuWbmT1jtcFEGcKuBcxTOPwiFKcVz5efHGyX6Mg5j0hB4KpF5vDSZ6xj7yBxiu0AIBZZn1eBlCWDXjob5mEW+sLaC+KLc9hiDsEVaW5S4A/zxGirRL6y6F5IerO4S0Vrpbj1Nlu5+3PjhE+6yZ4D8Wb02GRUW50hGUISCEq3Kxhu0By1EqM0bUry8BfQOdT5vdEVE8CPILRy6QuENDzXg/HVtuUS1vgT9m7V3V5a9M6w9YDZw5HU/qvkgzOxGL+BqUazOSjHIcptK/Qwi6iCdx6ziOM/qQsyn0nRMdAL2CgwuZV+hLPTcadmUiDi+CgeKlZ2C/L+GiZwGymu0BEFgF5TajOXTYSalaA/oex3V/LU7L7UedvCQBG08PXb/YSZw==
  • Spamdiagnosticmetadata: NSPM
  • Spamdiagnosticoutput: 1:99

If you're not interested in that [flattened changes], you can actually disable it.  

> That would improve performance and you can still get immediate membership changes.


Just fyi...
PSPNG provisions based on the flattened changes, so you'd be stuck with only full-syncs if change-flattening were disabled.

--Bert


From: <> on behalf of Shilen Patel <>
Sent: Tuesday, February 14, 2017 11:18 AM
To: Michael R Gettes
Cc: grouper-users
Subject: Re: [grouper-users] changelogTemp to changelog main processing
 
Maybe.  This was created back when the loader was single threaded and PSP
was the main provisioner.  Both of which were often the bottlenecks.  Now
with the loader being multi-threaded and PSPNG being much faster, the
change log increasingly becomes an issue when you're loading a lot of data.

The records are processed one at a time and in a single thread mainly to
make sure that flattened membership and privilege changes are represented
correctly in the change log.  E.g. If you're added to a group in two
different ways, there's only one flattened membership add in the change
log.  And it also ensures that changes are committed sequentially and in
the right order.

Also, from what I've seen in the past, the most expensive part of the
process is determining the flattened changes.  If you're not interested in
that, you can actually disable it.  That would improve performance and you
can still get immediate membership changes.

We can take another look to see what improvements may be possible.  In the
past, I think there were other thoughts floated around like using hooks to
speed up provisioning.  In any case, do you have more info on what
specifically you're seeing?

Thanks!

- Shilen



On 2/13/17, 3:14 PM, "Michael R Gettes" <> wrote:

>From what I can tell, moving data from the temp changelog to the main
>changelog is handled by
>
>select from temp, insert to main, delete from temp
>
>loop - one record at a time.
>
>Can this be handled more efficiently?  This seems to be the slowest part
>of processing with Grouper.  If there are good reasons for it being the
>way it is - education is appreciated.
>
>Thanks!
>
>/mrg
>




Archive powered by MHonArc 2.6.19.

Top of Page