grouper-study

[Gail H Lift <>]

Forwarding this discussion to the larger group for input/comment/thoughts:

---------- Forwarded message ----------
From: Coleman, Erik C <>
Date: Tue, Nov 28, 2017 at 3:01 PM
Subject: RE: Grouper Real-time Loader
To: Gail H Lift <>
Cc: "" <>

Hi Gail,

 

Yea, I’m still wringing my hands over this—on the one hand I see the potential for performance improvement on the LDAP side, particularly if it could leverage a simple LDAP changelog, rather than a full sync. But on the other, I see an advantage over doing the SQL query instead of LDAP, which we could do in the form of a query to our ‘data warehouse’ of Banner identities. Unfortunately, this excludes a good chunk of our affiliate users (non-student, non-staff), so it’s not a clean break.

 

Further, even if they fix the loader on the LDAP side, I still face the outbound PSPNG connector being the bottleneck syncing those group objects back into our target directory. I’m not convinced that is being done most-efficiently either, particularly for large groups. A message queue may be the answer there.

 

I plan to do some more prototyping later this week with Ester, who is our resident data warehouse guru, to see if switching to SQL-based loader jobs is a viable path. But I will still gladly support and co-sign any such joint request.

 

-Erik

 

 

 

From: Gail H Lift [mailto:]
Sent: Tuesday, November 28, 2017 12:19
To: Coleman, Erik C <>
Cc:
Subject: Re: Grouper Real-time Loader

 

Michigan also has an LDAP primary subject source, with affiliation-related person data changing throughout the day. We have the same concerns about prioritization and queueing of jobs. We are concerned about load, and whether jobs would be re-triggered multiple times while still running. This could also result in provisioning delays.

 

Our IAM system is event-driven. We could easily send changed subject data to grouper in real time.

 

This page:

https://spaces.internet2.edu/display/Grouper/Grouper+loader+real+time+updates

says 

"Support for LDAP jobs is minimal right now.  If a message is added to the message queue for an LDAP job, a full sync is simply performed.  If you plan to use this feature and would like to see this improved, please contact the Grouper Team."

 

Perhaps we should make a joint request?

 

 

On Thu, Nov 16, 2017 at 10:34 AM, Coleman, Erik C <> wrote:

Bert was promoting the real-time loader in the last conference call, so I’m starting to play around with it. I see already that it is primarily designed for SQL loader jobs not LDAP jobs, but our primary subject source is LDAP-based.  What sort of options do I have for improving our automatic LDAP-based loader processes? We have hundreds of groups, several with up to 100K+ members, and we have a concern with the prioritization and queueing of jobs if we end up stacking a lot of these, or we have other departments on campus creating loader jobs for their groups.  We also sync these large groups to a downstream Active Directory OU as well.

 

Thanks for any tips!

 

Erik Coleman

University of Illinois at Urbana-Champaign

 

 

 

 

 

--

Gail H Lift
MCommunity, IAM-IIA, ITS, University of Michigan

--

Gail H Lift
MCommunity, IAM-IIA, ITS, University of Michigan