Grouper Users - Open Discussion List

["Black, Carey M." <>]

I have started putting  the loader through some testing and I have observed a 
few "partial load error" conditions.

An example:

A job runs every 35 minutes, and syncs some data (a single group on the order 
of 300k members) from an LDAP_GROUPS_FROM_ATTRIBUTES job.
        Sometimes the job takes as little as about 15 minutes to run. (and 
appears to do all of the right things)
        Other times the job takes much longer. Sometimes over 3 hours. ( and 
does not do all of the right things)

        Sometimes the job add/removes a few people as expected.
        Other times the job will remove 1/3 of the population and then add 
them back in on the next one, and sometime two, runs. 
                The data in the LDAP source is not changing this drastically 
over this time period.

I have not yet found the right knob to detect the "LDAP search/session/handle 
failed" during the loader process.
And this churn is causing pain at the RDBMS layer due to large, and 
unnecessary, volumes of audit and change log activities.


I think the only reasonable interpretation of the data that I am seeing is 
that the read of the LDAP data is failing at some point. Or maybe the loader 
is failing to "record" all of the data it gets back from the LDAP data set. 
        I am not clear as to how the "diff" process works in the loader.

        Does it get a set of data from LDAP, store it in a temp table in 
grouper, then diff the memberships with SQL?
        Does it get a set of data from LDAP, get a set of data from grouper 
and diff it "in memory" then update grouper?
        Does it do something else?
        ( The inner workings of the loader are opaque, at this moment, to me. 
I have not yet really dug into that code....)

If it helps, I attached is a pdf file with some results of running this job 
over time.

-- 
Carey Matthew 


-----Original Message-----
From: 

 
[mailto:]
 On Behalf Of Hyzer, Chris
Sent: Wednesday, September 13, 2017 9:14 AM
To: Emilio Recio 
<>;
 Grouper Users Mailing List 
<>
Subject: [grouper-users] RE: grouper-loader fail safe / threshold for failed 
connections

If there is an error running the query, the loader will not remove all 
members.  Only if the database successfully returns no records.  Are you sure 
that is what happened?

-----Original Message-----
From: 

 
[mailto:]
 On Behalf Of Emilio Recio
Sent: Wednesday, September 13, 2017 7:20 AM
To: Grouper Users Mailing List 
<>
Subject: [grouper-users] grouper-loader fail safe / threshold for failed 
connections

So we had a network outage, and the connection between grouper-loader and our 
oracle database failed. This caused zero entries to be returned when loader 
ran. When pspng provisioned out to our ldap, it undid the attributes for the 
people we expected from the back end oracle database.

Is there a way I can tell grouper that if grouper-loader experiences a
*connection* problem with the back end database, do not do anything; just 
exit with an appropriate rc? Is there a way I can kick off a script before to 
check for connectivity that stops the loader process based on its rc?

I've seen the threshold value, but don't know if it would make sense. We 
expected only 40 peoples' data elements from the back end database. Say I put 
the threshold to 40, then as far as I've seen in the docs, reaching this just 
drops it into fullsynch mode. How does fullsynch mode know to apply a 
failsafe? Is there a property for that?

--

Thanks,
E. Recio
The information contained in this transmission contains privileged and 
confidential information. It is intended only for the use of the person named 
above. If you are not the intended recipient, you are hereby notified that 
any review, dissemination, distribution or duplication of this communication 
is strictly prohibited. If you are not the intended recipient, please contact 
the sender by reply email and destroy all copies of the original message.

CAUTION: Intended recipients should NOT use email communication for emergent 
or urgent health care matters.

Attachment: 2017-09-13.Loader_times.pdf
Description: 2017-09-13.Loader_times.pdf