Skip to Content.
Sympa Menu

grouper-users - [grouper-users] RE: grouper-loader fail safe / threshold for failed connections

Subject: Grouper Users - Open Discussion List

List archive

[grouper-users] RE: grouper-loader fail safe / threshold for failed connections

Chronological Thread 
  • From: "Black, Carey M." <>
  • To: Grouper Users Mailing List <>
  • Cc: "Hyzer, Chris" <>, Emilio Recio <>
  • Subject: [grouper-users] RE: grouper-loader fail safe / threshold for failed connections
  • Date: Wed, 13 Sep 2017 19:55:44 +0000
  • Accept-language: en-US
  • Authentication-results: spf=pass (sender IP is;; dkim=none (message not signed) header.d=none;; dmarc=pass action=none;
  • Ironport-phdr: 9a23:9vRz5RyREtBpra7XCy+O+j09IxM/srCxBDY+r6Qd2uwTIJqq85mqBkHD//Il1AaPBtSLraocw8Pt8InYEVQa5piAtH1QOLdtbDQizfssogo7HcSeAlf6JvO5JwYzHcBFSUM3tyrjaRsdF8nxfUDdrWOv5jAOBBr/KRB1JuPoEYLOksi7ze6/9pnQbglSmDaxfa55IQmrownWqsQYm5ZpJLwryhvOrHtIeuBWyn1tKFmOgRvy5dq+8YB6/ShItP0v68BPUaPhf6QlVrNYFygpM3o05MLwqxbOSxaE62YGXWUXlhpIBBXF7A3/U5zsvCb2qvZx1S+HNsDtU7s6RSqt4LtqSB/wiScIKTg58H3MisdtiK5XuQ+tqwBjz4LRZoyeKfhwcb7Hfd4CRWRPQNtfVzBPDI2/YYsADesBMvpXoITmolsCsR+zCBOwCO/zyDJFgGL9060g0+QmFAHLxAIsEs8KsHvOsNr1N78eWv2rwabS1zXMcfNX0ir65oTSfBwqvPaBUql0ccXL1UYvFBnJgkiOpYHrJD6V0f8Ns3WB4+V+SO2vlncqpgdsqTas3schkpfFiZgJxlzZ8Ch13Zs5KcC9RU51btOoDIdcuiSUN4RoTc4uX2RltSM4x7EatpO3ZDUGxIokyhLFdvCKfImF7gjnWeueOzt0mnxodbynixa870etyfHwW8yx3VtKsiVKjtfMu3UT2xHc68WIV/5w80ih1DuN2Q3e7/1LLlsvmqXBLZMq36Q+mYAJsUvZGy/7gEX2g7GSdkUj4uWm8/jqbLL6qpOBLoN6lxnwPrk3lsy4Gus3LBICX2+G+eSgz7Lj+lD5QLNXgfEsiqnZqpfaJdgFqaGlHw9V04Ej6xClAzehzdQYgX0HLFVCeBKElYTmJ1bOIPXgAfe+hVSjjitryujYMrL7HpnBM33OnKr8cbpg7kNcxgU+wcxD659RBLEOPv3+VlP0udHdDBI1LwO5z/7iCNpn14MeXWyPArWeMKPXqVKH/eYvLPOQa48WojrxNuYp6vD1gH8+gl8dYLOl0oUKZ3ClBvhmOVmWYWLwgtcdFmcHpgU+TPbtiF2fST5ceWyyU7sh5jEgFo2mF5zDS5upgLyAxye7AoZWan5cBlCNF3foa5uLW+0KaC2MPs9tjCYIWqa8RI88hlmSs1rV0b16NufOshADuIj4nP185unSkx560T1vE4zJ3HuKUnl5hCYVXDIsx4h+p1Bw0FGOzfI+jvBFQ499/fRMB00QMZfXzKgyINnoVxOJW5HDAAKsRtytAndoFIkZxMQTJUtxBoPx3Vj4wyO2DupNxPSwD5su//eZhiCpKg==
  • Spamdiagnosticmetadata: NSPM
  • Spamdiagnosticoutput: 1:99

I have started putting the loader through some testing and I have observed a
few "partial load error" conditions.

An example:

A job runs every 35 minutes, and syncs some data (a single group on the order
of 300k members) from an LDAP_GROUPS_FROM_ATTRIBUTES job.
Sometimes the job takes as little as about 15 minutes to run. (and
appears to do all of the right things)
Other times the job takes much longer. Sometimes over 3 hours. ( and
does not do all of the right things)

Sometimes the job add/removes a few people as expected.
Other times the job will remove 1/3 of the population and then add
them back in on the next one, and sometime two, runs.
The data in the LDAP source is not changing this drastically
over this time period.

I have not yet found the right knob to detect the "LDAP search/session/handle
failed" during the loader process.
And this churn is causing pain at the RDBMS layer due to large, and
unnecessary, volumes of audit and change log activities.

I think the only reasonable interpretation of the data that I am seeing is
that the read of the LDAP data is failing at some point. Or maybe the loader
is failing to "record" all of the data it gets back from the LDAP data set.
I am not clear as to how the "diff" process works in the loader.

Does it get a set of data from LDAP, store it in a temp table in
grouper, then diff the memberships with SQL?
Does it get a set of data from LDAP, get a set of data from grouper
and diff it "in memory" then update grouper?
Does it do something else?
( The inner workings of the loader are opaque, at this moment, to me.
I have not yet really dug into that code....)

If it helps, I attached is a pdf file with some results of running this job
over time.

Carey Matthew

-----Original Message-----

On Behalf Of Hyzer, Chris
Sent: Wednesday, September 13, 2017 9:14 AM
To: Emilio Recio
Grouper Users Mailing List
Subject: [grouper-users] RE: grouper-loader fail safe / threshold for failed

If there is an error running the query, the loader will not remove all
members. Only if the database successfully returns no records. Are you sure
that is what happened?

-----Original Message-----

On Behalf Of Emilio Recio
Sent: Wednesday, September 13, 2017 7:20 AM
To: Grouper Users Mailing List
Subject: [grouper-users] grouper-loader fail safe / threshold for failed

So we had a network outage, and the connection between grouper-loader and our
oracle database failed. This caused zero entries to be returned when loader
ran. When pspng provisioned out to our ldap, it undid the attributes for the
people we expected from the back end oracle database.

Is there a way I can tell grouper that if grouper-loader experiences a
*connection* problem with the back end database, do not do anything; just
exit with an appropriate rc? Is there a way I can kick off a script before to
check for connectivity that stops the loader process based on its rc?

I've seen the threshold value, but don't know if it would make sense. We
expected only 40 peoples' data elements from the back end database. Say I put
the threshold to 40, then as far as I've seen in the docs, reaching this just
drops it into fullsynch mode. How does fullsynch mode know to apply a
failsafe? Is there a property for that?


E. Recio
The information contained in this transmission contains privileged and
confidential information. It is intended only for the use of the person named
above. If you are not the intended recipient, you are hereby notified that
any review, dissemination, distribution or duplication of this communication
is strictly prohibited. If you are not the intended recipient, please contact
the sender by reply email and destroy all copies of the original message.

CAUTION: Intended recipients should NOT use email communication for emergent
or urgent health care matters.

Attachment: 2017-09-13.Loader_times.pdf
Description: 2017-09-13.Loader_times.pdf

Archive powered by MHonArc 2.6.19.

Top of Page