Grouper Users - Open Discussion List

["Redman, Chad" <>]

You could turn up the LDAP debugging in log4j.properties:

log4j.logger.edu.vt.middleware.ldap = DEBUG  (or DEBUG, your-appender-name, 
etc.)

You could even turn it up to TRACE, which some of the LDAP classes may 
utilize.

-Chad




-----Original Message-----
From: 

 
[mailto:]
 On Behalf Of Black, Carey M.
Sent: Thursday, September 14, 2017 3:19 PM
To: grouper-users 
<>
Cc: Emilio Recio 
<>;
 Waldbieser, Carl 
<>;
 Chris Hyzer 
<>
Subject: RE: [grouper-users] RE: grouper-loader fail safe / threshold for 
failed connections

The ldap service, in my case, can return all rows for the filter that I am 
using.
        I can do the search with another ldap client (Apache Directory 
Studio) and it consistently works.

The filter is a "simple" one. Or's of discrete values to get the set for the 
loader job.
        (|(Attr=A) (Attr=B) (Attr=Z) (Attr=U) (Attr=Q)...) 
        Each value becomes a group in grouper.

        Some of the single values are, however, 300k in size. So those run in 
a job by themselves.


I will be splitting the total size for the jobs down to the 30k size members 
and see if I can get all of those to run consistently.
        If that works,  then that still leaves a few values (2?) to "fix" 
some other way.
        If the magic number is <20k, then that leaves 4 values.

        I am thinking of finding ways to subdivide the larger group by... 
First letter (or two) of last name?
                Load 26 ( or more) smaller sets then "add them together" in 
grouper.

But the part the bothers me is the lack of error msg. That is unsettling.

-- 
Carey Matthew 



-----Original Message-----
From: Waldbieser, Carl 
[mailto:]
 
Sent: Thursday, September 14, 2017 2:27 PM
To: Chris Hyzer 
<>
Cc: Black, Carey M. 
<>;
 grouper-users 
<>;
 Emilio Recio 
<>
Subject: Re: [grouper-users] RE: grouper-loader fail safe / threshold for 
failed connections


Do the symptoms happen regardless of the client used?
E.g. if you run `ldapsearch` with the filter 100x, do the results show a 
similar pattern of incompleteness?  Keeping all other things constant (e.g. 
BIND user, base DN, etc.).

Could be a bug in the client.

Anything odd about the filter?

Thanks,
Carl Waldbieser
ITS Systems Programmer
Lafayette College

----- Original Message -----
From: "Chris Hyzer" 
<>
To: "waldbiec" 
<>
Cc: "Carey M. Black" 
<>,
 "grouper-users" 
<>,
 "Emilio Recio" 
<>
Sent: Thursday, September 14, 2017 2:14:21 PM
Subject: RE: [grouper-users] RE: grouper-loader fail safe / threshold for 
failed connections

Sometimes all, sometimes most, not a consistent number of results returned 
when its most...

-----Original Message-----
From: Waldbieser, Carl 
[mailto:]
 
Sent: Thursday, September 14, 2017 2:13 PM
To: Hyzer, Chris 
<>
Cc: Black, Carey M. 
<>;
 grouper-users 
<>;
 Emilio Recio 
<>
Subject: Re: [grouper-users] RE: grouper-loader fail safe / threshold for 
failed connections


I believe some directories have limits on how many entries they will return.

For example:  http://www.openldap.org/doc/admin24/limits.html

Thanks,
Carl Waldbieser
ITS Systems Programmer
Lafayette College

----- Original Message -----
From: "Chris Hyzer" 
<>
To: "Black, Carey M." 
<>,
 "grouper-users" 
<>
Cc: "Emilio Recio" 
<>
Sent: Thursday, September 14, 2017 10:06:53 AM
Subject: [grouper-users] RE: grouper-loader fail safe / threshold for failed 
connections

Btw, it takes that long since it is doing so much work on the memberships.

Does anyone know why a search filter in ldap would return successfully and 
not return all data (~2/3)?

Thanks
Chris

-----Original Message-----
From: Black, Carey M. 
[mailto:]
 
Sent: Wednesday, September 13, 2017 3:56 PM
To: Grouper Users Mailing List 
<>
Cc: Hyzer, Chris 
<>;
 Emilio Recio 
<>
Subject: RE: grouper-loader fail safe / threshold for failed connections

I have started putting  the loader through some testing and I have observed a 
few "partial load error" conditions.

An example:

A job runs every 35 minutes, and syncs some data (a single group on the order 
of 300k members) from an LDAP_GROUPS_FROM_ATTRIBUTES job.
        Sometimes the job takes as little as about 15 minutes to run. (and 
appears to do all of the right things)
        Other times the job takes much longer. Sometimes over 3 hours. ( and 
does not do all of the right things)

        Sometimes the job add/removes a few people as expected.
        Other times the job will remove 1/3 of the population and then add 
them back in on the next one, and sometime two, runs. 
                The data in the LDAP source is not changing this drastically 
over this time period.

I have not yet found the right knob to detect the "LDAP search/session/handle 
failed" during the loader process.
And this churn is causing pain at the RDBMS layer due to large, and 
unnecessary, volumes of audit and change log activities.


I think the only reasonable interpretation of the data that I am seeing is 
that the read of the LDAP data is failing at some point. Or maybe the loader 
is failing to "record" all of the data it gets back from the LDAP data set. 
        I am not clear as to how the "diff" process works in the loader.

        Does it get a set of data from LDAP, store it in a temp table in 
grouper, then diff the memberships with SQL?
        Does it get a set of data from LDAP, get a set of data from grouper 
and diff it "in memory" then update grouper?
        Does it do something else?
        ( The inner workings of the loader are opaque, at this moment, to me. 
I have not yet really dug into that code....)

If it helps, I attached is a pdf file with some results of running this job 
over time.

-- 
Carey Matthew 


-----Original Message-----
From: 

 
[mailto:]
 On Behalf Of Hyzer, Chris
Sent: Wednesday, September 13, 2017 9:14 AM
To: Emilio Recio 
<>;
 Grouper Users Mailing List 
<>
Subject: [grouper-users] RE: grouper-loader fail safe / threshold for failed 
connections

If there is an error running the query, the loader will not remove all 
members.  Only if the database successfully returns no records.  Are you sure 
that is what happened?

-----Original Message-----
From: 

 
[mailto:]
 On Behalf Of Emilio Recio
Sent: Wednesday, September 13, 2017 7:20 AM
To: Grouper Users Mailing List 
<>
Subject: [grouper-users] grouper-loader fail safe / threshold for failed 
connections

So we had a network outage, and the connection between grouper-loader and our 
oracle database failed. This caused zero entries to be returned when loader 
ran. When pspng provisioned out to our ldap, it undid the attributes for the 
people we expected from the back end oracle database.

Is there a way I can tell grouper that if grouper-loader experiences a
*connection* problem with the back end database, do not do anything; just 
exit with an appropriate rc? Is there a way I can kick off a script before to 
check for connectivity that stops the loader process based on its rc?

I've seen the threshold value, but don't know if it would make sense. We 
expected only 40 peoples' data elements from the back end database. Say I put 
the threshold to 40, then as far as I've seen in the docs, reaching this just 
drops it into fullsynch mode. How does fullsynch mode know to apply a 
failsafe? Is there a property for that?

--

Thanks,
E. Recio
The information contained in this transmission contains privileged and 
confidential information. It is intended only for the use of the person named 
above. If you are not the intended recipient, you are hereby notified that 
any review, dissemination, distribution or duplication of this communication 
is strictly prohibited. If you are not the intended recipient, please contact 
the sender by reply email and destroy all copies of the original message.

CAUTION: Intended recipients should NOT use email communication for emergent 
or urgent health care matters.