| Tom, I have to agree with Chris. I tweaked the ldappcng standard configuration to get the provisioning to work for our instance of grouper and ldap but it was mostly through trial and error.
Lynn On Sep 20, 2011, at 4:24 PM, Chris Hyzer wrote: People familiar with ldappng and shibboleth probably understand it better than I do, I would need a lot more comments to understand what it means :) Thanks, Chris -----Original Message----- From: [mailto:] On Behalf Of Tom Zeller Sent: Tuesday, September 20, 2011 12:59 PM To: Grouper Dev Subject: Re: [grouper-dev] ldapp incremental Reviewing the work I had previously done regarding incremental provisioning, I am curious if there are opinions on the attribute resolver configuration I mocked up. Is it ok ? horrible ? https://spaces.internet2.edu/display/Grouper/Incremental+Provisioning+DevelopmentOn Fri, Sep 16, 2011 at 3:15 PM, LYNN GARRISON <> wrote: Chris, Tom,
The first scenario would work for Penn State for incremental provisioning. Our goal is to have ldap sourced only from grouper. We typically create our course groups at the beginning of every semester and post incremental changes daily. So our massive group deletes/add don't occur that often. Generally they are scheduled for the weekend so I think we could deal with lots of transactions in the queue.
Lynn Garrison
----- Original Message -----
From: "Chris Hyzer" <>
To: "Tom Zeller" <>, "JAMES VUCCOLO" <>
Cc: "Grouper Dev" <>
Sent: Friday, September 16, 2011 3:05:35 PM
Subject: RE: [grouper-dev] ldapp incremental
Are we tuning a full sync, or looking into incremental?
Ie. For incremental, it could be as easy as:
1. member is added to a group in grouper. (this could be manually or via loader, or rule, or WS, or whatever). Note the member might be in 100 groups, and the group might have 100k members
2. the change log decides if this is a flattened change. Ie. If the member was in the group already e.g. via another path, then it will not send an event. If the user was not previously in the group by any path, and is now, then the change log event is sent. Note, this is already how it works, no code needed here.
3. if ldap is sourced only from grouper, then you know the member was not already in the group, just do the ldap operations to resolve the user and group, add the user to the hasMembers of the group object, and add the group to the memberOf of the user object.
Or, if you want it to be like the iamonline one, it gets to #3, and:
3. <alternate> get the change from the change log notification, then check a grouper hasMember to see if the user is a member, do an LDAP query to see if the group is in the memberOf and/or see if the user is in the hasMembers, and adjust if not consistent, but not by replacing the full member list, just by adding an attribute value or removing.
In neither of these approaches are we getting or replacing the full multi value attributes...
Note, that this approach is not good when you add a huge group to another group, or if you change memberships - delete the group - recreate the group - etc all in a matter of seconds. Also, if there are a ton of changes in the queue, which will happen from time to time but should be infrequent or off hours, then there will be more delay than 1 minute. But as we said, it is not perfect, but it handles the steady state requirements... right? Maybe down the line we can make it more efficient for these types of things...
The iamonline logic is very simple, seen here:
http://anonsvn.internet2.edu/cgi-bin/viewvc.cgi/i2mi/trunk/grouper-misc/poc_secureUserData/src/edu/internet2/middleware/poc_secureUserData/SudRealTime.java?revision=7553&view=co
Note, you would be using the grouper api, not WS, but same principle:
/**
* sync a group and user in incremental fashion
* @param groupExtension
* @param subjectId
*/
public static void syncGroupAndUser(String groupExtension, String subjectId) {
//get the memberships from Grouper
String rowGroupsFolder = GrouperClientUtils.propertiesValue("sud.rootFolder", true) + ":rowGroups";
//is there a member?
WsHasMemberResults wsHasMemberResults = new GcHasMember().addSubjectLookup(new WsSubjectLookup(subjectId, "jdbc", null))
.assignGroupName(rowGroupsFolder + ":" + groupExtension).execute();
boolean hasGrouperMembership = StringUtils.equals("IS_MEMBER",
wsHasMemberResults.getResults()[0].getResultMetadata().getResultCode());
//add all from DB
Set<String> groupExtensionsFromDb = new TreeSet<String>(GcDbUtils.listSelect(String.class,
"select distinct group_extension from secureuserdata_memberships where personid = ? and group_extension = ?",
GrouperClientUtils.toList(DbType.STRING), GrouperClientUtils.toList((Object)subjectId, groupExtension)));
boolean hasDbMembership = groupExtensionsFromDb.contains(groupExtension);
if (hasGrouperMembership && !hasDbMembership) {
SudMembership sudMembership = new SudMembership();
sudMembership.setGroupExtension(groupExtension);
sudMembership.setPersonid(subjectId);
sudMembership.store();
if (LOG.isInfoEnabled()) {
LOG.info("Add mship for group: "
+ groupExtension + ", personid: " + subjectId);
}
} else if (!hasGrouperMembership && hasDbMembership) {
//delete by query in case multiple
int rows = GcDbUtils.executeUpdate("delete from secureuserdata_memberships where group_extension = ? and personid = ?",
GrouperClientUtils.toList((Object)groupExtension, subjectId));
if (LOG.isInfoEnabled()) {
LOG.info("Del " + rows + " mships of group: "
+ groupExtension + ", personid: " + subjectId);
}
}
}
Thanks,
Chris
-----Original Message-----
From: [mailto:] On Behalf Of Tom Zeller
Sent: Friday, September 16, 2011 2:20 PM
To: JAMES VUCCOLO
Cc: Grouper Dev
Subject: Re: [grouper-dev] ldapp incremental
By profiling, I confirmed to myself that the bottleneck when
provisioning a group's memberships via ldappcng is typically IO, e.g.
ldap searches.
Before revisiting incremental provisioning from where I left off
before I changed jobs, I thought I should take a fresh look at
reducing the number of ldap searches.
On Wednesday, I realized that in a typical grouper installation with
an ldap subject source, we were searching ldap twice for every
membership. One search was performed by the grouper subject finder and
a second by the ldappcng ldap target. So, I added a shibboleth data
connector which wraps the grouper subject finder and the number of
ldap searches were reduced by half, great.
Tuning the grouper subject cache large enough to hold all subjects
resulted in some nice times to provision large groups after the first
run. For a group with 50k members in my test environment, the first
provisioning required ~30s, most of which was ldap searches.
Subsequent provisionings, because of caching, were ~5s.
So, with appropriate ehcache settings and by merging the subject ldap
adapter and target ldap adapter, member identifier resolution
performance is improved. Not exactly RTP (real-time provisioning), but
desirable regardless.
Next on my list is to look at caching target state, meaning, cache the
representation of a group as it is represented on a target, e.g. cache
ldap group entries. This is the other kind of ldap search bottleneck
after member identifier resolution.
Sizing questions for Penn State : are your groups currently in ldap ?
if so, what is the size of your DIT ? if you exported ou=people and
ou=groups to an ldif file, what is that file size (uncompressed) ?
Hopefully RTP won't be my SLO :-)
On Thu, Sep 15, 2011 at 1:56 PM, JAMES VUCCOLO <> wrote:
----- Original Message -----
From: "Chris Hyzer" <>
To: "Grouper Dev" <>
Sent: Thursday, September 15, 2011 2:00:25 PM
Subject: [grouper-dev] ldapp incremental
TomZ,
Do you mind adding sections to the grouper incremental provisioning
doc from 4 months ago:
https://spaces.internet2.edu/display/Grouper/Incremental+Provisioning+Development
Including the use cases supported, the use cases not supported (since
we don't want it to be "perfect"), and how the changes will
propagate from the change log to ldap (i.e. which type of operation
will occur, what is being cached, etc). Then Penn State and others
can review.
We are very interested in hearing about this too, definitely what operations will happen in the directory. Listening in on the PACCMAN call today, I am not sure the caching described will solve our problems. Our users expect that when they make a change to a group in a small period of time (did not want to use the word real-time), will see that change in LDAP. Most if not all of our applications today go against the directory and probably in the future will continue to do so. Our hope is for the incremental provisioning to LDAP as opposed to what exists in LDAPPCNG. We have a large number of groups > 40K and a number of them have memberships in the 30K vicinity.
Thanks JimmyV
Thanks,
Chris
--
James "Jimmy" Vuccolo,
Technical Manager, Identity and Access Management
The Pennsylvania State University
215B Computer Building, University Park, PA 16802
Office: 814-865-5635
http://www.personal.psu.edu/jvuccolo/
Lynn Garrison IAM Technical Architect Group 215A Computer Building University Park, PA 16802 (814) 865-4864
|
