Grouper Users - Open Discussion List

["Klug, Lawrence" <>]

>>In Grouper 2.1 you can use a findByIdentifier() to resolve loader subjects with a subject_identifier column, but still it is inefficient.

 

I wonder if the 2.1 loader will be much of an improvement?

 

Thanks,

 

Lawrence

 

 

 

From: Chris Hyzer
Sent: Saturday, February 11, 2012 12:04 PM
To: Klug, Lawrence;
Subject: RE: Grouper Loader

 

Ugh… yeah, I guess that patch wont work.  The deltas assume the col is subject_id… so…  any chance you can wait a couple of weeks for 2.1?  J

 

Thanks,

Chris

 

 

From: Klug, Lawrence
Sent: Friday, February 10, 2012 4:46 PM
To: Chris Hyzer;
Subject: RE: Grouper Loader

 

Chris,

 

Okay, I patched the code and the loader was able to get the subjects loaded with uclaPPID.  It is very slow.  It also removed  a membership that I entered manually. 

 

gsh 2% loaderRunOneJob(group);

loader ran successfully, inserted 47441 memberships, deleted 47427 memberships, total membership

count: 66610

 

This was the second run.  Why did it delete and recreate so many memberships?  I thought it was using deltas for the process.

 

Thanks,

 

Lawrence

 

 

From: Chris Hyzer
Sent: Thursday, January 26, 2012 1:25 PM
To: Klug, Lawrence;
Subject: RE: Grouper Loader

 

Im bringing this back to the list to share with others.

 

Basically the problem is that your loader query is returning uclaUniversityId, and that is not the subject_id, the subject_id is uclaPPID.

 

It is *very * inefficient to do it this way… since each time the loader runs a query, it will have to resolve all those subjects (LDAP lookup for each membership).  If you can have the query join to a lookup table to resolve to the uclaPPID, then you wont have to do lookups since the uclaPPID is the subjectId in the grouper_members table.  This is the best way to go.

 

In Grouper 2.1 you can use a findByIdentifier() to resolve loader subjects with a subject_identifier column, but still it is inefficient.  If you want to go down the uclaUniversityId route pre-2.1, we can patch the code… its right here in GrouperLoaderResultset.java:

 

FROM:

 

          this.subject = SubjectFinder.getSource(subjectSourceId).getSubject(subjectId, true);

          //CH 20091013: we need the loader to be based on subjectId to eliminate lookups...

          //this.subject = SubjectFinder.getSource(subjectSourceId).getSubject(subjectId, false);

          //if (this.subject == null) {

          //  this.subject = SubjectFinder.getSource(subjectSourceId).getSubjectByIdentifier(subjectId, true);

          //}

 

TO:

 

          this.subject = SubjectFinder.getSource(subjectSourceId).getSubject(subjectId, false);

          if (this.subject == null) {

            this.subject = SubjectFinder.getSource(subjectSourceId).getSubjectByIdentifier(subjectId, true);

          }

 

Thanks,

Chris

 

 

From: Klug, Lawrence
Sent: Thursday, January 26, 2012 10:15 AM
To: Chris Hyzer
Subject: RE: Grouper Loader

 

Here is the output.  Yes, searchSubject is configured to use uclaPPID and that’s incompatible with uclaUniversityId – so what is the proper configuration to  solve the issue?  I’ve tried the things that seem obvious.

 

Thanks,

 

Lawrence

 

Gsh cmd:

 

gsh 0%  grouperSession = GrouperSession.startRootSession();

edu.internet2.middleware.grouper.GrouperSession: e2d999891ac14e68b5a031bebdf813d6,'GrouperSystem

','application'

gsh 1% SubjectFinder.findByIdAndSource("6032xxxxxx", "ldap", true);

// Error: unable to evaluate command: Sourced file: inline evaluation of: ``SubjectFinder.findBy

IdAndSource("6032xxxxxx", "ldap", true);'' : Method Invocation SubjectFinder.findByIdAndSource

// See error log for full stacktrace

// caused by: edu.internet2.middleware.subject.SubjectNotFoundException:

// No results: searchSubject filter:(& (uclaPPID=%TERM%)(objectclass=person)) searchValue: 6032xxxxxx

gsh 2%

 

error log:

 

2012-01-26 07:04:23,485: [main] INFO  EventLog.info(141) -  - [ef2992efb8354d26ba4b25d6f046ddc6,'GrouperSystem','application'] add group type: 'grouperLoader' (2ms)

2012-01-26 07:04:41,461: [main] INFO  EventLog.info(141) -  - [e2d999891ac14e68b5a031bebdf813d6,'GrouperSystem','application'] session: start (8ms)

2012-01-26 07:04:55,838: [main] ERROR ShellHelper.eval(76) -  - Sourced file: inline evaluation of: ``SubjectFinder.findByIdAndSource("6032xxxxxx", "ldap", true);'' : Method Invocation SubjectFinder.findByIdAndSource

edu.internet2.middleware.subject.SubjectNotFoundException: No results: searchSubject filter:(& (uclaPPID=%TERM%)(objectclass=person)) searchValue: 6032xxxxxx

     at edu.internet2.middleware.subject.provider.JNDISourceAdapter.getLdapUnique(JNDISourceAdapter.java:469)

     at edu.internet2.middleware.subject.provider.JNDISourceAdapter.getSubject(JNDISourceAdapter.java:132)

     at edu.internet2.middleware.grouper.subj.SourcesXmlResolver.find(SourcesXmlResolver.java:107)

     at edu.internet2.middleware.grouper.subj.CachingResolver.find(CachingResolver.java:117)

     at edu.internet2.middleware.grouper.subj.ValidatingResolver.find(ValidatingResolver.java:85)

     at edu.internet2.middleware.grouper.SubjectFinder.findByIdAndSource(SubjectFinder.java:235)

     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

     at java.lang.reflect.Method.invoke(Method.java:597)

     at bsh.Reflect.invokeMethod(Unknown Source)

     at bsh.Reflect.invokeStaticMethod(Unknown Source)

     at bsh.Name.invokeMethod(Unknown Source)

     at bsh.BSHMethodInvocation.eval(Unknown Source)

     at bsh.BSHPrimaryExpression.eval(Unknown Source)

     at bsh.BSHPrimaryExpression.eval(Unknown Source)

     at bsh.Interpreter.eval(Unknown Source)

     at bsh.Interpreter.eval(Unknown Source)

     at bsh.Interpreter.eval(Unknown Source)

     at edu.internet2.middleware.grouper.app.gsh.ShellHelper.eval(ShellHelper.java:48)

     at edu.internet2.middleware.grouper.app.gsh.GrouperShell.run(GrouperShell.java:409)

     at edu.internet2.middleware.grouper.app.gsh.GrouperShell.grouperShellHelper(GrouperShell.java:216)

     at edu.internet2.middleware.grouper.app.gsh.GrouperShell.main(GrouperShell.java:146)

     at edu.internet2.middleware.grouper.app.gsh.GrouperShellWrapper.main(GrouperShellWrapper.java:16)

 

 

 

 

 

 

From: Chris Hyzer
Sent: Wednesday, January 25, 2012 9:11 PM
To: Klug, Lawrence
Subject: RE: Grouper Loader

 

What happens when you run this in gsh?

 

      grouperSession = GrouperSession.startRootSession();

      SubjectFinder.findByIdAndSource("6032xxxxxx", "ldap", true);