Skip to Content.
Sympa Menu

grouper-users - [grouper-users] RE: Grouper Loader

Subject: Grouper Users - Open Discussion List

List archive

[grouper-users] RE: Grouper Loader

Chronological Thread 
  • From: Chris Hyzer <>
  • To: "Klug, Lawrence" <>, "" <>
  • Subject: [grouper-users] RE: Grouper Loader
  • Date: Sat, 11 Feb 2012 20:04:10 +0000
  • Accept-language: en-US

Ugh… yeah, I guess that patch wont work.  The deltas assume the col is subject_id… so…  any chance you can wait a couple of weeks for 2.1?  J






From: Klug, Lawrence [mailto:]
Sent: Friday, February 10, 2012 4:46 PM
To: Chris Hyzer;
Subject: RE: Grouper Loader




Okay, I patched the code and the loader was able to get the subjects loaded with uclaPPID.  It is very slow.  It also removed  a membership that I entered manually. 


gsh 2% loaderRunOneJob(group);

loader ran successfully, inserted 47441 memberships, deleted 47427 memberships, total membership

count: 66610


This was the second run.  Why did it delete and recreate so many memberships?  I thought it was using deltas for the process.







From: Chris Hyzer [mailto:]
Sent: Thursday, January 26, 2012 1:25 PM
To: Klug, Lawrence;
Subject: RE: Grouper Loader


Im bringing this back to the list to share with others.


Basically the problem is that your loader query is returning uclaUniversityId, and that is not the subject_id, the subject_id is uclaPPID.


It is *very * inefficient to do it this way… since each time the loader runs a query, it will have to resolve all those subjects (LDAP lookup for each membership).  If you can have the query join to a lookup table to resolve to the uclaPPID, then you wont have to do lookups since the uclaPPID is the subjectId in the grouper_members table.  This is the best way to go.


In Grouper 2.1 you can use a findByIdentifier() to resolve loader subjects with a subject_identifier column, but still it is inefficient.  If you want to go down the uclaUniversityId route pre-2.1, we can patch the code… its right here in




          this.subject = SubjectFinder.getSource(subjectSourceId).getSubject(subjectId, true);

          //CH 20091013: we need the loader to be based on subjectId to eliminate lookups...

          //this.subject = SubjectFinder.getSource(subjectSourceId).getSubject(subjectId, false);

          //if (this.subject == null) {

          //  this.subject = SubjectFinder.getSource(subjectSourceId).getSubjectByIdentifier(subjectId, true);





          this.subject = SubjectFinder.getSource(subjectSourceId).getSubject(subjectId, false);

          if (this.subject == null) {

            this.subject = SubjectFinder.getSource(subjectSourceId).getSubjectByIdentifier(subjectId, true);







From: Klug, Lawrence
Sent: Thursday, January 26, 2012 10:15 AM
To: Chris Hyzer
Subject: RE: Grouper Loader


Here is the output.  Yes, searchSubject is configured to use uclaPPID and that’s incompatible with uclaUniversityId – so what is the proper configuration to  solve the issue?  I’ve tried the things that seem obvious.






Gsh cmd:


gsh 0%  grouperSession = GrouperSession.startRootSession();

edu.internet2.middleware.grouper.GrouperSession: e2d999891ac14e68b5a031bebdf813d6,'GrouperSystem


gsh 1% SubjectFinder.findByIdAndSource("6032xxxxxx", "ldap", true);

// Error: unable to evaluate command: Sourced file: inline evaluation of: ``SubjectFinder.findBy

IdAndSource("6032xxxxxx", "ldap", true);'' : Method Invocation SubjectFinder.findByIdAndSource

// See error log for full stacktrace

// caused by: edu.internet2.middleware.subject.SubjectNotFoundException:

// No results: searchSubject filter:(& (uclaPPID=%TERM%)(objectclass=person)) searchValue: 6032xxxxxx

gsh 2%


error log:


2012-01-26 07:04:23,485: [main] INFO -  - [ef2992efb8354d26ba4b25d6f046ddc6,'GrouperSystem','application'] add group type: 'grouperLoader' (2ms)

2012-01-26 07:04:41,461: [main] INFO -  - [e2d999891ac14e68b5a031bebdf813d6,'GrouperSystem','application'] session: start (8ms)

2012-01-26 07:04:55,838: [main] ERROR ShellHelper.eval(76) -  - Sourced file: inline evaluation of: ``SubjectFinder.findByIdAndSource("6032xxxxxx", "ldap", true);'' : Method Invocation SubjectFinder.findByIdAndSource

edu.internet2.middleware.subject.SubjectNotFoundException: No results: searchSubject filter:(& (uclaPPID=%TERM%)(objectclass=person)) searchValue: 6032xxxxxx

     at edu.internet2.middleware.subject.provider.JNDISourceAdapter.getLdapUnique(

     at edu.internet2.middleware.subject.provider.JNDISourceAdapter.getSubject(

     at edu.internet2.middleware.grouper.subj.SourcesXmlResolver.find(

     at edu.internet2.middleware.grouper.subj.CachingResolver.find(

     at edu.internet2.middleware.grouper.subj.ValidatingResolver.find(

     at edu.internet2.middleware.grouper.SubjectFinder.findByIdAndSource(

     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

     at sun.reflect.NativeMethodAccessorImpl.invoke(

     at sun.reflect.DelegatingMethodAccessorImpl.invoke(

     at java.lang.reflect.Method.invoke(

     at bsh.Reflect.invokeMethod(Unknown Source)

     at bsh.Reflect.invokeStaticMethod(Unknown Source)

     at bsh.Name.invokeMethod(Unknown Source)

     at bsh.BSHMethodInvocation.eval(Unknown Source)

     at bsh.BSHPrimaryExpression.eval(Unknown Source)

     at bsh.BSHPrimaryExpression.eval(Unknown Source)

     at bsh.Interpreter.eval(Unknown Source)

     at bsh.Interpreter.eval(Unknown Source)

     at bsh.Interpreter.eval(Unknown Source)












From: Chris Hyzer
Sent: Wednesday, January 25, 2012 9:11 PM
To: Klug, Lawrence
Subject: RE: Grouper Loader


What happens when you run this in gsh?


      grouperSession = GrouperSession.startRootSession();

      SubjectFinder.findByIdAndSource("6032xxxxxx", "ldap", true);





Archive powered by MHonArc 2.6.16.

Top of Page