Grouper Users - Open Discussion List

[Tom Barton <>]

I think I agree too, finally. Though we will face new problems when
grouper-resident subject data is not in sync with authoritative subject
data. Perhaps having new problems is progress...

Tom

On 10/3/2010 10:41 PM, Chris Hyzer wrote:
> I think we are agreeing... the only difference is whether grouper does the 
> caching/aggregation or not.  And I think there are more advantages for 
> Grouper doing it than for institutions to have to do it.  Right?
> 
> Thanks,
> Chris
> 
> -----Original Message-----
> From: Jim Fox 
> [mailto:]
>  
> Sent: Sunday, October 03, 2010 11:35 PM
> To: Chris Hyzer
> Cc: Jim Fox; GW Brown, Information Systems and Computing; Peter DiCamillo; 
> 
> Subject: Re: [grouper-users] Grouper 1.6.1 UI membership list sorting
> 
> 
> My point is this:  There are certain operations in grouper that are 
> intolerably slow.  This is a common and valid complaint.  Most of these 
> situations would be solved, or at least mitigated, by a local subject 
> source.  I think this is a fundamental flaw of grouper.  I think it is a 
> necessary correction.  
> 
> From an external point of view there can be several subject sources.  At UW 
> we have at least four.  However, when cached they can very easily be 
> represented as a single source --- at least as a single table, obviating a 
> lot of the database unions you speak of.  This is our own experience. 
> 
> Imagine how much more efficient grouper might be if you could assume a 
> local subject source.
> 
> Jim
> 
> 
> On Oct 3, 2010, at 7:48 PM, Chris Hyzer wrote:
> 
>> I think another common use case is finding a subject in a group.  i.e. we 
>> have a subject picker that wants to do a subject search (in one source) 
>> where the subject is an employee.  This takes a long time since it has to 
>> do the search of 500k subjects, then for the results, see which ones are 
>> employees (which is batched, but still takes a while).  Since I am a jdbc 
>> source guy, I would agree that adding more capabilities into the subject 
>> source would be nice (could solve this problem).  However, when 
>> searching/sorting over multiple sources, it seems more difficult even if 
>> the all the sources are jdbc ones... so that is why keeping a sort string 
>> in the member table (for all subjects in all sources which are members of 
>> groups) seems like a good thing to do.  If we kept a sort string in the 
>> member table, then we could also keep a lowered search string, then we 
>> only need to do that instead of give the JDBC sources extra features over 
>> JNDI.  And if we do that, we might as well k
eep a name/description so if they become unresolvable, we can still show who 
they are (were).  Though I don't see keeping other attributes in there (full 
replication) :)
>>
>> Thanks,
>> Chris
>>
>> -----Original Message-----
>> From: Jim Fox 
>> [mailto:]
>>  
>> Sent: Sunday, October 03, 2010 12:20 PM
>> To: GW Brown, Information Systems and Computing
>> Cc: Chris Hyzer; Peter DiCamillo; 
>> 
>> Subject: Re: [grouper-users] Grouper 1.6.1 UI membership list sorting
>>
>>
>>
>> It is for this reason more than any other that we cache all our subjects 
>> in tables in grouper's database.  We can use simple database joins to 
>> allow the DBMS do all the selecting and sorting, by subject id or name or 
>> whatever.  
>>
>> It seems obvious enough that caching a subject sort string is not much 
>> easier than caching the entire subject record.  Possibly version 2.0 could 
>> require, or at least optimize for, local caches of subjects.   
>>
>> Jim
>>
>> On Oct 3, 2010, at 3:49 AM, GW Brown, Information Systems and Computing 
>> wrote:
>>
>>> The problem is that the API is now doing the paging - but it has no means 
>>> to sort the data because, as Chris says, we don't store subject 
>>> attributes. 
>>> API paging was added because even without sorting, processing very large 
>>> memberships would bring the UI to a grinding halt. So, unfortunately the 
>>> comparator.sort.limit has become redundant.
>>>
>>> Going forward I think there are a coupe of options.
>>>
>>> 1) The UI checks how many results there will be and if under 
>>> comparator.sort.limit sets this as the pagesize for the API.
>>> 2) The API stores a sort String for subjects so it can add an order by 
>>> clause to its query. Some mechanism would be needed to refresh sort 
>>> Strings.
>>>
>>> I can have a look at option 1, but we have talked about option 2 before 
>>> so 
>>> I'll wait and see what the consensus is. Either way it is likely to be 
>>> 2.0 
>>> before we resolve it.
>>>
>>> Gary
>>>
>>> --On 03 October 2010 01:59 -0400 Chris Hyzer 
>>> <>
>>>  wrote:
>>>
>>>> I can reproduce that.  The problem is that subject data is stored
>>>> externally to Grouper (in sources), so if there are 50000 subjects in a
>>>> group, that is 50000 subject lookups (assuming none in cache), which
>>>> doesn't scale.  Granted if you had 356 with 250 on a page, that should
>>>> sort.  :)   What use case do you have?  Trying to see if someone is in a
>>>> group?  Just curious, what is the workaround?  Anyways, Im sure Gary will
>>>> weigh in here as well...
>>>>
>>>> Thanks,
>>>> Chris
>>>>
>>>> -----Original Message-----
>>>> From: 
>>>> 
>>>> [mailto:]
>>>>  On Behalf Of Peter DiCamillo
>>>> Sent: Saturday, October 02, 2010 11:14 PM
>>>> To: 
>>>> 
>>>> Subject: [grouper-users] Grouper 1.6.1 UI membership list sorting
>>>>
>>>> I'm having a problem where membership lists I view in the full Grouper
>>>> UI are not sorted as I would expect.  When the size of the membership
>>>> list does not exceed the page size the list is sorted.  But when more
>>>> than one page is needed, the list on each page is not sorted.  The value
>>>> of comparator.sort.limit in media.properties is 50000, larger than the
>>>> size of any of our groups, because even if it's slow, we always want
>>>> sorted lists.
>>>>
>>>> For example, I viewed a list with 2 direct members and 356 indirect
>>>> members.  When I viewed the indirect members or all members with a page
>>>> size of 500 the list was sorted.  But with a page size of 250 the lists
>>>> on the pages were not sorted.
>>>>
>>>> I didn't install Grouper 1.3, but we may have had a local fix for this.
>>>> But in any case, it appears that since 1.3 the code that for handling
>>>> pages and sorting has changed
>>>>
>>>> Peter
>>>>
>>>
>>>
>>>
>>> ----------------------
>>> GW Brown, Information Systems and Computing
>>> 
>>
>