Grouper Users - Open Discussion List

[Chris Hyzer <>]

I think we are agreeing... the only difference is whether grouper does the 
caching/aggregation or not.  And I think there are more advantages for 
Grouper doing it than for institutions to have to do it.  Right?

Thanks,
Chris

-----Original Message-----
From: Jim Fox 
[mailto:]
 
Sent: Sunday, October 03, 2010 11:35 PM
To: Chris Hyzer
Cc: Jim Fox; GW Brown, Information Systems and Computing; Peter DiCamillo; 

Subject: Re: [grouper-users] Grouper 1.6.1 UI membership list sorting


My point is this:  There are certain operations in grouper that are 
intolerably slow.  This is a common and valid complaint.  Most of these 
situations would be solved, or at least mitigated, by a local subject source. 
 I think this is a fundamental flaw of grouper.  I think it is a necessary 
correction.  

From an external point of view there can be several subject sources.  At UW 
we have at least four.  However, when cached they can very easily be 
represented as a single source --- at least as a single table, obviating a 
lot of the database unions you speak of.  This is our own experience. 

Imagine how much more efficient grouper might be if you could assume a local 
subject source.

Jim


On Oct 3, 2010, at 7:48 PM, Chris Hyzer wrote:

> I think another common use case is finding a subject in a group.  i.e. we 
> have a subject picker that wants to do a subject search (in one source) 
> where the subject is an employee.  This takes a long time since it has to 
> do the search of 500k subjects, then for the results, see which ones are 
> employees (which is batched, but still takes a while).  Since I am a jdbc 
> source guy, I would agree that adding more capabilities into the subject 
> source would be nice (could solve this problem).  However, when 
> searching/sorting over multiple sources, it seems more difficult even if 
> the all the sources are jdbc ones... so that is why keeping a sort string 
> in the member table (for all subjects in all sources which are members of 
> groups) seems like a good thing to do.  If we kept a sort string in the 
> member table, then we could also keep a lowered search string, then we only 
> need to do that instead of give the JDBC sources extra features over JNDI.  
> And if we do that, we might as well keep a name/description so if they 
> become unresolvable, we can still show who they are (were).  Though I don't 
> see keeping other attributes in there (full replication) :)
> 
> Thanks,
> Chris
> 
> -----Original Message-----
> From: Jim Fox 
> [mailto:]
>  
> Sent: Sunday, October 03, 2010 12:20 PM
> To: GW Brown, Information Systems and Computing
> Cc: Chris Hyzer; Peter DiCamillo; 
> 
> Subject: Re: [grouper-users] Grouper 1.6.1 UI membership list sorting
> 
> 
> 
> It is for this reason more than any other that we cache all our subjects in 
> tables in grouper's database.  We can use simple database joins to allow 
> the DBMS do all the selecting and sorting, by subject id or name or 
> whatever.  
> 
> It seems obvious enough that caching a subject sort string is not much 
> easier than caching the entire subject record.  Possibly version 2.0 could 
> require, or at least optimize for, local caches of subjects.   
> 
> Jim
> 
> On Oct 3, 2010, at 3:49 AM, GW Brown, Information Systems and Computing 
> wrote:
> 
>> The problem is that the API is now doing the paging - but it has no means 
>> to sort the data because, as Chris says, we don't store subject 
>> attributes. 
>> API paging was added because even without sorting, processing very large 
>> memberships would bring the UI to a grinding halt. So, unfortunately the 
>> comparator.sort.limit has become redundant.
>> 
>> Going forward I think there are a coupe of options.
>> 
>> 1) The UI checks how many results there will be and if under 
>> comparator.sort.limit sets this as the pagesize for the API.
>> 2) The API stores a sort String for subjects so it can add an order by 
>> clause to its query. Some mechanism would be needed to refresh sort 
>> Strings.
>> 
>> I can have a look at option 1, but we have talked about option 2 before so 
>> I'll wait and see what the consensus is. Either way it is likely to be 2.0 
>> before we resolve it.
>> 
>> Gary
>> 
>> --On 03 October 2010 01:59 -0400 Chris Hyzer 
>> <>
>>  wrote:
>> 
>>> I can reproduce that.  The problem is that subject data is stored
>>> externally to Grouper (in sources), so if there are 50000 subjects in a
>>> group, that is 50000 subject lookups (assuming none in cache), which
>>> doesn't scale.  Granted if you had 356 with 250 on a page, that should
>>> sort.  :)   What use case do you have?  Trying to see if someone is in a
>>> group?  Just curious, what is the workaround?  Anyways, Im sure Gary will
>>> weigh in here as well...
>>> 
>>> Thanks,
>>> Chris
>>> 
>>> -----Original Message-----
>>> From: 
>>> 
>>> [mailto:]
>>>  On Behalf Of Peter DiCamillo
>>> Sent: Saturday, October 02, 2010 11:14 PM
>>> To: 
>>> 
>>> Subject: [grouper-users] Grouper 1.6.1 UI membership list sorting
>>> 
>>> I'm having a problem where membership lists I view in the full Grouper
>>> UI are not sorted as I would expect.  When the size of the membership
>>> list does not exceed the page size the list is sorted.  But when more
>>> than one page is needed, the list on each page is not sorted.  The value
>>> of comparator.sort.limit in media.properties is 50000, larger than the
>>> size of any of our groups, because even if it's slow, we always want
>>> sorted lists.
>>> 
>>> For example, I viewed a list with 2 direct members and 356 indirect
>>> members.  When I viewed the indirect members or all members with a page
>>> size of 500 the list was sorted.  But with a page size of 250 the lists
>>> on the pages were not sorted.
>>> 
>>> I didn't install Grouper 1.3, but we may have had a local fix for this.
>>> But in any case, it appears that since 1.3 the code that for handling
>>> pages and sorting has changed
>>> 
>>> Peter
>>> 
>> 
>> 
>> 
>> ----------------------
>> GW Brown, Information Systems and Computing
>> 
>