Grouper Users - Open Discussion List

[Jeff McCullough <>]

Hi Chris,

Thank you for doing this work. Overall it makes sense, though I have a couple of comments relating to sortString and the way individual groups are paged. Is there anyway that the sort string can relate to what a JNDI or LDAP adapter might return? I can see why that might not work, but just checking. For paging multiple groups is it possible to page through one group after another rather than paging them all at the same time? For example,  the subjects from group #1 will be paged until empty, then the paging moves to group #2, etc. I can see why you chose the way you did it, so again just checking.

Thank you,

Jeff

On Sep 21, 2012, at 4:14 PM, Chris Hyzer wrote:

https://bugs.internet2.edu/jira/browse/GRP-845

 

This is done. You can add these params to getMembers and getMembersLite in 2.1.3+. Note, if paging/sorting is set on a non-lite call with multiple groups, then the paging sorting will apply to all the individual member lists, not the concatenation of the result

 

   * @param pageSize page size if paging

   * @param pageNumber page number 1 indexed if paging

   * @param sortString must be an hql query field, e.g.

   * can sort on uuid, subjectId, sourceId, name, description, sortString0, sortString1, sortString2, sortString3, sortString4

   * @param ascending T or null for ascending, F for descending.

 

I cut a 2.1.3 release which is not official and not even really  candidate release, but there is not much different in it than 2.1.2 so it is pretty stable,  feel free to try it out and give me feedback

 

http://www.internet2.edu/grouper/release/2.1.3/

 

Thanks,

Chris

 

Ps. there is no schedule for 2.1.3 at the moment

 

From:  [mailto:] On Behalf Of Chris Hyzer
Sent: Monday, September 17, 2012 1:33 AM
To: Jeff McCullough
Cc: ; CalNet Administration
Subject: RE: [grouper-users] REST query string

 

It sounds like you are doing a combination of #3 and #4 below.  Yes, you can create those groups, and get the members, and calculate things on the client.  We can think of the long term plan of having something like this in the Grouper API/WS.  I will continue to add paging to the getMembers operation then…

 

Thanks,

Chris

 

Ps. btw, include/exclude groups is intended for adhoc one-off changes to groups loaded from a system of record.  Otherwise you can add a group to a group (for union) or do a composite for intersection or minus…

 

From: Jeff McCullough [] 
Sent: Monday, September 17, 2012 12:35 AM
To: Chris Hyzer
Cc: ; CalNet Administration
Subject: Re: [grouper-users] REST query string

 

Chris,

 

Okay. The work around follows. Here is the target list group structure I've setup:

 

::Staff:groups:dept1..N

 

::Student:groups:dept1..N

 

From what you have said, it sounds like paging/sorting can only be done by group as grouper currently is coded. An extension might be to return a uniquely sorted paged list of all the members of the supplied group list. Grouper does this now with Include/Exclude groups. 

 

If the first part is all that is possible now, then so be it. The client app will need to pull the list, paged if it likes, and then sort the list uniquely.  From grouper's perspective it is just pulling data from a list of groups. From the client app's perspective it is pulling data from two or more target lists that are members of two or more departmental group lists. Make sense?

 

Jeff

 

On Sep 16, 2012, at 6:37 PM, Chris Hyzer wrote:

 

The WS (or the underlying Grouper API for that matter) doesn’t have a way to do what you describe.  I would suggest:

 

1.       Using LDAP provisioned from WS

2.       –or- Using SQL (using another schema with a grant to grouper_memberships_lw_v)

3.       –or- Getting all the members and do the calculations on the client (probably need caching)

4.       –or- Create all the calculated groups in grouper that you need (might be too many groups)

5.       –or- We can add this to Grouper, though its not as easy as adding paging/sorting, so we would need to discuss the specifics and prioritize with other enhancements…

 

If anyone on the Grouper team wants to elaborate, feel free.

 

Jeff, sorry this is not what you were hoping for, but let me know how you want to proceed.

 

Thanks,

Chris

 

From: Jeff McCullough [] 
Sent: Sunday, September 16, 2012 6:50 PM
To: Chris Hyzer
Cc: ; CalNet Administration
Subject: Re: [grouper-users] REST query string

 

You are correct in all points. The client app owners would prefer to use WS. 

 

It is fairly easy to compose the target lists into departmental groups, and a subset of those groups can be returned. If filtering, a union of two target lists could be filtered by departmental codes on the fly. There is also the issue of uniqueness. For example, if the target lists were staff and students, there will be overlap.

 

Jeff

 

 

On Sep 15, 2012, at 6:35 PM, Chris Hyzer wrote:

Just to be clear on what you are asking… you want to send a filter in to the WS and get a list of members back?

 

e.g. send in:

 

((STAFF UNION AFFILIATES) INTERSECTION (DEPT_A UNION DEPT_B))

 

Return the list of members (in pages of 500 at a time)…

 

Is that right?

 

Just curious, is Grouper provisioned to LDAP and an LDAP filter could do this, but you would prefer they used WS?

 

Thanks,

Chris

 

From: Jeff McCullough [] 
Sent: Friday, September 14, 2012 6:05 PM
To: Chris Hyzer
Cc: ; CalNet Administration
Subject: RE: [grouper-users] REST query string

 

The app is used to send messages to various target lists. For example staff (plus maybe affiliates) in several departments. So it is an AND of the target list(s) and selected departments. The target lists aren't always the same so the idea is to let grouper create the target list by creating departmental groups within that target list folder. Then the apparently can pick a target list by selecting the members from the appropriate groups/depts within the target list folder.

Jeff

Chris Hyzer <> wrote:

Do you want to get the members who are in any of the groups, or all of the groups (AND or OR)?
How many groups are there?  Can you add them to another overall group or use composites to have one overall group?

Thanks,
Chris

-----Original Message-----
From: Jeff McCullough [] 
Sent: Friday, September 14, 2012 2:52 PM
To: Chris Hyzer
Cc: ; CalNet Administration
Subject: Re: [grouper-users] REST query string

The customer for the client app wants to look at this as one uniquely sorted query, more like a JOIN across the specified groups. Is that at all possible?

Jeff

On Sep 14, 2012, at 10:28 AM, Chris Hyzer wrote:

I don't know what you are expecting it to return...  Note the paging is for each group, since there is a separate query for each group.  When you do a getMembers for multiple groups, it returns a list for each group...  can you call it for one group at a time when paging?  :)

Example:

GroupA:
memberA0
memberA1
memberA2
memberA3

GroupB:
memberB0
memberB1

################

Get first page of size 2 for groups GroupA,GroupB:

Results
 - GroupA
  - memberA0
  - memberA1
 - GroupB
  - memberB0
  - memberB1

###############

Get the second page of size 2 for groups GroupA,GroupB:

Not sure if this throw an error since there is no second page of size 2 for GroupB, or if it is an empty list...  I would think it would be an empty list:

Results
 - GroupA
  - memberA2
  - memberA3
 - GroupB
  <no members returned>


Is this what you expect?

Thanks,
Chris

-----Original Message-----
From: Jeff McCullough [] 
Sent: Friday, September 14, 2012 12:59 PM
To: Chris Hyzer
Cc: ; CalNet Administration
Subject: Re: [grouper-users] REST query string

You are awesome. Thank you sooo much. :) A quick question about sorting. Will it check for uniqueness across the listed groups?

Thank you,
Jeff

On Sep 14, 2012, at 5:33 AM, Chris Hyzer wrote:

ok, I have paging/sorting working in the logic, now I need to test more, document, expose in the WS transfer objects, REST/SOAP/WSDL/etc, implement in client, test client, change the configs for a 2.1.3 build, do a build, and put on server...  :)  Will let you know

Chris

From: Jeff McCullough []
Sent: Thursday, September 13, 2012 2:21 PM
To: Chris Hyzer
Cc: ; CalNet Administration
Subject: Re: [grouper-users] REST query string

Hi Chris,

I understand your concern, but the groups being accessed are reference groups that won't be changing at the same time. The SQL interface idea is interesting, but I'm very reluctant to give them access to the underlying db. It is probably best to stick with the REST interface paging if you are game.

Thank you,
Jeff

On Sep 13, 2012, at 11:15 AM, Chris Hyzer wrote:

If you are downloading all memberss from a group, I worry about paging...  I think paging is great for UIs though.

If a member is added while the page queries are being executing, a member could be missed.

e.g.

Grouper members:

1,2,3,4,5,6,7,8,9,10

App gets the page of size 2 starting at index 0: 1,2
App gets the page of size 2 starting at index 2: 3,4

Then item #2 is removed:

1,3,4,5,6,7,8,9,10

App gets the page of size 2 starting at index 4: 6,7

Note that member 5 was missed, even though they are in the group.

So... do you think they can just get all the members at once without paging (even though size of 10k)?  There is also the Grouper SQL interface of reading grouper_memberships_v (filter on field and group) which can be used for large exports...

If you still want paging in WS, no problem, it looks like I could add this to getMembers without much trouble, let me know.

Thanks,
Chris

-----Original Message-----
From: Jeff McCullough []
Sent: Thursday, September 13, 2012 1:56 PM
To: Chris Hyzer
Cc: ; CalNet Administration
Subject: Re: [grouper-users] REST query string

Hi Chris,

Thanks for the info. We were just hooking our first real client app, and the paging thing is going to be a deal breaker for them. Putting part of the query in the request body is workable. The client app will be pulling group data that will frequently be in the 10s of thousands, and they would prefer to pull about 500 members at a time. Do you have a sense of how much effort adding paging to getMembers might be, and a rough time frame? I can maybe steer them to using LDAP groups, but the window of opportunity they have to change to using grouper for the app is fairly short. I'm a little worried they may be lost on other projects for a very long time. Okay... Enough pleading. :) Let me know.

Jeff



On Sep 13, 2012, at 10:07 AM, Chris Hyzer wrote:

A couple of gaps unfortunately... for the params to be in the URL, it has to be a Lite operation, and you can only get members from one group at a time in a Lite getMembers or getMemberships operation.  If you want to pass a JSON/XML/XHTML string in the body of the request, you can specify multiple groups.  I don't see where there is paging on getMembers or getMemberships... I think we need to add that.

https://bugs.internet2.edu/jira/browse/GRP-845

Thanks,
Chris


-----Original Message-----
From:  [] On Behalf Of Jeff McCullough
Sent: Wednesday, September 12, 2012 6:48 PM
To: 
Subject: [grouper-users] REST query string

Quick question. Is it possible to compose a REST like query in one URL that uses paging and requests subjects from more than one group?

Thanks,
Jeff