Grouper Developers Forum

[Chris Hyzer <>]

Yes, that is always what is mentioned as what to do for this.

Here are some more advantages, though I agree with you that replication might 
be the way to go...

I think it would add value if Grouper (assuming you need highly available web 
services) were shipped as a product which did not require database 
replication, since it might not be something the institution has experience 
with.  I think it would be nice if the only server process Grouper needs to 
function in read queries is the servlet container (no DB process to worry 
about, even if there are multiple).

Another reason to have an in-memory cache of the data is to have high 
performant attribute read queries for privilege management (which can be 
complex and high volume)... e.g. does ID 12345 have rights to read english110 
data?  It could be a direct assignment, an assignment through a role, and 
assignment through a privilege SET (e.g. an assignment to all of the English 
department), an assignment of a role to a SET, an assignment to a role which 
is inherited from another role which the user is in, etc.  Of course you 
could also benefit from high volume group membership queries, though these 
have simpler SQL/logic...

Regards!
Chris

> -----Original Message-----
> From: Jim Fox 
> [mailto:]
> Sent: Sunday, June 14, 2009 11:09 PM
> To: Chris Hyzer
> Cc: 
> 
> Subject: Re: [grouper-dev] highly available web services
> 
> 
> These steps you outline are a lot more complicated than you let on.
> Why not just replicate the database?
> 
> Jim
> 
> On Jun 14, 2009, at 2:25 PM, Chris Hyzer wrote:
> 
> > On the call and on the list we have been discussing highly available
> > web services (Im thinking about reads here not writes).  Some use
> > cases:
> >
> > 1. Queries that aren't conducive to LDAP: e.g. if the limits are
> > exceeded of LDAP, or maybe complex queries
> >
> > 2. If a deployment of grouper doesn't have LDAP, it doesn't need it
> >
> > 3. Queries that require logic, e.g. the attribute framework will
> > need privilege decision point queries which require logic to get the
> > answer
> >
> > So I think web services should:
> >
> > 1. Slurp all data from the database into objects, and serialize them
> > to disk.
> >
> > 2. For read operations, use those structures to make the decision
> >
> > 3. If web services are shutdown, deserialize from disk (and run
> > recent change logs from the time it was last serialized)
> >
> > 4. If a certain amount of time has passed since the last slurp,
> > slurp again.  E.g. 24 hours
> >
> > 5. Have a thread which reads the change log every so often (e.g.
> > every 2 minutes, configurable), update the object structures.  Every
> > so often (30 minutes), serialize to disk
> >
> > 6. There could be an option of configuration for if you want
> > readonly to do this, or just hit the db, or or this if the DB is
> > unavailable
> >
> > 7. Yes, if you have this mode on, it will require a lot of memory
> > (proportional to the size of the registry)...  we could swap this
> > out for a local database or some other storage if we like, this part
> > could even be pluggable
> >
> > This way you should be able to run multiple web services instances,
> > with a load balancer, and be highly available for readonly
> > operations (when DB goes down)...
> >
> > Anyways, I don't think any of this is all that complicated or hard
> > or time consuming to build...  also, Im not going to work on this
> > soon, just throwing out ideas...
> >
> > Thanks,
> > Chris