Grouper Developers Forum

[Niels van Dijk <>]

Hi Chris,

I really admire your wish to make life of the local admin easy :)
However, I feel that your life will become *very* complicated by this :(

We may not be a typical setup, but for us, redundant MySQL was rather
the logical way to go. This is also a very well documented solution, so
I feel grouper should not have to worry about this. erhaps the grouper
community would also be served well by just documenting very well how to
setup such a HA enviroment with grouper?

If you require a good in-memory cache, take a look at memcache. This is
a very good, highly scaleable solution.

just my 2 cents,

regards,
Niels


Chris Hyzer wrote:
> Yes, that is always what is mentioned as what to do for this.
> 
> Here are some more advantages, though I agree with you that replication 
> might be the way to go...
> 
> I think it would add value if Grouper (assuming you need highly available 
> web services) were shipped as a product which did not require database 
> replication, since it might not be something the institution has experience 
> with.  I think it would be nice if the only server process Grouper needs to 
> function in read queries is the servlet container (no DB process to worry 
> about, even if there are multiple).
> 
> Another reason to have an in-memory cache of the data is to have high 
> performant attribute read queries for privilege management (which can be 
> complex and high volume)... e.g. does ID 12345 have rights to read 
> english110 data?  It could be a direct assignment, an assignment through a 
> role, and assignment through a privilege SET (e.g. an assignment to all of 
> the English department), an assignment of a role to a SET, an assignment to 
> a role which is inherited from another role which the user is in, etc.  Of 
> course you could also benefit from high volume group membership queries, 
> though these have simpler SQL/logic...
> 
> Regards!
> Chris
> 
>> -----Original Message-----
>> From: Jim Fox 
>> [mailto:]
>> Sent: Sunday, June 14, 2009 11:09 PM
>> To: Chris Hyzer
>> Cc: 
>> 
>> Subject: Re: [grouper-dev] highly available web services
>>
>>
>> These steps you outline are a lot more complicated than you let on.
>> Why not just replicate the database?
>>
>> Jim
>>
>> On Jun 14, 2009, at 2:25 PM, Chris Hyzer wrote:
>>
>>> On the call and on the list we have been discussing highly available
>>> web services (Im thinking about reads here not writes).  Some use
>>> cases:
>>>
>>> 1. Queries that aren't conducive to LDAP: e.g. if the limits are
>>> exceeded of LDAP, or maybe complex queries
>>>
>>> 2. If a deployment of grouper doesn't have LDAP, it doesn't need it
>>>
>>> 3. Queries that require logic, e.g. the attribute framework will
>>> need privilege decision point queries which require logic to get the
>>> answer
>>>
>>> So I think web services should:
>>>
>>> 1. Slurp all data from the database into objects, and serialize them
>>> to disk.
>>>
>>> 2. For read operations, use those structures to make the decision
>>>
>>> 3. If web services are shutdown, deserialize from disk (and run
>>> recent change logs from the time it was last serialized)
>>>
>>> 4. If a certain amount of time has passed since the last slurp,
>>> slurp again.  E.g. 24 hours
>>>
>>> 5. Have a thread which reads the change log every so often (e.g.
>>> every 2 minutes, configurable), update the object structures.  Every
>>> so often (30 minutes), serialize to disk
>>>
>>> 6. There could be an option of configuration for if you want
>>> readonly to do this, or just hit the db, or or this if the DB is
>>> unavailable
>>>
>>> 7. Yes, if you have this mode on, it will require a lot of memory
>>> (proportional to the size of the registry)...  we could swap this
>>> out for a local database or some other storage if we like, this part
>>> could even be pluggable
>>>
>>> This way you should be able to run multiple web services instances,
>>> with a load balancer, and be highly available for readonly
>>> operations (when DB goes down)...
>>>
>>> Anyways, I don't think any of this is all that complicated or hard
>>> or time consuming to build...  also, Im not going to work on this
>>> soon, just throwing out ideas...
>>>
>>> Thanks,
>>> Chris
> 

-- 
Niels van Dijk
Advanced Services

T: +31 302 305 337 / M: +31 651 347 657
SURFnet - PO Box 19035 - NL-3501 DA Utrecht - The Netherlands -

http://www.surfnet.nl
SURFnet grensverleggend netwerk voor hoger onderwijs en onderzoek