Grouper Users - Open Discussion List

[Kathryn Huxtable <>]

I'm not going to work on this today, but I think the thing to do is for me
to get some measurements on how much time is being spent in the subject API
versus the grouper API. I'm guessing that it's the subject resolution that
is taking time. I've looked over the grouper code and it looks fine and
shouldn't take much time.

I'm guessing adding my own JDBC provider to handle some customization will
take care of this...

-K


On 8/13/06 9:21 AM, "Kathryn Huxtable" 
<>
 wrote:

> Here's what "top" returns. I'm currently running an update.
> 
> CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
>            total    0.2%    0.0%    0.0%   0.0%     0.0%    0.7%   98.9%
>            cpu00    0.0%    0.0%    0.0%   0.0%     0.0%    0.7%   99.2%
>            cpu01    0.9%    0.0%    0.0%   0.0%     0.0%    0.7%   98.2%
>            cpu02    0.0%    0.0%    0.1%   0.0%     0.0%    0.7%   99.0%
>            cpu03    0.0%    0.0%    0.0%   0.0%     0.0%    0.7%   99.2%
> Mem:  2061448k av, 1754356k used,  307092k free,     0k shrd,  224164k buff
>                     877208k actv,  529504k in_d, 16904k in_c
> Swap: 4192924k av,    4952k used, 4187972k free                835376k
> cached
> 
> -K
> 
> 
> On 8/13/06 9:15 AM, "Kathryn Huxtable" 
> <>
>  wrote:
> 
>> I should add that it looks as if I'm able to add or remove one subject per
>> 50 seconds. That's averaged over about 100,000 of them.
>> 
>> -K
>> 
>> 
>> On 8/13/06 9:06 AM, "Kathryn Huxtable" 
>> <>
>>  wrote:
>> 
>>> Sure. First of all, I miscounted. There were 50,000 subjects. There were
>>> 240,000 memberships, all JNDI subjects added as members to 638 groups.
>>> 
>>> I generated the XML import file using a Perl script that ran against my
>>> Oracle database in about 100 seconds.
>>> 
>>> I'm afraid I don't really know the hardware specs for my test server, but
>>> it's the same as my production server. It's a Dell Poweredge running 
>>> RedHat
>>> Linux. I'm using Java1.5.
>>> 
>>> The subject connector is JNDI running against an extremely oversized 
>>> Solaris
>>> box running SunJava 5.2. Would JDBC against Oracle be faster? I can do 
>>> that,
>>> but I'll have to worry about FERPA suppressing the name and description, 
>>> so
>>> I'll have to make my own JDBC connector to add some code. I get that for
>>> free with my ACIs in LDAP.
>>> 
>>> The grouper database is Oracle 9i running on some sort of SunFire box.
>>> 
>>> I made the mistake of running this in a shell inside XEmacs, so I had to
>>> leave my laptop on for all that time.
>>> 
>>> Anyway, does adding 240,000 memberships take 47.5 hours normally?
>>> 
>>> I've rewritten _processMembershipLists in XmlImporter so that instead of
>>> removing the old memberships and adding the new ones, it gets the old
>>> membership IDs into a HashSet, does the same with the new membership IDs 
>>> and
>>> then only removes or adds those that aren't present in both. This does
>>> updates faster, since from day to day most of these groups' memberships
>>> won't change that much (except for three times per year at semester 
>>> breaks).
>>> 
>>> But that was after running this.
>>> 
>>> -K
>>> 
>>> 
>>> On 8/12/06 8:56 AM, "Tom Barton" 
>>> <>
>>>  wrote:
>>> 
>>>> Can you supply further details of the operation, the execution
>>>> environment, and exactly how it was conducted? That does sound absurdly
>>>> slow, and far off from my own experience.
>>>> 
>>>> Tom
>>>> 
>>>> Kathryn Huxtable wrote:
>>>>> I just added external provisioning for all the base groups we might want
>>>>> to
>>>>> use in composite operations. It took almost 48 hours. This is 
>>>>> ridiculous.
>>>>> My
>>>>> home-grown group management can populate from empty in about four hours.
>>>>> My
>>>>> nightly update only does adds/removes and so runs very quickly.
>>>>> 
>>>>> I can obviously write similar update code, but really, what all is going
>>>>> on
>>>>> that adding 20,000 subjects takes so long? Is a table not indexed on 
>>>>> some
>>>>> attribute, or are you creating new subject records from the JNDI data in
>>>>> my
>>>>> subject configs?
>>>>> 
>>>>> Inquiring minds want to know (fnord).
>>>>> 
>>>>> -K
>>>>> 
>>>>> 
>>>>> On 8/4/06 3:43 PM, "Tom Barton" 
>>>>> <>
>>>>>  wrote:
>>>>> 
>>>>>> You should be able to use the XML import/export tool for this purpose.
>>>>>> See 
>>>>>> 
>>> 
>> 
> 
<https://wiki.internet2.edu/confluence/display/GrouperWG/Import-Export+v1.0>>>>
>
>> 
>>> 
>>> .
>>>>>> 
>>>>>> <hint>
>>>>>> It'd be good to include an example of how to "prune & graft", like
>>>>>> you're wanting to do, in that wiki page.
>>>>>> </hint>
>>>>>> 
>>>>>> Tom
>>>>>> 
>>>>>> Will Norris wrote:
>>>>>>> can groups or stems be moved to somewhere else in the group hierarchy?
>>>>>>> Looking at the database I can't think of any technical reason why this
>>>>>>> wouldn't be possible, unless I'm overlooking something.
>>>>>>> 
>>>>>>> -will
>>>>> 
>>> 
>> 
>