Grouper Users - Open Discussion List

[Kathryn Huxtable <>]

Here's what "top" returns. I'm currently running an update.

CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total    0.2%    0.0%    0.0%   0.0%     0.0%    0.7%   98.9%
           cpu00    0.0%    0.0%    0.0%   0.0%     0.0%    0.7%   99.2%
           cpu01    0.9%    0.0%    0.0%   0.0%     0.0%    0.7%   98.2%
           cpu02    0.0%    0.0%    0.1%   0.0%     0.0%    0.7%   99.0%
           cpu03    0.0%    0.0%    0.0%   0.0%     0.0%    0.7%   99.2%
Mem:  2061448k av, 1754356k used,  307092k free,     0k shrd,  224164k buff
                    877208k actv,  529504k in_d, 16904k in_c
Swap: 4192924k av,    4952k used, 4187972k free                835376k
cached

-K


On 8/13/06 9:15 AM, "Kathryn Huxtable" 
<>
 wrote:

> I should add that it looks as if I'm able to add or remove one subject per
> 50 seconds. That's averaged over about 100,000 of them.
> 
> -K
> 
> 
> On 8/13/06 9:06 AM, "Kathryn Huxtable" 
> <>
>  wrote:
> 
>> Sure. First of all, I miscounted. There were 50,000 subjects. There were
>> 240,000 memberships, all JNDI subjects added as members to 638 groups.
>> 
>> I generated the XML import file using a Perl script that ran against my
>> Oracle database in about 100 seconds.
>> 
>> I'm afraid I don't really know the hardware specs for my test server, but
>> it's the same as my production server. It's a Dell Poweredge running RedHat
>> Linux. I'm using Java1.5.
>> 
>> The subject connector is JNDI running against an extremely oversized 
>> Solaris
>> box running SunJava 5.2. Would JDBC against Oracle be faster? I can do 
>> that,
>> but I'll have to worry about FERPA suppressing the name and description, so
>> I'll have to make my own JDBC connector to add some code. I get that for
>> free with my ACIs in LDAP.
>> 
>> The grouper database is Oracle 9i running on some sort of SunFire box.
>> 
>> I made the mistake of running this in a shell inside XEmacs, so I had to
>> leave my laptop on for all that time.
>> 
>> Anyway, does adding 240,000 memberships take 47.5 hours normally?
>> 
>> I've rewritten _processMembershipLists in XmlImporter so that instead of
>> removing the old memberships and adding the new ones, it gets the old
>> membership IDs into a HashSet, does the same with the new membership IDs 
>> and
>> then only removes or adds those that aren't present in both. This does
>> updates faster, since from day to day most of these groups' memberships
>> won't change that much (except for three times per year at semester 
>> breaks).
>> 
>> But that was after running this.
>> 
>> -K
>> 
>> 
>> On 8/12/06 8:56 AM, "Tom Barton" 
>> <>
>>  wrote:
>> 
>>> Can you supply further details of the operation, the execution
>>> environment, and exactly how it was conducted? That does sound absurdly
>>> slow, and far off from my own experience.
>>> 
>>> Tom
>>> 
>>> Kathryn Huxtable wrote:
>>>> I just added external provisioning for all the base groups we might want 
>>>> to
>>>> use in composite operations. It took almost 48 hours. This is ridiculous.
>>>> My
>>>> home-grown group management can populate from empty in about four hours. 
>>>> My
>>>> nightly update only does adds/removes and so runs very quickly.
>>>> 
>>>> I can obviously write similar update code, but really, what all is going 
>>>> on
>>>> that adding 20,000 subjects takes so long? Is a table not indexed on some
>>>> attribute, or are you creating new subject records from the JNDI data in 
>>>> my
>>>> subject configs?
>>>> 
>>>> Inquiring minds want to know (fnord).
>>>> 
>>>> -K
>>>> 
>>>> 
>>>> On 8/4/06 3:43 PM, "Tom Barton" 
>>>> <>
>>>>  wrote:
>>>> 
>>>>> You should be able to use the XML import/export tool for this purpose.
>>>>> See 
>>>>> 
>> 
> 
<https://wiki.internet2.edu/confluence/display/GrouperWG/Import-Export+v1.0>>>>
>
>> 
>> .
>>>>> 
>>>>> <hint>
>>>>> It'd be good to include an example of how to "prune & graft", like
>>>>> you're wanting to do, in that wiki page.
>>>>> </hint>
>>>>> 
>>>>> Tom
>>>>> 
>>>>> Will Norris wrote:
>>>>>> can groups or stems be moved to somewhere else in the group hierarchy?
>>>>>> Looking at the database I can't think of any technical reason why this
>>>>>> wouldn't be possible, unless I'm overlooking something.
>>>>>> 
>>>>>> -will
>>>> 
>> 
>