Grouper Users - Open Discussion List

[Chris Hyzer <>]

Are you on the latest 1.4 before upgrading to 1.5?  We test our upgrades from 
version to version.  So if you are on 1.3, you first need to go to 1.4, then 
to 1.5... 1.4 should have a grouper_memberships.via_id

Thanks,
Chris

-----Original Message-----
From: Colin Hudler 
[mailto:]
 
Sent: Tuesday, December 22, 2009 12:14 PM
To: 

Subject: Re: [grouper-users] gsh xmlimport memory usage

Thanks for all the suggestions.  I was hoping to be able to partition 
the import this way.  In the meantime, I have been trying "Upgrade 
Method 2".  I eventually got to a point in the script where it runs:

update grouper_memberships
set owner_group_id = (select gg.id from grouper_groups gg where gg.id = 
owner_id),
owner_stem_id = (select gs.id from grouper_stems gs where gs.id = owner_id),
via_composite_id = (select gc.id from grouper_composites gc where gc.id 
= via_id),
via_id_bak = via_id,
via_id = null,
depth_bak = depth,
depth = 0,
parent_membership_bak = parent_membership,
parent_membership = null
where owner_group_id is null and owner_stem_id is null and 
owner_attr_def_id is null
and via_composite_id is null and via_id_bak is null;
commit;


Which dies with
Error running script: 
/srv/grouper/api-1.5.0/ddlScripts/grouperDdl_20091217_11_01_02_600.sql, 
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown 
column 'via_id' in 'field list'

Indeed, I have no column by that name, but there is
via_composite_id and via_id_bak.  It is possible I did something wrong. 
  I rolled back the database a few times and could have made a mistake. 
  I will probably roll back again and start over, but I thought I would 
mention it here in case it is a real error.

Tom Zeller wrote:
> In addition to importing stems separately, it might also be
> interesting to import first without memberships or permissions, then a
> second time with memberships and permissions.
> 
> To ignore memberships and privileges, you would do something like
> 
>  bin/gsh.sh -xmlimport GrouperSystem export.xml import.properties
> 
> import.properties :
> 
>  import.data.lists=ignore
>  import.data.privileges=ignore
> 
> And, of course, there's running with a profiler.
> 
> TomZ
> 
> On Thu, Dec 17, 2009 at 3:48 PM, Chris Hyzer 
> <>
>  wrote:
>> Is this a 32bit machine or 64?  2gig might be the max that Java can use.
>>
>> We are planning to rewrite the import/export, and I hope to improve 
>> performance in the rewrite.  In the meantime, maybe if this is not a 64 
>> bit machine, you could try from another machine which is 64 bit.  Or maybe 
>> you could export twice and partition the stems so that you have two files 
>> (or more) of half the size (or less).  Kind of a pain I know, but just 
>> brainstorming here.
>>
>> Thanks,
>> chris
>>
>> -----Original Message-----
>> From: Colin Hudler 
>> [mailto:]
>> Sent: Thursday, December 17, 2009 10:18 AM
>> To: 
>> 
>> Subject: Re: [grouper-users] gsh xmlimport memory usage
>>
>> The backup file is 256M.  Memberships are 1507480, in 90908 groups.  The
>> subject source is
>> edu.internet2.middleware.grouper.subj.GrouperJndiSourceAdapter, but
>> maybe I will change it and compare.
>>
>> Tom Zeller wrote:
>>> What is the filesize of backup.xml ? How many memberships do you have ?
>>>
>>> I've been able to xmlimport with Xmx=1024m, and we have ~1 million
>>> memberships (select count(*) from grouper_memberships), and our xml
>>> export is ~175MB.
>>>
>>> What source adapters are you using ?
>>>
>>> This doesn't answer your question, of course.
>>>
>>> TomZ
>>>
>>> On Thu, Dec 17, 2009 at 9:00 AM, Colin Hudler 
>>> <>
>>>  wrote:
>>>> Hi,
>>>>
>>>>  Congrats on the fantastic new release, nice stuff.  I am doing upgrade
>>>> rehearsals under Method 1,
>>>>
>>>> "./bin/gsh.sh -xmlimport GrouperSystem backup.xml"
>>>>
>>>> But it always uses all of the memory.  I increased it to 4GB, but it 
>>>> still
>>>> actually commits that much.  Before moving on to method 2, I thought I'd
>>>> ask, is there anything I can do reduce it?
>>>>
>>>> Obviously, this is the actual error:
>>>>
>>>> java.lang.OutOfMemoryError: Java heap space
>>>>
>