Skip to Content.
Sympa Menu

grouper-users - RE: [grouper-users] gsh xmlimport memory usage

Subject: Grouper Users - Open Discussion List

List archive

RE: [grouper-users] gsh xmlimport memory usage

Chronological Thread 
  • From: Chris Hyzer <>
  • To: Colin Hudler <>, "" <>
  • Subject: RE: [grouper-users] gsh xmlimport memory usage
  • Date: Tue, 22 Dec 2009 12:22:22 -0500
  • Accept-language: en-US
  • Acceptlanguage: en-US

Are you on the latest 1.4 before upgrading to 1.5? We test our upgrades from
version to version. So if you are on 1.3, you first need to go to 1.4, then
to 1.5... 1.4 should have a grouper_memberships.via_id


-----Original Message-----
From: Colin Hudler

Sent: Tuesday, December 22, 2009 12:14 PM

Subject: Re: [grouper-users] gsh xmlimport memory usage

Thanks for all the suggestions. I was hoping to be able to partition
the import this way. In the meantime, I have been trying "Upgrade
Method 2". I eventually got to a point in the script where it runs:

update grouper_memberships
set owner_group_id = (select from grouper_groups gg where =
owner_stem_id = (select from grouper_stems gs where = owner_id),
via_composite_id = (select from grouper_composites gc where
= via_id),
via_id_bak = via_id,
via_id = null,
depth_bak = depth,
depth = 0,
parent_membership_bak = parent_membership,
parent_membership = null
where owner_group_id is null and owner_stem_id is null and
owner_attr_def_id is null
and via_composite_id is null and via_id_bak is null;

Which dies with
Error running script:
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown
column 'via_id' in 'field list'

Indeed, I have no column by that name, but there is
via_composite_id and via_id_bak. It is possible I did something wrong.
I rolled back the database a few times and could have made a mistake.
I will probably roll back again and start over, but I thought I would
mention it here in case it is a real error.

Tom Zeller wrote:
> In addition to importing stems separately, it might also be
> interesting to import first without memberships or permissions, then a
> second time with memberships and permissions.
> To ignore memberships and privileges, you would do something like
> bin/ -xmlimport GrouperSystem export.xml
> :
> And, of course, there's running with a profiler.
> TomZ
> On Thu, Dec 17, 2009 at 3:48 PM, Chris Hyzer
> <>
> wrote:
>> Is this a 32bit machine or 64? 2gig might be the max that Java can use.
>> We are planning to rewrite the import/export, and I hope to improve
>> performance in the rewrite. In the meantime, maybe if this is not a 64
>> bit machine, you could try from another machine which is 64 bit. Or maybe
>> you could export twice and partition the stems so that you have two files
>> (or more) of half the size (or less). Kind of a pain I know, but just
>> brainstorming here.
>> Thanks,
>> chris
>> -----Original Message-----
>> From: Colin Hudler
>> [mailto:]
>> Sent: Thursday, December 17, 2009 10:18 AM
>> To:
>> Subject: Re: [grouper-users] gsh xmlimport memory usage
>> The backup file is 256M. Memberships are 1507480, in 90908 groups. The
>> subject source is
>> edu.internet2.middleware.grouper.subj.GrouperJndiSourceAdapter, but
>> maybe I will change it and compare.
>> Tom Zeller wrote:
>>> What is the filesize of backup.xml ? How many memberships do you have ?
>>> I've been able to xmlimport with Xmx=1024m, and we have ~1 million
>>> memberships (select count(*) from grouper_memberships), and our xml
>>> export is ~175MB.
>>> What source adapters are you using ?
>>> This doesn't answer your question, of course.
>>> TomZ
>>> On Thu, Dec 17, 2009 at 9:00 AM, Colin Hudler
>>> <>
>>> wrote:
>>>> Hi,
>>>> Congrats on the fantastic new release, nice stuff. I am doing upgrade
>>>> rehearsals under Method 1,
>>>> "./bin/ -xmlimport GrouperSystem backup.xml"
>>>> But it always uses all of the memory. I increased it to 4GB, but it
>>>> still
>>>> actually commits that much. Before moving on to method 2, I thought I'd
>>>> ask, is there anything I can do reduce it?
>>>> Obviously, this is the actual error:
>>>> java.lang.OutOfMemoryError: Java heap space

Archive powered by MHonArc 2.6.16.

Top of Page