Skip to Content.
Sympa Menu

grouper-users - Re: [grouper-users] gsh xmlimport memory usage

Subject: Grouper Users - Open Discussion List

List archive

Re: [grouper-users] gsh xmlimport memory usage

Chronological Thread 
  • From: Colin Hudler <>
  • To: "" <>
  • Subject: Re: [grouper-users] gsh xmlimport memory usage
  • Date: Wed, 23 Dec 2009 11:42:37 -0600

I am sorry. I forgot about that!

On 12/23/09 11:21 AM, Chris Hyzer wrote:
Did you do this part?

gsh 2% new

It is required to see memberships.


-----Original Message-----
From: Colin Hudler
Sent: Wednesday, December 23, 2009 12:19 PM

Subject: Re: [grouper-users] gsh xmlimport memory usage

I started over and it went pretty well. Towards the end of the
migration, it died trying to create the view "grouper_memberships_v". I
dropped the view myself and continued the script where it left off. I
then had to repeate the process for "grouper_rpt_group_field_v", but the
rest completed successfully, and both of the views exist now.

Here's the strange part. The API doesn't think I belong to groups, even
though grouper_memberships has entries for me (I took a peek at the db).

gsh 1% ses = GrouperSession.startRootSession();
gsh 2% s = findSubject("21882177X")
subject: id='21882177X' type='person' source='ucmcdb' name='Colin Hudler'
gsh 3% member = MemberFinder.findBySubject(ses, s)
member: id='21882177X' type='person' source='ucmcdb'
gsh 4% member.getGroups()
gsh 5%

The UI confirms that (all?) stems and groups are present, but none of
the groups have members. Side note, the grouper_membership_v is an
empty result hmm. This is all only a test system. I am going to
continue prodding it, and I welcome troubleshooting suggestions. Thanks
for the help so far.

On 12/22/09 11:33 AM, Colin Hudler wrote:
I looked harder and found that the column is present in backups and in
production and various dev dbs, but since the migrating system doesn't
have it, the migration script must have dropped it even though this
happens *after* the part I get the error for. Hmm. In any case, I am
going to take two steps back and start from a pristine database and see
what happens.

Chris Hyzer wrote:
Are you on the latest 1.4 before upgrading to 1.5? We test our
upgrades from version to version. So if you are on 1.3, you first need
to go to 1.4, then to 1.5... 1.4 should have a grouper_memberships.via_id


-----Original Message-----
From: Colin Hudler
Sent: Tuesday,
December 22, 2009 12:14 PM

Subject: Re: [grouper-users] gsh xmlimport memory usage

Thanks for all the suggestions. I was hoping to be able to partition
the import this way. In the meantime, I have been trying "Upgrade
Method 2". I eventually got to a point in the script where it runs:

update grouper_memberships
set owner_group_id = (select from grouper_groups gg where
= owner_id),
owner_stem_id = (select from grouper_stems gs where =
via_composite_id = (select from grouper_composites gc where = via_id),
via_id_bak = via_id,
via_id = null,
depth_bak = depth,
depth = 0,
parent_membership_bak = parent_membership,
parent_membership = null
where owner_group_id is null and owner_stem_id is null and
owner_attr_def_id is null
and via_composite_id is null and via_id_bak is null;

Which dies with
Error running script:
Unknown column 'via_id' in 'field list'

Indeed, I have no column by that name, but there is
via_composite_id and via_id_bak. It is possible I did something wrong.
I rolled back the database a few times and could have made a mistake.
I will probably roll back again and start over, but I thought I would
mention it here in case it is a real error.

Tom Zeller wrote:
In addition to importing stems separately, it might also be
interesting to import first without memberships or permissions, then a
second time with memberships and permissions.

To ignore memberships and privileges, you would do something like

bin/ -xmlimport GrouperSystem export.xml :

And, of course, there's running with a profiler.


On Thu, Dec 17, 2009 at 3:48 PM, Chris
Is this a 32bit machine or 64? 2gig might be the max that Java can use.

We are planning to rewrite the import/export, and I hope to improve
performance in the rewrite. In the meantime, maybe if this is not a
64 bit machine, you could try from another machine which is 64 bit.
Or maybe you could export twice and partition the stems so that you
have two files (or more) of half the size (or less). Kind of a pain
I know, but just brainstorming here.


-----Original Message-----
From: Colin Hudler
Sent: Thursday, December 17, 2009 10:18 AM

Subject: Re: [grouper-users] gsh xmlimport memory usage

The backup file is 256M. Memberships are 1507480, in 90908 groups. The
subject source is
edu.internet2.middleware.grouper.subj.GrouperJndiSourceAdapter, but
maybe I will change it and compare.

Tom Zeller wrote:
What is the filesize of backup.xml ? How many memberships do you
have ?

I've been able to xmlimport with Xmx=1024m, and we have ~1 million
memberships (select count(*) from grouper_memberships), and our xml
export is ~175MB.

What source adapters are you using ?

This doesn't answer your question, of course.


On Thu, Dec 17, 2009 at 9:00 AM, Colin Hudler

Congrats on the fantastic new release, nice stuff. I am doing upgrade
rehearsals under Method 1,

"./bin/ -xmlimport GrouperSystem backup.xml"

But it always uses all of the memory. I increased it to 4GB, but
it still
actually commits that much. Before moving on to method 2, I
thought I'd
ask, is there anything I can do reduce it?

Obviously, this is the actual error:

java.lang.OutOfMemoryError: Java heap space

Archive powered by MHonArc 2.6.16.

Top of Page