Skip to Content.
Sympa Menu

grouper-users - Re: [grouper-users] gsh xmlimport memory usage

Subject: Grouper Users - Open Discussion List

List archive

Re: [grouper-users] gsh xmlimport memory usage


Chronological Thread 
  • From: Colin Hudler <>
  • To: "" <>
  • Subject: Re: [grouper-users] gsh xmlimport memory usage
  • Date: Tue, 22 Dec 2009 11:33:44 -0600

I looked harder and found that the column is present in backups and in production and various dev dbs, but since the migrating system doesn't have it, the migration script must have dropped it even though this happens *after* the part I get the error for. Hmm. In any case, I am going to take two steps back and start from a pristine database and see what happens.

Chris Hyzer wrote:
Are you on the latest 1.4 before upgrading to 1.5? We test our upgrades from
version to version. So if you are on 1.3, you first need to go to 1.4, then
to 1.5... 1.4 should have a grouper_memberships.via_id

Thanks,
Chris

-----Original Message-----
From: Colin Hudler [mailto:] Sent: Tuesday, December 22, 2009 12:14 PM
To:

Subject: Re: [grouper-users] gsh xmlimport memory usage

Thanks for all the suggestions. I was hoping to be able to partition the import this way. In the meantime, I have been trying "Upgrade Method 2". I eventually got to a point in the script where it runs:

update grouper_memberships
set owner_group_id = (select gg.id from grouper_groups gg where gg.id = owner_id),
owner_stem_id = (select gs.id from grouper_stems gs where gs.id = owner_id),
via_composite_id = (select gc.id from grouper_composites gc where gc.id = via_id),
via_id_bak = via_id,
via_id = null,
depth_bak = depth,
depth = 0,
parent_membership_bak = parent_membership,
parent_membership = null
where owner_group_id is null and owner_stem_id is null and owner_attr_def_id is null
and via_composite_id is null and via_id_bak is null;
commit;


Which dies with
Error running script: /srv/grouper/api-1.5.0/ddlScripts/grouperDdl_20091217_11_01_02_600.sql, com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column 'via_id' in 'field list'

Indeed, I have no column by that name, but there is
via_composite_id and via_id_bak. It is possible I did something wrong. I rolled back the database a few times and could have made a mistake. I will probably roll back again and start over, but I thought I would mention it here in case it is a real error.

Tom Zeller wrote:
In addition to importing stems separately, it might also be
interesting to import first without memberships or permissions, then a
second time with memberships and permissions.

To ignore memberships and privileges, you would do something like

bin/gsh.sh -xmlimport GrouperSystem export.xml import.properties

import.properties :

import.data.lists=ignore
import.data.privileges=ignore

And, of course, there's running with a profiler.

TomZ

On Thu, Dec 17, 2009 at 3:48 PM, Chris Hyzer
<>
wrote:
Is this a 32bit machine or 64? 2gig might be the max that Java can use.

We are planning to rewrite the import/export, and I hope to improve
performance in the rewrite. In the meantime, maybe if this is not a 64 bit
machine, you could try from another machine which is 64 bit. Or maybe you
could export twice and partition the stems so that you have two files (or
more) of half the size (or less). Kind of a pain I know, but just
brainstorming here.

Thanks,
chris

-----Original Message-----
From: Colin Hudler
[mailto:]
Sent: Thursday, December 17, 2009 10:18 AM
To:

Subject: Re: [grouper-users] gsh xmlimport memory usage

The backup file is 256M. Memberships are 1507480, in 90908 groups. The
subject source is
edu.internet2.middleware.grouper.subj.GrouperJndiSourceAdapter, but
maybe I will change it and compare.

Tom Zeller wrote:
What is the filesize of backup.xml ? How many memberships do you have ?

I've been able to xmlimport with Xmx=1024m, and we have ~1 million
memberships (select count(*) from grouper_memberships), and our xml
export is ~175MB.

What source adapters are you using ?

This doesn't answer your question, of course.

TomZ

On Thu, Dec 17, 2009 at 9:00 AM, Colin Hudler
<>
wrote:
Hi,

Congrats on the fantastic new release, nice stuff. I am doing upgrade
rehearsals under Method 1,

"./bin/gsh.sh -xmlimport GrouperSystem backup.xml"

But it always uses all of the memory. I increased it to 4GB, but it still
actually commits that much. Before moving on to method 2, I thought I'd
ask, is there anything I can do reduce it?

Obviously, this is the actual error:

java.lang.OutOfMemoryError: Java heap space






Archive powered by MHonArc 2.6.16.

Top of Page