Grouper Users - Open Discussion List

[Colin Hudler <>]

Thanks for all the suggestions.  I was hoping to be able to partition 
the import this way.  In the meantime, I have been trying "Upgrade 
Method 2".  I eventually got to a point in the script where it runs:

update grouper_memberships
set owner_group_id = (select gg.id from grouper_groups gg where gg.id = 
owner_id),
owner_stem_id = (select gs.id from grouper_stems gs where gs.id = owner_id),
via_composite_id = (select gc.id from grouper_composites gc where gc.id 
= via_id),
via_id_bak = via_id,
via_id = null,
depth_bak = depth,
depth = 0,
parent_membership_bak = parent_membership,
parent_membership = null
where owner_group_id is null and owner_stem_id is null and 
owner_attr_def_id is null
and via_composite_id is null and via_id_bak is null;
commit;


Which dies with
Error running script: 
/srv/grouper/api-1.5.0/ddlScripts/grouperDdl_20091217_11_01_02_600.sql, 
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown 
column 'via_id' in 'field list'

Indeed, I have no column by that name, but there is
via_composite_id and via_id_bak.  It is possible I did something wrong. 
 I rolled back the database a few times and could have made a mistake. 
 I will probably roll back again and start over, but I thought I would 
mention it here in case it is a real error.

Tom Zeller wrote:
> In addition to importing stems separately, it might also be
> interesting to import first without memberships or permissions, then a
> second time with memberships and permissions.
>
> To ignore memberships and privileges, you would do something like
>
>  bin/gsh.sh -xmlimport GrouperSystem export.xml import.properties
>
> import.properties :
>
>  import.data.lists=ignore
>  import.data.privileges=ignore
>
> And, of course, there's running with a profiler.
>
> TomZ
>
> On Thu, Dec 17, 2009 at 3:48 PM, Chris Hyzer 
> <>
>  wrote:
> > Is this a 32bit machine or 64?  2gig might be the max that Java can use.
> >
> > We are planning to rewrite the import/export, and I hope to improve 
> > performance in the rewrite.  In the meantime, maybe if this is not a 64 bit 
> > machine, you could try from another machine which is 64 bit.  Or maybe you 
> > could export twice and partition the stems so that you have two files (or 
> > more) of half the size (or less).  Kind of a pain I know, but just 
> > brainstorming here.
> >
> > Thanks,
> > chris
> >
> > -----Original Message-----
> > From: Colin Hudler 
> > [mailto:]
> > Sent: Thursday, December 17, 2009 10:18 AM
> > To: 
> >
> > Subject: Re: [grouper-users] gsh xmlimport memory usage
> >
> > The backup file is 256M.  Memberships are 1507480, in 90908 groups.  The
> > subject source is
> > edu.internet2.middleware.grouper.subj.GrouperJndiSourceAdapter, but
> > maybe I will change it and compare.
> >
> > Tom Zeller wrote:
> > > What is the filesize of backup.xml ? How many memberships do you have ?
> > >
> > > I've been able to xmlimport with Xmx=1024m, and we have ~1 million
> > > memberships (select count(*) from grouper_memberships), and our xml
> > > export is ~175MB.
> > >
> > > What source adapters are you using ?
> > >
> > > This doesn't answer your question, of course.
> > >
> > > TomZ
> > >
> > > On Thu, Dec 17, 2009 at 9:00 AM, Colin Hudler 
> > > <>
> > >  wrote:
> > > > Hi,
> > > >
> > > >  Congrats on the fantastic new release, nice stuff.  I am doing upgrade
> > > > rehearsals under Method 1,
> > > >
> > > > "./bin/gsh.sh -xmlimport GrouperSystem backup.xml"
> > > >
> > > > But it always uses all of the memory.  I increased it to 4GB, but it still
> > > > actually commits that much.  Before moving on to method 2, I thought I'd
> > > > ask, is there anything I can do reduce it?
> > > >
> > > > Obviously, this is the actual error:
> > > >
> > > > java.lang.OutOfMemoryError: Java heap space