Grouper Users - Open Discussion List

[Colin Hudler <>]

I looked harder and found that the column is present in backups and in 
production and various dev dbs, but since the migrating system doesn't 
have it, the migration script must have dropped it even though this 
happens *after* the part I get the error for.  Hmm.  In any case, I am 
going to take two steps back and start from a pristine database and see 
what happens.

Chris Hyzer wrote:
> Are you on the latest 1.4 before upgrading to 1.5?  We test our upgrades from 
> version to version.  So if you are on 1.3, you first need to go to 1.4, then 
> to 1.5... 1.4 should have a grouper_memberships.via_id
>
> Thanks,
> Chris
>
> -----Original Message-----
> From: Colin Hudler [mailto:] 
> Sent: Tuesday, December 22, 2009 12:14 PM
> To: 
>
> Subject: Re: [grouper-users] gsh xmlimport memory usage
>
> Thanks for all the suggestions.  I was hoping to be able to partition 
> the import this way.  In the meantime, I have been trying "Upgrade 
> Method 2".  I eventually got to a point in the script where it runs:
>
> update grouper_memberships
> set owner_group_id = (select gg.id from grouper_groups gg where gg.id = 
> owner_id),
> owner_stem_id = (select gs.id from grouper_stems gs where gs.id = owner_id),
> via_composite_id = (select gc.id from grouper_composites gc where gc.id 
> = via_id),
> via_id_bak = via_id,
> via_id = null,
> depth_bak = depth,
> depth = 0,
> parent_membership_bak = parent_membership,
> parent_membership = null
> where owner_group_id is null and owner_stem_id is null and 
> owner_attr_def_id is null
> and via_composite_id is null and via_id_bak is null;
> commit;
>
>
> Which dies with
> Error running script: 
> /srv/grouper/api-1.5.0/ddlScripts/grouperDdl_20091217_11_01_02_600.sql, 
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown 
> column 'via_id' in 'field list'
>
> Indeed, I have no column by that name, but there is
> via_composite_id and via_id_bak.  It is possible I did something wrong. 
>   I rolled back the database a few times and could have made a mistake. 
>   I will probably roll back again and start over, but I thought I would 
> mention it here in case it is a real error.
>
> Tom Zeller wrote:
> > In addition to importing stems separately, it might also be
> > interesting to import first without memberships or permissions, then a
> > second time with memberships and permissions.
> >
> > To ignore memberships and privileges, you would do something like
> >
> >  bin/gsh.sh -xmlimport GrouperSystem export.xml import.properties
> >
> > import.properties :
> >
> >  import.data.lists=ignore
> >  import.data.privileges=ignore
> >
> > And, of course, there's running with a profiler.
> >
> > TomZ
> >
> > On Thu, Dec 17, 2009 at 3:48 PM, Chris Hyzer 
> > <>
> >  wrote:
> > > Is this a 32bit machine or 64?  2gig might be the max that Java can use.
> > >
> > > We are planning to rewrite the import/export, and I hope to improve 
> > > performance in the rewrite.  In the meantime, maybe if this is not a 64 bit 
> > > machine, you could try from another machine which is 64 bit.  Or maybe you 
> > > could export twice and partition the stems so that you have two files (or 
> > > more) of half the size (or less).  Kind of a pain I know, but just 
> > > brainstorming here.
> > >
> > > Thanks,
> > > chris
> > >
> > > -----Original Message-----
> > > From: Colin Hudler 
> > > [mailto:]
> > > Sent: Thursday, December 17, 2009 10:18 AM
> > > To: 
> > >
> > > Subject: Re: [grouper-users] gsh xmlimport memory usage
> > >
> > > The backup file is 256M.  Memberships are 1507480, in 90908 groups.  The
> > > subject source is
> > > edu.internet2.middleware.grouper.subj.GrouperJndiSourceAdapter, but
> > > maybe I will change it and compare.
> > >
> > > Tom Zeller wrote:
> > > > What is the filesize of backup.xml ? How many memberships do you have ?
> > > >
> > > > I've been able to xmlimport with Xmx=1024m, and we have ~1 million
> > > > memberships (select count(*) from grouper_memberships), and our xml
> > > > export is ~175MB.
> > > >
> > > > What source adapters are you using ?
> > > >
> > > > This doesn't answer your question, of course.
> > > >
> > > > TomZ
> > > >
> > > > On Thu, Dec 17, 2009 at 9:00 AM, Colin Hudler 
> > > > <>
> > > >  wrote:
> > > > > Hi,
> > > > >
> > > > >  Congrats on the fantastic new release, nice stuff.  I am doing upgrade
> > > > > rehearsals under Method 1,
> > > > >
> > > > > "./bin/gsh.sh -xmlimport GrouperSystem backup.xml"
> > > > >
> > > > > But it always uses all of the memory.  I increased it to 4GB, but it still
> > > > > actually commits that much.  Before moving on to method 2, I thought I'd
> > > > > ask, is there anything I can do reduce it?
> > > > >
> > > > > Obviously, this is the actual error:
> > > > >
> > > > > java.lang.OutOfMemoryError: Java heap space