Grouper Users - Open Discussion List

[Colin Hudler <>]

I started over and it went pretty well.  Towards the end of the 
migration, it died trying to create the view "grouper_memberships_v".  I 
dropped the view myself and continued the script where it left off.  I 
then had to repeate the process for "grouper_rpt_group_field_v", but the 
rest completed successfully, and both of the views exist now.

Here's the strange part.  The API doesn't think I belong to groups, even 
though grouper_memberships has entries for me (I took a peek at the db).

gsh 1% ses = GrouperSession.startRootSession();
edu.internet2.middleware.grouper.GrouperSession: 
0cb3aeeb887449c59832e60da816735d,'GrouperSystem','application'
gsh 2% s = findSubject("21882177X")
subject: id='21882177X' type='person' source='ucmcdb' name='Colin Hudler'
gsh 3% member = MemberFinder.findBySubject(ses, s)
member: id='21882177X' type='person' source='ucmcdb' 
uuid='599eb506-e655-43d9-bbf5-f38988444b3d'
gsh 4% member.getGroups()
gsh 5%

The UI confirms that (all?) stems and groups are present, but none of 
the groups have members.  Side note, the grouper_membership_v is an 
empty result hmm.  This is all only a test system.  I am going to 
continue prodding it, and I welcome troubleshooting suggestions.  Thanks 
for the help so far.

On 12/22/09 11:33 AM, Colin Hudler wrote:
> I looked harder and found that the column is present in backups and in
> production and various dev dbs, but since the migrating system doesn't
> have it, the migration script must have dropped it even though this
> happens *after* the part I get the error for. Hmm. In any case, I am
> going to take two steps back and start from a pristine database and see
> what happens.
>
> Chris Hyzer wrote:
> > Are you on the latest 1.4 before upgrading to 1.5? We test our
> > upgrades from version to version. So if you are on 1.3, you first need
> > to go to 1.4, then to 1.5... 1.4 should have a grouper_memberships.via_id
> >
> > Thanks,
> > Chris
> >
> > -----Original Message-----
> > From: Colin Hudler 
> > [mailto:]
> >  Sent: Tuesday,
> > December 22, 2009 12:14 PM
> > To: 
> >
> > Subject: Re: [grouper-users] gsh xmlimport memory usage
> >
> > Thanks for all the suggestions. I was hoping to be able to partition
> > the import this way. In the meantime, I have been trying "Upgrade
> > Method 2". I eventually got to a point in the script where it runs:
> >
> > update grouper_memberships
> > set owner_group_id = (select gg.id from grouper_groups gg where gg.id
> > = owner_id),
> > owner_stem_id = (select gs.id from grouper_stems gs where gs.id =
> > owner_id),
> > via_composite_id = (select gc.id from grouper_composites gc where
> > gc.id = via_id),
> > via_id_bak = via_id,
> > via_id = null,
> > depth_bak = depth,
> > depth = 0,
> > parent_membership_bak = parent_membership,
> > parent_membership = null
> > where owner_group_id is null and owner_stem_id is null and
> > owner_attr_def_id is null
> > and via_composite_id is null and via_id_bak is null;
> > commit;
> >
> >
> > Which dies with
> > Error running script:
> > /srv/grouper/api-1.5.0/ddlScripts/grouperDdl_20091217_11_01_02_600.sql, 
> > com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
> > Unknown column 'via_id' in 'field list'
> >
> > Indeed, I have no column by that name, but there is
> > via_composite_id and via_id_bak. It is possible I did something wrong.
> > I rolled back the database a few times and could have made a mistake.
> > I will probably roll back again and start over, but I thought I would
> > mention it here in case it is a real error.
> >
> > Tom Zeller wrote:
> > > In addition to importing stems separately, it might also be
> > > interesting to import first without memberships or permissions, then a
> > > second time with memberships and permissions.
> > >
> > > To ignore memberships and privileges, you would do something like
> > >
> > > bin/gsh.sh -xmlimport GrouperSystem export.xml import.properties
> > >
> > > import.properties :
> > >
> > > import.data.lists=ignore
> > > import.data.privileges=ignore
> > >
> > > And, of course, there's running with a profiler.
> > >
> > > TomZ
> > >
> > > On Thu, Dec 17, 2009 at 3:48 PM, Chris Hyzer 
> > > <>
> > > wrote:
> > > > Is this a 32bit machine or 64? 2gig might be the max that Java can use.
> > > >
> > > > We are planning to rewrite the import/export, and I hope to improve
> > > > performance in the rewrite. In the meantime, maybe if this is not a
> > > > 64 bit machine, you could try from another machine which is 64 bit.
> > > > Or maybe you could export twice and partition the stems so that you
> > > > have two files (or more) of half the size (or less). Kind of a pain
> > > > I know, but just brainstorming here.
> > > >
> > > > Thanks,
> > > > chris
> > > >
> > > > -----Original Message-----
> > > > From: Colin Hudler 
> > > > [mailto:]
> > > > Sent: Thursday, December 17, 2009 10:18 AM
> > > > To: 
> > > >
> > > > Subject: Re: [grouper-users] gsh xmlimport memory usage
> > > >
> > > > The backup file is 256M. Memberships are 1507480, in 90908 groups. The
> > > > subject source is
> > > > edu.internet2.middleware.grouper.subj.GrouperJndiSourceAdapter, but
> > > > maybe I will change it and compare.
> > > >
> > > > Tom Zeller wrote:
> > > > > What is the filesize of backup.xml ? How many memberships do you
> > > > > have ?
> > > > >
> > > > > I've been able to xmlimport with Xmx=1024m, and we have ~1 million
> > > > > memberships (select count(*) from grouper_memberships), and our xml
> > > > > export is ~175MB.
> > > > >
> > > > > What source adapters are you using ?
> > > > >
> > > > > This doesn't answer your question, of course.
> > > > >
> > > > > TomZ
> > > > >
> > > > > On Thu, Dec 17, 2009 at 9:00 AM, Colin Hudler
> > > > > <>
> > > > >  wrote:
> > > > > > Hi,
> > > > > >
> > > > > > Congrats on the fantastic new release, nice stuff. I am doing upgrade
> > > > > > rehearsals under Method 1,
> > > > > >
> > > > > > "./bin/gsh.sh -xmlimport GrouperSystem backup.xml"
> > > > > >
> > > > > > But it always uses all of the memory. I increased it to 4GB, but
> > > > > > it still
> > > > > > actually commits that much. Before moving on to method 2, I
> > > > > > thought I'd
> > > > > > ask, is there anything I can do reduce it?
> > > > > >
> > > > > > Obviously, this is the actual error:
> > > > > >
> > > > > > java.lang.OutOfMemoryError: Java heap space