Grouper Users - Open Discussion List

[Chris Hyzer <>]

Did you do this part?

gsh 2% new 
edu.internet2.middleware.grouper.misc.AddMissingGroupSets().addAllMissingGroupSets()


It is required to see memberships.


Thanks,
Chris

-----Original Message-----
From: Colin Hudler 
[mailto:]
 
Sent: Wednesday, December 23, 2009 12:19 PM
To: 

Subject: Re: [grouper-users] gsh xmlimport memory usage

I started over and it went pretty well.  Towards the end of the 
migration, it died trying to create the view "grouper_memberships_v".  I 
dropped the view myself and continued the script where it left off.  I 
then had to repeate the process for "grouper_rpt_group_field_v", but the 
rest completed successfully, and both of the views exist now.

Here's the strange part.  The API doesn't think I belong to groups, even 
though grouper_memberships has entries for me (I took a peek at the db).

gsh 1% ses = GrouperSession.startRootSession();
edu.internet2.middleware.grouper.GrouperSession: 
0cb3aeeb887449c59832e60da816735d,'GrouperSystem','application'
gsh 2% s = findSubject("21882177X")
subject: id='21882177X' type='person' source='ucmcdb' name='Colin Hudler'
gsh 3% member = MemberFinder.findBySubject(ses, s)
member: id='21882177X' type='person' source='ucmcdb' 
uuid='599eb506-e655-43d9-bbf5-f38988444b3d'
gsh 4% member.getGroups()
gsh 5%

The UI confirms that (all?) stems and groups are present, but none of 
the groups have members.  Side note, the grouper_membership_v is an 
empty result hmm.  This is all only a test system.  I am going to 
continue prodding it, and I welcome troubleshooting suggestions.  Thanks 
for the help so far.

On 12/22/09 11:33 AM, Colin Hudler wrote:
> I looked harder and found that the column is present in backups and in
> production and various dev dbs, but since the migrating system doesn't
> have it, the migration script must have dropped it even though this
> happens *after* the part I get the error for. Hmm. In any case, I am
> going to take two steps back and start from a pristine database and see
> what happens.
>
> Chris Hyzer wrote:
>> Are you on the latest 1.4 before upgrading to 1.5? We test our
>> upgrades from version to version. So if you are on 1.3, you first need
>> to go to 1.4, then to 1.5... 1.4 should have a grouper_memberships.via_id
>>
>> Thanks,
>> Chris
>>
>> -----Original Message-----
>> From: Colin Hudler 
>> [mailto:]
>>  Sent: Tuesday,
>> December 22, 2009 12:14 PM
>> To: 
>> 
>> Subject: Re: [grouper-users] gsh xmlimport memory usage
>>
>> Thanks for all the suggestions. I was hoping to be able to partition
>> the import this way. In the meantime, I have been trying "Upgrade
>> Method 2". I eventually got to a point in the script where it runs:
>>
>> update grouper_memberships
>> set owner_group_id = (select gg.id from grouper_groups gg where gg.id
>> = owner_id),
>> owner_stem_id = (select gs.id from grouper_stems gs where gs.id =
>> owner_id),
>> via_composite_id = (select gc.id from grouper_composites gc where
>> gc.id = via_id),
>> via_id_bak = via_id,
>> via_id = null,
>> depth_bak = depth,
>> depth = 0,
>> parent_membership_bak = parent_membership,
>> parent_membership = null
>> where owner_group_id is null and owner_stem_id is null and
>> owner_attr_def_id is null
>> and via_composite_id is null and via_id_bak is null;
>> commit;
>>
>>
>> Which dies with
>> Error running script:
>> /srv/grouper/api-1.5.0/ddlScripts/grouperDdl_20091217_11_01_02_600.sql, 
>> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
>> Unknown column 'via_id' in 'field list'
>>
>> Indeed, I have no column by that name, but there is
>> via_composite_id and via_id_bak. It is possible I did something wrong.
>> I rolled back the database a few times and could have made a mistake.
>> I will probably roll back again and start over, but I thought I would
>> mention it here in case it is a real error.
>>
>> Tom Zeller wrote:
>>> In addition to importing stems separately, it might also be
>>> interesting to import first without memberships or permissions, then a
>>> second time with memberships and permissions.
>>>
>>> To ignore memberships and privileges, you would do something like
>>>
>>> bin/gsh.sh -xmlimport GrouperSystem export.xml import.properties
>>>
>>> import.properties :
>>>
>>> import.data.lists=ignore
>>> import.data.privileges=ignore
>>>
>>> And, of course, there's running with a profiler.
>>>
>>> TomZ
>>>
>>> On Thu, Dec 17, 2009 at 3:48 PM, Chris Hyzer 
>>> <>
>>> wrote:
>>>> Is this a 32bit machine or 64? 2gig might be the max that Java can use.
>>>>
>>>> We are planning to rewrite the import/export, and I hope to improve
>>>> performance in the rewrite. In the meantime, maybe if this is not a
>>>> 64 bit machine, you could try from another machine which is 64 bit.
>>>> Or maybe you could export twice and partition the stems so that you
>>>> have two files (or more) of half the size (or less). Kind of a pain
>>>> I know, but just brainstorming here.
>>>>
>>>> Thanks,
>>>> chris
>>>>
>>>> -----Original Message-----
>>>> From: Colin Hudler 
>>>> [mailto:]
>>>> Sent: Thursday, December 17, 2009 10:18 AM
>>>> To: 
>>>> 
>>>> Subject: Re: [grouper-users] gsh xmlimport memory usage
>>>>
>>>> The backup file is 256M. Memberships are 1507480, in 90908 groups. The
>>>> subject source is
>>>> edu.internet2.middleware.grouper.subj.GrouperJndiSourceAdapter, but
>>>> maybe I will change it and compare.
>>>>
>>>> Tom Zeller wrote:
>>>>> What is the filesize of backup.xml ? How many memberships do you
>>>>> have ?
>>>>>
>>>>> I've been able to xmlimport with Xmx=1024m, and we have ~1 million
>>>>> memberships (select count(*) from grouper_memberships), and our xml
>>>>> export is ~175MB.
>>>>>
>>>>> What source adapters are you using ?
>>>>>
>>>>> This doesn't answer your question, of course.
>>>>>
>>>>> TomZ
>>>>>
>>>>> On Thu, Dec 17, 2009 at 9:00 AM, Colin Hudler
>>>>> <>
>>>>>  wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Congrats on the fantastic new release, nice stuff. I am doing upgrade
>>>>>> rehearsals under Method 1,
>>>>>>
>>>>>> "./bin/gsh.sh -xmlimport GrouperSystem backup.xml"
>>>>>>
>>>>>> But it always uses all of the memory. I increased it to 4GB, but
>>>>>> it still
>>>>>> actually commits that much. Before moving on to method 2, I
>>>>>> thought I'd
>>>>>> ask, is there anything I can do reduce it?
>>>>>>
>>>>>> Obviously, this is the actual error:
>>>>>>
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>
>>
>
>