Grouper Users - Open Discussion List

[Reid Watson <>]

Hi Shilen, 

Thank you for the information but is there anything that can be done in this 
situation to minimise the database size or improve import performance, I 
tested stopping the Grouper Loader while doing the bulk import but we noticed 
a performance delay when completing the bulk upload… 

Would you suggest migrating at a DB level for the group import (from DB to 
Grouper DB) instead of using Grouper Web service — is this worth an 
investigation ? 

Just really looking for advice on few tips and tricks with Bulk Importing 

Cheers

Reid 


> On 22/11/2017, at 6:46 AM, Shilen Patel 
> <>
>  wrote:
> 
> In general, you can’t truncate those tables unless you’re doing it 
> temporarily (e.g. migrating databases) and plan to sync it back up.  But 
> old data can be deleted after the objects in the non-PIT tables are 
> deleted.  There’s some information about that here:
>  
> https://spaces.internet2.edu/display/Grouper/Point+in+Time+Auditing
>  
> - Shilen
>  
> On 11/21/17, 1:18 AM, 
> "
>  on behalf of Reid Watson" 
> <
>  on behalf of 
> >
>  wrote:
>  
>     Hi Everyone, 
>     
>     Im a newbie to the Grouper Application but I was looking for some 
> advice around Grouper PIT tables, were currently in the process of 
> importing over 900 000 groups / sub groups into Grouper, were noticing the 
> following tables are
>     consuming all of the free disk space (72Gb) , for example
>     
>     select count(*) from grouper_pit_groups;
>     617648
>     
>     select count(*) from grouper_pit_group_set;
>     7232276
>     
>     select count(*) from grouper_pit_members;
>     617671
>     
>     select count(*) from grouper_pit_memberships;
>     1586755
>     
>     Questions 
>     
>     - Before we start our bulk import of the groups is it possible to 
> disable the writes to the PIT tables — we would enable after the import was 
> complete  (I don’t believe this will be possible but wanted to check)
>     - Or after the import is complete is it advisable to truncate the PIT 
> tables, if so is there be a simple way to mange the constraints by either 
> dropping / disabling the constraints on the following tables
>     
>     - GROUPER_PIT_ATTR_ASSN_ACTN 
>     - GROUPER_PUT_ATTR_ASSN_ACTN_SET 
>     - GROUPER_PIT_ATTR_ASSN_VALUE 
>     - GROUPER_PIT_ATTR_DEF_NAME 
>     - GROUPER_PIT_ATTR_DEF_NAME_SET 
>     - GROUPER_PIT_ATTRIBUTE_ASSIGN 
>     - GROUPER_PIT_ATTRIBUTE_DEF 
>     - GROUPER_PIT_FIELDS 
>     - GROUPER_PIT_GROUP_SET 
>     - GROUPER_PIT_GROUPS 
>     - GROUPER_PIT_MEMBERS 
>     - GROUPER_PIT_MEMBERSHIPS 
>     - GROUPER_PIT_ROLE_SETS 
>     - GROUPER_PIT_STEMS
>     
>     - Or is it a better option to stop the Grouper Loader so it won’t write 
> to the PIT tables while were running a large bulk import
>     
>     Cheers
>     
>     Reid