Grouper Users - Open Discussion List

[Shilen Patel <>]

Hi,

Can you tell us more about the delays that you’re seeing?  Also, are you 
migrating from one Grouper DB to another?  If the daemon isn’t running while 
you’re importing, then nothing is written to the grouper_pit_* tables until 
after the daemon is started up again.  So are you seeing delays while 
importing even when the daemon is off?  

Also, if you’re seeing that the import is significantly slower while the 
daemon is running, that may suggest opportunities to tune the database server 
I think.

Also note that if you don’t care about any of the existing point in time 
audit data and you don’t care about notifications and change log updates for 
the data that you’re importing, then you could leave the daemon off while 
importing, truncate the temp change log when done, truncate all the pit 
tables, and then sync the pit tables.  I can give details if you want to try 
that.  It’ll at least speed up population of the grouper_pit_* tables.

Thanks!

- Shilen

On 11/22/17, 1:51 PM, "Reid Watson" 
<>
 wrote:

    Hi Shilen, 
    
    
    
    Thank you for the information but is there anything that can be done in 
this situation to minimise the database size or improve import performance, I 
tested stopping the Grouper Loader while doing the bulk import but we noticed 
a performance delay when completing the bulk upload… 
    
    
    
    Would you suggest migrating at a DB level for the group import (from DB 
to Grouper DB) instead of using Grouper Web service — is this worth an 
investigation ? 
    
    
    
    Just really looking for advice on few tips and tricks with Bulk Importing 
    
    
    
    Cheers
    
    
    
    Reid 
    
    
    
    
    
    > On 22/11/2017, at 6:46 AM, Shilen Patel 
<>
 wrote:
    
    > 
    
    > In general, you can’t truncate those tables unless you’re doing it 
temporarily (e.g. migrating databases) and plan to sync it back up.  But old 
data can be deleted after the objects in the non-PIT tables are deleted.  
There’s some information about that here:
    
    >  
    
    > 
https://urldefense.proofpoint.com/v2/url?u=https-3A__spaces.internet2.edu_display_Grouper_Point-2Bin-2BTime-2BAuditing&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=sWqutME58phurE0oO57Icg&m=3QErfYwCmPhhurHCQ3Oht3cdWJCGg4efjD9SEVb9oKs&s=i95T9Jh6tI9kUb1AbJUv_sRJDNmSJVmiUOcSccHCljQ&e=
 
    
    >  
    
    > - Shilen
    
    >  
    
    > On 11/21/17, 1:18 AM, 
"
 on behalf of Reid Watson" 
<
 on behalf of 
>
 wrote:
    
    >  
    
    >     Hi Everyone, 
    
    >     
    
    >     Im a newbie to the Grouper Application but I was looking for some 
advice around Grouper PIT tables, were currently in the process of importing 
over 900 000 groups / sub groups into Grouper, were noticing the following 
tables are
    
    >     consuming all of the free disk space (72Gb) , for example
    
    >     
    
    >     select count(*) from grouper_pit_groups;
    
    >     617648
    
    >     
    
    >     select count(*) from grouper_pit_group_set;
    
    >     7232276
    
    >     
    
    >     select count(*) from grouper_pit_members;
    
    >     617671
    
    >     
    
    >     select count(*) from grouper_pit_memberships;
    
    >     1586755
    
    >     
    
    >     Questions 
    
    >     
    
    >     - Before we start our bulk import of the groups is it possible to 
disable the writes to the PIT tables — we would enable after the import was 
complete  (I don’t believe this will be possible but wanted to check)
    
    >     - Or after the import is complete is it advisable to truncate the 
PIT tables, if so is there be a simple way to mange the constraints by either 
dropping / disabling the constraints on the following tables
    
    >     
    
    >     - GROUPER_PIT_ATTR_ASSN_ACTN 
    
    >     - GROUPER_PUT_ATTR_ASSN_ACTN_SET 
    
    >     - GROUPER_PIT_ATTR_ASSN_VALUE 
    
    >     - GROUPER_PIT_ATTR_DEF_NAME 
    
    >     - GROUPER_PIT_ATTR_DEF_NAME_SET 
    
    >     - GROUPER_PIT_ATTRIBUTE_ASSIGN 
    
    >     - GROUPER_PIT_ATTRIBUTE_DEF 
    
    >     - GROUPER_PIT_FIELDS 
    
    >     - GROUPER_PIT_GROUP_SET 
    
    >     - GROUPER_PIT_GROUPS 
    
    >     - GROUPER_PIT_MEMBERS 
    
    >     - GROUPER_PIT_MEMBERSHIPS 
    
    >     - GROUPER_PIT_ROLE_SETS 
    
    >     - GROUPER_PIT_STEMS
    
    >     
    
    >     - Or is it a better option to stop the Grouper Loader so it won’t 
write to the PIT tables while were running a large bulk import
    
    >     
    
    >     Cheers
    
    >     
    
    >     Reid