Grouper Users - Open Discussion List

[Reid Watson <>]

Hi Shilen 

> Can you tell us more about the delays that you’re seeing?

- Shutdown Loader Daemon 
- Post a request to the WebService 
- Check the Group table for the entry (Not the PIT table the standard froiups 
table) 

To be honest you need more details (repose and insert times) than this so I 
will debug further 

> then you could leave the daemon off while importing, truncate the temp 
> change log when done, truncate all the pit tables, and then sync the pit 
> tables. 

That would be great (we don’t care about the bulk uploaded data — only new 
data when we go live) 

Thank you for the support 

Cheers 

Reid 



> On 24/11/2017, at 12:55 AM, Shilen Patel 
> <>
>  wrote:
> 
> Hi,
> 
> Can you tell us more about the delays that you’re seeing?  Also, are you 
> migrating from one Grouper DB to another?  If the daemon isn’t running 
> while you’re importing, then nothing is written to the grouper_pit_* tables 
> until after the daemon is started up again.  So are you seeing delays while 
> importing even when the daemon is off?  
> 
> Also, if you’re seeing that the import is significantly slower while the 
> daemon is running, that may suggest opportunities to tune the database 
> server I think.
> 
> Also note that if you don’t care about any of the existing point in time 
> audit data and you don’t care about notifications and change log updates 
> for the data that you’re importing, then you could leave the daemon off 
> while importing, truncate the temp change log when done, truncate all the 
> pit tables, and then sync the pit tables.  I can give details if you want 
> to try that.  It’ll at least speed up population of the grouper_pit_* 
> tables.
> 
> Thanks!
> 
> - Shilen
> 
> On 11/22/17, 1:51 PM, "Reid Watson" 
> <>
>  wrote:
> 
>    Hi Shilen, 
> 
> 
> 
>    Thank you for the information but is there anything that can be done in 
> this situation to minimise the database size or improve import performance, 
> I tested stopping the Grouper Loader while doing the bulk import but we 
> noticed a performance delay when completing the bulk upload… 
> 
> 
> 
>    Would you suggest migrating at a DB level for the group import (from DB 
> to Grouper DB) instead of using Grouper Web service — is this worth an 
> investigation ? 
> 
> 
> 
>    Just really looking for advice on few tips and tricks with Bulk 
> Importing 
> 
> 
> 
>    Cheers
> 
> 
> 
>    Reid 
> 
> 
> 
> 
> 
>> On 22/11/2017, at 6:46 AM, Shilen Patel 
>> <>
>>  wrote:
> 
>> 
> 
>> In general, you can’t truncate those tables unless you’re doing it 
>> temporarily (e.g. migrating databases) and plan to sync it back up.  But 
>> old data can be deleted after the objects in the non-PIT tables are 
>> deleted.  There’s some information about that here:
> 
>> 
> 
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__spaces.internet2.edu_display_Grouper_Point-2Bin-2BTime-2BAuditing&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=sWqutME58phurE0oO57Icg&m=3QErfYwCmPhhurHCQ3Oht3cdWJCGg4efjD9SEVb9oKs&s=i95T9Jh6tI9kUb1AbJUv_sRJDNmSJVmiUOcSccHCljQ&e=
>>  
> 
>> 
> 
>> - Shilen
> 
>> 
> 
>> On 11/21/17, 1:18 AM, 
>> "
>>  on behalf of Reid Watson" 
>> <
>>  on behalf of 
>> >
>>  wrote:
> 
>> 
> 
>>    Hi Everyone, 
> 
>> 
> 
>>    Im a newbie to the Grouper Application but I was looking for some 
>> advice around Grouper PIT tables, were currently in the process of 
>> importing over 900 000 groups / sub groups into Grouper, were noticing the 
>> following tables are
> 
>>    consuming all of the free disk space (72Gb) , for example
> 
>> 
> 
>>    select count(*) from grouper_pit_groups;
> 
>>    617648
> 
>> 
> 
>>    select count(*) from grouper_pit_group_set;
> 
>>    7232276
> 
>> 
> 
>>    select count(*) from grouper_pit_members;
> 
>>    617671
> 
>> 
> 
>>    select count(*) from grouper_pit_memberships;
> 
>>    1586755
> 
>> 
> 
>>    Questions 
> 
>> 
> 
>>    - Before we start our bulk import of the groups is it possible to 
>> disable the writes to the PIT tables — we would enable after the import 
>> was complete  (I don’t believe this will be possible but wanted to check)
> 
>>    - Or after the import is complete is it advisable to truncate the PIT 
>> tables, if so is there be a simple way to mange the constraints by either 
>> dropping / disabling the constraints on the following tables
> 
>> 
> 
>>    - GROUPER_PIT_ATTR_ASSN_ACTN 
> 
>>    - GROUPER_PUT_ATTR_ASSN_ACTN_SET 
> 
>>    - GROUPER_PIT_ATTR_ASSN_VALUE 
> 
>>    - GROUPER_PIT_ATTR_DEF_NAME 
> 
>>    - GROUPER_PIT_ATTR_DEF_NAME_SET 
> 
>>    - GROUPER_PIT_ATTRIBUTE_ASSIGN 
> 
>>    - GROUPER_PIT_ATTRIBUTE_DEF 
> 
>>    - GROUPER_PIT_FIELDS 
> 
>>    - GROUPER_PIT_GROUP_SET 
> 
>>    - GROUPER_PIT_GROUPS 
> 
>>    - GROUPER_PIT_MEMBERS 
> 
>>    - GROUPER_PIT_MEMBERSHIPS 
> 
>>    - GROUPER_PIT_ROLE_SETS 
> 
>>    - GROUPER_PIT_STEMS
> 
>> 
> 
>>    - Or is it a better option to stop the Grouper Loader so it won’t write 
>> to the PIT tables while were running a large bulk import
> 
>> 
> 
>>    Cheers
> 
>> 
> 
>>    Reid 
> 
> 
> 
> 
>