Skip to Content.
Sympa Menu

grouper-users - Re: [grouper-users] Maintenance on Grouper PIT tables

Subject: Grouper Users - Open Discussion List

List archive

Re: [grouper-users] Maintenance on Grouper PIT tables


Chronological Thread 
  • From: Shilen Patel <>
  • To: Reid Watson <>
  • Cc: "" <>
  • Subject: Re: [grouper-users] Maintenance on Grouper PIT tables
  • Date: Thu, 23 Nov 2017 11:55:55 +0000
  • Accept-language: en-US
  • Authentication-results: oit.duke.edu; spf=none
  • Ironport-phdr: 9a23:rff+FxKJKtntlsFt1tmcpTZWNBhigK39O0sv0rFitYgfK/TxwZ3uMQTl6Ol3ixeRBMOAuqIC07KempujcFRI2YyGvnEGfc4EfD4+ouJSoTYdBtWYA1bwNv/gYn9yNs1DUFh44yPzahANS47xaFLIv3K98yMZFAnhOgppPOT1HZPZg9iq2+yo9ZDeZwZFiCChbb9uMR67sRjfus4KjIV4N60/0AHJonxGe+RXwWNnO1eelAvi68mz4ZBu7T1et+ou+MBcX6r6eb84TaFDAzQ9L281/szrugLdQgaJ+3ART38ZkhtMAwjC8RH6QpL8uTb0u+ZhxCWXO9D9QKsqUjq+8ahkVB7oiD8GNzEn9mHXltdwh79frB64uhBz35LYbISTOfFjfK3SYMkaSHJBUMhPSiJBHo2yYYgBD+UDM+lYrpXyqVQBohalGQmjH+bvxiNUinLs36A31fkqHwHc3AwnGtIDqGnarNX0NKcWT++1yrTDwy/ZYPNKxzj97JXDfxc6ofGMXLJwddTeyU80GgzYk1WQqpLqMymL2esQrmiW9uxtXv+hhW4grgF+uDmvxsE0h4nRnIIVzVfE9SthwIYyJN24TlJ7bcS4H5tXsiGWL5N2QsQjQ2FupCY206MKtoW6fCgM0Zgr2QXQa+CGc4iN/B3vTfidLixii3J4Zb2/hgqy8Ummyu3zUMS/zVVErjJdn9XRtn0ByQbf58adRvdg/Uqs3SyD2x7d5+xGOUw4iKTWJp87zrMwl5cfq0fOEjX2lUj0lKOaaFgo9+my5+npfLnqvIKQOoxphg3kLKgjlMqyCvkiPAcURWiU4+G82aXj/ULnRLVKieU7kqzDv5DbIcQXv6G5AgFM3oo+9hqzFTCm0NUEnXUdMV1FZRGHgJbzN17QPf/4EO+zg1WqkDh12/DLJqDtD5XTInTZnrrtZ6tx5k1YxQYp09xS6ZBZBqkEIP3pW0/xsNLYDgU+Mwyx2+vnE9V91oIaWW2RBK+UK77dsUKL5u0xOeWDepIauCvlJ/g/+/HulWM5mUMafaSx0psXcne4Hul+I0qHe3rgm8oBEXwUsQokV+znk1mCUT9IZ3auRKIw+Cs3CIOgDYffWI+tmrqB0zmnHpFIfGxJFE2DEWq7P7mDDs0BaS7aHcZ9iTsCHey7Ro4tkw6nqBT9zZJuM+HX/iQeudTq355z9buAuws18GlPD8WZ3mfFd2hok2dAEyQ23aRyp2Rx11SOl6V0nqoLRpRo+/pVX1JiZtbnxOtgBoW3A1qZcw==
  • Spamdiagnosticmetadata: NSPM
  • Spamdiagnosticoutput: 1:99

Hi,

Can you tell us more about the delays that you’re seeing? Also, are you
migrating from one Grouper DB to another? If the daemon isn’t running while
you’re importing, then nothing is written to the grouper_pit_* tables until
after the daemon is started up again. So are you seeing delays while
importing even when the daemon is off?

Also, if you’re seeing that the import is significantly slower while the
daemon is running, that may suggest opportunities to tune the database server
I think.

Also note that if you don’t care about any of the existing point in time
audit data and you don’t care about notifications and change log updates for
the data that you’re importing, then you could leave the daemon off while
importing, truncate the temp change log when done, truncate all the pit
tables, and then sync the pit tables. I can give details if you want to try
that. It’ll at least speed up population of the grouper_pit_* tables.

Thanks!

- Shilen

On 11/22/17, 1:51 PM, "Reid Watson"
<>
wrote:

Hi Shilen,



Thank you for the information but is there anything that can be done in
this situation to minimise the database size or improve import performance, I
tested stopping the Grouper Loader while doing the bulk import but we noticed
a performance delay when completing the bulk upload…



Would you suggest migrating at a DB level for the group import (from DB
to Grouper DB) instead of using Grouper Web service — is this worth an
investigation ?



Just really looking for advice on few tips and tricks with Bulk Importing



Cheers



Reid





> On 22/11/2017, at 6:46 AM, Shilen Patel
<>
wrote:

>

> In general, you can’t truncate those tables unless you’re doing it
temporarily (e.g. migrating databases) and plan to sync it back up. But old
data can be deleted after the objects in the non-PIT tables are deleted.
There’s some information about that here:

>

>
https://urldefense.proofpoint.com/v2/url?u=https-3A__spaces.internet2.edu_display_Grouper_Point-2Bin-2BTime-2BAuditing&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=sWqutME58phurE0oO57Icg&m=3QErfYwCmPhhurHCQ3Oht3cdWJCGg4efjD9SEVb9oKs&s=i95T9Jh6tI9kUb1AbJUv_sRJDNmSJVmiUOcSccHCljQ&e=


>

> - Shilen

>

> On 11/21/17, 1:18 AM,
"
on behalf of Reid Watson"
<
on behalf of
>
wrote:

>

> Hi Everyone,

>

> Im a newbie to the Grouper Application but I was looking for some
advice around Grouper PIT tables, were currently in the process of importing
over 900 000 groups / sub groups into Grouper, were noticing the following
tables are

> consuming all of the free disk space (72Gb) , for example

>

> select count(*) from grouper_pit_groups;

> 617648

>

> select count(*) from grouper_pit_group_set;

> 7232276

>

> select count(*) from grouper_pit_members;

> 617671

>

> select count(*) from grouper_pit_memberships;

> 1586755

>

> Questions

>

> - Before we start our bulk import of the groups is it possible to
disable the writes to the PIT tables — we would enable after the import was
complete (I don’t believe this will be possible but wanted to check)

> - Or after the import is complete is it advisable to truncate the
PIT tables, if so is there be a simple way to mange the constraints by either
dropping / disabling the constraints on the following tables

>

> - GROUPER_PIT_ATTR_ASSN_ACTN

> - GROUPER_PUT_ATTR_ASSN_ACTN_SET

> - GROUPER_PIT_ATTR_ASSN_VALUE

> - GROUPER_PIT_ATTR_DEF_NAME

> - GROUPER_PIT_ATTR_DEF_NAME_SET

> - GROUPER_PIT_ATTRIBUTE_ASSIGN

> - GROUPER_PIT_ATTRIBUTE_DEF

> - GROUPER_PIT_FIELDS

> - GROUPER_PIT_GROUP_SET

> - GROUPER_PIT_GROUPS

> - GROUPER_PIT_MEMBERS

> - GROUPER_PIT_MEMBERSHIPS

> - GROUPER_PIT_ROLE_SETS

> - GROUPER_PIT_STEMS

>

> - Or is it a better option to stop the Grouper Loader so it won’t
write to the PIT tables while were running a large bulk import

>

> Cheers

>

> Reid








Archive powered by MHonArc 2.6.19.

Top of Page