Skip to Content.
Sympa Menu

grouper-users - Re: [grouper-users] Memory Issues While Importing

Subject: Grouper Users - Open Discussion List

List archive

Re: [grouper-users] Memory Issues While Importing

Chronological Thread 
  • From:
  • To: Tom Barton <>, Shilen Patel <>
  • Cc:
  • Subject: Re: [grouper-users] Memory Issues While Importing
  • Date: Thu, 7 Jun 2007 10:24:07 -0400

At 8:53 AM -0500 6/7/07, Tom Barton wrote:
This is probably due to DOM processing memory needs, a drawback to dealing with very large object models. To xml-import, the size of the DOM and available memory will need to be balanced. So, either assign more ram, if possible, or break the xml into pieces. If your database is logically partitioned by stems, one approach is to xml-export then import by stem. Or resort to editing that huge file to produce logically complete chunks.

We ran into the same problem. We have an existing Brown Grouper install, which has tens of thousands of groups. We felt we needed to migrate many of these into MACE Grouper.

In addition, on an ongoing basis, many of our "base" group definitions are provisioned from our business systems (eg courses, departments, various slices of the community (eg faculty:on-campus, etc). We use Brown Grouper's set arithmetic functionality to build "effective" groups from the base definitions + some hand-entered "tuning"; we plan to do the same with MACE Grouper. So, we have an ongoing need to maintain the membership of many very large base groups.

We tried using the supplied xml-import tool, but ran into a number of problems; in the end, we judged these problems to be intractable.

We ended up writing our own program to load/maintain the membership of these groups. So far, this seems to be working well. The program is still in development (eg we recently added functionality to set ACLs on the groups in MACE Grouper). And we're still trying to improve its performance.

We're willing to share this program with the community, if people are interested. Obviously, the "input" side of this program is currently somewhat Brown specific. However, all of the logic is isolated from that, so it should be somewhat portable.

Archive powered by MHonArc 2.6.16.

Top of Page