Grouper Developers Forum

["McDermott, Michael" <>]

There are libraries in Perl and Ruby at least that do this sort of thing (generate fake data), would they be helpful?

On Thu, Jan 5, 2012 at 11:23 PM, Chris Hyzer <> wrote:

Because the prop100k is not the proportion of the top 1000 names, it is the proportion of all names.  So if all names that exist, if you take a sample of 100k, then 880 of them are Smith.  If you look at 100k of the top 1000 names, then 2000 of them are Smith.  The names you have in the list are only ~40% of all names.  So if you take 2000 and multiple by 40% you get ~800...  in this case, I think the 2000 is fine, (well, actually I think a cartesian product of 200 first and 500 last names is fine, but if we want to go weighted, then 2000 smiths works for me :) )

Thanks,
Chris

-----Original Message-----
From: [mailto:] On Behalf Of Tom Zeller
Sent: Thursday, January 05, 2012 10:18 PM
To: Grouper Dev
Subject: [grouper-dev] help with math homework ?

So I have an AI to grab 100,000 names from the Internet and provision
them to ldap for grouperdemo.

Fortunately google exists, who clued me in that the census provides us
with a spreadsheet of the most common 1,000 surnames and their
proportion per 100,000 names. It looks like :

# From http://www.census.gov/genealogy/www/data/2000surnames
# name,rank,count,prop100k,cum_prop100k,pctwhite,pctblack,pctapi,pctaian,pct2prace,pcthispanic
SMITH,1,2376206,880.85,880.85,73.35,22.22,0.4,0.85,1.63,1.56
...

I do not know what "rank" is, but the technical documentation from the
census data website may be of interest to identity matchers needing to
clean up name data. They had to clean up OCR.

Anyway, my issue is that the weighted random collection I googled for
returns approximately 2,000 SMITHs every time I generate 100,000
random surnames, when it should return 880.05 (prop100k from the
census data).

The weighted random collection is from

 http://stackoverflow.com/questions/6409652/random-weighted-selection-java-framework

which lists the question as

 "closed as not a real question"

which is perhaps why I get 2,000 SMITHs instead of 880.05.

public class RandomCollection<E> {
   private final NavigableMap<Double, E> map = new TreeMap<Double, E>();
   private final Random random;
   private double total = 0;

   public RandomCollection() {
       this(new Random());
   }

   public RandomCollection(Random random) {
       this.random = random;
   }

   public void add(double weight, E result) {
       if (weight <= 0) return;
       total += weight;
       map.put(total, result);
   }

   public E next() {
       double value = random.nextDouble() * total;
       return map.ceilingEntry(value).getValue();
   }
}

which I populated it from the census data

 RandomCollection<String> surnames ...
  surnames.add(Double.parseDouble(prop100k)/100000.0, name);

where prop100k and name match the columns of the census data.

Any ideas ?

Thanks,
TomZ

P.S. I thought this would help with real-time provisioning testing as well.

--
Michael J. McDermott
Lead Developer, Identity and Access Management
Brown University