Grouper Developers Forum

[Chris Hyzer <>]

Because the prop100k is not the proportion of the top 1000 names, it is the 
proportion of all names.  So if all names that exist, if you take a sample of 
100k, then 880 of them are Smith.  If you look at 100k of the top 1000 names, 
then 2000 of them are Smith.  The names you have in the list are only ~40% of 
all names.  So if you take 2000 and multiple by 40% you get ~800...  in this 
case, I think the 2000 is fine, (well, actually I think a cartesian product 
of 200 first and 500 last names is fine, but if we want to go weighted, then 
2000 smiths works for me :) )

Thanks,
Chris

-----Original Message-----
From: 

 
[mailto:]
 On Behalf Of Tom Zeller
Sent: Thursday, January 05, 2012 10:18 PM
To: Grouper Dev
Subject: [grouper-dev] help with math homework ?

So I have an AI to grab 100,000 names from the Internet and provision
them to ldap for grouperdemo.

Fortunately google exists, who clued me in that the census provides us
with a spreadsheet of the most common 1,000 surnames and their
proportion per 100,000 names. It looks like :

# From http://www.census.gov/genealogy/www/data/2000surnames
# 
name,rank,count,prop100k,cum_prop100k,pctwhite,pctblack,pctapi,pctaian,pct2prace,pcthispanic
SMITH,1,2376206,880.85,880.85,73.35,22.22,0.4,0.85,1.63,1.56
...

I do not know what "rank" is, but the technical documentation from the
census data website may be of interest to identity matchers needing to
clean up name data. They had to clean up OCR.

Anyway, my issue is that the weighted random collection I googled for
returns approximately 2,000 SMITHs every time I generate 100,000
random surnames, when it should return 880.05 (prop100k from the
census data).

The weighted random collection is from

 
http://stackoverflow.com/questions/6409652/random-weighted-selection-java-framework

which lists the question as

 "closed as not a real question"

which is perhaps why I get 2,000 SMITHs instead of 880.05.

public class RandomCollection<E> {
    private final NavigableMap<Double, E> map = new TreeMap<Double, E>();
    private final Random random;
    private double total = 0;

    public RandomCollection() {
        this(new Random());
    }

    public RandomCollection(Random random) {
        this.random = random;
    }

    public void add(double weight, E result) {
        if (weight <= 0) return;
        total += weight;
        map.put(total, result);
    }

    public E next() {
        double value = random.nextDouble() * total;
        return map.ceilingEntry(value).getValue();
    }
}

which I populated it from the census data

 RandomCollection<String> surnames ...
   surnames.add(Double.parseDouble(prop100k)/100000.0, name);

where prop100k and name match the columns of the census data.

Any ideas ?

Thanks,
TomZ

P.S. I thought this would help with real-time provisioning testing as well.