Skip to Content.
Sympa Menu

grouper-dev - RE: [grouper-dev] help with math homework ?

Subject: Grouper Developers Forum

List archive

RE: [grouper-dev] help with math homework ?

Chronological Thread 
  • From: Chris Hyzer <>
  • To: Tom Zeller <>, Grouper Dev <>
  • Subject: RE: [grouper-dev] help with math homework ?
  • Date: Fri, 6 Jan 2012 04:23:03 +0000
  • Accept-language: en-US

Because the prop100k is not the proportion of the top 1000 names, it is the
proportion of all names. So if all names that exist, if you take a sample of
100k, then 880 of them are Smith. If you look at 100k of the top 1000 names,
then 2000 of them are Smith. The names you have in the list are only ~40% of
all names. So if you take 2000 and multiple by 40% you get ~800... in this
case, I think the 2000 is fine, (well, actually I think a cartesian product
of 200 first and 500 last names is fine, but if we want to go weighted, then
2000 smiths works for me :) )


-----Original Message-----

On Behalf Of Tom Zeller
Sent: Thursday, January 05, 2012 10:18 PM
To: Grouper Dev
Subject: [grouper-dev] help with math homework ?

So I have an AI to grab 100,000 names from the Internet and provision
them to ldap for grouperdemo.

Fortunately google exists, who clued me in that the census provides us
with a spreadsheet of the most common 1,000 surnames and their
proportion per 100,000 names. It looks like :

# From

I do not know what "rank" is, but the technical documentation from the
census data website may be of interest to identity matchers needing to
clean up name data. They had to clean up OCR.

Anyway, my issue is that the weighted random collection I googled for
returns approximately 2,000 SMITHs every time I generate 100,000
random surnames, when it should return 880.05 (prop100k from the
census data).

The weighted random collection is from

which lists the question as

"closed as not a real question"

which is perhaps why I get 2,000 SMITHs instead of 880.05.

public class RandomCollection<E> {
private final NavigableMap<Double, E> map = new TreeMap<Double, E>();
private final Random random;
private double total = 0;

public RandomCollection() {
this(new Random());

public RandomCollection(Random random) {
this.random = random;

public void add(double weight, E result) {
if (weight <= 0) return;
total += weight;
map.put(total, result);

public E next() {
double value = random.nextDouble() * total;
return map.ceilingEntry(value).getValue();

which I populated it from the census data

RandomCollection<String> surnames ...
surnames.add(Double.parseDouble(prop100k)/100000.0, name);

where prop100k and name match the columns of the census data.

Any ideas ?


P.S. I thought this would help with real-time provisioning testing as well.

Archive powered by MHonArc 2.6.16.

Top of Page