Grouper Developers Forum

[caleb racey <>]

Years back when we were setting up our data infrastructure I put a fair bit of effort into looking for a good torture test identity data set to test our data feeds

e.g. a data set with names with apostrophes, accents, really long names, names without surnames, hyphenated names, multiple firstnames, multiple last names etc etc

 

I has hoped that since pretty much everyone who has processed name date will have been bitten by false assumptions someone would have grouped together example of all the edge cases so you could test against them. Unfortunately all my research ever uncovered where numerous blog posts  talking about how pretty much every assumption you make about name data format turns out to be false.

 

I still think a good torture test data set would be immensely valuable but have yet to find an example.   Generating fake data is a good approach but is unlikely to uncover all the nasty rare examples.

 

cal

 

 

 

 

 

 

 

From: [mailto:] On Behalf Of McDermott, Michael
Sent: 06 January 2012 12:46 PM
To: Chris Hyzer
Cc: Tom Zeller; Grouper Dev
Subject: Re: [grouper-dev] help with math homework ?

 

There are libraries in Perl and Ruby at least that do this sort of thing (generate fake data), would they be helpful?

 

On Thu, Jan 5, 2012 at 11:23 PM, Chris Hyzer <> wrote:

Because the prop100k is not the proportion of the top 1000 names, it is the proportion of all names.  So if all names that exist, if you take a sample of 100k, then 880 of them are Smith.  If you look at 100k of the top 1000 names, then 2000 of them are Smith.  The names you have in the list are only ~40% of all names.  So if you take 2000 and multiple by 40% you get ~800...  in this case, I think the 2000 is fine, (well, actually I think a cartesian product of 200 first and 500 last names is fine, but if we want to go weighted, then 2000 smiths works for me :) )

Thanks,
Chris

-----Original Message-----
From: [mailto:] On Behalf Of Tom Zeller
Sent: Thursday, January 05, 2012 10:18 PM
To: Grouper Dev
Subject: [grouper-dev] help with math homework ?

So I have an AI to grab 100,000 names from the Internet and provision
them to ldap for grouperdemo.

Fortunately google exists, who clued me in that the census provides us
with a spreadsheet of the most common 1,000 surnames and their
proportion per 100,000 names. It looks like :

# From http://www.census.gov/genealogy/www/data/2000surnames
# name,rank,count,prop100k,cum_prop100k,pctwhite,pctblack,pctapi,pctaian,pct2prace,pcthispanic
SMITH,1,2376206,880.85,880.85,73.35,22.22,0.4,0.85,1.63,1.56
...

I do not know what "rank" is, but the technical documentation from the
census data website may be of interest to identity matchers needing to
clean up name data. They had to clean up OCR.

Anyway, my issue is that the weighted random collection I googled for
returns approximately 2,000 SMITHs every time I generate 100,000
random surnames, when it should return 880.05 (prop100k from the
census data).

The weighted random collection is from

 http://stackoverflow.com/questions/6409652/random-weighted-selection-java-framework

which lists the question as

 "closed as not a real question"

which is perhaps why I get 2,000 SMITHs instead of 880.05.

public class RandomCollection<E> {
   private final NavigableMap<Double, E> map = new TreeMap<Double, E>();
   private final Random random;
   private double total = 0;

   public RandomCollection() {
       this(new Random());
   }

   public RandomCollection(Random random) {
       this.random = random;
   }

   public void add(double weight, E result) {
       if (weight <= 0) return;
       total += weight;
       map.put(total, result);
   }

   public E next() {
       double value = random.nextDouble() * total;
       return map.ceilingEntry(value).getValue();
   }
}

which I populated it from the census data

 RandomCollection<String> surnames ...
  surnames.add(Double.parseDouble(prop100k)/100000.0, name);

where prop100k and name match the columns of the census data.

Any ideas ?

Thanks,
TomZ

P.S. I thought this would help with real-time provisioning testing as well.

 

--
Michael J. McDermott
Lead Developer, Identity and Access Management
Brown University