Sources of Surname Data

From edegan.com
Revision as of 20:50, 8 July 2009 by imported>Ed
Jump to navigation Jump to search

We are primarily interested in sources of surname data that contain both surnames and countries of birth for surnames.

Olympic Athletes

The Olympic Athletes data was taken from the relevant pages on wikipedia. Local copies of all of the individual country pages from the 2004 Summer Games were retrieved by a script (Olympics-RetrievePageURLs.pl) that uses an offline version (Olympics-OfflineSource.html) of the wikipedia Nations at the 2004 Summer Olympics page. This script also constructs a list of participating countries (Olympics-ParticipatingCountries.txt).

The offline pages were then parsed by another script (Olympics-ExtractOlypiads.pl) and the resulting output (Olympics-RawOutput.txt) was checked by hand. This output is the basic names set with countries for the 2004 Olympic Athletes used here. Because some individuals competed in multiple events, identical full name strings were collapsed to produce a single record with a count. It seems unlikely that many John Joe Smiths entered, making such a reduction erroneous. Users of these scripts should the wikipedia source files have likely changed and should check results carefully.

The country names were then corrected to the UN GeoRegions and coded using SQL scripts, and country with idiosyncratic name reversals were marked to produce a normalization input file (Olympics-RawOutputWithUNReversal.txt). The NormalizeSurnames.pl script was with following options (and defaults):

perl -i=Olympics-RawOutputWithUNReversal.txt -ncol=1 -rcol=3

The resultant output (Olympics-RawOutputWithUNReversal-Normalized.txt) was used to create the n-gram variables.

Internet Movie Database (IMDB)

A list of all actors and their birth countries was extracted from the IMDB biographies file using a simple one-off script (IMDB-ExtractNamesAndCountries.pl). This produced the input file http://www.edegan.com/repository/IMDB-BioData.txt IMDB-BioData.txt]. Much as with the Olympics data the country names were then corrected to the UN GeoRegions, with individuals who were born in non-recognized jurisdictions, such as on a cruise ship at sea, excluded (see IMDB-BiosUNCountryCodes.txt).

The NormalizeSurnames.pl script was with following options (and defaults):

perl -i=IMDB-BiosUNCountryCodes.txt -ncol=1 -comma=1

The resultant output (IMDB-BiosUNCountryCodes-Normalized.txt) was used to create the n-gram variables.

World Leaders