Sources of Surname Data - Revision history

imported>Ed at 22:09, 15 July 2009

2009-07-15T22:09:08Z

imported>Ed at 22:08, 15 July 2009

2009-07-15T22:08:02Z

imported>Ed at 22:07, 15 July 2009

2009-07-15T22:07:37Z

imported>Ed at 23:38, 10 July 2009

2009-07-10T23:38:33Z

imported>Ed at 23:37, 10 July 2009

2009-07-10T23:37:46Z

imported>Ed at 23:37, 10 July 2009

2009-07-10T23:37:20Z

imported>Ed at 02:19, 9 July 2009

2009-07-09T02:19:57Z

imported>Ed at 00:50, 9 July 2009

2009-07-09T00:50:21Z

imported>Ed at 00:48, 9 July 2009

2009-07-09T00:48:51Z

imported>Ed at 00:36, 9 July 2009

2009-07-09T00:36:52Z

@@ Line 33: / Line 33: @@
 A request to the CIA to use a web-bot to scrap data from the HTML version of the [https://www.cia.gov/library/publications/the-world-factbook CIA World Factbook] recieved no response. World leader information was downloaded in pdf format from the [https://www.cia.gov/library/publications/world-leaders-1/pdf-version/pdf-version.html CIA World Leaders PDF site] for April 2008, and converted into a plain-text file ([http://www.edegan.com/repository/WorldLeaders-Raw.txt WorldLeaders-Raw.txt]).
-The raw file was then reprocessed by a one-off script to produce ([http://www.edegan.com/repository/WorldLeaders-Extracted.txt WorldLeaders-Extracted.txt]). The country codes were corrected using a look-up table ([http://www.edegan.com/repository/WorldLeaders-UNCountryLookup.txt WorldLeaders-UNCountryLookup.txt]) to produce the resulting basic dataset ([http://www.edegan.com/repository/WorldLeaders-ExtractedUNCountry.txt WorldLeaders-ExtractedUNCountry.txt]). Users should note that a very small number of individuals had invalidly coded countries (mostly countries that were not recognized by the UN but were recognized by the CIA) and were excluded in this process.
+The raw file was then reprocessed by a one-off script to produce ([http://www.edegan.com/repository/WorldLeaders-Extracted.txt WorldLeaders-Extracted.txt]). The country codes were corrected using a look-up table ([http://www.edegan.com/repository/WorldLeaders-UNCountryLookup.txt WorldLeaders-UNCountryLookup.txt]) to produce the resulting basic dataset ([http://www.edegan.com/repository/WorldLeaders-ExtractedUNCountry.txt WorldLeaders-ExtractedUNCountry.txt]). Users should note that a very small number of individuals had invalidly coded countries (mostly countries that were not recognized by the UN but were recognized by the CIA) and were excluded in this process. Furthermore, some leaders held multiple positions in their governments and had multiple listings - these were collapsed to a single record.
 The NormalizeSurnames.pl script was with following options (and defaults):

@@ Line 15: / Line 15: @@
 The resultant output ([http://www.edegan.com/repository/Olympics-RawOutputWithUNReversal-Normalized.txt Olympics-RawOutputWithUNReversal-Normalized.txt]) was used to create the n-gram variables.
-Other classification (aside from UN GeoRegions) can be added using their UN Country name lookup tables. For example, the output with the (Egan 2009) CultureClass classification is [http://www.edegan.com/repository/Olympics-RawOutputWithUNReversalCultureclass-Normalized.txt Olympics-RawOutputWithUNReversalCultureclass-Normalized.txt].
+Other classifications(aside from UN GeoRegions) can be added using their UN Country name lookup tables. For example, the output with the (Egan 2009) CultureClass classification is [http://www.edegan.com/repository/Olympics-RawOutputWithUNReversalCultureclass-Normalized.txt Olympics-RawOutputWithUNReversalCultureclass-Normalized.txt].
 ==Internet Movie Database (IMDB)==

← Older revision		Revision as of 22:07, 15 July 2009
Line 14:		Line 14:

	The resultant output ([http://www.edegan.com/repository/Olympics-RawOutputWithUNReversal-Normalized.txt Olympics-RawOutputWithUNReversal-Normalized.txt]) was used to create the n-gram variables.		The resultant output ([http://www.edegan.com/repository/Olympics-RawOutputWithUNReversal-Normalized.txt Olympics-RawOutputWithUNReversal-Normalized.txt]) was used to create the n-gram variables.
		+
		+	Other classification (aside from UN GeoRegions) can be added using their UN Country name lookup tables. For example, the output with the (Egan 2009) CultureClass classification is [http://www.edegan.com/repository/Olympics-RawOutputWithUNReversalCultureclass-Normalized.txt Olympics-RawOutputWithUNReversalCultureclass-Normalized.txt].

	==Internet Movie Database (IMDB)==		==Internet Movie Database (IMDB)==

← Older revision		Revision as of 23:38, 10 July 2009
Line 1:		Line 1:
−	~~:->~~This page is a part of series in [[Classifying Names by Culture]]	+	*This page is a part of series in [[Classifying Names by Culture]]

	We are primarily interested in sources of surname data that contain both surnames and countries of birth for surnames, for training and testing purposes.		We are primarily interested in sources of surname data that contain both surnames and countries of birth for surnames, for training and testing purposes.

← Older revision		Revision as of 23:37, 10 July 2009
Line 1:		Line 1:
−	::This page is a part of [Classifying Names by Culture]	+	:->This page is a part of series in [[Classifying Names by Culture]]

	We are primarily interested in sources of surname data that contain both surnames and countries of birth for surnames, for training and testing purposes.		We are primarily interested in sources of surname data that contain both surnames and countries of birth for surnames, for training and testing purposes.

@@ Line 1: / Line 1: @@
-We are primarily interested in sources of surname data that contain both surnames and countries of birth for surnames.
+We are primarily interested in sources of surname data that contain both surnames and countries of birth for surnames, for training and testing purposes.
 ==Olympic Athletes==
@@ Line 16: / Line 16: @@
 A list of all actors and their birth countries was extracted from the [http://www.imdb.com/interfaces#plain IMDB biographies file] using a simple one-off script ([http://www.edegan.com/repository/IMDB-ExtractNamesAndCountries.pl IMDB-ExtractNamesAndCountries.pl]). This produced the input file http://www.edegan.com/repository/IMDB-BioData.txt IMDB-BioData.txt]. Much as with the Olympics data the country names were then corrected to the [[UN GeoRegion standard | UN GeoRegions]], with individuals who were born in non-recognized jurisdictions, such as on a cruise ship at sea, excluded (see [http://www.edegan.com/repository/IMDB-BiosUNCountryCodes.txt IMDB-BiosUNCountryCodes.txt]).
 The NormalizeSurnames.pl script was with following options (and defaults):
@@ Line 24: / Line 26: @@
 ==World Leaders==

@@ Line 15: / Line 15: @@
 <tt>perl -i=Olympics-RawOutputWithUNReversal.txt -ncol=1 -rcol=3</tt>
-The resultant output ([http://www.edegan.com/repository/Olympics-RawOutputWithUNReversalShort-Normalized.txt Olympics-RawOutputWithUNReversalShort-Normalized.txt) can be used to create the n-gram variables.
+The resultant output ([http://www.edegan.com/repository/Olympics-RawOutputWithUNReversalShort-Normalized.txt Olympics-RawOutputWithUNReversalShort-Normalized.txt]) can be used to create the n-gram variables.
 ==World Leaders==