Normalizing Surnames - Revision history

imported>Ed: /* Name Orders and Formats */

2009-07-30T06:49:45Z

Name Orders and Formats

2009-07-30T06:49:07Z

2009-07-10T23:39:28Z

2009-07-10T23:39:16Z

2009-07-09T00:39:07Z

2009-07-06T23:33:14Z

Tussenvoegsel

2009-07-06T23:32:04Z

2009-07-06T23:31:28Z

2009-07-06T23:30:57Z

2009-07-02T20:38:56Z

@@ Line 43: / Line 43: @@
 |}
-Note: The US census Address Data Content Standard was managed by the [http://www.fgdc.gov/standards/projects/FGDC-standards-projects/addressstandard/ | Federal Geographic Data Committee] but is now discontinued.
+Note: The US census Address Data Content Standard was managed by the [http://www.fgdc.gov/standards/projects/FGDC-standards-projects/addressstandard/ Federal Geographic Data Committee] but is now discontinued.
 The phone book format is most commonly encountered as: Surname, Firstname I. In this instance we refer to it as a comma format name.

@@ Line 42: / Line 42: @@
 |Phone Book (Hardcopy) || Last Name || First Name || Middle Initial || ||
 |}
 The phone book format is most commonly encountered as: Surname, Firstname I. In this instance we refer to it as a comma format name.

@@ Line 1: / Line 1: @@
 *This page is a part of series in [[Classifying Names by Culture]]
 ==Encodings==
 Surnames can be represented in many different encodings. For comparison purposes, it is convenient to have surnames encoded in single standard encoding, such as the Latin alphabet.

← Older revision		Revision as of 23:39, 10 July 2009
Line 1:		Line 1:
		+	*This page is a part of series in [[Classifying Names by Culture]]
	==Encodings==		==Encodings==
	Surnames can be represented in many different encodings. For comparison purposes, it is convenient to have surnames encoded in single standard encoding, such as the Latin alphabet.		Surnames can be represented in many different encodings. For comparison purposes, it is convenient to have surnames encoded in single standard encoding, such as the Latin alphabet.

@@ Line 40: / Line 40: @@
 |Phone Book (Hardcopy) || Last Name || First Name || Middle Initial || ||
 |}
 ==The Normalization Script==
@@ Line 45: / Line 47: @@
 # Force the encoding to Latin
 # Remove Stop Words (default uses: [http://www.edegan.com/repository/Names-Stopwords.txt Names-Stopwords.txt])
 # Mark discards
 # Extract "Surname"
 # Extract "Firstname Surname" pair

@@ Line 8: / Line 8: @@
 ==Tussenvoegsel==
-[http://en.wikipedia.org/wiki/Tussenvoegsel Tussenvoegsel] are surname prefixes, specifically in the Dutch language but used here generically, such as the words Van and De. A custom compiled [http://www.edegan.com/repository/Surnames-Tussenvoegsel.txt list of Tussenvoegsel]is used in the normalization process. Tussenvoegsel can be removed (and recorded with a binary variable) or concatenated with the surname. Note that in some sources Tussenvoegsel can be identified by their lack of capitalization.
+[http://en.wikipedia.org/wiki/Tussenvoegsel Tussenvoegsel] are surname prefixes, specifically in the Dutch language but used here generically, such as the words Van and De. A custom compiled [http://www.edegan.com/repository/Names-Tussenvoegsel.txt list of Tussenvoegsel]is used in the normalization process. Tussenvoegsel can be removed (and recorded with a binary variable) or concatenated with the surname. Note that in some sources Tussenvoegsel can be identified by their lack of capitalization.
 ==Double-barrelled Surnames==

← Older revision		Revision as of 23:31, 6 July 2009
Line 51:		Line 51:
	# Extract "Firstname Surname" pair		# Extract "Firstname Surname" pair

−	An example command line is: <tt>perl NormalizeSurnames.pl -i=sourcefile.txt -ncol=1 -rcol=3 </tt> where the ncol specifies the name column and rcol specifies whether the name is in reversed format (use -r=1 to force reversals for the entire dataset). Basic script help on options is available through <tt>perl NormalizeSurnames.pl -h</tt>	+	An example command line is:
		+
		+	<tt>perl NormalizeSurnames.pl -i=sourcefile.txt -ncol=1 -rcol=3 </tt>
		+
		+	where the ncol specifies the name column and rcol specifies whether the name is in reversed format (use -r=1 to force reversals for the entire dataset). Basic script help on options is available through
		+
		+	<tt>perl NormalizeSurnames.pl -h</tt>

← Older revision		Revision as of 23:30, 6 July 2009
Line 40:		Line 40:
	\|Phone Book (Hardcopy) \|\| Last Name \|\| First Name \|\| Middle Initial \|\| \|\|		\|Phone Book (Hardcopy) \|\| Last Name \|\| First Name \|\| Middle Initial \|\| \|\|
	\|}		\|}
		+
		+	=The Normalization Script=
		+	A [http://www.edegan.com/repository/NormalizeSurnames.pl script for conducting the normalization] takes all of the above points into consideration. The sequence of normalization is:
		+	# Force the encoding to Latin
		+	# Remove Stop Words (default uses: [http://www.edegan.com/repository/Names-Stopwords.txt Names-Stopwords.txt])
		+	# Mark discards
		+	# Remove or concatenate (default) Tussenvoegsel (default uses: [http://www.edegan.com/repository/Names-Tussenvoegsel.txt Names-Tussenvoegsel.txt])
		+	# Remove first barrel (default) or concatenate double-barrelled names
		+	# Extract "Surname"
		+	# Extract "Firstname Surname" pair
		+
		+	An example command line is: <tt>perl NormalizeSurnames.pl -i=sourcefile.txt -ncol=1 -rcol=3 </tt> where the ncol specifies the name column and rcol specifies whether the name is in reversed format (use -r=1 to force reversals for the entire dataset). Basic script help on options is available through <tt>perl NormalizeSurnames.pl -h</tt>