Merging Existing Data with Crunchbase - Revision history

Ed at 17:41, 21 September 2020

2020-09-21T17:41:44Z

Ed at 17:34, 21 September 2020

2020-09-21T17:34:19Z

Ed at 17:20, 20 September 2020

2020-09-20T17:20:06Z

Ed at 17:12, 20 September 2020

2020-09-20T17:12:19Z

Maxine.tao at 19:53, 28 July 2018

2018-07-28T19:53:51Z

Maxine.tao: /* Finding Company URLs */

2018-07-27T18:24:35Z

Finding Company URLs

Maxine.tao at 18:24, 27 July 2018

2018-07-27T18:24:18Z

Maxine.tao: /* Finding Company URLs */

2018-07-27T17:27:35Z

Finding Company URLs

Maxine.tao: /* Finding Company URLs */

2018-07-27T17:27:16Z

Finding Company URLs

Maxine.tao: /* Finding Company URLs */

2018-07-27T17:26:34Z

Finding Company URLs

@@ Line 1: / Line 1: @@
 {{Project
-|Has project output=
+|Has project output=Data
 |Has sponsor=McNair Center
 |Has title=Merging Existing Data with Crunchbase

@@ Line 1: / Line 1: @@
 {{Project
 |Has sponsor=McNair Center
 |Has title=Merging Existing Data with Crunchbase

@@ Line 1: / Line 1: @@
 {{Project
 |Has title=Merging Existing Data with Crunchbase
 |Has owner=Connor Rothschild

@@ Line 1: / Line 1: @@
-{{McNair Projects
+{{Project
 |Has title=Merging Existing Data with Crunchbase
 |Has owner=Connor Rothschild

← Older revision		Revision as of 19:53, 28 July 2018
Line 146:		Line 146:

	Doing training data - 2,600 pages and are a little bit more than 1/2 way (~1500-1600).		Doing training data - 2,600 pages and are a little bit more than 1/2 way (~1500-1600).
−
−	~~==Finding Company URLs==~~
−	~~Excel master datasets are in:~~
−	~~E:\McNair\Projects\Accelerators\Summer 2018~~
−
−	~~Code and files specific to this URL finder are in:~~
−	~~E:\McNair\Projects\Accelerators\Summer 2018\url finder~~
−
−	~~====Results====~~
−	~~I used STEP1_crawl.py and STEP2_findcorrecturl.py to add approximately 1000 more URLs into 'The File to Rule Them All.xlsx'.~~
−
−	~~====Testing====~~
−
−	~~In this file (sheet: 'Most Recent Merged Data' note that this is just a copy of 'Cohorts Final' in 'The File to Rule Them All'):~~
−	~~E:\McNair\Projects\Accelerators\Summer 2018\Merged W Crunchbase Data as of July 17.xlx~~
−
−	~~We filter for companies (~4000) that did not receive VC, are not in crunchbase, and do not have URLs.~~
−	~~Using a Google crawler(STEP1_crawl.py) and URL matching script(STEP2_findcorrecturl.py), we will try to find as many URLs as possible.~~
−
−	To test, I ran about 40 companies from "smallcompanylist.txt", using only the company name as a search term and taking the first 4 valid results (see don't collect list in code). The google crawler and URL matcher was able to correctly identify around 20 URLs. It also misidentifies some URLs that look really similar to the company name, but it is accurate for the most part if the name is not too generic. I then tried to run the 20 unfound company names through the crawler again, but this time I used company name + startup as the search term. This did not identify any more correct URLs.
−
−	It seems reasonable to assume that if the company URL cannot be found within the first 4 valid search results, then that company probably does not have URL at all. This is the case for many of the unfound 20 URLs from my test run above.
−
−	~~====Actual Run Info====~~
−
−	~~The companies we needed to find URLs for are in a file called 'ACTUALNEEDEDCOMPANIES.txt'.~~
−
−	~~The first four results for every company, as found by STEP1_crawl.py, are in 'ACTUAL_crawled_company_urls.txt'.~~
−
−	~~The results after the matching done by STEP2_findcorrecturl.py, are in 'ACTUAL_finalurls.txt'.~~
−
−	~~Note that in the end, I decided to only take URLs that were given a match score of greater than 0.9.~~

@@ Line 153: / Line 153: @@
 Code and files specific to this URL finder are in:
   E:\McNair\Projects\Accelerators\Summer 2018\url finder
 ====Results====

@@ Line 155: / Line 155: @@
-RESULTS: I used STEP1_crawl.py and STEP2_findcorrecturl.py to add approximately 1000 more URLs into 'The File to Rule Them All.xlsx'.
+====Results====
 In this file (sheet: 'Most Recent Merged Data' note that this is just a copy of 'Cohorts Final' in 'The File to Rule Them All'):
@@ Line 167: / Line 170: @@
 It seems reasonable to assume that if the company URL cannot be found within the first 4 valid search results, then that company probably does not have URL at all. This is the case for many of the unfound 20 URLs from my test run above.
-'''Actual Run Info'''
+====Actual Run Info====
 The companies we needed to find URLs for are in a file called 'ACTUALNEEDEDCOMPANIES.txt'.

← Older revision		Revision as of 17:27, 27 July 2018
Line 168:		Line 168:

	'''Actual Run Info'''		'''Actual Run Info'''
		+
	The companies we needed to find URLs for are in a file called 'ACTUALNEEDEDCOMPANIES.txt'.		The companies we needed to find URLs for are in a file called 'ACTUALNEEDEDCOMPANIES.txt'.