{{McNair Projects
|Has title=Accelerator Demo Day
|Has owner=Minh Le,
|Has start date=06/18/2018
|Has project status=Active
|Does subsume=Demo Day Page Parser, Demo Day Page Google Classifier
}}
==Project Introduction==
This project that utilizes Selenium and Machine Learning to get good candidate web pages and classify web pages as a demo day page containing a list of cohort companies, ultimately to gather good candidates to push to Mechanical Turk. The code is written using Python 3 using Selenium and Tensorflow (Keras). This article will also preliminaries of the Mechanical Turk tool and how it can be used to collect data.

==Project Goal==
The goal of this project is to find good "Demo Day" candidate web pages and to submit these pages to Amazon Mechanical Turk for data collecting. A good candidate is defined as a page containing a list of cohort companies associated with an accelerator. Through observation, good candidates usually containing time and location information about the demo day as well and thus is sufficient to be pushed to MTurk to collect data.

==Code Location==
The source code and relevant files for the project can be found here:
E:\McNair\Projects\Accelerator Demo Day\
The current working model using RF is in:
E:\McNair\Projects\Accelerator Demo Day\Test Run
The RNN model is in:
E:\McNair\Projects\Accelerator Demo Day\Experiment
The RNN is still under much development. Modifying anything in this folder is not recommended

All the other folders are used for experimenting purposes, please don't touch them. If you want to understand more about the files as a general user, go to the section A Quick Glance through the File in The Directory below. If you are a developer, go to the Advance User Guide section.

==General User Guide: How to Use this Project (Random Forest model)==

First, change your directory to the working folder:
cd E:\McNair\Projects\Accelerator Demo Day\Test Run
Then you need to specify the list of accelerators you want to crawl by modifying the following file:
ListOfAccsToCrawl.txt
The first line must remain fixed as "Accelerator". Then the next several rows are the Accelerators name. The name needs not to be case sensitive, but it is preferable that the case remains sensitive if possible.

All necessary preparations are now complete. Now onto running the code!

Running the project is as simple as executing the code in the correct order. The files are named in the format "STEPX_name", where as X is the order of execution. To be more specific, run the following 4 commands:
''# Crawl Google to get the data for the demo day pages for the accelerator stored in ListOfAccsToCrawl.txt''
python3 STEP1_crawl.py
''# Preprocess data using a bag of word approach: each page is characterized by the frequencies of chosen keywords. Chosen keywords are stored in words.txt. This script reates a file called feature_matrix.txt''
python3 STEP2_preprocessing_feature_matrix_generator.py
''# Train the RF model''
python3 STEP3_train_rf.py
''# Run the model to predict on the HTML of the crawled HTMLs.''
python3 STEP4_classify_rf.py

The result is stored in CrawledHTMLFull folder and is classified in two folder: positive and negative. The positive folder contains HTMLs that the classifier thought of as "good candidate." The negative contains the opposite. There is also a txt file called prediction.txt that lists everything. feature.txt is an irrelevant file for the general user, please ignore it. Its sole purpose is for analyzing and debugging.

NEVER touch the TrainingHTML folder, datareader.py or the classifier.txt. These are used internally to train data.

==A Quick Glance through the File in The Directory==
All working file is stored in this folder:
E:\McNair\Projects\Accelerator Demo Day\Test Run
The file

==Amazon Mechanical Turk==
Login info:
username: mcnair@rice.edu
password: amount

There's a file in the folder
CrawledHTMLFull
called
FinalResultWithURL
that was manually created by combining the file
crawled_demoday_page_list.txt
in the mother folder and the file
predicted.txt
This file combined the predictions to the actual url of the websites.

Since MTurk makes it hard for us to display the downloaded HTML, it is much faster to just copy the url into the question box rather than trying to display the downloaded HTML.

The advantage to this is that some websites, such as techcrunch.com behaves abnormally when downloaded as HTML so opening these kinds of websites in the browser would actually be more beneficial because the UI would not be messed up. Moreover, if certain websites has paywall or pop-up ads, the user can also click out of it. Since most of the times, paywall or pop-ups are scripts within HTMLs, the classifier can't rule them out because the body of the HTMLs may still contain useful information we are looking for. Major paywalls or websites that required log-ins such as linkedin have been black-listed in the crawler. More detail in the crawler section below.

However. there is a disadvantage to this: websites are ever changing, so there is a possibility that in the future, the URL may not be usable, or has changed to something else; on the other hand, downloaded HTMLs remain the same because it does not require any internet connection to render and thus, the content is static.

To create the MTurk for this project, follow this tutorial in [[Mechanical Turk (Tool)]]. For testing and development purpose, use https://requestersandbox.mturk.com/

Test account:
email: mcboatfaceboaty670@gmail.com
password: sameastheoneforemail2018

For this project, all the fields that was asked of the user is:

*Whether the page had a list of companies going through an accelerator
*The month and year of the demo day (or article)
*Accelerator name
*Companies going through accelerator

Layout:

[[File:Demodayfinal]]

==Hand Collecting Data==

To crawl, we only looked for data on accelerators which did not receive venture capital data (which Ed found via VentureXpert) and lacked timing info. The purpose of this crawl is to find timing info where we cannot find it otherwise, and if a company received VC we can find timing info via that investment. The file we used to find instances in which we lack timing info and lacked VC is:
/bulk/McNair/Projects/Accelerators/Summer 2018/Merged W Crunchbase Data as of July 17.xlsx

We filtered this sheet in Excel (and checked our work by filtering in SQL) and found 809 companies that lacked timing info and didn't receive VC. From this, we found 74 accelerators which we needed to crawl for.

We used the crawler to search for cohort companies listed for these accelerators.

During the initial test run, the number of good pages was 359. The data is then handled by hand by fellow interns.

The file for hand-coding is in:
/bulk/McNair/Projects/Accelerator Demo Day/Test Run/CrawledDemoDayHTMLFull/'''FinalResultWithURL'''

For the sake of collaboration, the team copied this information to a Google Sheet, accessible here:
https://docs.google.com/spreadsheets/d/16Suyp364lMkmUuUmK2dy_9MeSoS1X4DfFl3dYYDGPT4/edit?usp=sharing

We split the process into four parts. Each interns will do the following:

1. Go to the given URL.

2. Record whether the page is good data (column F); this can later be used by [[Minh Le]] to refine/fine-tune training data.

3. Record whether the page is announcing a cohort or recapping/explaining a demo day (column G). This variable will be used to decide if we should subtract weeks from the given date (e.g. if it is recapping a demo day, the cohort went through the accelerator for the past ~12 weeks, and we should subtract weeks as such).

4. Record date, month, year, and the companies listed for that given accelerator.

5. Note any any information, such as a cohort's special name.

Once this process is finished, we will filter only the 1s in Column F, and [[Connor Rothschild]] and [[Maxine Tao]] will work to populate empty cells in The File to Rule Them All with that data.

==Advance User Guide: An in-depth look into the project and the various settings==

===Accelerators needed to Crawl===
The name lists of Accelerators to crawl is stored in the file:
E:\McNair\Projects\Accelerator Demo Day\Test Run\ListOfAccsToCrawl.txt

===Training Data===
Training data is stored in the folder:
E:\McNair\Projects\Accelerator Demo Day\Test Run\TrainingHTML

===The Crawler Functionality===
The crawler functionality is stored in the file:
STEP1_crawl.py
The crawler was optimized for improved speed, improved performance and improved filtration while remain functional over the large set of data.

BUG REPORT by Maxine Tao (FIXED): update the crawler with this line of code:
search_results = driver.find_elements_by_xpath("//div[@class='g']/div/div/div/h3/a") + driver.find_elements_by_xpath("//div[@class='g']/div/div/h3/a")
Because apparently for some reason it stopped grabbing the first web page (I think because google may have modified how their website looks.

===The Classifier===

===Input (Features)===
The input (features) right now is the frequency of X_NUMBER of words appearing in each documents. The word choice is hand selected. This is the naive bag-of-word approach.

Idea: Create a matrix with the first col being the file BiBTex, and the following columns are the words, and the value at (file, word) is the frequency of that word in the file.
Then, split the matrix into an array of row vectors, and each vector is then feed into the RNN)

This seems to not give really high accuracy with our LSTM RNN, so I will consider a word2vec approach

==Development Notes==
Right now I am working on two different classifier: Kyran's old Random Forest model - optimizing it by tweaking parameters and different combination of features - and my RNN text classifier.

The RF model has a ~92% accuracy on the training data and ~70% accuracy on the test data.

The RNN currently has a ~50% accuracy on both train and est data, which is rather concerning.

Test : train ratio is 1:3 (25/75)

Both model is currently using the Bag-of-word approach to preprocess data, but I will try to use Yang's code in the industry classifier to preprocess using word2vec. I'm not familiar with this approach, but I will try to learn this.

==Reading resources==
http://www.fit.vutbr.cz/research/groups/speech/publi/2010/mikolov_interspeech2010_IS100722.pdf

File:Demodayfinal.png

2018-07-31T15:21:50Z

Connorrothschild:

Accelerator Demo Day

2018-07-31T15:20:45Z

Connorrothschild: /* Amazon Mechanical Turk */

{{McNair Projects
|Has title=Accelerator Demo Day
|Has owner=Minh Le,
|Has start date=06/18/2018
|Has project status=Active
|Does subsume=Demo Day Page Parser, Demo Day Page Google Classifier
}}
==Project Introduction==
This project that utilizes Selenium and Machine Learning to get good candidate web pages and classify web pages as a demo day page containing a list of cohort companies, ultimately to gather good candidates to push to Mechanical Turk. The code is written using Python 3 using Selenium and Tensorflow (Keras). This article will also preliminaries of the Mechanical Turk tool and how it can be used to collect data.

==Project Goal==
The goal of this project is to find good "Demo Day" candidate web pages and to submit these pages to Amazon Mechanical Turk for data collecting. A good candidate is defined as a page containing a list of cohort companies associated with an accelerator. Through observation, good candidates usually containing time and location information about the demo day as well and thus is sufficient to be pushed to MTurk to collect data.

==Code Location==
The source code and relevant files for the project can be found here:
E:\McNair\Projects\Accelerator Demo Day\
The current working model using RF is in:
E:\McNair\Projects\Accelerator Demo Day\Test Run
The RNN model is in:
E:\McNair\Projects\Accelerator Demo Day\Experiment
The RNN is still under much development. Modifying anything in this folder is not recommended

All the other folders are used for experimenting purposes, please don't touch them. If you want to understand more about the files as a general user, go to the section A Quick Glance through the File in The Directory below. If you are a developer, go to the Advance User Guide section.

==General User Guide: How to Use this Project (Random Forest model)==

First, change your directory to the working folder:
cd E:\McNair\Projects\Accelerator Demo Day\Test Run
Then you need to specify the list of accelerators you want to crawl by modifying the following file:
ListOfAccsToCrawl.txt
The first line must remain fixed as "Accelerator". Then the next several rows are the Accelerators name. The name needs not to be case sensitive, but it is preferable that the case remains sensitive if possible.

All necessary preparations are now complete. Now onto running the code!

Running the project is as simple as executing the code in the correct order. The files are named in the format "STEPX_name", where as X is the order of execution. To be more specific, run the following 4 commands:
''# Crawl Google to get the data for the demo day pages for the accelerator stored in ListOfAccsToCrawl.txt''
python3 STEP1_crawl.py
''# Preprocess data using a bag of word approach: each page is characterized by the frequencies of chosen keywords. Chosen keywords are stored in words.txt. This script reates a file called feature_matrix.txt''
python3 STEP2_preprocessing_feature_matrix_generator.py
''# Train the RF model''
python3 STEP3_train_rf.py
''# Run the model to predict on the HTML of the crawled HTMLs.''
python3 STEP4_classify_rf.py

The result is stored in CrawledHTMLFull folder and is classified in two folder: positive and negative. The positive folder contains HTMLs that the classifier thought of as "good candidate." The negative contains the opposite. There is also a txt file called prediction.txt that lists everything. feature.txt is an irrelevant file for the general user, please ignore it. Its sole purpose is for analyzing and debugging.

NEVER touch the TrainingHTML folder, datareader.py or the classifier.txt. These are used internally to train data.

==A Quick Glance through the File in The Directory==
All working file is stored in this folder:
E:\McNair\Projects\Accelerator Demo Day\Test Run
The file

==Amazon Mechanical Turk==
Login info:
username: mcnair@rice.edu
password: amount

There's a file in the folder
CrawledHTMLFull
called
FinalResultWithURL
that was manually created by combining the file
crawled_demoday_page_list.txt
in the mother folder and the file
predicted.txt
This file combined the predictions to the actual url of the websites.

Since MTurk makes it hard for us to display the downloaded HTML, it is much faster to just copy the url into the question box rather than trying to display the downloaded HTML.

The advantage to this is that some websites, such as techcrunch.com behaves abnormally when downloaded as HTML so opening these kinds of websites in the browser would actually be more beneficial because the UI would not be messed up. Moreover, if certain websites has paywall or pop-up ads, the user can also click out of it. Since most of the times, paywall or pop-ups are scripts within HTMLs, the classifier can't rule them out because the body of the HTMLs may still contain useful information we are looking for. Major paywalls or websites that required log-ins such as linkedin have been black-listed in the crawler. More detail in the crawler section below.

However. there is a disadvantage to this: websites are ever changing, so there is a possibility that in the future, the URL may not be usable, or has changed to something else; on the other hand, downloaded HTMLs remain the same because it does not require any internet connection to render and thus, the content is static.

To create the MTurk for this project, follow this tutorial in [[Mechanical Turk (Tool)]]. For testing and development purpose, use https://requestersandbox.mturk.com/

Test account:
email: mcboatfaceboaty670@gmail.com
password: sameastheoneforemail2018

For this project, all the fields that was asked of the user is:

*Whether the page had a list of companies going through an accelerator
*The month and year of the demo day (or article)
*Accelerator name
*Companies going through accelerator

Layout:

[[File:demodayfinal]]

==Hand Collecting Data==

To crawl, we only looked for data on accelerators which did not receive venture capital data (which Ed found via VentureXpert) and lacked timing info. The purpose of this crawl is to find timing info where we cannot find it otherwise, and if a company received VC we can find timing info via that investment. The file we used to find instances in which we lack timing info and lacked VC is:
/bulk/McNair/Projects/Accelerators/Summer 2018/Merged W Crunchbase Data as of July 17.xlsx

We filtered this sheet in Excel (and checked our work by filtering in SQL) and found 809 companies that lacked timing info and didn't receive VC. From this, we found 74 accelerators which we needed to crawl for.

We used the crawler to search for cohort companies listed for these accelerators.

During the initial test run, the number of good pages was 359. The data is then handled by hand by fellow interns.

The file for hand-coding is in:
/bulk/McNair/Projects/Accelerator Demo Day/Test Run/CrawledDemoDayHTMLFull/'''FinalResultWithURL'''

For the sake of collaboration, the team copied this information to a Google Sheet, accessible here:
https://docs.google.com/spreadsheets/d/16Suyp364lMkmUuUmK2dy_9MeSoS1X4DfFl3dYYDGPT4/edit?usp=sharing

We split the process into four parts. Each interns will do the following:

1. Go to the given URL.

2. Record whether the page is good data (column F); this can later be used by [[Minh Le]] to refine/fine-tune training data.

3. Record whether the page is announcing a cohort or recapping/explaining a demo day (column G). This variable will be used to decide if we should subtract weeks from the given date (e.g. if it is recapping a demo day, the cohort went through the accelerator for the past ~12 weeks, and we should subtract weeks as such).

4. Record date, month, year, and the companies listed for that given accelerator.

5. Note any any information, such as a cohort's special name.

Once this process is finished, we will filter only the 1s in Column F, and [[Connor Rothschild]] and [[Maxine Tao]] will work to populate empty cells in The File to Rule Them All with that data.

==Advance User Guide: An in-depth look into the project and the various settings==

===Accelerators needed to Crawl===
The name lists of Accelerators to crawl is stored in the file:
E:\McNair\Projects\Accelerator Demo Day\Test Run\ListOfAccsToCrawl.txt

===Training Data===
Training data is stored in the folder:
E:\McNair\Projects\Accelerator Demo Day\Test Run\TrainingHTML

===The Crawler Functionality===
The crawler functionality is stored in the file:
STEP1_crawl.py
The crawler was optimized for improved speed, improved performance and improved filtration while remain functional over the large set of data.

BUG REPORT by Maxine Tao (FIXED): update the crawler with this line of code:
search_results = driver.find_elements_by_xpath("//div[@class='g']/div/div/div/h3/a") + driver.find_elements_by_xpath("//div[@class='g']/div/div/h3/a")
Because apparently for some reason it stopped grabbing the first web page (I think because google may have modified how their website looks.

===The Classifier===

===Input (Features)===
The input (features) right now is the frequency of X_NUMBER of words appearing in each documents. The word choice is hand selected. This is the naive bag-of-word approach.

Idea: Create a matrix with the first col being the file BiBTex, and the following columns are the words, and the value at (file, word) is the frequency of that word in the file.
Then, split the matrix into an array of row vectors, and each vector is then feed into the RNN)

This seems to not give really high accuracy with our LSTM RNN, so I will consider a word2vec approach

==Development Notes==
Right now I am working on two different classifier: Kyran's old Random Forest model - optimizing it by tweaking parameters and different combination of features - and my RNN text classifier.

The RF model has a ~92% accuracy on the training data and ~70% accuracy on the test data.

The RNN currently has a ~50% accuracy on both train and est data, which is rather concerning.

Test : train ratio is 1:3 (25/75)

Both model is currently using the Bag-of-word approach to preprocess data, but I will try to use Yang's code in the industry classifier to preprocess using word2vec. I'm not familiar with this approach, but I will try to learn this.

==Reading resources==
http://www.fit.vutbr.cz/research/groups/speech/publi/2010/mikolov_interspeech2010_IS100722.pdf

Connor Rothschild (Work Log)

2018-07-30T22:29:10Z

Connorrothschild: /* Summer 2018 */

===Summer 2018===

<onlyinclude>

7/30/2018 -
*Recoded founders' education
*In the process of recoding founders' job experience
*Worked with Minh to test MTurk survey
*Talked through MTurk logistics and strategy with Minh
*Recoded equity and investment variables given new SeedDB data
*Renormalized investment amount based on midpoint of ranges, and upper bounds (upon Hira's request)

7/29/2018 -
*Finalized multiple campuses work, refined addresses
*Upon Hira's request, recoded dead/alive variable for updated accuracy

7/27/2018 -
*Recoded founders
*Fixed multiple campuses and cohorts
*Fixed the Google Sheet

7/26/2018 -
*Cleaned up and fixed the Google Sheet with timing info.
*Recoded the employee count variable.
*Normalized investment amount

7/25/2018 -
*Created a comprehensive Google Sheet with new timing info, collaborated with other interns to find data. Cleaned up sheet.

7/24/2018 -
Sick day :(

7/23/2018 -
*Helped Minh with Demo Day information.

7/19/2018 -
*Helped Minh with training data for Demo Day Crawler

7/18/2018 -
*Helped Augi with MA cleaning
*Talked to Minh about Demo Day progress

7/17/2018 -
*Worked with Ed to add/merge data from Crunchbase to existing data. This was a replication of the process but done by Ed in SQL, not Excel. New data can be found in
/McNair/Projects/Accelerators/Summer 2018/'''Merged With Crunchbase Info as of July 17.xlsx'''

NOTE: Use this data rather than the sheet mentioned in yesterday's entry.

7/16/2018 -
*Merged cohort company data with Crunchbase data, by doing a Vlookup then cleaning up data. I used a =IF(A2="",B2,A2) formula to merge cells only when blanks were present. This provided us updated data for four columns:
**colocation (removed 6324 blanks)
**codescription (removed 5151 blanks)
**costatus (removed 7342 blanks)
**courl (removed 6670 blanks)
and new columns:
**address
**founded_on date
**employee_count
**linkedin_url

These new variables can be found in:
/McNair/Projects/Accelerators/Summer 2018/'''Crunchbase Info Populated Empty Cells.xlsx''' (OUTDATED:: DON'T USE)

Upon Ed's approval, I'll move this sheet to replace Cohort Companies in '''The File to Rule Them All'''.

7/13/2018 -
*Using SQL, matched our cohort companies with information from Crunchbase. This gave us a lot of new information, like employee counts, company status, the date founded, and the location of the company. This data can be found here:
/McNair/Projects/Accelerators/Summer 2018/'''Cohort Companies With Crunchbase Info.xlsx'''

7/12/2018 -

*Created 'The File to Rule them All' with finalized info on accelerators, cohort companies, and founders.
*Attempted to match our company data to Crunchbase data with SQL to get more info on companies.

7/11/2018 -

*Worked on LinkedIn Founders data. Cleaned up data, removed duplicates, checked for fidelity.
*Worked with Maxine to finish Crunchbase matching.

7/10/2018 -
*Merged Clean Cohort Data (Veeral) and Cohort List (new) in the Accelerator Master Variable List file. Cross-referenced this list with Ed's data sent last week, titled accelerator_data_noflag.txt. We found that there are 4866 more entries in the new merged file, meaning Ed's merging may have dropped valid entries. (This was after filtering the list so we only looked at the accelerators on our list).

7/9/2018 -
*Worked with Maxine to remove duplicates/gather clean data for Crunchbase matching

06/29/2018 -
*Finished manually coding an equity variable in Master Variable List sheet (with the help of [[Maxine Tao]]).
*Finished editing terms of joining accelerator:
*Given the above two tasks, there are five new columns in our Master Variable List sheet:
**Terms of joining - terms of joining accelerator and important details about program
**equity (1/0) - cells contain a 1 if the accelerator take equity, a 0 if an accelerator definitively does not, and is blank if we could not find that information
**equity amount - the % of equity the accelerator will take (can sometimes be a range (eg. 5-7%))
**investment - the $ the accelerator invests in a company to begin, if relevant (also could be a range or a "up to $######")
**notes - anything to comment on previous 4 columns
*Taught [[Maxine Tao]] how to VLookup :D

06/28/2018 -
*Began manually coding an equity variable in Master Variable List sheet.
*Edited terms of joining accelerator.
*Helped Grace with LinkedIn crawler.

06/27/2018 -
*Finished coding duplicates. Final file can be found at:
/bulk/McNair/Projects/Accelerators/Summer 2018/Duplicate Companies.xlsx

*Dylan taught interns Excel skills

06/26/2018 -
*Began coding duplicates in CohortMainBaseWCounts.txt file that Ed sent. Sorted by company name alphabetically, then used conditional formatting to highlight when an accelerator had the same name as the accelerator above. This narrowed down the results to instances in which a company would go through the same accelerator twice. Most of the time, this was due to an error with the normalizer, so I moved those un-normalized company names to their own sheet and deleted them from the file.

06/25/2018 -
*Went through and manually fixed discrepancies between our accelerator data and the Crunchbase data, found at
/bulk/McNair/Projects/Accelerators/Summer 2018/Accelerators Matched by Name and Homepage URL.xlsx

*Finalized a sheet with a list of accelerator names as we code them, as Crunchbase codes them, and the appropriate UUID for each accelerator. I recommend updating the names in our spreadsheet of accelerators to the Crunchbase list so that we will be able to look up that name without having an in-between. The list can be found in the rightmost columns here:
/bulk/McNair/Projects/Accelerators/Summer 2018/Accelerator Master Variable List - Revised by Ed V2.xlsx
and here:
https://docs.google.com/spreadsheets/d/1n1sX5DqZrm_0vbUXG9ZaZIagF9sa0Kva9PAno-6H854/edit?usp=sharing

*Worked with [[Minh Le]] to better understand and begin documenting the [[Demo Day Page Parser]] project.

06/22/2018 -
*Finished going through Accelerator Master Variable List to refine industry classification and update addresses/accelerator statuses.

06/21/2018 -
*Began manually editing entries in Accelerator Master Variable List.
*Reached out to Grace and Maxine and sent them the necessary sheets/txt files so they could begin on their Crunchbase project.
*I also made these graphics to better represent what our collaborative work would look like, and what the final project would include:
https://docs.google.com/document/d/13Mb7lOLydm9r-ENYxSlZJVGgY9wxClATR6Hy8F9YK1Y/edit?usp=sharing

06/20/2018 -
*Talked with Ed about project details.
*Began looking through the Accelerator Master List to better understand project description.
*Sent Grace and Maxine the relevant company names listed in the Accelerator Master Spreadsheet so they could begin using their relevant parsers and tools to sort through data.

06/19/2018 -
*Set up work stations on balcony, trained

06/18/2018 -
*Trained, met other interns

</onlyinclude>

[[Category:Work Log]]

U.S. Seed Accelerators

2018-07-30T17:39:30Z

Connorrothschild: /* Update for Hira */

{{McNair Projects
|Has title=U.S. Seed Accelerators
|Has owner=Connor Rothschild,
|Has start date=06/18/2018
|Has keywords=accelerators, data
|Has notes=Continuation of [[Accelerator Data]]
|Has project status=Active
|Is dependent on=Industry Classifier, Demo Day Page Parser, Accelerator Demo Day, Demo Day Page Google Classifier, Crunchbase Accelerator Equity, Crunchbase Accelerator Founders, Crunchbase Data,
|Has image=Seed-Accelerator.jpg
|Does subsume=Accelerator Data, Accelerator Seed List (Data),
}}

==Project Location==
The master file can be found at
/bulk/McNair/Projects/Accelerators/Summer 2018/'''The File to Rule Them All.xlsx'''

==Relevant Former Projects==
This page serves as an updated and tidied version of the data and work presented on the [[Accelerator Seed List (Data)]] Project, which subsumed [[Accelerator Data]].
Both of these projects (and as a corollary, this project) are dependent on the [[Demo Day Page Parser]], [[Industry Classifier]], and the [[Whois Parser]].

==Update for Hira==

===Recoded Founders Education===

I have recoded two components of the founders' education sheet:

1) Degree name has been reclassified into nine categories:
*High School
*Associates
*Bachelors
*Masters
*Certificate
*JD
*MBA
*PhD
*Other

2) Majors have also been recoded into nine categories:
*H = Humanities
*SS = Social Sciences
*NS = Natural Sciences
*E = Engineering (includes computer science)
*B = Business and Economics
*L = Leadership
*MBA
*JD
*O = Other

The Google Sheet I used to reclassify can be found here:
https://docs.google.com/spreadsheets/d/1XWtCTeaof8WxAuOCbn3XEFaK0sh-ZIif2JqcdbkH72I/edit?usp=sharing

With the Sheets '''OLD''' containing old, outdated data, '''WORK''' containing the process of reclassifying, and '''Updated Info''' containing only good, updated data.

The data has been merged into '''The File to Rule Them All'''. Old data has not yet been deleted but can at any time, considering we have it saved in the Google Sheet '''OLD''' (just wanted the go ahead from Hira or Ed).

===Multiple campuses and cohorts===

The File to Rule Them All contains an updated address variable. 139 of our 166 accelerators have addresses that are available online. The ones we could not find information for are left blank.

The collaborative sheet that Hira, Maxine, and I worked on to update this list can be accessed here:
https://docs.google.com/spreadsheets/d/1nktgJZfm3L8IsSCHgYbPasSdvKb7QHKxZp8K5Si2iNo/edit?usp=sharing

We have also added a new cohort list. Under The File To Rule Them All, the sheet '''Multiple Campuses''' contains the different locations for accelerators with multiple campuses. Column B can be used to filter out multiple location accelerators.

===Fixed Google Sheet===

We created a new sheet with only data we want to keep, and cleaned it up. That sheet is called "Good Data Only", at the same link:
https://docs.google.com/spreadsheets/d/16Suyp364lMkmUuUmK2dy_9MeSoS1X4DfFl3dYYDGPT4/edit?usp=sharing

I first used our "recap" and "announced" classification to standardize and fix the dates.

*Columns N-R contain our new data. Please note that all of these columns are based on formulas and will be made erroneous if edited.
*Column N is the # of weeks for an accelerator program, gathered via VLookup from The File to Rule Them All.
*Column O is the Actual Date we want to record, and was gathered by subtracting the # of weeks from a date '''if''' the listed page was a '''recap'''.
*Columns P and Q are the Month and Years stripped from Column O.
*Finally, Column R is the season variable, as Ed said it should be coded.

We have also gone through and removed all bad data, all duplicates, and all rows without timing info. These is the most complete list possible.

===Recode employee count===

I have added a new column in Cohorts Final (of the File to Rule Them All) yet left the old column in case you would prefer to edit/classify differently.
Column AB (emp_count_scale) contains a variable coded on a scale of 1 to 9, with each number corresponding to one of the employee_count classifications (1 the lowest, 9 the highest). The exact output can be modified (1 could instead be tiny, 2 be small... 9 be huge).

The employee count column is standardized and can easily be edited given some modification of the Excel formula.

===Normalized investment amount===

I've been trying to fix the investment amount. But I think its smart we discuss this before I move forward. I've done some tentative standardization (finding the average in a range if given), but so many accelerators "take up to __%" equity and "invest up to $___" that I think its smartest we think hard about how to standardize first. Also notable is that some accelerators say they provide "$____ up front and another $___ in follow up funding for each stage." How do we deal with these? Message me if you'd like to talk more about this.

I refrained from creating max and min investment columns lest spending time on the data before we discuss it.

===Remaning to do===

===Recoding Founders Experience===

I've began recoding the Founders variable. I can categorize 232 of the 824 listed job titles (there are 466 distinct job titles)into 9 categories (the most frequent results as given in a PivotTable found in the sheet).

We will need to talk about how to categorize the most extraneous job titles.

==Recent Work==

Here's a project update on the work that has been done since coming to McNair. The most recent file is
/bulk/McNair/Projects/Accelerators/Summer 2018/'''The File to Rule Them All.xlsx'''

===Merging Cohort Companies with Crunchbase Info===

More information on this part of the project can be found on the page [[Merging Existing Data with Crunchbase]].

The newest updated sheet of cohort company info is under the '''Cohorts Final''' sheet of '''The File to Rule Them All.xlsx'''.

Working with [[Maxine Tao]], we have matched companies to their respective pages and information found in Crunchbase (via UUID). We ensured single matches by doing a 1-1-1-1 match with our data and with Crunchbase (using the Matcher). We then received additional information on 8092 companies. The following new information (on top of what we already had) is included in the sheet:
*short_description
*long_description
*category_list (details the company's category)
*category_group_list (a less refined, more all-encompassing category classification)
*founded_on date
*employee_count
*linkedin_url
*address

And the following information was also pulled from Crunchbase and '''merged''' with our existing data:
*URL (was merged with courl cells)
*city (was merged with colocation)
*state_code (was merged with colocation)
*country_code (was merged with colocation)
*status (was merged with costatus)

===The Equity Variables===

[[Maxine Tao]] and I have added six new variables to the '''Accelerators Final''' sheet. Those variables are:

*Terms of joining - terms of joining accelerator and important details about program
*Equity? (1/0) - cells contain a 1 if the accelerator take equity, a 0 if an accelerator definitively does not, and is blank if we could not find that information
*Equity Amount - the % of equity the accelerator will take (can sometimes be a range (eg. 5-7%))
*Equity Amount Normalized - this copies the previous column but only keeps %>0, and if a range was given (e.g. 5-7%) it returns the average (e.g. 6%)
*Investment Amount - the $ the accelerator invests in a company to begin, if relevant (also could be a range or a "up to $######")
*Investment Notes - anything to comment on previous 4 columns

These six variables tell us more about the characteristics of accelerators; specifically, which ones take equity and which ones do not, and how much equity accelerators take.

Relevant information:
*82 accelerators take equity, 42 do not, and we lack information for 37.
*The average % of equity among accelerators who take equity (rough estimate--do not use for anything official) is 6.49% (got this number by only looking at accelerators who take equity, averaging equity amount for accelerators who report a range (e.g. 4%-10% equity would be coded as 7% equity) and took mean.

===Matching Accelerators to UUIDs via Crunchbase===

We've also added UUIDs for 163 of our 166 accelerators. The UUIDs can be found in Column AE of the '''Accelerators Final''' sheet.

The file with accelerators matched to Crunchbase UUIDs can be found at:
/bulk/McNair/Projects/Accelerators/Summer 2018/Accelerators and UUIDs.xlsx

This is the master file and should never be modified unless we find a UUID changed. ALL OTHER SHEETS with UUIDs are linked to this sheet so its changes will be reflected elsewhere.

More information can be found on the [[Crunchbase Data]] page.

===Linking Accelerators to Founders/LinkedIn Crawling===

[[Grace Tan]] got the [[LinkedIn Crawler (Python)]] to work, which means we currently have the following information about accelerator founders:
*Current Job Title
*Location
*Employer
*Job(s) Title
*Dates Employed
*Time Employed
*Location of jobs
*Extra Description
*School Name
*Degree Name
*Major
*Attended
*Graduated
*Societies

This information can be found in the various '''Founders''' sheets in '''The File to Rule Them All'''.

===Finding Company URLs===
Excel master datasets are in:
E:\McNair\Projects\Accelerators\Summer 2018

Code and files specific to this URL finder are in:
E:\McNair\Projects\Accelerators\Summer 2018\url finder

====Results====
I used STEP1_crawl.py and STEP2_findcorrecturl.py to add approximately 1000 more URLs into 'The File to Rule Them All.xlsx'.

====Testing====

In this file (sheet: 'Most Recent Merged Data' note that this is just a copy of 'Cohorts Final' in 'The File to Rule Them All'):
E:\McNair\Projects\Accelerators\Summer 2018\Merged W Crunchbase Data as of July 17.xlx

We filter for companies (~4000) that did not receive VC, are not in crunchbase, and do not have URLs.
Using a Google crawler(STEP1_crawl.py) and URL matching script(STEP2_findcorrecturl.py), we will try to find as many URLs as possible.

To test, I ran about 40 companies from "smallcompanylist.txt", using only the company name as a search term and taking the first 4 valid results (see don't collect list in code). The google crawler and URL matcher was able to correctly identify around 20 URLs. It also misidentifies some URLs that look really similar to the company name, but it is accurate for the most part if the name is not too generic. I then tried to run the 20 unfound company names through the crawler again, but this time I used company name + startup as the search term. This did not identify any more correct URLs.

It seems reasonable to assume that if the company URL cannot be found within the first 4 valid search results, then that company probably does not have URL at all. This is the case for many of the unfound 20 URLs from my test run above.

====Actual Run Info====

The companies we needed to find URLs for are in a file called 'ACTUALNEEDEDCOMPANIES.txt'.

The first four results for every company, as found by STEP1_crawl.py, are in 'ACTUAL_crawled_company_urls.txt'.

The results after the matching done by STEP2_findcorrecturl.py, are in 'ACTUAL_finalurls.txt'.

Note that in the end, I decided to only take URLs that were given a match score of greater than 0.9 by setting this restriction in STEP2_findcorrecturl.py. Then I manually removed any duplicates/inaccurate results. If you want, you can set the threshold lower in STEP2 and use STEP3_clean.py to find the URL with the highest score for each company.

The point of this URL finder is to find timing information for companies. Timing information can be found on Whois. See the page http://mcnair.bakerinstitute.org/wiki/Whois_Parser#Summer_2018_Work for information on running the whois parser.

====Using Python files====
'''To use STEP1_crawl.py''':
INPUT: a list of company names (or anything) you would like to find websites for by searching on google
OUTPUT: a list of company names and the top X number of results from google

1. Change LIST_FILEPATH in line 26 to be the name of the file that contains the list of things you would like to search.

2. Change NUMRESULT to be however many results you would like from Google.

3. Adjust DONT_COLLECT to include any websites that you don't want.

4. If you would like to add another search keyword, add this in line 87 which is queries.append(name + "whatever you want here")

5. Change line 127 to be the name of your output file.

'''To use STEP2_findcorrecturl.py''':
INPUT: output file from STEP1
OUTPUT: a file formatted the same as the output of STEP1, but URLs that do not match over the threshold value you set will be replaced with "no match"

1. Change file f to be the output file name from STEP1. Change g to be the desired name of the output file for this part.

2. In the if statement on line 44, set your desired threshold. Note that anything greater than 0.6 is generally considered a decent match. It might be safer to use 0.75 and my use of 0.9 ensures almost exact matches. However, you should consider that if your list of companies (or whatever you are searching) includes really common names, then a 90%+ match might not be exactly what you're looking for.

'''To use STEP3_clean.py''':

Note this is an optional step to use depending on the accuracy level you need and what kind of data you crawled earlier. I chose not to use this and instead set a more restrictive threshold in STEP2.

1. Change file f to be the output file from STEP2 (you should delete anything that says "no match", and when you use STEP2, you must also write the ratio score to the text file). Change g to be the desired name of the output file for this part.

Your output should be a text file containing the company name and the URL that had the highest assigned score in STEP2. In case of more than 1 URL with the highest score, the script should take the first one.

==An Overview==
This project will be used to determine which accelerators are the most effective at churning out successful startups, as well as what characteristics are exhibited by these accelerators. First, we need to gather as much data as we can about as many accelerators as we can in order to look at factors that differentiate successful vs. unsuccessful ventures. Next, we need to create a web crawling program which will gather information about accelerators across the world by accessing their websites and extracting information. I believe that our overall goal with this research project is to gain insight into the methods of successful accelerators, as well as to find out what exactly differentiates very successful accelerators from dead accelerators.

Helpful Links: http://seedrankings.com/

==Remaining To Dos==

The last update on [[Accelerator Seed List (Data)]] said the following needed to be done:

*Cross-reference sheet with data from Peter's old accelerator consolidation file ("accelerator_data_noflag" and "accelerator_data" in "All Relevant Files") and fill in missing data
*Variables that are 100% NOT in these 2 files:
**Cohort Breakout?
**Subtype
**Designed for Students?
**Campuses
**Stage
**Software Tech
**What stage do they look for?

==(Outdated) Necessary Steps==

'''Acquiring the necessary data to complete the Accelerator Master Variable List and the Cohort List will require the following (not necessarily in this order):'''

===Step Zero: Connect to Crunchbase and Link Data===

Complete - more info: [[Crunchbase Data]]

===Step One: LinkedIn Founders Data===

This project will begin by working with [[Grace Tan]] and [[Maxine Tao]] to connect accelerators to their founders and cohort companies using Crunchbase and LinkedIn crawlers. Grace and Maxine will go through [http://crunchbase.com Crunchbase] and find the UUID for companies and their founders (reference [[Crunchbase Data]], [[Crunchbase Accelerator Founders]], [[Crunchbase Accelerator Equity]]). Connect them using SQL and feed the names of founders into our LinkedIn crawler (headed by [[Grace Tan]]).

The list of founders for accelerators can be found at
McNair/Projects/Accelerators/Fall 2017/founders_linkedin.txt

The '''Unfound Founders''' file codes a 0 for all companies '''''not listed''''' within the LinkedIn Founders file, and a 1 for those that do have founders listed.

'''Given the founders' names, we will then be able to use the [[LinkedIn Crawler (Python)]] to find the relevant details of an accelerator founder (education, work experience, etc.) This data on founders will help us solve the horse, jockey, racetrack question to detect what variables affect a startup's success (the accelerator, the founders, the environment/city).'''

===Step Two: Linking Accelerators to Cohorts Using Investments on Crunchbase===

In this step we focus on accelerators who take equity from the companies that engage in their program. We do this to prevent looking at accelerators who may also run funds/invest in various companies but do not take equity. This would provide us misleading results and lead us to believe some companies are in cohorts at accelerators that they are really not.

Maxine will acquire the list of accelerators who take equity from companies from the following sheet:

E://McNair/Projects/Accelerators/All Relevant Files/accelerator_data_noflag.txt

Looking at the file, however, shows that very few are actually categorized well and the equity variable is messy. Moving forward, we need to check/refine/fix this classification.

This file has 266 rows. The most recent, actual version of our accelerator database (found at E://McNair/Projects/Accelerators/Summer 2018/Connor Accelerator Work/Accelerator Master Variable List - Revised by Ed V2.xlsx under the sheet Master Variable List) only has 167 rows, meaning the accelerator_data_noflag.txt file has too many rows.

We will need to do a left join of Accelerator Master List with accelerator_data_noflag.txt to get rid of the accelerator names that are in accelerator_data_noflag but NOT in Accelerator Master List.

Once this is finished, we should have an “Equity” classification variable for every accelerator in Accelerator Master List. The accelerators that have a Y (or maybe it’s a 1) are companies that do take equity. These are the companies we’ll be able to do your Crunchbase work on to see when accelerators take equity.

We then look at the accelerators investments (or companies and the entities which invested in them), cross-reference the list of companies/accelerators, and once we find a match, we know that a company went through an accelerator and during which year they went through a cohort.

'''From this, we get the following data:'''
*Accelerator a given company went through
*Year said company went through a cohort/Specific cohort company went through

===Step Three: Demo Day Crawler===

This part of the project relies on the contributions of the wonderful [[Minh Le]]. Better documentation for the project can be found on the [[Demo Day Page Parser]], [[Demo Day Page Google Classifier]], and [[Accelerator Demo Day]] project pages.

Essentially, this part of the accelerator data project will use the [[Demo Day Page Parser]] to look through accelerator websites for pages which list a cohort's 'Demo Day', or the day in which accelerators present their companies to a group of special investors ([https://www.ycombinator.com/demoday/faq/ here's an example FAQ page from Y Combinator]). The [[Demo Day Page Google Classifier]] will then determine if the page is, in fact, a demo day page.

'''Given a cohort's demo day, we can gather a few pieces of key information (check with Ed to make sure this is the correct information to gather from Demo Days):'''
*The date a cohort began/the season the cohort went through the accelerator
**This is acquired by looking at the cohort's demo day date, and subtracting the number of weeks/months of a cohort for that given accelerator. The length of a cohort can be found in the file:
E://McNair/Projects/Accelerators/Summer 2018/Connor Accelerator Work/Accelerator Master Variable List - Revised by Ed V2.xlsx
*I assume we can also acquire the companies in a specific cohort, as in we will have the list of all companies in the (for example) Spring 2018 Cohort of Techstars.

===Step Four: Non-profit Finder===

More at [[Non-profit Finder]]

Another important step in this project is finding which accelerators are non-profits.

A comprehensive list of nonprofits taken from the [https://www.irs.gov/charities-non-profits/tax-exempt-organization-search-bulk-data-downloads IRS] can be found here:

E://McNair/Projects/Accelerators/Summer 2018/Connor Accelerator Work/Nonprofits in US.xlsx
Warning: this file has 1 million rows

This file should be cross-referenced with the list of accelerators to find which ones are listed--those accelerators are non-profits. We'll need someone good in SQL to do this with a match.

Potential problem:
*The names of listed non-profits will likely be somewhat different than the names of accelerators, because companies often file for tax exemption with different names than they show the public.

==Workflow Image==

(the color coding for this image is a very rough and preliminary estimate subject to change) I imagine the project will look something like this, in that it will require the following information to fully complete the project:

[[File:image1.png]]

Connor Rothschild (Work Log)

2018-07-27T21:54:21Z

2018-07-25T20:12:11Z

Connorrothschild: /* Hand Collecting Data */

{{McNair Projects
|Has title=Accelerator Demo Day
|Has owner=Minh Le,
|Has start date=06/18/2018
|Has project status=Active
|Does subsume=Demo Day Page Parser, Demo Day Page Google Classifier
}}
==Project Introduction==
This project that utilizes Selenium and Machine Learning to get good candidate web pages and classify web pages as a demo day page containing a list of cohort companies, ultimately to gather good candidates to push to Mechanical Turk. The code is written using Python 3 using Selenium and Tensorflow (Keras). This article will also preliminaries of the Mechanical Turk tool and how it can be used to collect data.

==Project Goal==
The goal of this project is to find good "Demo Day" candidate web pages and to submit these pages to Amazon Mechanical Turk for data collecting. A good candidate is defined as a page containing a list of cohort companies associated with an accelerator. Through observation, good candidates usually containing time and location information about the demo day as well and thus is sufficient to be pushed to MTurk to collect data.

==Code Location==
The source code and relevant files for the project can be found here:
E:\McNair\Projects\Accelerator Demo Day\
The current working model using RF is in:
E:\McNair\Projects\Accelerator Demo Day\Test Run
The RNN model is in:
E:\McNair\Projects\Accelerator Demo Day\Experiment
The RNN is still under much development. Modifying anything in this folder is not recommended

All the other folders are used for experimenting purposes, please don't touch them. If you want to understand more about the files as a general user, go to the section A Quick Glance through the File in The Directory below. If you are a developer, go to the Advance User Guide section.

==General User Guide: How to Use this Project (Random Forest model)==

First, change your directory to the working folder:
cd E:\McNair\Projects\Accelerator Demo Day\Test Run
Then you need to specify the list of accelerators you want to crawl by modifying the following file:
ListOfAccsToCrawl.txt
The first line must remain fixed as "Accelerator". Then the next several rows are the Accelerators name. The name needs not to be case sensitive, but it is preferable that the case remains sensitive if possible.

All necessary preparations are now complete. Now onto running the code!

Running the project is as simple as executing the code in the correct order. The files are named in the format "STEPX_name", where as X is the order of execution. To be more specific, run the following 4 commands:
''# Crawl Google to get the data for the demo day pages for the accelerator stored in ListOfAccsToCrawl.txt''
python3 STEP1_crawl.py
''# Preprocess data using a bag of word approach: each page is characterized by the frequencies of chosen keywords. Chosen keywords are stored in words.txt. This script reates a file called feature_matrix.txt''
python3 STEP2_preprocessing_feature_matrix_generator.py
''# Train the RF model''
python3 STEP3_train_rf.py
''# Run the model to predict on the HTML of the crawled HTMLs.''
python3 STEP4_classify_rf.py

The result is stored in CrawledHTMLFull folder and is classified in two folder: positive and negative. The positive folder contains HTMLs that the classifier thought of as "good candidate." The negative contains the opposite. There is also a txt file called prediction.txt that lists everything. feature.txt is an irrelevant file for the general user, please ignore it. Its sole purpose is for analyzing and debugging.

NEVER touch the TrainingHTML folder, datareader.py or the classifier.txt. These are used internally to train data.

==A Quick Glance through the File in The Directory==
All working file is stored in this folder:
E:\McNair\Projects\Accelerator Demo Day\Test Run
The file

==Amazon Mechanical Turk==
There's a file in the folder
CrawledHTMLFull
called
FinalResultWithURL
that was manually created by combining the file
crawled_demoday_page_list.txt
in the mother folder and the file
predicted.txt
This file combined the predictions to the actual url of the websites.

Since MTurk makes it hard for us to display the downloaded HTML, it is much faster to just copy the url into the question box rather than trying to display the downloaded HTML.

The advantage to this is that some websites, such as techcrunch.com behaves abnormally when downloaded as HTML so opening these kinds of websites in the browser would actually be more beneficial because the UI would not be messed up. Moreover, if certain websites has paywall or pop-up ads, the user can also click out of it. Since most of the times, paywall or pop-ups are scripts within HTMLs, the classifier can't rule them out because the body of the HTMLs may still contain useful information we are looking for. Major paywalls or websites that required log-ins such as linkedin have been black-listed in the crawler. More detail in the crawler section below.

However. there is a disadvantage to this: websites are ever changing, so there is a possibility that in the future, the URL may not be usable, or has changed to something else; on the other hand, downloaded HTMLs remain the same because it does not require any internet connection to render and thus, the content is static.

To create the MTurk for this project, follow this tutorial in [[Mechanical Turk (Tool)]]. For testing and development purpose, use https://requestersandbox.mturk.com/

Test account:
email: mcboatfaceboaty670@gmail.com
password: sameastheoneforemail2018

For this project, all the fields that was asked of the user is:

*Whether the page had a list of companies going through an accelerator
*The month and year of the demo day (or article)
*Accelerator name
*Companies going through accelerator

Layout:

[[File:Screen Shot 2018-07-25 at 11.37.02 AM.png]]

==Hand Collecting Data==

To crawl, we only looked for data on accelerators which did not receive venture capital data (which Ed found via VentureXpert) and lacked timing info. The purpose of this crawl is to find timing info where we cannot find it otherwise, and if a company received VC we can find timing info via that investment. The file we used to find instances in which we lack timing info and lacked VC is:
/bulk/McNair/Projects/Accelerators/Summer 2018/Merged W Crunchbase Data as of July 17.xlsx

We filtered this sheet in Excel (and checked our work by filtering in SQL) and found 809 companies that lacked timing info and didn't receive VC. From this, we found 74 accelerators which we needed to crawl for.

We used the crawler to search for cohort companies listed for these accelerators.

During the initial test run, the number of good pages was 359. The data is then handled by hand by fellow interns.

The file for hand-coding is in:
/bulk/McNair/Projects/Accelerator Demo Day/Test Run/CrawledDemoDayHTMLFull/'''FinalResultWithURL'''

For the sake of collaboration, the team copied this information to a Google Sheet, accessible here:
https://docs.google.com/spreadsheets/d/16Suyp364lMkmUuUmK2dy_9MeSoS1X4DfFl3dYYDGPT4/edit?usp=sharing

We split the process into four parts. Each interns will do the following:

1. Go to the given URL.

2. Record whether the page is good data (column F); this can later be used by [[Minh Le]] to refine/fine-tune training data.

3. Record whether the page is announcing a cohort or recapping/explaining a demo day (column G). This variable will be used to decide if we should subtract weeks from the given date (e.g. if it is recapping a demo day, the cohort went through the accelerator for the past ~12 weeks, and we should subtract weeks as such).

4. Record date, month, year, and the companies listed for that given accelerator.

5. Note any any information, such as a cohort's special name.

Once this process is finished, we will filter only the 1s in Column F, and [[Connor Rothschild]] and [[Maxine Tao]] will work to populate empty cells in The File to Rule Them All with that data.

==Advance User Guide: An in-depth look into the project and the various settings==

===Accelerators needed to Crawl===
The name lists of Accelerators to crawl is stored in the file:
E:\McNair\Projects\Accelerator Demo Day\Test Run\ListOfAccsToCrawl.txt

===Training Data===
Training data is stored in the folder:
E:\McNair\Projects\Accelerator Demo Day\Test Run\TrainingHTML

===The Crawler Functionality===
The crawler functionality is stored in the file:
STEP1_crawl.py
The crawler was optimized for improved speed, improved performance and improved filtration while remain functional over the large set of data.

BUG REPORT by Maxine Tao (FIXED): update the crawler with this line of code:
search_results = driver.find_elements_by_xpath("//div[@class='g']/div/div/div/h3/a") + driver.find_elements_by_xpath("//div[@class='g']/div/div/h3/a")
Because apparently for some reason it stopped grabbing the first web page (I think because google may have modified how their website looks.

===The Classifier===

===Input (Features)===
The input (features) right now is the frequency of X_NUMBER of words appearing in each documents. The word choice is hand selected. This is the naive bag-of-word approach.

Idea: Create a matrix with the first col being the file BiBTex, and the following columns are the words, and the value at (file, word) is the frequency of that word in the file.
Then, split the matrix into an array of row vectors, and each vector is then feed into the RNN)

This seems to not give really high accuracy with our LSTM RNN, so I will consider a word2vec approach

==Development Notes==
Right now I am working on two different classifier: Kyran's old Random Forest model - optimizing it by tweaking parameters and different combination of features - and my RNN text classifier.

The RF model has a ~92% accuracy on the training data and ~70% accuracy on the test data.

The RNN currently has a ~50% accuracy on both train and est data, which is rather concerning.

Test : train ratio is 1:3 (25/75)

Both model is currently using the Bag-of-word approach to preprocess data, but I will try to use Yang's code in the industry classifier to preprocess using word2vec. I'm not familiar with this approach, but I will try to learn this.

==Reading resources==
http://www.fit.vutbr.cz/research/groups/speech/publi/2010/mikolov_interspeech2010_IS100722.pdf

Accelerator Demo Day

2018-07-25T20:04:03Z

{{McNair Projects
|Has title=Merging Existing Data with Crunchbase
|Has owner=Connor Rothschild
}}

This page details the process of merging existing data with data pulled from Crunchbase.

==Project Location==

For the merge detailed in this page, our data was from:
/bulk/McNair/Projects/Accelerators/Summer 2018/The File to Rule Them All.xlsx

and Crunchbase info can be found in:
/bulk/McNair/Projects/Accelerators/Summer 2018/Cohort Companies with Crunchbase Info.xlsx

The data from Crunchbase, organized into tables, is in a script found at:
/bulk/McNair/Software/Database Scripts/Crunchbase2/LoadTables.sql

==Process==

===Step One: Creating UUID Matches===

We began by making sure our company names were unique; creating a 1-1-1-1 relationship (only one instance of a company name in our data, and in Crunchbase data). We did so using the Matcher. We matched our sheet against itself, and Crunchbase info (pulled from organizations table detailed below) against itself, to remove duplicates and only leave unique values. [[[More here: http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data#Collecting_Company_Information]]]

Upon Ed's instruction, we then looked at companies ''in Crunchbase'' which had more than one UUID associated with the company name. Of the 670,000 companies in Crunchbase, only 15,000 had duplicate UUIDs. From this list of 15,000, we used recursive filtering to determine if any companies could be properly matched to the company in our data by looking at additional variables (such as company location).

Upon refining our list based on recursive filtering, we found __ companies which match our data, and added UUIDs appropriately.

===Step Two: Pulling Data===

The necessary tables for this pull can be found at:
/bulk/McNair/Software/Database Scripts/Crunchbase2/'''LoadTables.sql'''

We pulled the relevant data from Crunchbase based on unique UUID matches. In the crunchbase2 database, we used the table ''organizations''.

We pull based on the unique UUIDs found by Maxine, which can be found in the file:
/bulk/McNair/Projects/Accelerators/Summer 2018/'''The File to Rule Them All.xlsx''', in column W.

The table we get most Crunchbase data from looks like this:

DROP TABLE organizations;
CREATE TABLE organizations (
company_name varchar(100),
role varchar(255),
permalink varchar(255),
domain varchar(5000),
homepage_url varchar(5000),
country_code varchar(10),
state_code varchar(2),
region varchar(50),
city varchar(100),
address text,
status varchar(50),
short_description text,
category_list text,
category_group_list text,
funding_rounds integer,
funding_total_usd money,
founded_on date, --yyyy-mm-dd
last_funding_on date, --yyyy-mm-dd
closed_on date, --yyyy-mm-dd
employee_count varchar(255),
email varchar(255),
phone text,
facebook_url varchar(5000),
linkedin_url varchar(5000),
cb_url varchar(5000),
logo_url varchar(5000),
twitter_url varchar(5000),
alias varchar(10000),
uuid varchar(255),
created_at date, --yyyy-mm-dd-hh-mm-s.s
updated_at date, --yyyy-mm-dd-hh-mm-s.s
primary_role varchar(255),
type varchar(255)
);

'''From this list, we care about the following:
*company_name,
*domain,
*country_code,
*state_code,
*city,
*address,
*status,
*short_description,
*category_list,
*category_group_list,
*founded_on,
*employee_count,
*linkedin_url,
*uuid -- our primary key'''

We also want to get more information on organization descriptions. To do so, we can pull ''description'' from the table ''organization_descriptions'', matching based on UUID.

We also, for the purposes of industry classification, want to pull ''category_name'' from the table ''category_groups'', matching based on UUID.

Finally, it may be worthwhile to pull variables such as name, description, and started_on from the ''events'' table, in the hopes of finding Cohort years, or potentially demo days. This can also be matched based on UUID.

Given the aforementioned information, we now have much data that can be used to populate empty cells in our existing data, as well as to create new columns.

===Step Three: Merging===

Of the data we've pulled from Crunchbase, we're interested in merging ''four'' columns with our existing data:
*domain (to be merged with the empty cells of courl)
*city, state_code, and country_code (some combination of this is to be merged with the empty cells of colocation)
*status (to be merged with the empty cells of costatus)
*short_description and description '''''from the table organization_descriptions''''' (some combination to be merged with empty cells of codescription)

Note: we may also be able to merge some combination of category_list, category_group_list, and (from category_groups table) category_name, to merge with cosector in our data, and use it for [[Maxine Tao]]'s industry classifier.

The other columns can be added to the end of our sheet as supplemental data.

Merging Existing Data with Crunchbase

2018-07-17T17:15:03Z

Connorrothschild:

{{McNair Projects
|Has title=Merging Existing Data with Crunchbase
|Has owner=Connor Rothschild
}}

This page details the process of merging existing data with data pulled from Crunchbase.

==Project Location==

For the merge detailed in this page, our data was from:
/bulk/McNair/Projects/Accelerators/Summer 2018/The File to Rule Them All.xlsx

and Crunchbase info can be found in:
/bulk/McNair/Projects/Accelerators/Summer 2018/Cohort Companies with Crunchbase Info.xlsx

The data from Crunchbase, organized into tables, is in a script found at:
/bulk/McNair/Software/Database Scripts/Crunchbase2/LoadTables.sql

==Process==

===Step One: Creating UUID Matches===

We began by making sure our company names were unique; creating a 1-1-1-1 relationship (only one instance of a company name in our data, and in Crunchbase data). We did so using the Matcher. We matched our sheet against itself, and Crunchbase info (pulled from organizations table detailed below) against itself, to remove duplicates and only leave unique values. [More here: http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data#Collecting_Company_Information]

Upon Ed's instruction, we then looked at companies ''in Crunchbase'' which had more than one UUID associated with the company name. Of the 670,000 companies in Crunchbase, only 15,000 had duplicate UUIDs. From this list of 15,000, we used recursive filtering to determine if any companies could be properly matched to the company in our data by looking at additional variables (such as company location).

Upon refining our list based on recursive filtering, we found __ companies which match our data, and added UUIDs appropriately.

===Step Two: Pulling Data===

The necessary tables for this pull can be found at:
/bulk/McNair/Software/Database Scripts/Crunchbase2/'''LoadTables.sql'''

We pulled the relevant data from Crunchbase based on unique UUID matches. In the crunchbase2 database, we used the table ''organizations''.

We pull based on the unique UUIDs found by Maxine, which can be found in the file:
/bulk/McNair/Projects/Accelerators/Summer 2018/'''The File to Rule Them All.xlsx''', in column W.

The table we get most Crunchbase data from looks like this:

DROP TABLE organizations;
CREATE TABLE organizations (
company_name varchar(100),
role varchar(255),
permalink varchar(255),
domain varchar(5000),
homepage_url varchar(5000),
country_code varchar(10),
state_code varchar(2),
region varchar(50),
city varchar(100),
address text,
status varchar(50),
short_description text,
category_list text,
category_group_list text,
funding_rounds integer,
funding_total_usd money,
founded_on date, --yyyy-mm-dd
last_funding_on date, --yyyy-mm-dd
closed_on date, --yyyy-mm-dd
employee_count varchar(255),
email varchar(255),
phone text,
facebook_url varchar(5000),
linkedin_url varchar(5000),
cb_url varchar(5000),
logo_url varchar(5000),
twitter_url varchar(5000),
alias varchar(10000),
uuid varchar(255),
created_at date, --yyyy-mm-dd-hh-mm-s.s
updated_at date, --yyyy-mm-dd-hh-mm-s.s
primary_role varchar(255),
type varchar(255)
);

'''From this list, we care about the following:
*company_name,
*domain,
*country_code,
*state_code,
*city,
*address,
*status,
*short_description,
*category_list,
*category_group_list,
*founded_on,
*employee_count,
*linkedin_url,
*uuid -- our primary key'''

We also want to get more information on organization descriptions. To do so, we can pull ''description'' from the table ''organization_descriptions'', matching based on UUID.

We also, for the purposes of industry classification, want to pull ''category_name'' from the table ''category_groups'', matching based on UUID.

Finally, it may be worthwhile to pull variables such as name, description, and started_on from the ''events'' table, in the hopes of finding Cohort years, or potentially demo days. This can also be matched based on UUID.

Given the aforementioned information, we now have much data that can be used to populate empty cells in our existing data, as well as to create new columns.

===Step Three: Merging===

Of the data we've pulled from Crunchbase, we're interested in merging ''four'' columns with our existing data:
*domain (to be merged with the empty cells of courl)
*city, state_code, and country_code (some combination of this is to be merged with the empty cells of colocation)
*status (to be merged with the empty cells of costatus)
*short_description and description '''''from the table organization_descriptions''''' (some combination to be merged with empty cells of codescription)

Note: we may also be able to merge some combination of category_list, category_group_list, and (from category_groups table) category_name, to merge with cosector in our data, and use it for [[Maxine Tao]]'s industry classifier.

The other columns can be added to the end of our sheet as supplemental data.