Difference between revisions of "Accelerator Demo Day"
|Line 89:||Line 89:|
==Hand Collecting Data==
==Hand Collecting Data==
Revision as of 11:28, 31 July 2018
|Accelerator Demo Day|
|Project Title||Accelerator Demo Day|
|Has project status||Active|
|Subsumes:||Demo Day Page Parser, Demo Day Page Google Classifier|
|Copyright © 2016 edegan.com. All Rights Reserved.|
- 1 Project Introduction
- 2 Project Goal
- 3 Code Location
- 4 General User Guide: How to Use this Project (Random Forest model)
- 5 A Quick Glance through the File in The Directory
- 6 Amazon Mechanical Turk
- 7 Hand Collecting Data
- 8 Advance User Guide: An in-depth look into the project and the various settings
- 9 Development Notes
- 10 Reading resources
This project that utilizes Selenium and Machine Learning to get good candidate web pages and classify web pages as a demo day page containing a list of cohort companies, ultimately to gather good candidates to push to Mechanical Turk. The code is written using Python 3 using Selenium and Tensorflow (Keras). This article will also preliminaries of the Mechanical Turk tool and how it can be used to collect data.
The goal of this project is to find good "Demo Day" candidate web pages and to submit these pages to Amazon Mechanical Turk for data collecting. A good candidate is defined as a page containing a list of cohort companies associated with an accelerator. Through observation, good candidates usually containing time and location information about the demo day as well and thus is sufficient to be pushed to MTurk to collect data.
The source code and relevant files for the project can be found here:
E:\McNair\Projects\Accelerator Demo Day\
The current working model using RF is in:
E:\McNair\Projects\Accelerator Demo Day\Test Run
The RNN model is in:
E:\McNair\Projects\Accelerator Demo Day\Experiment
The RNN is still under much development. Modifying anything in this folder is not recommended
All the other folders are used for experimenting purposes, please don't touch them. If you want to understand more about the files as a general user, go to the section A Quick Glance through the File in The Directory below. If you are a developer, go to the Advance User Guide section.
General User Guide: How to Use this Project (Random Forest model)
First, change your directory to the working folder:
cd E:\McNair\Projects\Accelerator Demo Day\Test Run
Then you need to specify the list of accelerators you want to crawl by modifying the following file:
The first line must remain fixed as "Accelerator". Then the next several rows are the Accelerators name. The name needs not to be case sensitive, but it is preferable that the case remains sensitive if possible.
All necessary preparations are now complete. Now onto running the code!
Running the project is as simple as executing the code in the correct order. The files are named in the format "STEPX_name", where as X is the order of execution. To be more specific, run the following 4 commands:
# Crawl Google to get the data for the demo day pages for the accelerator stored in ListOfAccsToCrawl.txt python3 STEP1_crawl.py # Preprocess data using a bag of word approach: each page is characterized by the frequencies of chosen keywords. Chosen keywords are stored in words.txt. This script reates a file called feature_matrix.txt python3 STEP2_preprocessing_feature_matrix_generator.py # Train the RF model python3 STEP3_train_rf.py # Run the model to predict on the HTML of the crawled HTMLs. python3 STEP4_classify_rf.py
The result is stored in CrawledHTMLFull folder and is classified in two folder: positive and negative. The positive folder contains HTMLs that the classifier thought of as "good candidate." The negative contains the opposite. There is also a txt file called prediction.txt that lists everything. feature.txt is an irrelevant file for the general user, please ignore it. Its sole purpose is for analyzing and debugging.
NEVER touch the TrainingHTML folder, datareader.py or the classifier.txt. These are used internally to train data.
A Quick Glance through the File in The Directory
All working file is stored in this folder:
E:\McNair\Projects\Accelerator Demo Day\Test Run
Amazon Mechanical Turk
username: email@example.com password: amount
There's a file in the folder
that was manually created by combining the file
in the mother folder and the file
This file combined the predictions to the actual url of the websites.
Since MTurk makes it hard for us to display the downloaded HTML, it is much faster to just copy the url into the question box rather than trying to display the downloaded HTML.
The advantage to this is that some websites, such as techcrunch.com behaves abnormally when downloaded as HTML so opening these kinds of websites in the browser would actually be more beneficial because the UI would not be messed up. Moreover, if certain websites has paywall or pop-up ads, the user can also click out of it. Since most of the times, paywall or pop-ups are scripts within HTMLs, the classifier can't rule them out because the body of the HTMLs may still contain useful information we are looking for. Major paywalls or websites that required log-ins such as linkedin have been black-listed in the crawler. More detail in the crawler section below.
However. there is a disadvantage to this: websites are ever changing, so there is a possibility that in the future, the URL may not be usable, or has changed to something else; on the other hand, downloaded HTMLs remain the same because it does not require any internet connection to render and thus, the content is static.
Test account: email: firstname.lastname@example.org password: sameastheoneforemail2018
For this project, all the fields that was asked of the user is:
- Whether the page had a list of companies going through an accelerator
- The month and year of the demo day (or article)
- Accelerator name
- Companies going through accelerator
Connor and Minh talked about how to price MTurk so that it is not too generous nor too stingy for workers. Connor could complete four MTurk HITs in 12 minutes (3min/HIT). He then asked his friends who were unfamiliar with MTurk to complete a few surveys, and found they completed around 3 in 15-20 minutes (5-7min/HIT). Given this, we think an upper limit of 10 min/HIT is appropriate. If this is the case, we should price each HIT at $1.50, which leads to an appropriate $9.00/hour rate for workers.
Hand Collecting Data
To crawl, we only looked for data on accelerators which did not receive venture capital data (which Ed found via VentureXpert) and lacked timing info. The purpose of this crawl is to find timing info where we cannot find it otherwise, and if a company received VC we can find timing info via that investment. The file we used to find instances in which we lack timing info and lacked VC is:
/bulk/McNair/Projects/Accelerators/Summer 2018/Merged W Crunchbase Data as of July 17.xlsx
We filtered this sheet in Excel (and checked our work by filtering in SQL) and found 809 companies that lacked timing info and didn't receive VC. From this, we found 74 accelerators which we needed to crawl for.
We used the crawler to search for cohort companies listed for these accelerators.
During the initial test run, the number of good pages was 359. The data is then handled by hand by fellow interns.
The file for hand-coding is in:
/bulk/McNair/Projects/Accelerator Demo Day/Test Run/CrawledDemoDayHTMLFull/FinalResultWithURL
For the sake of collaboration, the team copied this information to a Google Sheet, accessible here:
We split the process into four parts. Each interns will do the following:
1. Go to the given URL.
2. Record whether the page is good data (column F); this can later be used by Minh Le to refine/fine-tune training data.
3. Record whether the page is announcing a cohort or recapping/explaining a demo day (column G). This variable will be used to decide if we should subtract weeks from the given date (e.g. if it is recapping a demo day, the cohort went through the accelerator for the past ~12 weeks, and we should subtract weeks as such).
4. Record date, month, year, and the companies listed for that given accelerator.
5. Note any any information, such as a cohort's special name.
Advance User Guide: An in-depth look into the project and the various settings
Accelerators needed to Crawl
The name lists of Accelerators to crawl is stored in the file:
E:\McNair\Projects\Accelerator Demo Day\Test Run\ListOfAccsToCrawl.txt
Training data is stored in the folder:
E:\McNair\Projects\Accelerator Demo Day\Test Run\TrainingHTML
The Crawler Functionality
The crawler functionality is stored in the file:
The crawler was optimized for improved speed, improved performance and improved filtration while remain functional over the large set of data.
BUG REPORT by Maxine Tao (FIXED): update the crawler with this line of code:
search_results = driver.find_elements_by_xpath("//div[@class='g']/div/div/div/h3/a") + driver.find_elements_by_xpath("//div[@class='g']/div/div/h3/a")
Because apparently for some reason it stopped grabbing the first web page (I think because google may have modified how their website looks.
The input (features) right now is the frequency of X_NUMBER of words appearing in each documents. The word choice is hand selected. This is the naive bag-of-word approach.
Idea: Create a matrix with the first col being the file BiBTex, and the following columns are the words, and the value at (file, word) is the frequency of that word in the file. Then, split the matrix into an array of row vectors, and each vector is then feed into the RNN)
This seems to not give really high accuracy with our LSTM RNN, so I will consider a word2vec approach
Right now I am working on two different classifier: Kyran's old Random Forest model - optimizing it by tweaking parameters and different combination of features - and my RNN text classifier.
The RF model has a ~92% accuracy on the training data and ~70% accuracy on the test data.
The RNN currently has a ~50% accuracy on both train and est data, which is rather concerning.
Test : train ratio is 1:3 (25/75)
Both model is currently using the Bag-of-word approach to preprocess data, but I will try to use Yang's code in the industry classifier to preprocess using word2vec. I'm not familiar with this approach, but I will try to learn this.