Changes

{{Project|Has project output=Tool|Has sponsor=McNair ProjectsCenter

This project that utilizes Selenium and Machine Learning to get good candidate web pages and classify webpages web pages as a demo day page containing a list of cohort companies, currently ultimately to gather good candidates to push to Mechanical Turk. The code is written using scikit learn's random forest model Python 3 using Selenium and Tensorflow (Keras). This article will also preliminaries of the Mechanical Turk tool and how it can be used to collect data. ==Project Goal==The goal of this project is to find good "Demo Day" candidate web pages and to submit these pages to Amazon Mechanical Turk for data collecting. A good candidate is defined as a bag page containing a list of words approachcohort companies associated with an accelerator. Through observation, good candidates usually containing time and location information about the demo day as well and thus is sufficient to be pushed to MTurk to collect data.

The file for hand-coding is in: /bulk/McNair/Projects/Accelerator Demo Day/Test Run/CrawledDemoDayHTMLFull/'''FinalResultWithURL''' For the sake of collaboration, the team copied this information to a Google Sheet, accessible here: https://docs.google.com/spreadsheets/d/16Suyp364lMkmUuUmK2dy_9MeSoS1X4DfFl3dYYDGPT4/edit?usp==Development Notes==sharing Right now I am working on two different classifierWe split the process into four parts. Each interns will do the following: Kyran's old Random Forest model - optimizing it  1. Go to the given URL. 2. Record whether the page is good data (column F); this can later be used by tweaking parameters and different combination of features [[Minh Le]] to refine/fine- and my RNN text classifiertune training data.The RF model has 3. Record whether the page is announcing a cohort or recapping/explaining a demo day (column G). This variable will be used to decide if we should subtract weeks from the given date (e.g. if it is recapping a demo day, the cohort went through the accelerator for the past ~92% accuracy on the training data 12 weeks, and we should subtract weeks as such). 4. Record date, month, year, and ~70% accuracy on the test datacompanies listed for that given accelerator.The RNN currently has 5. Note any any information, such as a ~50% accuracy on both train and est data, which is rather concerningcohort's special name. Test : train ration is 1:3 (25/75)Both model Once this process is currently using finished, we will filter only the Bag-of-word approach to preprocess data1s in Column F, but I and [[Connor Rothschild]] and [[Maxine Tao]] will try work to use Yang's code populate empty cells in The File to Rule Them All with that data.  ==Advance User Guide: An in -depth look into the industry classifier project and the various settings== ===Accelerators needed to preprocess using word2vec. I'm not familiar with this approach, but I will try Crawl===The name lists of Accelerators to learn thiscrawl is stored in the file: E:\McNair\Projects\Accelerator Demo Day\Test Run\ListOfAccsToCrawl.txt ===Training Data===Training data is stored in the folder: E:\McNair\Projects\Accelerator Demo Day\Test Run\TrainingHTML

===The Crawler Functionality===The crawler functionality is stored in the file:To be updated STEP1_crawl.pyThe crawler was optimized for improved speed, improved performance and improved filtration while remain functional over the large set of data.

BUG REPORT by Maxine Tao (FIXED): update the crawler with this line of code: search_results = driver.find_elements_by_xpath("//div[@class='g']/div/div/div/h3/a") + driver.find_elements_by_xpath("//div[@class='g']/div/div/h3/a")Because apparently for some reason it stopped grabbing the first web page (I think because google may have modified how their website looks. ===The Classifier===