Changes

{{Project|Has project output=Tool|Has sponsor=McNair ProjectsCenter

|Has project status=ActiveSubsume

This is a tensorflow ML project that classifies webpages as a demo day page containing a list of cohort companies, currently using scikit learn's random forest modeland a bag of words approach. Currently about 80% accuracy, though this would be vastly improved with more training data. The classifier currently really overfits the training data. The classifier itself takes:

A<strong>Input features: The number of times each word in words.txt occurs in a webpage. </strong> This is calculated by web_demo_features.py in the same directoryand output to a tsv file. It also takes : the number frequencies of occurrences of years each word from 1900-2099words.txt, and month words group grouped in seasons, and phrases of the form "# startups". It also takes the number of simple links (links in the form www.abc.com or www.abc.org) and the number of those that are attached to images. It also takes the number of "strong" html tags in the body. These features can be extended by adding words to words.txt or regexes to the PATTERNS variable in web_demo_features.py. There is also unused code for generating monogram/bigram tfidf frequencies, this might improve the classifier if we had more data.

B<strong>Training data: </strong> A set of webpages hand-classified as to whether they contain a list of cohort companies. This The classification is stored in classification.txt, which is a tsv equivalent of Demo Day URLS.xlsx. Keep in mind that this txt file must be utf-8 encoded. In textpad, one can convert a file to utf-8 by pressing save-as, and changing the encoding at the bottom. The HTML pages themselves are stored in DemoDayHTMLFull.

<strong>Project location:</strong>

Training data: E:\McNair\Projects\Accelerators\Spring 2018\demo_day_classifier\DemoDayHTMLFull\Demo Day URLs.xlsx 2018-04-09: Wrote the code to put everything together. It runs the google crawler, creates the features matrix from the results, and then runs the classifier on it. This can be used to increase the size of the dataset and improve the accuracy of the classifier.  * Steps to train add training data to the model: Put all of the html files to be used in DemoDayHTMLFull. Then run web_demo_featuresPut corresponding entries into demo_day_cohort_lists.py to generate xlsx (only the features matrixcolumns "URL" and "Cohort" are necessary, training_featuresbut they must be in alphabetical order.txtdata_reader. Thenpy will throw error otherwise), run demo_day_classifier_randforest.py then export it to generate the model, classifierclassification.pkltxt. Make sure that in demo_day_classifier_randforest.pyConvert this to utf-8 (textpad can do this, USE_CROSS_VALIDATION is set to False in order to generate the modeljust save as -> encoding:utf-8). * Steps to Then run: In the file crawl_and_classify python3 web_demo_features.py, set the variables #to whatever is wanted. Then, run crawl_and_classify using python3. It will download all of generate the html files into the directory CrawledHTMLPages, and then it will generate a features matrix of features, CrawledHTMLPages\featureshand_training_features.txt python3 demo_day_classifier_randforest. It will then run py #to generate the trained model saved in , classifier.pkl to predict whether these pages are demo day pages, and then it will save the results to predicted.txt.

Google bindings ==Files and Directories==* CrawledHTMLPages** Contains the classified html file results from crawl_and_classify.py, stored in positive and negative folders based on how they are classified.* DemoDayHTMLFull** Contains the training data for pythonthe classifier. demo_day_cohort_lists.xlsx is the classification (same as classification.txt, which is actually used by the program, but the excel file has hyperlinks), and the html files are used for generating the features matrix.* demo_day_classifier_randforest.py** The classifier itself. A pickled version of the classifier should be saved in classifier.pkl.* web_demo_features.py** Generates the features matrix from a directory of html files to be used in the classifier. See input features.* words.txt** The words for the features. The frequency of each word is used as a feature (maybe change this to tfidf?)* data_reader.py** Helper functions to read in the data for the classifier.* crawl_and_classify.py** Googles a bunch of results for a given query and list of accelerators and their years, and then classifies the html pages into CrawledHTMLPages.

HTML Change from Bag-Of-Words model to text converteda more powerful neural network, perhaps an RNN, or use full tfidf monogram/bigram frequencies. This would need even more data, though. The best way to collect more data would probably be to automate/make easier the process of data collection, and just have a few people collect a few thousand points of data, or use mechanical turk. This would likely improve accuracy a lot, and allow for more sophisticated classification methods.

*httpshttp://statsscikit-learn.stackexchange.comorg/stable/questionstutorial/181text_analytics/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netwworking_with_text_data.html