Changes

# Build a site map generator: output every internal link of input websitesa website# Build a tool that captures a screenshot screenshots of individual web pages# Build a CNN classifier using Python and TensorFlow

===Approaches (IN PROGRESS)===[[Listing Page Classifier Progress|Progress Log (updated on 4/15/2019)]]====Site Map Generator====

'''Part I: ====URL Extraction from HTML'''====

The goal here is to identify url links from the HTML code of a website. We can solve this by finding the place holder, which is the anchor tag <a>, for a hyperlink. Within the anchor tag, we may locate the href attribute that contains the url link that we are looking for (see example below). <code><a href="/wiki/Listing_Page_Classifier_Progress" title="Listing Page Classifier Progress"> Progress Log (updated on 4/15/2019)</a></code>

* The href may not give us the full url, like above example it excludes the domain name: "http://www.edegan.com"

'''Note: ''' the [https://www.crummy.com/software/BeautifulSoup/bs4/doc/ beautifulsoup] package is used for pulling data out of HTML

 '''Part II: ====Distinguish Internal Links'''====

* If the href is in a full url format, but it does not contain the domain name, then it is an external link (see example below, assuming the domain name is not facebook.com)

'''Part III[[File: Algorithm on Collecting Internal Links'''WebPageTree.png|500px|thumb|center|Site Map Tree]]

[[File'''Intuitions:WebPageTree.png|700px|thumb|center|Site Map Tree]]'''*We treat each internal page as a tree node*Each node can have multiple linked children or none*Taking the above picture as an example, the homepage is the first tree node (at depth = 0) that we will be given as an input to our function, and it has 4 children (at depth = 1): page 1, page 2, page 3, and page 4*Given the above idea, we have built 2 following algorithms to find all internal links of a web page with 2 user inputs: homepage url and depth

Intuition'''Note:*We treat each internal page as a tree node*Each node can have multiple children or none*Taking ''' the '''recommended maximum depth''' input is '''2'''. Since our primary goal is to capture the above picture as an example, screenshot of the homepage is portfolio page (client listing page) and this page often appears at the first tree node that we depth, if not, second depth will be given as an input enough to our function, and it has 4 children: page 1, page 2, page 3, and page 4*Given achieve the above ideagoal, we have built 2 following algorithms no need to find all internal links of a web page with 2 given user inputs: homepage url and dive deeper than the second depth.

we We examine all pages(nodes) at the same depth before going down to the next depth.

For example E:\projects\listing page identifier\Internal_url_BFS.py ===Web Page Screenshot Tool===This tool reads two text files: test.txt and train.txt, assuming and outputs a full screenshot (see sample output on the furthest depth a user wants to dig right) of each url in is these 2text files.[[File:screenshotEx.png|200x400px|thumb|right|Sample Output]] ====Browser Automation Tool====The initial idea was to use the [https://www.seleniumhq.org/ selenium] package to set up a browser window that fits the web page size, we will start with our homepage and then examine its first child node "capture the whole window to get a full screenshot of the page 1". After several test runs on different websites, this method worked great for most web pages but with some exceptions. Therefore, then visiting page 1's children until we meet the maximum depth[https://splinter. Then we move onto homepage's second child "page 2" and visit page 2's children until we reach readthedocs.io/en/latest/why.html splinter] package is chosen as the maximum depthfinal browser automation tool to assist our screenshot tool  ====Used Browser====The picked browser for taking screenshot is Firefox. A geckodriver v0.24. Next we visit homepage's next child and so on0 was downloaded for setting up the browser during browser automation. E'''Note:\projects\listing page identifier\Internal_Link\Internal_url_DFS''' initial plan was to use Chrome, but encountered some issues with switching different versions(v73 to v74) of chromedriver during the browser automation.py

====Web Page Screenshot Tool (IN PROGRESS)====This tool will take 2 user input: the url and the output file(.png)'s name. It will output a png file that has the full screen shot of a web page (see output Python file example on the right)[[File:screenshotEx.png|50px|thumb|right|Output File Example]]saved in E:\projects\listing page identifier\screen_shot\screen_shot_tool.py

This method would likely rely '''''NOTE: '''''[https://keras.io/ Keras] package (with TensorFlow backend) is used for setting up the model '''Current condition/issue''' of the model:* loss: 0.9109, accuracy: 0.9428* The model runs with no problem, however, it does not make classification. All predictions on the test set are the same Some '''factors/problems''' to consider for '''future implementation''' on a the model:* Class label is highly imbalanced: o (not cohort) is way more than 1 (cohort) class**may cause our model favoring the larger class, then the accuracy metric is not reliable**several suggestions to fix this: A) under-sampling the larger class B)over-sampling the smaller class* Convert image data into same format: [https://enwww.wikipediaoreilly.orgcom/library/view/linux-multimedia-hacks/0596100760/wikich01s04.html Make image thumbnail]**we can modify image target size in our CNN, but we don't know if Keras library crop or re-scale image with given target size*I chose to group images into cohort folder or not_cohort folder to let our CNN model detect the class label of an image. There are certainly other ways to detect class label and one may want to modify the Screenshot Tool and <code>cnn.py</Convolutional_neural_network code> to assist with other approaches  Useful rescource:*Image generator in Keras: https://keras.io/preprocessing/image/*Keras tutorial for builindg a CNN: https://adventuresinmachinelearning.com/keras-tutorial-cnn-11-lines/ https://towardsdatascience.com/building-a-convolutional -neural -network -cnn-in-keras-329fbbadc5f5 https://becominghuman.ai/building-an-image-classifier-using-deep-learning-in-python-totally-from-a-beginners-perspective-be8dbaf22dd8 ===Workflow===This section summarizes a general process of utilizing above tools to get appropriate input for our CNN model, also serves as a guidance for anyone who wants to implement upon those tools. # Feed raw data (CNNas for now, our raw data is the <code>The File to Rule Them All.csv</code>)] into <code> generate_dataset.py</code> to classify HTML elements present get text files (<code>train.txt</code> and<code>text.txt</code>) that contain a list of all internal urls with their corresponding indicator (class label)# Create 2 folders: train and test, located in web page screenshotsthe same directory as <code>train.txt</code> and <code>text. Implementation could be achieved by combining txt</code>, also create 2 sub-folders: cohort and not_cohort within these 2 folders# Feed the VGG16 model or ResNet architecture with batch normalization to increase accuracy directory/path of <code>train.txt</code> and <code>text.txt</code> into <code>screen_shot_tool.py</code>. This process will automatically group images into their corresponding folders that we just created in this context.step 2