Changes

# Build a site map generator: output every internal link of input websitesa website# Build a tool that captures a screenshot screenshots of individual web pages# Build a CNN classifier using Python and TensorFlow

===Approaches (IN PROGRESS)===[[Listing Page Classifier Progress|Progress Log (updated on 4/15/2019)]]====Site Map Generator====

[[File:WebPageTreeThe goal here is to identify url links from the HTML code of a website. We can solve this by finding the place holder, which is the anchor tag <a>, for a hyperlink. Within the anchor tag, we may locate the href attribute that contains the url (see example below).png|900px]]

Intuition:*We treat each internal page as <code><a tree node. Each node can have multiple children. *Taking the above picture as an example, the homepage is the first tree node that we will be given as an input to our function, and it has href="/wiki/Listing_Page_Classifier_Progress" title="Listing Page Classifier Progress"> Progress Log (updated on 4 children: page 1, page 2, page 3, and page 4*Given the above idea, we have built 2 following algorithms to find all internal links of /15/2019)</a web page with 2 given user inputs: homepage url and depth></code>

'''Breadth-First Search(BFS)approach'''Issues may occur: * The href may not give us the full url, like above example it excludes the domain name: http://www.edegan.com * Some may not exclude the domain name and we should take consideration of both cases when extracting the url

we examine all pages(nodes) at '''Note:''' the same depth before going down to the next depth[https://www.crummy.com/software/BeautifulSoup/bs4/doc/ beautifulsoup] package is used for pulling data out of HTML

====Distinguish Internal Links====* If the href is not presented in a full url format (referring to the example above), then it is for sure an internal link* If the href is in a full url format, but it does not contain the domain name, then it is an external link (see example below, assuming the domain name is not facebook.com) E<code><a href = https:\projects\listing page identifier\Internal_Link\Internal_url_BFS//www.pyfacebook.com/...></a></code>'''Depth-First Search (DFS) approach'''====Algorithm on Collecting Internal Links==== [[File: WebPageTree.png|500px|thumb|center|Site Map Tree]]

we visit '''Intuitions:'''*We treat each internal page as a pagetree node*Each node can have multiple linked children or none*Taking the above picture as an example, the homepage is the first tree node (nodeat depth = 0)"A" that we will be given as an input to our function, and then all its it has 4 children on (at depth = 1): page 1, page 2, page 3, and page 4*Given the current path will be visited before above idea, we visit A's neighbor node "B".have built 2 following algorithms to find all internal links of a web page with 2 user inputs: homepage url and depth

For example, assuming '''Note:''' the furthest '''recommended maximum depth a user wants to dig in ''' input is '''2, we will start with '''. Since our homepage primary goal is to capture the screenshot of the portfolio page (client listing page) and then examine its this page often appears at the first children "page 1"depth, if not, second depth will be enough to achieve the goal, then visiting page 1's children until we meet no need to dive deeper than the maximum second depth. Then we move onto homepage '''''Breadth-First Search (BFS) approach''''s second children "page 2" and visit page 2's children until we reach :  We examine all pages(nodes) at the maximum same depth. Next we visit homepage before going down to the next children page 3 and so ondepth.  Python file saved in  E:\projects\listing page identifier\Internal_Link\Internal_url_DFSInternal_url_BFS.py ===Web Page Screenshot Tool===This tool reads two text files: test.txt and train.txt, and outputs a full screenshot (see sample output on the right) of each url in these 2 text files.[[File:screenshotEx.png|200x400px|thumb|right|Sample Output]] ====Browser Automation Tool====The initial idea was to use the [https://www.seleniumhq.org/ selenium] package to set up a browser window that fits the web page size, then capture the whole window to get a full screenshot of the page. After several test runs on different websites, this method worked great for most web pages but with some exceptions. Therefore, the [https://splinter.readthedocs.io/en/latest/why.html splinter] package is chosen as the final browser automation tool to assist our screenshot tool  ====Used Browser====The picked browser for taking screenshot is Firefox. A geckodriver v0.24.0 was downloaded for setting up the browser during browser automation. '''Note:''' initial plan was to use Chrome, but encountered some issues with switching different versions(v73 to v74) of chromedriver during the browser automation.

====Web Page Screenshot Tool (IN PROGRESS)====This tool will take 2 user input: the url and the output Python file(.png)'s name. It will output a png file that has the full screen shot of a web pagesaved in E:\projects\listing page identifier\screen_shot\screen_shot_tool.py

This method would likely rely Python file saved in E:\projects\listing page identifier\generate_dataset.py '''''Generate and Label Image Data: ''''' feed paths/directories of <code>train.txt</code> and <code>text.txt</code> into Screenshot Tool to get our image data*Results are split into two folders: train and test** Also separated into sub-folders: cohort and not_cohort[[File:autoName.png|250px]]** Make sure to create train and test folders (in the '''same directory''' as <code>train.txt</code> and <code>text.txt</code>), and their sub-folders cohort and not_cohort '''BEFORE''' running the Screenshot Tool ====CNN Model====Python file saved in E:\projects\listing page identifier\cnn.py '''''NOTE: '''''[https://keras.io/ Keras] package (with TensorFlow backend) is used for setting up the model '''Current condition/issue''' of the model:* loss: 0.9109, accuracy: 0.9428* The model runs with no problem, however, it does not make classification. All predictions on the test set are the same Some '''factors/problems''' to consider for '''future implementation''' on a the model:* Class label is highly imbalanced: o (not cohort) is way more than 1 (cohort) class**may cause our model favoring the larger class, then the accuracy metric is not reliable**several suggestions to fix this: A) under-sampling the larger class B)over-sampling the smaller class* Convert image data into same format: [https://enwww.oreilly.com/library/view/linux-multimedia-hacks/0596100760/ch01s04.html Make image thumbnail]**we can modify image target size in our CNN, but we don't know if Keras library crop or re-scale image with given target size*I chose to group images into cohort folder or not_cohort folder to let our CNN model detect the class label of an image. There are certainly other ways to detect class label and one may want to modify the Screenshot Tool and <code>cnn.py</code> to assist with other approaches  Useful rescource:*Image generator in Keras: https://keras.wikipediaio/preprocessing/image/*Keras tutorial for builindg a CNN: https://adventuresinmachinelearning.orgcom/keras-tutorial-cnn-11-lines/ https://wikitowardsdatascience.com/Convolutional_neural_network building-a-convolutional -neural -network -cnn-in-keras-329fbbadc5f5 https://becominghuman.ai/building-an-image-classifier-using-deep-learning-in-python-totally-from-a-beginners-perspective-be8dbaf22dd8 ===Workflow===This section summarizes a general process of utilizing above tools to get appropriate input for our CNN model, also serves as a guidance for anyone who wants to implement upon those tools. # Feed raw data (CNNas for now, our raw data is the <code>The File to Rule Them All.csv</code>)] into <code> generate_dataset.py</code> to classify HTML elements present get text files (<code>train.txt</code> and<code>text.txt</code>) that contain a list of all internal urls with their corresponding indicator (class label)# Create 2 folders: train and test, located in web page screenshotsthe same directory as <code>train.txt</code> and <code>text. Implementation could be achieved by combining txt</code>, also create 2 sub-folders: cohort and not_cohort within these 2 folders# Feed the VGG16 model or ResNet architecture with batch normalization to increase accuracy directory/path of <code>train.txt</code> and <code>text.txt</code> into <code>screen_shot_tool.py</code>. This process will automatically group images into their corresponding folders that we just created in this context.step 2