Changes

[[Listing Page Classifier Progress|Progress Log (updated on 45/2217/2019)]]

# Build a CNN classifier using Python and TensorFlow

The goal here is to identify url links from the HTML code of a website. We can solve this by finding the place holder, which is the anchor tag <a>, for a hyperlink. Within the anchor tag, we may locate the href attribute that contains the url that we look for (see example below). <code><a href="/wiki/Listing_Page_Classifier_Progress" title="Listing Page Classifier Progress"> Progress Log (updated on 4/15/2019)</a></code>

<code><a href = https://www.facebook.com/...></a></code>

[[File:WebPageTree.png|700px500px|thumb|center|Site Map Tree]]

Intuition'''Intuitions:'''

E:\projects\listing page identifier\Internal_Link\Internal_url_BFS.py'''''Depth-First Search (DFS) approach''''':

We visit ===Web Page Screenshot Tool===This tool reads two text files: test.txt and train.txt, and outputs a pagefull screenshot (node) "A" and then all A's children see sample output on the current path will be visited before we visit A's neighbor node "B"right) of each url in these 2 text files.[[File:screenshotEx.png|200x400px|thumb|right|Sample Output]]

For example====Browser Automation Tool====The initial idea was to use the [https://www.seleniumhq.org/ selenium] package to set up a browser window that fits the web page size, assuming then capture the furthest depth whole window to get a user wants to dig in is 2full screenshot of the page. After several test runs on different websites, we will start this method worked great for most web pages but with some exceptions. Therefore, the [https://splinter.readthedocs.io/en/latest/why.html splinter] package is chosen as the final browser automation tool to assist our homepage and then examine its first child node "page 1", then visiting page 1's children until we meet screenshot tool  ====Used Browser====The picked browser for taking screenshot is Firefox. A geckodriver v0.24.0 was downloaded for setting up the maximum depthbrowser during browser automation. Then we move onto homepage '''Note:''s second child "page 2" and visit page 2's children until we reach initial plan was to use Chrome, but encountered some issues with switching different versions(v73 to v74) of chromedriver during the maximum depth. Next we visit homepage's next child and so onbrowser automation.

Python file saved in E:\projects\listing page identifier\Internal_Link\Internal_url_DFSgenerate_dataset.py

===Web Page '''''Generate and Label Image Data: ''''' feed paths/directories of <code>train.txt</code> and <code>text.txt</code> into Screenshot Tool (IN PROGRESS)===to get our image dataThis tool will take 2 user input*Results are split into two folders: the url train and the output file(.png)'s name. It will output a png file that has the full screen shot of a web page (see output file example on the right)test** Also separated into sub-folders: cohort and not_cohort[[File:screenshotExautoName.png|50px|thumb|right|Sample Output250px]]** Make sure to create train and test folders (in the '''same directory''' as <code>train.txt</code> and <code>text.txt</code>), and their sub-folders cohort and not_cohort '''BEFORE''' running the Screenshot Tool

E:\projects\listing page identifier\screen_shot\screen_shot_toolcnn.py

'''''NOTE: '''''[https://keras.io/ Keras] package (with TensorFlow backend) is used for setting up the model '''Current condition/issue''' of the model:* loss: 0.9109, accuracy: 0.9428* The model runs with no problem, however, it does not make classification. All predictions on the test set are the same Some '''factors/problems''' to consider for '''future implementation''' on the model:* Class label is highly imbalanced: o (not cohort) is way more than 1 (cohort) class**may cause our model favoring the larger class, then the accuracy metric is not reliable**several suggestions to fix this: A) under-sampling the larger class B)over-sampling the smaller class* Convert image data into same format: [https://www.oreilly.com/library/view/linux-multimedia-hacks/0596100760/ch01s04.html Make image thumbnail]**we can modify image target size in our CNN, but we don't know if Keras library crop or re-scale image with given target size*I chose to group images into cohort folder or not_cohort folder to let our CNN model detect the class label of an image. There are certainly other ways to detect class label and one may want to modify the Screenshot Tool and <code>cnn.py</code> to assist with other approaches  Useful rescource:*Image generator in Keras: https://keras.io/preprocessing/image/*Keras tutorial for builindg a CNN: https://adventuresinmachinelearning.com/keras-tutorial-cnn-11-lines/ https://towardsdatascience.com/building-a-convolutional-neural-network-cnn-in-keras-329fbbadc5f5 https://becominghuman.ai/building-an-image-classifier-using-deep-learning-in-python-totally-from-a-beginners-perspective-be8dbaf22dd8 ===Image ProcessingWorkflow===This section summarizes a general process of utilizing above tools to get appropriate input for our CNN model, also serves as a guidance for anyone who wants to implement upon those tools.

This method would likely rely on a [https:# Feed raw data (as for now, our raw data is the <code>The File to Rule Them All.csv</code>) into <code> generate_dataset.py</encode> to get text files (<code>train.wikipediatxt</code> and<code>text.orgtxt</wiki/Convolutional_neural_network convolutional neural network code>) that contain a list of all internal urls with their corresponding indicator (CNNclass label)] to classify HTML elements present # Create 2 folders: train and test, located in web page screenshotsthe same directory as <code>train. Implementation could be achieved by combining txt</code> and <code>text.txt</code>, also create 2 sub-folders: cohort and not_cohort within these 2 folders# Feed the VGG16 model or ResNet architecture with batch normalization to increase accuracy directory/path of <code>train.txt</code> and <code>text.txt</code> into <code>screen_shot_tool.py</code>. This process will automatically group images into their corresponding folders that we just created in this context.step 2