7,612
edits
Changes
Jump to navigation
Jump to search
==Overview of Possible Methods==
All BibTex citations are in a text file in folder E:\projects\Kauffman Incubator Project\03 Automate the extraction of information\RajanLasya_ExtractionProtocols_03.21
* [http://www.numpy.org/ NumPy]
: NumPy is a computing package that includes a N-dimensional array object (useful in encoding) and many computational functions to process data. Is required for pix2code.
→DSL Encoder
|Has project status=Active
}}
==Summary==<onlyinclude>The [[LP Extractor Protocol]] currently envisages marking data locations on webpages, converting webpages into a simplified Domain Specific Language (DSL), and then encoding the DSL into a matrix. The markings of data locations would be encoded into a companion matrix. Both matrices will then be fed into a neural network, which is trained to produce the markings given the DSL. To date, we have conducted a literature review that has found papers describing similar "paired input" networks, and are in the process refining our understanding of the pre-existing code and work related to each step.</onlyinclude> Files location: E:\projects\Kauffman Incubator Project\03 Automate the extraction of information\RajanLasya_ExtractionProtocols_03.21 ==Proposed Method== According to “Project Goal V2,” (E:\mcnair\Projects\Incubators) there are we considered three proposed broadly defined methods to organize and extract useful information from an HTML web page. The first method is text processing, analyzing and classifying the textual content of the HTML page. The second method is to use image based pattern recognition, likely through an off-the-shelf model that can extrapolate key HTML elements from web page screenshots. The third, and most novel method is to structurally analyze the HTML tree structure, and express that simplified HTML structure in a Domain Specific Language (DSL). We are currently pursuing this third method.
=== HTML Tree Structure Analysis ===
==Literature==
=== HTML Tree Structure Analysis ===
* [https://www.crummy.com/software/BeautifulSoup/bs4/doc/ Beautiful Soup]
: A simple Python library that can parse HTML files into "Beautiful Soup objects," which are basically tree structure objects. Likely too limited in functionality for automated extraction, but might be useful in developing/testing DSL.
*[https://docs.scrapy.org/en/latest/index.html Scrapy]
: Similar to Beautiful Soup, but has more extensive and efficient functionality. Extracts HTML data by creating "selectors" specified by CSS or XPath expressions.
==== pix2code ====
* [https://github.com/tonybeltramelli/pix2code pix2Code]
: Github repo that contains original reference implementation of pix2code architecture. See above pix2code paper. [https://www.youtube.com/watch?v=pqKeXkhFA3I&feature=youtu.be Video] demo of trained neural network.
* [https://github.com/fjbriones/pix2code2 pix2code2]
: An attempt to improve pix2code through the use of autoencoders between the two LSTM layers.
* [https://github.com/emilwallner/Screenshot-to-code Screenshot-to-code]
: Another version of pix2code with a Bootstrap version that converts web page screenshots to HTML, with the potential to generalize on new design mock-ups.
* [https://github.com/andrewsoohwanlee/pix2code-pytorch pix2code PyTorch]
: pix2code implemented in PyTorch, also not ready for general usage yet.
* [https://github.com/ngundotra/code2pix code2pix]
: A project to recreate an inverse architecture to pix2code, with the objective of creating a GAN (Generative Adversarial Network) to replace pix2code.
=== DFS Encoding ===
: Github repo that contains reference implementation of node2vec algorithm as a python module. See above node2vec paper.
: A [https://github.com/thunlp/OpenNE toolkit] containing node2vec implemented in a framework based on tensorflow
: [https://towardsdatascience.com/node2vec-embeddings-for-graph-data-32a866340fef Here] is a very good and elementary introduction to node2vec
* [https://networkx.github.io/documentation/stable/index.html NetworkX]
: NetworkX is a Python package for loading, visualizing, and processing graph data. Includes built in functions for DFS encoding, and constructing adjacency and edges to vertices matrices.
*[https://radimrehurek.com/gensim/ Gensim]
: Gensim is a Python library used to analyze plain-text documents for semantic structure. Is required for node2vec.
* [http://www.numpy.org/ NumPy]
: NumPy is a computing package that includes a N-dimensional array object (useful in encoding) and many other functions to process data. Is required for pix2code.
=== DSL Development ===
* [http://hackage.haskell.org/package/lucid Lucid]
: Lucid is a DSL implemented with Haskell for writing HTML. It represents DOM elements as functions, and uses specific notation to differentiate between data elements and code elements.
=== General ===
* [https://keras.io/ Keras]
: In conjunction with tensorflow, Keras will support the deep learning components of the project. Is required for pix2code.
* [https://github.com/ziyan/spider SVM Classifier Training Algorithm ]
: From the Yao, Zuo paper, this Github repo contains an algorithm for labelling the collected dataset using clustering, training the SVM with the labeled dataset, and using SVM model to extract content from new webpages. Implemented in JavaScript, CoffeeScript, and Python.
* [https://www.h5py.org/ H5PY]
: The h5py package can be used to store large amounts of numerical data, and integrates well with NumPy
=== Useful tutorials ===
: Since we will be using a two-layer LSTMs in tensorflow, this [https://medium.com/@erikhallstrm/using-the-tensorflow-multilayered-lstm-api-f6e7da7bbe40 article] might be useful.
=== Proposed Model ===
: Here is a visualization of the model that we might want to use for our extractor
[[File: Extractor-Model.png| first diagram of extractor model]]
==DSL Encoder==
To encode the structure of the DSL scripts, we can try using one-hot vector. More details can be found [https://blog.floydhub.com/turning-design-mockups-into-code-with-deep-learning/ here] and on the [[DSL Encoding]] page.
