Pix2code - Revision history

Ed at 17:34, 21 September 2020

2020-09-21T17:34:40Z

Ed: /* Usage of pix2code on RDP */

2019-05-30T17:13:02Z

Usage of pix2code on RDP

Ed: /* Usage of pix2code on RDP */

2019-05-30T17:12:02Z

Usage of pix2code on RDP

Ed: /* Usage of pix2code on RDP */

2019-05-30T17:07:21Z

Usage of pix2code on RDP

Hiep at 22:45, 5 April 2019

2019-04-05T22:45:14Z

Hiep at 22:43, 5 April 2019

2019-04-05T22:43:41Z

Hiep: /* Usage on RDP */

2019-04-05T22:42:15Z

Usage on RDP

Hiep: Created page with "{{Project |Has title=Pix2code experimentation |Has owner=Hiep Nguyen, }} ==Brief Introduction== Pix2code is an AI model that can convert GUI images to DSL codes and then uses..."

2019-04-05T22:41:53Z

Created page with "{{Project |Has title=Pix2code experimentation |Has owner=Hiep Nguyen, }} ==Brief Introduction== Pix2code is an AI model that can convert GUI images to DSL codes and then uses..."

New page

{{Project
|Has title=Pix2code experimentation
|Has owner=Hiep Nguyen,
}}

==Brief Introduction==
Pix2code is an AI model that can convert GUI images to DSL codes and then uses a compiler to convert DSL code to HTML, Android XML, and iOS Storyboard. More details can be found [https://arxiv.org/pdf/1705.07962.pdf here] in the original paper. Instructions to train and use the models can be found on the original [https://github.com/tonybeltramelli/pix2code github] page. There is an improved version of pix2code, which is [https://github.com/fjbriones/pix2code2 pix2code2]. It uses a Convolutional Neural Network (CNN) as an autoencoder for the GUI before training. The users also include a pre-trained model to experiment with.

==Usage on RDP==
Currently, source code and pre-trained model for pix2code are living on
E:/projects/pix2code_test

To generate DSL code from specific GUI images, first place the image on pix2code_test directory, then do the following

cd pix2code_test/model
./sample.py ../bin pix2code2 ../test_img.png ../code greedy #to use greedy algorithm, replace greedy with 1,2,3..,k to use beam search with size k.

The GUI code will be inside the pix2code_test/code directory

To generate GUI to HTML:
cd compiler
./web_compiler.py ./code/test_img.gui

==Discussion==
While pix2code can preserve the structure of the HTML page quite well, it cannot preserve the contents of the websites. Most of the texts from the original page are distorted in the generated DSL. Moreover, pix2code is extremely expensive to train and the current model only works for very simple GUIs that are similar to ones in the training set. Hence, pix2code model would not be suited for building an information extractor. However, we can learn from the source code how to input and structure GUI data and construct LSTM networks on top of GUI and output DSL code.

@@ Line 20: / Line 20: @@
 To generate GUI to HTML:
   cd compiler
-  ./web-compiler.py ./code/test.gui
+  ./web-compiler.py ../code/test.gui
 ==Discussion==
 While pix2code can preserve the structure of the HTML page quite well, it cannot preserve the contents of the website. Most of the texts from the original page are distorted in the generated DSL. Moreover, pix2code is extremely expensive to train and the current model only works for very simple GUIs that are similar to ones in the training set. Hence, pix2code model would not be suited for building an information extractor. However, we can learn from the source code how to input and structure GUI data and construct LSTM networks on top of GUI and output DSL code.

@@ Line 20: / Line 20: @@
 To generate GUI to HTML:
   cd compiler
-  ./web-compiler.py ./code/test_img.gui
+  ./web-compiler.py ./code/test.gui
 ==Discussion==
 While pix2code can preserve the structure of the HTML page quite well, it cannot preserve the contents of the website. Most of the texts from the original page are distorted in the generated DSL. Moreover, pix2code is extremely expensive to train and the current model only works for very simple GUIs that are similar to ones in the training set. Hence, pix2code model would not be suited for building an information extractor. However, we can learn from the source code how to input and structure GUI data and construct LSTM networks on top of GUI and output DSL code.

@@ Line 5: / Line 5: @@
 ==Brief Introduction==
-Pix2code is an AI model that can convert GUI images to DSL codes and then uses a compiler to convert DSL code to HTML, Android XML, and iOS Storyboard. More details can be found [https://arxiv.org/pdf/1705.07962.pdf here] in the original paper. Instructions to train and use the models can be found on the original [https://github.com/tonybeltramelli/pix2code github] page. There is an improved version of pix2code, which is [https://github.com/fjbriones/pix2code2 pix2code2]. It uses a Convolutional Neural Network  (CNN) as an autoencoder for the GUI before training. The users also include a pre-trained model to experiment with.
+Pix2code is an AI model that can convert GUI images to DSL codes and then uses a compiler to convert DSL code to HTML, Android XML, and iOS Storyboard. More details can be found [https://arxiv.org/pdf/1705.07962.pdf here] in the original paper. Instructions to train and use the models can be found on the original [https://github.com/tonybeltramelli/pix2code github] page. There is an improved version of pix2code, which is [https://github.com/fjbriones/pix2code2 pix2code2]. It uses a Convolutional Neural Network  (CNN) as an autoencoder for the GUI before training. The users also include a pre-trained model to experiment with. What we have in the RDP right now is pix2code2.
 ==Usage of pix2code on RDP==

← Older revision		Revision as of 22:43, 5 April 2019
Line 23:		Line 23:

	==Discussion==		==Discussion==
−	While pix2code can preserve the structure of the HTML page quite well, it cannot preserve the contents of the ~~websites~~. Most of the texts from the original page are distorted in the generated DSL. Moreover, pix2code is extremely expensive to train and the current model only works for very simple GUIs that are similar to ones in the training set. Hence, pix2code model would not be suited for building an information extractor. However, we can learn from the source code how to input and structure GUI data and construct LSTM networks on top of GUI and output DSL code.	+	While pix2code can preserve the structure of the HTML page quite well, it cannot preserve the contents of the website. Most of the texts from the original page are distorted in the generated DSL. Moreover, pix2code is extremely expensive to train and the current model only works for very simple GUIs that are similar to ones in the training set. Hence, pix2code model would not be suited for building an information extractor. However, we can learn from the source code how to input and structure GUI data and construct LSTM networks on top of GUI and output DSL code.

@@ Line 1: / Line 1: @@
 {{Project
 |Has title=Pix2code experimentation
 |Has owner=Hiep Nguyen,

@@ Line 7: / Line 7: @@
 Pix2code is an AI model that can convert GUI images to DSL codes and then uses a compiler to convert DSL code to HTML, Android XML, and iOS Storyboard. More details can be found [https://arxiv.org/pdf/1705.07962.pdf here] in the original paper. Instructions to train and use the models can be found on the original [https://github.com/tonybeltramelli/pix2code github] page. There is an improved version of pix2code, which is [https://github.com/fjbriones/pix2code2 pix2code2]. It uses a Convolutional Neural Network  (CNN) as an autoencoder for the GUI before training. The users also include a pre-trained model to experiment with.
-==Usage on RDP==
+==Usage of pix2code on RDP==
 Currently, source code and pre-trained model for pix2code are living on
   E:/projects/pix2code_test