edegan.com - User contributions [en]

Deep Text Classifier

2017-10-18T15:59:28Z

Yangzhang: /* Deep Text Classifier */

=Deep Text Classifier=

E:\McNair\Projects\Deep Text Classifier

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

'''To general users:'''

Your input file (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:

1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name will be the same as your input. Please change the name properly to indicate the label information as have been discussed above. And place this pickle file under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

'''To advanced users:'''

1. One important step in data preprocessing is to encode words (strings) into integers. The solution is to build a dictionary mapping words to their corresponding indices. For example, let's say "hello" is the 17th words in our dictionary and thus "hello" is encoded to 17. Our advanced dictionary is ordered by the words' frequency. Higher the frequency smaller the index. That is you should expect to see "the" and "a" these words with very small indices. Please also notice that 0 and 1 these two indices are not assigned to any words intentionally. The advantage here is that you can easily ignore those very common and meaningless words, like "the", by simply saying I only want to consider words with the indices > 20 for example. Notice that it's possible to encounter words that are not in our dictionary and we will alway assign them to index 1. These words are safe to ignore given that our dictionary is big enough.

2. Saving a pickle file is an very efficient way to retrieve the data so that you don't need to do data preprocessing every time when you want to run your classifier.

==Model Training/Prediction==

We write in [https://www.tensorflow.org/ Tensorflow] for all the classifiers. [https://keras.io/ Keras] is a good wrapper over the Tensorflow framework to allow you quickly build up a neural network and train it. ( if you are new to Deep Learning and Tensorflow, please do stay with Keras. )

* '''Embedding'''
[https://keras.io/layers/embeddings/ Keras Official Documentation]
[https://www.tensorflow.org/tutorials/word2vec Tensorflow : Vector Representations of Words]
[https://en.wikipedia.org/wiki/Word2vec Wiki : Word2vec]

* '''LSTM'''
[http://colah.github.io/posts/2015-08-Understanding-LSTMs/ A Nice Blog about LSTM]
[https://www.tensorflow.org/tutorials/recurrent Tensorflow : Recurrent Neural Networks]
[https://keras.io/layers/recurrent/ Keras Official Documentation]

Deep Text Classifier

2017-10-12T20:46:00Z

Yangzhang:

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

'''To general users:'''

Your input file (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:

1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name will be the same as your input. Please change the name properly to indicate the label information as have been discussed above. And place this pickle file under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

'''To advanced users:'''

1. One important step in data preprocessing is to encode words (strings) into integers. The solution is to build a dictionary mapping words to their corresponding indices. For example, let's say "hello" is the 17th words in our dictionary and thus "hello" is encoded to 17. Our advanced dictionary is ordered by the words' frequency. Higher the frequency smaller the index. That is you should expect to see "the" and "a" these words with very small indices. Please also notice that 0 and 1 these two indices are not assigned to any words intentionally. The advantage here is that you can easily ignore those very common and meaningless words, like "the", by simply saying I only want to consider words with the indices > 20 for example. Notice that it's possible to encounter words that are not in our dictionary and we will alway assign them to index 1. These words are safe to ignore given that our dictionary is big enough.

2. Saving a pickle file is an very efficient way to retrieve the data so that you don't need to do data preprocessing every time when you want to run your classifier.

==Model Training/Prediction==

We write in [https://www.tensorflow.org/ Tensorflow] for all the classifiers. [https://keras.io/ Keras] is a good wrapper over the Tensorflow framework to allow you quickly build up a neural network and train it. ( if you are new to Deep Learning and Tensorflow, please do stay with Keras. )

* '''Embedding'''
[https://keras.io/layers/embeddings/ Keras Official Documentation]
[https://www.tensorflow.org/tutorials/word2vec Tensorflow : Vector Representations of Words]
[https://en.wikipedia.org/wiki/Word2vec Wiki : Word2vec]

* '''LSTM'''
[http://colah.github.io/posts/2015-08-Understanding-LSTMs/ A Nice Blog about LSTM]
[https://www.tensorflow.org/tutorials/recurrent Tensorflow : Recurrent Neural Networks]
[https://keras.io/layers/recurrent/ Keras Official Documentation]

Deep Text Classifier

2017-10-12T20:45:22Z

Yangzhang: /* Model Training/Prediction */

Deep Text Classifier

2017-10-12T20:44:39Z

Yangzhang:

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

'''To general users:'''

Your input file (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:

1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name will be the same as your input. Please change the name properly to indicate the label information as have been discussed above. And place this pickle file under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

'''To advanced users:'''

1. One important step in data preprocessing is to encode words (strings) into integers. The solution is to build a dictionary mapping words to their corresponding indices. For example, let's say "hello" is the 17th words in our dictionary and thus "hello" is encoded to 17. Our advanced dictionary is ordered by the words' frequency. Higher the frequency smaller the index. That is you should expect to see "the" and "a" these words with very small indices. Please also notice that 0 and 1 these two indices are not assigned to any words intentionally. The advantage here is that you can easily ignore those very common and meaningless words, like "the", by simply saying I only want to consider words with the indices > 20 for example. Notice that it's possible to encounter words that are not in our dictionary and we will alway assign them to index 1. These words are safe to ignore given that our dictionary is big enough.

2. Saving a pickle file is an very efficient way to retrieve the data so that you don't need to do data preprocessing every time when you want to run your classifier.

==Model Training/Prediction==

We write in [https://www.tensorflow.org/ Tensorflow] for all the classifiers. [https://keras.io/ Keras] is a good wrapper over the Tensorflow framework to allow you quickly build up a neural network and train it. ( '''Suggestion:''' if you are new to Deep Learning and Tensorflow, please do stay with Keras. )

* '''Embedding'''
[https://keras.io/layers/embeddings/ Keras Official Documentation]
[https://www.tensorflow.org/tutorials/word2vec Tensorflow : Vector Representations of Words]
[https://en.wikipedia.org/wiki/Word2vec Wiki : Word2vec]

* '''LSTM'''
[http://colah.github.io/posts/2015-08-Understanding-LSTMs/ A Nice Blog about LSTM]
[https://www.tensorflow.org/tutorials/recurrent Tensorflow : Recurrent Neural Networks]
[https://keras.io/layers/recurrent/ Keras Official Documentation]

==General Guidelines for Tuning the Hyper-Parameters==

Deep Text Classifier

2017-10-12T20:35:43Z

Yangzhang: /* Model Training/Prediction */

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

'''To general users:'''

Your input file (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:

1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name will be the same as your input. Please change the name properly to indicate the label information as have been discussed above. And place this pickle file under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

'''To advanced users:'''

1. One important step in data preprocessing is to encode words (strings) into integers. The solution is to build a dictionary mapping words to their corresponding indices. For example, let's say "hello" is the 17th words in our dictionary and thus "hello" is encoded to 17. Our advanced dictionary is ordered by the words' frequency. Higher the frequency smaller the index. That is you should expect to see "the" and "a" these words with very small indices. Please also notice that 0 and 1 these two indices are not assigned to any words intentionally. The advantage here is that you can easily ignore those very common and meaningless words, like "the", by simply saying I only want to consider words with the indices > 20 for example. Notice that it's possible to encounter words that are not in our dictionary and we will alway assign them to index 1. These words are safe to ignore given that our dictionary is big enough.

2. Saving a pickle file is an very efficient way to retrieve the data so that you don't need to do data preprocessing every time when you want to run your classifier.

==Model Training/Prediction==

We write in [https://www.tensorflow.org/ Tensorflow] for all the classifiers. [https://keras.io/ Keras] is a good wrapper over the Tensorflow framework to allow you quickly build up a neural network and train it. ( '''Suggestion:''' if you are new to Deep Learning and Tensorflow, please do stay with Keras. )

==General Guidelines for Tuning the Hyper-Parameters==

Deep Text Classifier

2017-10-12T20:34:18Z

Yangzhang: /* Model Training/Prediction */

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

'''To general users:'''

Your input file (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:

1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name will be the same as your input. Please change the name properly to indicate the label information as have been discussed above. And place this pickle file under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

'''To advanced users:'''

1. One important step in data preprocessing is to encode words (strings) into integers. The solution is to build a dictionary mapping words to their corresponding indices. For example, let's say "hello" is the 17th words in our dictionary and thus "hello" is encoded to 17. Our advanced dictionary is ordered by the words' frequency. Higher the frequency smaller the index. That is you should expect to see "the" and "a" these words with very small indices. Please also notice that 0 and 1 these two indices are not assigned to any words intentionally. The advantage here is that you can easily ignore those very common and meaningless words, like "the", by simply saying I only want to consider words with the indices > 20 for example. Notice that it's possible to encounter words that are not in our dictionary and we will alway assign them to index 1. These words are safe to ignore given that our dictionary is big enough.

2. Saving a pickle file is an very efficient way to retrieve the data so that you don't need to do data preprocessing every time when you want to run your classifier.

==Model Training/Prediction==

We write in [https://www.tensorflow.org/ Tensorflow] for all the classifiers. [https://keras.io/ Keras] is a good wrapper over the Tensorflow framework to allow you quickly build up a neural network and train it. ('''Tip:''' If you are new to deep learning and Tensorflow, please do stay with Keras.)

==General Guidelines for Tuning the Hyper-Parameters==

Deep Text Classifier

2017-10-12T20:32:18Z

Yangzhang: /* Model Training/Prediction */

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

'''To general users:'''

Your input file (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:

1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name will be the same as your input. Please change the name properly to indicate the label information as have been discussed above. And place this pickle file under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

'''To advanced users:'''

1. One important step in data preprocessing is to encode words (strings) into integers. The solution is to build a dictionary mapping words to their corresponding indices. For example, let's say "hello" is the 17th words in our dictionary and thus "hello" is encoded to 17. Our advanced dictionary is ordered by the words' frequency. Higher the frequency smaller the index. That is you should expect to see "the" and "a" these words with very small indices. Please also notice that 0 and 1 these two indices are not assigned to any words intentionally. The advantage here is that you can easily ignore those very common and meaningless words, like "the", by simply saying I only want to consider words with the indices > 20 for example. Notice that it's possible to encounter words that are not in our dictionary and we will alway assign them to index 1. These words are safe to ignore given that our dictionary is big enough.

2. Saving a pickle file is an very efficient way to retrieve the data so that you don't need to do data preprocessing every time when you want to run your classifier.

==Model Training/Prediction==

We write in [https://www.tensorflow.org/ Tensorflow] for all the classifiers. [https://keras.io/ Keras] is a good wrapper over the Tensorflow framework to allow you quickly build up a neural network and train it. If you are new to deep learning and Tensorflow, please do stay with Keras.

==General Guidelines for Tuning the Hyper-Parameters==

Deep Text Classifier

2017-10-12T20:30:58Z

Yangzhang: /* Model Training/Prediction */

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

'''To general users:'''

Your input file (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:

1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name will be the same as your input. Please change the name properly to indicate the label information as have been discussed above. And place this pickle file under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

'''To advanced users:'''

1. One important step in data preprocessing is to encode words (strings) into integers. The solution is to build a dictionary mapping words to their corresponding indices. For example, let's say "hello" is the 17th words in our dictionary and thus "hello" is encoded to 17. Our advanced dictionary is ordered by the words' frequency. Higher the frequency smaller the index. That is you should expect to see "the" and "a" these words with very small indices. Please also notice that 0 and 1 these two indices are not assigned to any words intentionally. The advantage here is that you can easily ignore those very common and meaningless words, like "the", by simply saying I only want to consider words with the indices > 20 for example. Notice that it's possible to encounter words that are not in our dictionary and we will alway assign them to index 1. These words are safe to ignore given that our dictionary is big enough.

2. Saving a pickle file is an very efficient way to retrieve the data so that you don't need to do data preprocessing every time when you want to run your classifier.

==Model Training/Prediction==

We write in [https://www.tensorflow.org/ Tensorflow] for all the deep neural networks. [https://keras.io/ Keras] is a good wrapper over the Tensorflow framework to allow you quickly build up a neural network and train it. If you are new to deep learning and Tensorflow, please do stay with Keras.

==General Guidelines for Tuning the Hyper-Parameters==

Deep Text Classifier

2017-10-12T20:29:57Z

Yangzhang: /* Model Training/Prediction */

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

'''To general users:'''

Your input file (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:

1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name will be the same as your input. Please change the name properly to indicate the label information as have been discussed above. And place this pickle file under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

'''To advanced users:'''

1. One important step in data preprocessing is to encode words (strings) into integers. The solution is to build a dictionary mapping words to their corresponding indices. For example, let's say "hello" is the 17th words in our dictionary and thus "hello" is encoded to 17. Our advanced dictionary is ordered by the words' frequency. Higher the frequency smaller the index. That is you should expect to see "the" and "a" these words with very small indices. Please also notice that 0 and 1 these two indices are not assigned to any words intentionally. The advantage here is that you can easily ignore those very common and meaningless words, like "the", by simply saying I only want to consider words with the indices > 20 for example. Notice that it's possible to encounter words that are not in our dictionary and we will alway assign them to index 1. These words are safe to ignore given that our dictionary is big enough.

2. Saving a pickle file is an very efficient way to retrieve the data so that you don't need to do data preprocessing every time when you want to run your classifier.

==Model Training/Prediction==

We write in [https://www.tensorflow.org/ Tensorflow] for the classifier. [https://keras.io/ Keras] is a good wrapper over the Tensorflow framework to allow you quickly build up a neural network and train it. If you are new to deep learning and Tensorflow, please do stay with Keras.

==General Guidelines for Tuning the Hyper-Parameters==

Deep Text Classifier

2017-10-12T20:23:25Z

Yangzhang: /* Model Training/Prediction */

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

'''To general users:'''

Your input file (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:

1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name will be the same as your input. Please change the name properly to indicate the label information as have been discussed above. And place this pickle file under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

'''To advanced users:'''

1. One important step in data preprocessing is to encode words (strings) into integers. The solution is to build a dictionary mapping words to their corresponding indices. For example, let's say "hello" is the 17th words in our dictionary and thus "hello" is encoded to 17. Our advanced dictionary is ordered by the words' frequency. Higher the frequency smaller the index. That is you should expect to see "the" and "a" these words with very small indices. Please also notice that 0 and 1 these two indices are not assigned to any words intentionally. The advantage here is that you can easily ignore those very common and meaningless words, like "the", by simply saying I only want to consider words with the indices > 20 for example. Notice that it's possible to encounter words that are not in our dictionary and we will alway assign them to index 1. These words are safe to ignore given that our dictionary is big enough.

2. Saving a pickle file is an very efficient way to retrieve the data so that you don't need to do data preprocessing every time when you want to run your classifier.

==Model Training/Prediction==

I use [https://keras.io/ Keras] to build the entire neural network. Keras is an easy and quick way to write [https://www.tensorflow.org/ tensorflow] code.

==General Guidelines for Tuning the Hyper-Parameters==

Deep Text Classifier

2017-10-12T20:10:39Z

Yangzhang: /* Data Preprocessing */

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

'''To general users:'''

Your input file (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:

1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name will be the same as your input. Please change the name properly to indicate the label information as have been discussed above. And place this pickle file under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

'''To advanced users:'''

1. One important step in data preprocessing is to encode words (strings) into integers. The solution is to build a dictionary mapping words to their corresponding indices. For example, let's say "hello" is the 17th words in our dictionary and thus "hello" is encoded to 17. Our advanced dictionary is ordered by the words' frequency. Higher the frequency smaller the index. That is you should expect to see "the" and "a" these words with very small indices. Please also notice that 0 and 1 these two indices are not assigned to any words intentionally. The advantage here is that you can easily ignore those very common and meaningless words, like "the", by simply saying I only want to consider words with the indices > 20 for example. Notice that it's possible to encounter words that are not in our dictionary and we will alway assign them to index 1. These words are safe to ignore given that our dictionary is big enough.

2. Saving a pickle file is an very efficient way to retrieve the data so that you don't need to do data preprocessing every time when you want to run your classifier.

==Model Training/Prediction==

==General Guidelines for Tuning the Hyper-Parameters==

Deep Text Classifier

2017-10-12T20:06:09Z

Yangzhang: /* Data Preprocessing */

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

'''To general users:'''

Your input file (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:

1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name will be the same as your input. Please change the name properly to indicate the label information as have been discussed above. And place this pickle file under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

'''To advanced users:'''

1. One important step in data preprocessing is to encode words (strings) into integers. The solution is to build a dictionary mapping words to their corresponding indices. For example, let's say "hello" is the 17th words in our dictionary and thus "hello" is encoded to 17. Our advanced dictionary is ordered by the words' frequency. Higher the frequency smaller the index. That is you should expect to see "the" and "a" these words with very small indices. Please also notice that 0 and 1 these two indices are not assigned to any words intentionally. The advantage here is that you can easily ignore those very common and meaningless words, like "the", by simply saying I only want to consider words with the indices > 20 for example. Notice that it's possible to encounter words that are not in our dictionary and we will alway assign them to index 1. These words are safe to ignore given that our dictionary is big enough.

2. Saving a pickle file is an very efficient way to retrieve the data so that you don't need to do preprocessing every time you want to run your classifier.

==Model Training/Prediction==

==General Guidelines for Tuning the Hyper-Parameters==

Deep Text Classifier

2017-10-12T19:49:56Z

Yangzhang: /* Data Preprocessing */

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

'''To general users:'''

Your input file (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:

1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name will be the same as your input. Please change the name properly to indicate the label information as have been discussed above. And place this pickle file under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

'''To advanced users:'''

1. One important step in data preprocessing is to encode words (strings) into integers. The solution is to build a dictionary mapping words to their corresponding indices. For example, let's say "hello" is the 17th words in our dictionary and thus "hello" is encoded to 17. Our advanced dictionary is ordered by the words' frequency. Higher the frequency smaller the index, i.e. you should expect to see "the, a, ..." these words in the smallest 10 indices : 2, 3, 4, .... Please also notice that 0 and 1 these two indices are not assigned to any words intentionally. The advantage of doing this is that you can specify easily ignore those common and meaningless words by simply say I want to consider words with the index > 20 for example. And for any word that is not in our dictionary, code it with index 1, so again you can easily ignore it.

2. Saving a pickle file is an very efficient way to retrieve the data so that you don't need to do preprocessing every time you want to run your classifier.

==Model Training/Prediction==

==General Guidelines for Tuning the Hyper-Parameters==

Deep Text Classifier

2017-10-12T19:49:09Z

Yangzhang: /* Data Preprocessing */

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

'''To general users:'''

Your input file (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:

1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name will be the same as your input. Please change the name properly to indicate the label information as have been discussed above. And place this pickle file under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

'''To advanced users:'''

1. One important step in data preprocessing is to encode words (strings) into integers. The solution is to build a dictionary mapping words to their corresponding indices. For example, let's say "hello" is the 17th words in our dictionary and thus "hello" is encoded to 17. Our dictionary is ordered by the words' frequency. Higher the frequency smaller the index, i.e. you should expect to see "the, a, ..." these words in the smallest 10 indices : 2, 3, 4, .... Please also notice that 0 and 1 these two indices are not assigned to any words intentionally. The advantage of doing this is that you can specify easily ignore those common and meaningless words by simply say I want to consider words with the index > 20 for example. And for any word that is not in our dictionary, code it with index 1, so again you can easily ignore it.

2. Saving a pickle file is an very efficient way to retrieve the data so that you don't need to do preprocessing every time you want to run your classifier.

==Model Training/Prediction==

==General Guidelines for Tuning the Hyper-Parameters==

Deep Text Classifier

2017-10-12T19:47:55Z

Yangzhang: /* Data Preprocessing */

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

'''To general users:'''

Your input file (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:

1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name will be the same as your input. Please change the name properly to indicate the label information as have been discussed above. And place this pickle file under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

'''To advanced users:'''

1. One important step in data preprocessing is to encode words (strings) into integers. The solution is to build a dictionary mapping words to their corresponding indices. (say "hello" is the 17th words in the dictionary, so "hello" -> 17) Our dictionary is ordered by the words' frequency. Higher the frequency smaller the index, i.e. you should expect to see "the, a, ..." these words in the smallest 10 indices : 2, 3, 4, .... Please also notice that 0 and 1 these two indices are not assigned to any words intentionally. The advantage of doing this is that you can specify easily ignore those common and meaningless words by simply say I want to consider words with the index > 20 for example. And for any word that is not in our dictionary, code it with index 1, so again you can easily ignore it.

2. Saving a pickle file is an very efficient way to retrieve the data so that you don't need to do preprocessing every time you want to run your classifier.

==Model Training/Prediction==

==General Guidelines for Tuning the Hyper-Parameters==

Deep Text Classifier

2017-10-12T19:46:22Z

Yangzhang: /* Data Preprocessing */

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

'''To general users:'''

Your input file (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:

1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name will be the same as your input. Please change the name properly to indicate the label information as have been discussed above. And place this pickle file under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

'''To advanced users:'''

1. One important step in data preprocessing is to encode words (strings) into integers. The solution is to build a dictionary mapping words to their corresponding indices. Our dictionary is ordered by the words' frequency. Higher the frequency smaller the index, i.e. you should expect to see "the, a, ..." these words in the smallest 10 indices : 2, 3, 4, .... Please also notice that 0 and 1 these two indices are not assigned to any words intentionally. The advantage of doing this is that you can specify easily ignore those common and meaningless words by simply say I want to consider words with the index > 20 for example. And for any word that is not in our dictionary, code it with index 1, so again you can easily ignore it.

2. Saving a pickle file is an very efficient way to retrieve the data so that you don't need to do preprocessing every time you want to run your classifier.

==Model Training/Prediction==

==General Guidelines for Tuning the Hyper-Parameters==

Deep Text Classifier

2017-10-12T19:45:21Z

Yangzhang: /* Data Preprocessing */

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

'''To general users:'''

Your input file (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:

1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name will be the same as your input. Please change the name properly to indicate the label information as have been discussed above. And place this pickle file under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

'''To advanced users:'''

1. One important step in data preprocessing is to encode words (strings) into integers. That is we need to build a dictionary mapping words to their corresponding indices. Our dictionary is ordered by the words' frequency. Higher the frequency smaller the index, i.e. you should expect to see "the, a, ..." these words in the smallest 10 indices : 2, 3, 4, .... Please also notice that 0 and 1 these two indices are not assigned to any words intentionally. The advantage of doing this is that you can specify easily ignore those common and meaningless words by simply say I want to consider words with the index > 20 for example. And for any word that is not in our dictionary, code it with index 1, so again you can easily ignore it.

2. Saving a pickle file is an very efficient way to retrieve the data so that you don't need to do preprocessing every time you want to run your classifier.

==Model Training/Prediction==

==General Guidelines for Tuning the Hyper-Parameters==

Deep Text Classifier

2017-10-12T19:44:51Z

Yangzhang: /* Data Preprocessing */

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

'''To general users:'''

Your input file (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:

1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name will be the same as your input. Please change the name properly to indicate the label information as have been discussed above. And place this pickle file under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

'''To advanced users:'''

1. One important step in data preprocessing is to encode words (strings) to integers. That is we need to build a dictionary mapping words to their corresponding indices. Our dictionary is ordered by the words' frequency. Higher the frequency smaller the index, i.e. you should expect to see "the, a, ..." these words in the smallest 10 indices : 2, 3, 4, .... Please also notice that 0 and 1 these two indices are not assigned to any words intentionally. The advantage of doing this is that you can specify easily ignore those common and meaningless words by simply say I want to consider words with the index > 20 for example. And for any word that is not in our dictionary, code it with index 1, so again you can easily ignore it.

2. Saving a pickle file is an very efficient way to retrieve the data so that you don't need to do preprocessing every time you want to run your classifier.

==Model Training/Prediction==

==General Guidelines for Tuning the Hyper-Parameters==

Deep Text Classifier

2017-10-12T19:43:23Z

Yangzhang: /* Data Preprocessing */

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

'''To general users:'''

your input file (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:

1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name will be the same as your input. Please change the name properly to indicate the label information as have been discussed above. And place this pickle file under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

'''To advanced users:'''

1. one important step in data preprocessing is to convert words (strings) to integers. That is we need to build a dictionary mapping words to their corresponding indices. Our dictionary is ordered by the words' frequency. Higher the frequency smaller the index, i.e. you should expect to see "the, a, ..." these words in the smallest 10 indices : 2, 3, 4, .... Please also notice that 0 and 1 these two indices are not assigned to any words intentionally. The advantage of doing this is that you can specify easily ignore those common and meaningless words by simply say I want to consider words with the index > 20 for example. And for any word that is not in our dictionary, code it with index 1, so again you can easily ignore it.

2. Saving a pickle file is an very efficient way to retrieve the data so that you don't need to do preprocessing every time you want to run your classifier.

==Model Training/Prediction==

==General Guidelines for Tuning the Hyper-Parameters==

Deep Text Classifier

2017-10-12T19:39:55Z

Yangzhang: /* Data Preprocessing */

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

'''To general users:''' your input file (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:

1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name will be the same as your input. Please change the name properly to indicate the label information as have been discussed above. And place this pickle file under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

'''To advanced users:'''

1. one important step in data preprocessing is to convert words (strings) to integers. That is we need to build a dictionary mapping words to their corresponding indices. Our dictionary is ordered by the words' frequency. Higher the frequency smaller the index, i.e. you should expect to see "the, a, ..." these words in the smallest 10 indices : 2, 3, 4, .... Please also notice that 0 and 1 these two indices are not assigned to any words intentionally. The advantage of doing this is that you can specify easily ignore those common and meaningless words by simply say I want to consider words with the index > 20 for example. And for any word that is not in our dictionary, code it with index 1, so again you can easily ignore it.

2. Saving a pickle file is an very efficient way to retrieve the data so that you don't need to do preprocessing every time you want to run your classifier.

==Model Training/Prediction==

==General Guidelines for Tuning the Hyper-Parameters==

Deep Text Classifier

2017-10-12T19:35:02Z

Yangzhang: /* Data Preprocessing */

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

'''To general users:''' your input file (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:

1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name will be the same as your input. Please change the name properly to indicate the label information as have been discussed above. And place this pickle file under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

'''To advanced users:''' one important step in data preprocessing is to convert words (strings) to integers. That is we need to build a dictionary mapping words to their corresponding indices. Our dictionary is ordered by the words' frequency. Higher the frequency smaller the index, i.e. you should expect to see "the, a, ..." these words in the smallest 10 indices : 2, 3, 4, .... Please also notice that 0 and 1 these two indices are not assigned to any words intentionally. The advantage of doing this is that you can specify easily ignore those common and meaningless words by simply say I want to consider words with the index > 20 for example.

==Model Training/Prediction==

==General Guidelines for Tuning the Hyper-Parameters==

Deep Text Classifier

2017-10-12T19:19:10Z

Yangzhang: /* Data Preprocessing */

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

'''To general users:''' your input (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:
1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name the same as your input. Please change the name properly to indicate the target label information as have been discussed above. And make sure this pickle file is under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

==Model Training/Prediction==

==General Guidelines for Tuning the Hyper-Parameters==

Deep Text Classifier

2017-10-12T19:18:45Z

Yangzhang: /* Data Preprocessing */

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

# '''To general users:''' your input (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:
1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

The script will generate a pickle file with an ".pkl" extension and the name the same as your input. Please change the name properly to indicate the target label information as have been discussed above. And make sure this pickle file is under the same directory with your classification code, i.e. "classification_MMM_LLL.py"

==Model Training/Prediction==

==General Guidelines for Tuning the Hyper-Parameters==

Deep Text Classifier

2017-10-12T19:13:39Z

Yangzhang: /* Data Preprocessing */

=Deep Text Classifier=

==Problem Description==

We want to build a classifier for the text input. For example, we may want to classify a company's industry area based on its description. Or we may want to classify a company's IPO status based on its description.

==General Approach==

We will build a deep neural network to uniformly solve this problem. The traditional way of doing this is to hire a task specific expert to manually design some useful features, say to check if the text contains words "Internet" and "High-tech" at the same time, and to classify based on the observed features. Our way, by using the deep neural network, can automatically extract the features and most importantly achieve very high testing accuracy. However, the features that are used by the deep neural network are not human interpretable.

==About the Deep Models==

There are basically two big categories of deep neural networks - the convolutional neural networks [http://cs231n.github.io/convolutional-networks/ CNN] and the recurrent neural networks [https://en.wikipedia.org/wiki/Recurrent_neural_network RNN]. The first one, CNN, is more suitable for dealing with the image based classification tasks. The second one, RNN, is in general for sequential information (i.e. language, video ...) based classification tasks. I have tired both kinds of models and , as expected, the RNN is more robust in facing different text classification tasks

==Major Package Dependences==

* Tensorflow https://www.tensorflow.org/
* Numpy http://www.numpy.org/
* Keras https://keras.io/

==How to Run the Code==

The code contains two parts: Data Preprocessing and Model Training/Prediction.

'''Data Preprocessing (preprocessing.py)''' : this is where you transfer a text based "XXX.txt" input file into a numerical value based pickle file that the later part of the code can understand and use for training and prediction.

* Step 1 : specify the target file name in "main()"

# don't add ".txt" extension
file_name = 'ThicketDefCodingTestProcessed'

* Step 2 : specify the expected columns of your target file in "main()"

# expected number of columns, in case we have "None" in the table
expected_columns = 5

* Step 3 : specify the indices of the text and the label in "prepare_imdb_structure(file_name, expected_columns)"

# the index of the label in the tokens
label_index = 1
# the index of the text in the tokens
content_index = 4

* Step 4 : run the code

python preprocessing.py

* Step 5 : give your pickle file a more reasonable name

Attention: by default, the name of the pickle file is same as the original ".txt" file. But it's highly likely that you will use the same text inputs to predict different things. So it's important to give your pickle file a more reasonable name each time you run the above script. For example, from "longdescriptions.pkl" to "longdescriptions_indu.pkl" to indicate that we are predicting the industry areas and to "longdescriptions_ipo.pkl" to indicate that we are predicting the IPO status. If you don't do this, the later generated pickle files will overwrite the previously generated ones.

'''Model Training/Prediction (classification_MMM_LLL.py)''' : this is where the deep neural network is. The "MMM" represents the model. For example, currently I have "1DConvolution", "2DConvolution" and "LSTM". "LLL" represents the name of the label. Notice that for the same text inputs we can predict for different things using the same model literally. For example, "classification_LSTM_indu.py" is a LSTM model to predict the industray based on the descriptions. And "classification_LSTM_ipo.py" is a LSTM model to predict the IPO status based on the same descriptions. Again you need to name your files properly! Different tasks will have different hyper-parameter configurations though the model and the inputs can be totally the same. This Python file, no matter what the model is, will always load in a pickle file you generated in the previous step and train the neural network. At the end, the well trained neural network will predict on your test examples (the examples you don't see during the training) and print the accuracy.

* Step 1 : specify the name of the pickle file

with open('longdescription_ipo.pkl', 'rb') as file:

* Step 2 : specify the total number of possible labels

model.add(Dense(2, activation='softmax'))

* Step 3 : run the code

python classification_LSTM_ipo.py

==Data Preprocessing==

For data preprocessing, we adopt the same standard as in the [http://ai.stanford.edu/~amaas/data/sentiment/ IMDB] dataset.

# '''To general users:''' your input (usually a single ".txt" file contains many examples each as a row) will be split into a training set (80% by default) and a testing set (20% by default). The labels you want to predict will be the folder names. The content (usually a block of text) of the examples will go into separate ".txt" files. To run the script, you basically need to specify the following:
1. "File Name" : without the ".txt" extension,
2. "Expected Columns" : total number of columns in the input file
3. "Content Index" : the column index of the content
4. "Label Index" : the column index of the label

==Model Training/Prediction==

==General Guidelines for Tuning the Hyper-Parameters==

Deep Text Classifier

2017-10-12T19:12:58Z

Yangzhang: /* Data Preprocessing */

Deep Text Classifier

2017-10-12T18:59:59Z

Yangzhang: /* Data Preprocessing */

Deep Text Classifier

2017-10-12T18:59:12Z

Yangzhang: /* Data Preprocessing */

Deep Text Classifier

2017-10-10T22:31:15Z

Yangzhang: /* About the Deep Models */

Deep Text Classifier

2017-10-10T22:29:56Z

Yangzhang: /* About the Deep Models */

Deep Text Classifier

2017-10-10T22:28:22Z

Yangzhang: /* About the Deep Models */

Deep Text Classifier

2017-10-10T22:27:18Z

Yangzhang: /* About the Deep Models */

Deep Text Classifier

2017-10-10T22:12:04Z

Yangzhang: /* How to Run the Code */

Deep Text Classifier

2017-10-10T21:48:47Z

Yangzhang:

Deep Text Classifier

2017-10-10T21:48:04Z

Yangzhang:

Deep Text Classifier

2017-10-10T21:43:16Z

Yangzhang: /* How to Run the Code */

Deep Text Classifier

2017-10-10T21:42:16Z

Yangzhang: /* How to Run the Code */

Deep Text Classifier

2017-10-10T21:32:21Z

Yangzhang: /* How to Run the Code */

Deep Text Classifier

2017-10-10T21:30:05Z

Yangzhang: /* How to Run the Code */

Deep Text Classifier

2017-10-10T21:26:19Z

Yangzhang: /* How to Run the Code */

Deep Text Classifier

2017-10-10T21:18:58Z

Yangzhang: /* How to Run the Code */

Deep Text Classifier

2017-10-10T21:16:54Z

Yangzhang: /* How to Run the Code */

Deep Text Classifier

2017-10-10T21:13:10Z

Yangzhang: /* How to Run the Code */

Deep Text Classifier

2017-10-10T21:11:18Z

Yangzhang: /* How to Run the Code */

Deep Text Classifier

2017-10-10T21:10:43Z

Yangzhang: /* How to Run the Code */

Deep Text Classifier

2017-10-10T21:09:42Z

Yangzhang: /* How to Run the Code */

Deep Text Classifier

2017-10-10T20:56:28Z

Yangzhang: /* How to Run the Code */

Deep Text Classifier

2017-10-10T20:52:23Z

Yangzhang:

Deep Text Classifier

2017-10-10T20:50:39Z

Yangzhang: /* How to Run the Code */

Deep Text Classifier

2017-10-10T20:49:32Z

Yangzhang: /* How to Run the Code */

Deep Text Classifier

2017-10-10T20:34:02Z

Yangzhang: /* How to Run the Code */