{{McNair Projects
|Has title=ArcMap / ArcGIS Documentation
|Has owner=Jeemin Sim,
}}

= ArcMap =

== Add Text file (.txt) as data on ArcMap ==
1) Open ArcMap
2) Locate a a button with a black cross with yellow diamond underneath (below'Selection')
3) Click the tiny triangle drop-down button to its immediate right
4) Select 'Add Data'
5) Find location where local text file resides. For example: (Home - Documents\ArcGIS)
6) Select a text file of interest and click Add
- The text file should have at least two columns: X and Y coordinates
7) Locate the text file in the Table Of Contents tab in the left
8) Right click the text file
9) Select 'Display XY Data...'
10) In the pop-up window 'Display XY Data', choose XField and YField - the default should work
11) Check that in the same pop-up window the box labelled 'Coordinate System of Input Coordinates'
contains GCS_WGS_1984
- If not: click the 'Edit...' Button. --> Click the Plus sign for 'Geographic Coordinate Systems'
--> Click the Plus sign for 'World' --> Select 'WGS 1984' --> Press OK
12) Click 'OK'
13) Most likely a window titled 'Table Does Not Have Object-ID Field' will pop up,
but do not fret
14) The points will show on the big screen in the middle

==Manipulate Data With Queries==
1) Right click on data layer
2) Click on Properties
3) Click on Definition Query
4) There are two options, definition query or query builder
5) Type In Query and Hit Apply and Then OK

== Add Base layer or map on ArcMap ==

1) Locate a a button with a black cross with yellow diamond underneath (below'Selection')
2) Click the tiny triangle drop-down button to its immediate right
3) Select 'Add Basemap...'
4) Choose a base map of your liking. 'Light Gray Canvas' is frequently used.
5) Click 'Add'

ArcMap / ArcGIS Documentation

2017-11-29T20:18:44Z

Hbrown512:

Harrison Brown (Work Log)

2017-11-09T19:28:07Z

Hbrown512:

===Fall 2017===
<onlyinclude>
[[Harrison Brown]] [[Work Logs]] [[Harrison Brown (Work Log)|(log page)]]

2017-11-01:
*Looked at Oliver's code. Got git repository set up for the project on Bonobo. Started messing around with reading the XML documents in Java.

2017-10-30:
*Worked on seeing what data can be gathered from the CSV and XML files. Started project page for project.

2017-10-26:
*Met with Ed to talk about the direction of the project. Starting to work on extracting information from the XML files. Working on adding documentation to wiki and work log. Looking into work from other projects that may use XML.

2017-10-25:
*Found information about a USITC database that we could use. Added this information to the wiki, and updated information on USITC wiki page.

2017-10-19:
*Continued to look into NLTK. Talked with Ed about looking into alternative approaches to gathering this data.

2017-10-18:
*Trying to figure out the best way to extract respondents from the documents. Right now using exclusively NLTK will not get us any more accuracy that using regular expressions. Currently neither will allow us to match every entity correctly so trying to figure out alternate approaches.

2017-10-16:
*NLTK
** NLTK Information
*** Need to convert text to ascii. Had issues with my PDF texts and had to convert
*** Can use sent_tokenize() function to split document into sentences, easier that regular expressions
*** Use pos_tag() to tag the sentences. This can be used to extract proper nouns
**** Trying to figure out how to use this to grab location data from these documents
*** Worked with Peter to try to extract geographic information from the documents. We looked into tools Geograpy and GeoText. Geograpy does not have the functionality that we would like. GeoText looks to be better but we have issues with dependencies. Will try to resolve these next time.

2017-10-11:
*Started to use NLTK library for gathering information to extract respondents. See code in Projects/USITC/ProcessingTexts

2017-10-05:
*Made photos for the requested maps in ArcGIS with Peter and Jeemin.
To access:
Go to E:\McNair\Projects\Agglomeration\HarrisonPeterWorkArcGIS
The photos can be found in there
To generate the photos open ArcMap with the beginMapArc file
To generate a PNG Click, File, Export to export the photos
To adjust the data right click on the table name in the layers lab, and hit properties, then query builder

2017-10-04:
*Worked with Peter on connecting ArcGIS to the database and displaying different points in ArcGIS

2017-10-02:
*Started work with ArcGIS. Got the data with startups from Houston into the ArcGIS application. For notes see McNair/Porject/Agglomeration

2017-09-28:
*Helped Christy with set up on Postgres server. Looked through text documents to see what information I could gather. Looked at Stanford NLTK library for extracting the respondents from the documents.

2017-09-28:
*Got the PDFS parsed to text. Some of the formatting is off will need to determine if data can still be gathered.

2017-09-25:
*Got 3000 PDFS downloaded. Script works. Completed a task to get emails for people who had written papers about economics and entrepreneurship. Started work on pasring the PDFS to text

2017-09-20:
*Shell program did not work. Create Python program that can catch all exceptions (url does not exist, lost connection, and improperly formatted url) Hopefully it will complete with no problems. This program is found in the database server under the USITC folder.
*Got connected to the database server and mounted the drive onto my computer. Got the list of all the PDFS on the website and started a shell script on the database server to download all of the PDFs. I will leave it running overnight hopefully it completes by tomorrow.

2017-09-17:
*Added features to python program to pull the dates in numerical form. Worked on pulling the PDFs from the website. Currently working on pulling them in Python. The program can run and pull PDFs on my local machine but it doesn't work on the Remote Desktop. I will work on this next time.

2017-09-14:
*Have a python program that can scrape the entire webpage and navigate through all of the pages that contain section 337 documents. You can see these files and more information on the USITC project page. It can pull all of the information that is in the HTML that can be gathered for each case. The PDFs now need to be scraped; will start work on that next time. Generated a csv file with more than 4000 entries from the webpage. There is a small edge case I need to fix where the entry does not contain the Investigation No.

2017-09-13:
*Worked on parsing the USITC website Section 337 Notices. Nearly have all of the data I can scrape. Scraper works, but there are a few edges
*cases where information in the tables are part of a Notice but do not have Investigation Numbers. Will finish this hopefully next time. Also added my USITC project to the projects page I did not have it linked

2017-09-11:
Met with Dr. Egan and got assigned project. Set Up Project Page USITC, Started Coding in Python for the Web Crawler. Look in McNair/Projects/UISTC for project notes and code.

2017-09-07:
Set Up Work Log Pages, Slack, Microsoft Remote Desktop
</onlyinclude>
[[Category:Work Log]]

Harrison Brown (Work Log)

2017-11-01T20:42:37Z

Hbrown512:

[[Harrison Brown]] [[Work Logs]] [[Harrison Brown (Work Log)|(log page)]]

===Fall 2017 Work===
09/07/2017 2:20pm-3:50pm - Set Up Work Log Pages, Slack, Microsoft Remote Desktop

09/11/2017 1pm-5pm - Met with Dr. Egan and got assigned project. Set Up Project Page USITC, Started Coding in Python for the Web Crawler. Look in McNair/Projects/UISTC for project notes and code.

09/13/2017 1pm-3:50pm - Worked on parsing the USITC website Section 337 Notices. Nearly have all of the data I can scrape. Scraper works, but there are a few edges
cases where information in the tables are part of a Notice but do not have Investigation Numbers. Will finish this hopefully next time. Also added my USITC project to the projects page I did not have it linked

09/14/2017 1pm-3:50pm - Have a python program that can scrape the entire webpage and navigate through all of the pages that contain section 337 documents. You can see these files and more information on the USITC project page. It can pull all of the information that is in the HTML that can be gathered for each case. The PDFs now need to be scraped; will start work on that next time. Generated a csv file with more than 4000 entries from the webpage. There is a small edge case I need to fix where the entry does not contain the Investigation No.

09/18/2017 1pm-5:00pm - Added features to python program to pull the dates in numerical form. Worked on pulling the PDFs from the website. Currently working on pulling them in Python. The program can run and pull PDFs on my local machine but it doesn't work on the Remote Desktop. I will work on this next time.

09/20/2017 1pm- 3:50pm - Got connected to the database server and mounted the drive onto my computer. Got the list of all the PDFS on the website and started a shell script on the database server to download all of the PDFs. I will leave it running overnight hopefully it completes by tomorrow.

09/20/2017 1pm- 2:30pm - Shell program did not work. Create Python program that can catch all exceptions (url does not exist, lost connection, and improperly formatted url) Hopefully it will complete with no problems. This program is found in the database server under the USITC folder.

09/25/2017 1pm- 5:00pm - Got 3000 PDFS downloaded. Script works. Completed a task to get emails for people who had written papers about economics and entrepreneurship. Started work on pasring the PDFS to text

09/27/2017 1pm- 3:50pm - Got the PDFS parsed to text. Some of the formatting is off will need to determine if data can still be gathered.

09/28/2017 1pm- 3:50pm - Helped Christy with set up on Postgres server. Looked through text documents to see what information I could gather. Looked at Stanford NLTK library for extracting the respondents from the documents.

10/02/2017 1pm- 5:00pm - Started work with ArcGIS. Got the data with startups from Houston into the ArcGIS application. For notes see McNair/Porject/Agglomeration

10/04/2017 1pm- 3:50pm - Worked with Peter on connecting ArcGIS to the database and displaying different points in ArcGIS

10/05/2017 1pm- 3:50pm - Made photos for the requested maps in ArcGIS with Peter and Jeemin.
To access:
Go to E:\McNair\Projects\Agglomeration\HarrisonPeterWorkArcGIS
The photos can be found in there
To generate the photos open ArcMap with the beginMapArc file
To generate a PNG Click, File, Export to export the photos
To adjust the data right click on the table name in the layers lab, and hit properties, then query builder
10/11/2017 1pm- 3:50pm - Started to use NLTK library for gathering information to extract respondents. See code in Projects/USITC/ProcessingTexts

10/16/2017 1pm - 5:00pm - NLTK
* NLTK Information
** Need to convert text to ascii. Had issues with my PDF texts and had to convert
** Can use sent_tokenize() function to split document into sentences, easier that regular expressions
** Use pos_tag() to tag the sentences. This can be used to extract proper nouns
*** Trying to figure out how to use this to grab location data from these documents
** Worked with Peter to try to extract geographic information from the documents. We looked into tools Geograpy and GeoText. Geograpy does not have the functionality that we would like. GeoText looks to be better but we have issues with dependencies. Will try to resolve these next time.

10/18/2017 1pm - 3:50pm
Trying to figure out the best way to extract respondents from the documents. Right now using exclusively NLTK will not get us any more accuracy that using regular expressions. Currently neither will allow us to match every entity correctly so trying to figure out alternate approaches.

10/19/2017 1pm - 3:50pm
Continued to look into NLTK. Talked with Ed about looking into alternative approaches to gathering this data.

10/25/2017 1pm - 3:50pm
Found information about a USITC database that we could use. Added this information to the wiki, and updated information on USITC wiki page.

10/26/2017 1pm - 3:50pm
Met with Ed to talk about the direction of the project. Starting to work on extracting information from the XML files. Working on adding documentation to wiki and work log. Looking into work from other projects that may use XML.

10/30/2017 1pm - 5:00pm
Worked on seeing what data can be gathered from the CSV and XML files. Started project page for project.

11/31/2017 1pm - 3:50pm
Looked at Oliver's code. Got git repository set up for the project on Bonobo. Started messing around with reading the XML documents in Java.

[[Category:Work Log]]

XML Parsing

2017-11-01T19:43:00Z

Hbrown512:

{{McNair Projects
|Has title=Python Libraries
|Has owner=Harrison Brown
|Has deadline=Never
|Has keywords=XML
}}
This page is dedicated to XML parsing projects that have been done

==Projects Using XML==
This will hold the projects that use XML
Had Issues Checking out with HTTPS

* [[Reproducible_Patent_Data|Reproducible Patent Data]], uses Java

==Potentially Relevant Projects==
HTML Parsing/Potentially relevant projects

* [[Accelerator_Seed_List_(Data)|Accelerator Seed List (Data)]], look at F6S

==Information==
This will hold information on how to parse XML

XML Parsing

2017-11-01T19:02:13Z

Hbrown512:

{{McNair Projects
|Has title=Python Libraries
|Has owner=Harrison Brown
|Has deadline=Never
|Has keywords=XML
}}
This page is dedicated to XML parsing projects that have been done

==Projects Using XML==
This will hold the projects that use XML

* [[Reproducible_Patent_Data|Reproducible Patent Data]], uses Java

==Potentially Relevant Projects==
HTML Parsing/Potentially relevant projects

* [[Accelerator_Seed_List_(Data)|Accelerator Seed List (Data)]], look at F6S

==Information==
This will hold information on how to parse XML

2017-10-30T20:19:11Z

Hbrown512:

2017-10-16T18:04:55Z

Hbrown512:

[[Harrison Brown]] [[Work Logs]] [[Harrison Brown (Work Log)|(log page)]]

===Fall 2017 Work===
09/07/2017 2:20pm-3:50pm - Set Up Work Log Pages, Slack, Microsoft Remote Desktop

09/11/2017 1pm-5pm - Met with Dr. Egan and got assigned project. Set Up Project Page USITC, Started Coding in Python for the Web Crawler. Look in McNair/Projects/UISTC for project notes and code.

09/13/2017 1pm-3:50pm - Worked on parsing the USITC website Section 337 Notices. Nearly have all of the data I can scrape. Scraper works, but there are a few edges
cases where information in the tables are part of a Notice but do not have Investigation Numbers. Will finish this hopefully next time. Also added my USITC project to the projects page I did not have it linked

09/14/2017 1pm-3:50pm - Have a python program that can scrape the entire webpage and navigate through all of the pages that contain section 337 documents. You can see these files and more information on the USITC project page. It can pull all of the information that is in the HTML that can be gathered for each case. The PDFs now need to be scraped; will start work on that next time. Generated a csv file with more than 4000 entries from the webpage. There is a small edge case I need to fix where the entry does not contain the Investigation No.

09/18/2017 1pm-5:00pm - Added features to python program to pull the dates in numerical form. Worked on pulling the PDFs from the website. Currently working on pulling them in Python. The program can run and pull PDFs on my local machine but it doesn't work on the Remote Desktop. I will work on this next time.

09/20/2017 1pm- 3:50pm - Got connected to the database server and mounted the drive onto my computer. Got the list of all the PDFS on the website and started a shell script on the database server to download all of the PDFs. I will leave it running overnight hopefully it completes by tomorrow.

09/20/2017 1pm- 2:30pm - Shell program did not work. Create Python program that can catch all exceptions (url does not exist, lost connection, and improperly formatted url) Hopefully it will complete with no problems. This program is found in the database server under the USITC folder.

09/25/2017 1pm- 5:00pm - Got 3000 PDFS downloaded. Script works. Completed a task to get emails for people who had written papers about economics and entrepreneurship. Started work on pasring the PDFS to text

09/27/2017 1pm- 3:50pm - Got the PDFS parsed to text. Some of the formatting is off will need to determine if data can still be gathered.

09/28/2017 1pm- 3:50pm - Helped Christy with set up on Postgres server. Looked through text documents to see what information I could gather. Looked at Stanford NLTK library for extracting the respondents from the documents.

10/02/2017 1pm- 5:00pm - Started work with ArcGIS. Got the data with startups from Houston into the ArcGIS application. For notes see McNair/Porject/Agglomeration

10/04/2017 1pm- 3:50pm - Worked with Peter on connecting ArcGIS to the database and displaying different points in ArcGIS

10/05/2017 1pm- 3:50pm - Made photos for the requested maps in ArcGIS with Peter and Jeemin.
To access:
Go to E:\McNair\Projects\Agglomeration\HarrisonPeterWorkArcGIS
The photos can be found in there
To generate the photos open ArcMap with the beginMapArc file
To generate a PNG Click, File, Export to export the photos
To adjust the data right click on the table name in the layers lab, and hit properties, then query builder
10/11/2017 1pm- 3:50pm - Started to use NLTK library for gathering information to extract respondents. See code in Projects/USITC/ProcessingTexts

10/16/2017 1pm - 5:00pm - NLTK
* NLTK Information
** Can use sent_tokenize() function to split document into sentences, easier that regular expressions
** Use pos_tag() to tag the sentences. This can be used to extract proper nouns
*** Trying to figure out how to use this to grab location data from these documents

[[Category:Work Log]]

Harrison Brown (Work Log)

2017-10-16T18:03:23Z

Hbrown512:

Harrison Brown (Work Log)

2017-10-11T20:30:30Z

Hbrown512:

Harrison Brown (Work Log)

2017-10-05T20:35:07Z

Hbrown512: