7,613
edits
Changes
Jump to navigation
Jump to search
is a csv of the data that I have been able to scrape from the HTML
of https://www.usitc.gov/secretary/fed_reg_notices/337.htm
Downloaded most of There is a query builder for Section 337 Notices Here https://pubapps2.usitc.gov/337external/advancedTo use you must select fields from the GUI at the PDFs. There were errors download some bottom of the files.page
no edit summary
{{Project|Has project output=Data,Tool|Has sponsor=McNair ProjectsCenter
|Has title=USITC Data
|Has owner=Harrison Brown
My files are in 2 different places:
Did development work here
E:\McNair\Projects\USITC
I had to use to the Postgres SQL server to download the PDFS
The Postgres SQL Server:
128.42.44.182/bulk/USITC
==WorkAdditional Information==There is more information that the USITC provides besides 337 notices.
Here is information and a database on Section 701/731
https://www.usitc.gov/trade_remedy/trade_research_tools
https://pubapps2.usitc.gov/sunset/
=New Work=
==USITC 337 Cases Tab Delimited Text==
USITC patent information was gathered from the investigations.json file downloaded from the USITC website (https://pubapps2.usitc.gov/337external/, Click on Cases Instituted After 2008).
This contains information on 337 cases and their respondents/complainants and the patents that were part of the case.
The code and results for this program are here:
Projects/USITC/JSON_scraping_python
The program grabs the information, places it into lists of lists in Python, and then writes to the file names listed below. The files do not have headers and null values are set to be empty strings.
To create the tab delimited text files, run code.py in the JSON_scraping_python directory. This has all of the file names hard coded. It will create the following files
investigation_info.txt
Schema for this file is id, title, investigation number, investigation type, docket number, date of publication notice
complainant_info.txt
Schema for this file is investigation id, investigation number, Complaintant name, complainant outside party ID, comp_city, comp country
respondent_info.txt
Schema for this file is investigation id, investigation number, Respondent Outside Party ID , Respondent Name, Respondent City, Respondent Country
patent_info.txt
Schema for this file is Investigation Number, Patent ID, Patent Number, Active Date, Inactive Date,
==XML Information==
UPDATE: used JSON file of data to convert to tab-delimited text.
There is an XML file that contains information on investigations. To get it go to the link below and 'xml' link that is under the tab that under 'Cases instituted after October 2008'.
https://pubapps2.usitc.gov/337external/
The information that is found this file can be grabbed with an XML parser. For each investigation, we can find out
* Investigation Number ex - (<entry key="investigationNo">966</entry>)
* Date of publication Notice - (<entry key="dateOfPublicationFrNotice">2015-09-24T04:00:00.000Z</entry>)
* Title ex - <entry key="title">Silicon-on-Insulator Wafers</entry>
* There is an entry for patent numbers, ex - <entry key="patentNumbers">
* Investigation Type ex - <entry key="investigationType">Violation</entry>
* Respondents can be found under <entry key="respondent">
* Complainant can be found under <entry key="complainant">
Additional information can also be gathered from the XML document
To find information on cases prior to 2008, go to the link above and click on 'Looking for cases instituted prior to October 2008?', and it will
download a csv file.
* The investigation number, Title, Unfair Act Alleged, Patent Numbers,Complainants, Respondents, can be grabbed easily from the CSV
* Target Date, Beginning and Ending Dates contain notes (some cases are extended and dates are changed)and so we may need to do some text processing to grab this information
==Old Work==
This is the work that was done before I knew about the XML files provided by the USITC (see Alternative Solutions section below)
Here is a csv of the data that I have been able to scrape from the HTML of https://www.usitc.gov/secretary/fed_reg_notices/337.htm
The results.csv file found here,
E:\McNair\Projects\USITC
For every notice paper, there is a line in the CSV file that
There are PDFs where the parsing does not work completely and the text is scrambled.
I have started using NLTK to gather information about the filings. See code in Projects/USITC/ProcessingTextsI am working on extracting the respondents from these documents. ==StatusAlternative Solutions==Currently Here is where you can get data on the web scraper is able USITC 337 notices instead of extracting this information from PDFs 337Info - Unfair Import Investigations Information System https://pubapps2.usitc.gov/337external/ This link has links to gather all Complaints. Most of the data that I can gather from be gathered is there and it is in a nice format that could be easily web scraped if we needed to. They have a query builder (link below). There is also a link that download all of the HTMLraw data in JSON or XML.There Here are a few cases where the Investigation links to various statistics we could use:Section 337 Statistics: Settlement Rate Data https://www.usitc.gov/intellectual_property/337_statistics_settlement_rate_data.htm Section 337 Statistics https://www.usitc.gov/press_room/337_stats.htm Section 337 Statistics: Number is not listed of New, Completed, and I need Active Investigations by Fiscal Year (Updated Quarterly) https://www.usitc.gov/intellectual_property/337_statistics_number_new_completed_and_active.htm Section 337 StatisticsThis contains links to test for those andvarious other pages with statistics https://www.usitc.gov/intellectual_property/337_statistics.htm fix that Here is a dictionary of terms used in the codethese Documents https://www.usitc.gov/documents/337Info_ext_DataDictionary.pdf There are FAQs listed here https://www.usitc.gov/documents/337Info_FAQ.pdf
