Difference between revisions of "Patent Data"

From edegan.com
Jump to navigation Jump to search
imported>Leo
 
(103 intermediate revisions by 11 users not shown)
Line 1: Line 1:
The Patent Data page is for instructions on how to get the USPTO patent data, how to use the database, and for the documentation of our database.
+
{{Project
 +
|Has project output=Data,Tool,Content,Guide
 +
|Has sponsor=McNair Center
 +
|Has title=Patent Data
 +
|Has owner=Marcela Interiano
 +
|Has start date=Spring 2016
 +
||Has keywords=Patent, Database,Data
 +
|Has project status=Subsume
 +
|Due Date=NA
 +
}}
 +
This project is concerned with maintaining and updating patent data, to enable the McNair Center staff to extract meaningful data for academic papers and reports. Currently, there are two primary sources for this data - the US Patent and Trademark Office as well as the Harvard Dataverse. Data from the LexMachina online database has been added to provide information on [[Guide to Patent Litigation (Wiki Page) | patent litigation]].  All the acquired data is stored in normalized tables to be accessed and modified using PostgreSQL. The patent data has been separated into multiple databases based on data source or subject matter. Each database consists of several tables for which the known [[Patent Data Issues | issues]] have been recorded.  
  
== ER diagram ==
+
==Data Sources==
See [http://www.edegan.com/wiki/images/0/06/Patent_Data.png ER Diagram]
+
Data has been extracted from the '''[[USPTO Bulk Data]]''', the '''[[Harvard Dataverse]]''', and '''[[Lex Machina]]''', an online patent litigation database. {{#section:USPTO_Bulk_Data_Processing|bulk}}{{#section:Harvard_Dataverse|dataverse}} The sources used were intended to follow the overall [[Data Model]] established by the McNair Center.  
  
== Downloading the files ==
+
==Database Specifics==
The files (in xml format) for granted patent data can be obtained at [https://www.google.com/googlebooks/uspto-patents-grants-text.html granted patent]
+
The [[Patent|Patent Database]] contains the merged datasets from the USPTO bulk data and Harvard Dataverse using PostgreSQL. Specifics on how the datasets were merged are given in [[Patent Data Processing - SQL Steps]].
 +
Patent Database focuses on patents, patent litigation, patent maintenance, patent assignment, and other details on patent owners. The [[USPTOAssigneesData|USPTO Assignees Database (version 2)]] focuses on patent assignments, a transaction between one or more patent owners with one or more parties where ownership or interest in one or more patents is assigned or shared. The database consists of historical assignment data provided by the USPTO in XML files. Specifics on how the database are given on the [[USPTO Assignees Data Processing]] Page.
  
The files for patent application data can be obtained at [https://www.google.com/googlebooks/uspto-patents-applications-text.html patent applications]
+
==Academic Projects==
  
The files for maintenance fees data can be obtained at [https://www.google.com/googlebooks/uspto-patents-maintenance-fees.html maintenance]
+
===[[Little Guy Academic Paper|'Little Guy' Academic Paper]]===
 +
The first application of the refined database will be the [[Little Guy Academic Paper]]. {{#section:Little_Guy_Academic_Paper|Little Guy}}
 +
===Patent Trolls===
  
Scripts are available to perform a bulk download of all the above files:
+
Academic Paper: The patent database will also be used to explore the existence of patent trolls and characteristic litigation activity. An academic paper may be developed defining patent trolls and other entities often confused as patent trolls. The data from Lex Machina will be used to track troll behavior and associated outcomes as well as the impact of other patent intermediary and assertion bodies.
  
[http://www.edegan.com/wiki/index.php/Image:Applications_download_2001-2004.sh Script to download patent application data from 2001-2004]
+
Issue Brief: Based on an analysis of the litigation data from Lex Machina, an issue brief, tentatively titled [[The Truth Behind Patent Trolls Issue Brief| The Truth Behind Patent Trolls]], on patent troll activity may be written to report on how best to curve abuses through [[Innovation Policy| innovation policy]] and reform.
  
[http://www.edegan.com/wiki/index.php/Image:Applications_download_2005-2015.sh Script to download patent application data from 2005-2015]
 
  
[http://www.edegan.com/wiki/index.php/Image:Grant_download_1976-2000.sh Script to download patent grant data from 1976-2000]
+
<!-- flush -->
 
 
[http://www.edegan.com/wiki/index.php/Image:Grant_download_2001-2004.sh Script to download patent grant data from 2001-2004]
 
 
 
[http://www.edegan.com/wiki/index.php/Image:Grant_download_2005-2015.sh Script to download patent grant data from 2005-2015]
 
 
 
To use the scripts, save the scripts as shell scripts, then either
 
 
 
~ leodu$ sh Applications_download_2001-2004.sh
 
 
 
or first change the script to an executable and execute it
 
 
 
~ leodu$ chmod a+x Applications_download_2001-2004.sh
 
~ leodu$ ./Applications_download_2001-2004.sh
 
 
 
Notice there will be several hundreds of .zip files of size ~100mb getting downloaded so the process might take a while.
 
When all the files are downloaded, unzip all of them using
 
 
 
~ leodu$ unzip *.zip
 
 
 
==XML Schema Notes==
 
 
 
Tags we are using:
 
*CPC Classification: https://en.wikipedia.org/wiki/Cooperative_Patent_Classification
 
 
 
Tags we aren't using:
 
*Kind codes: http://www.uspto.gov/learning-and-resources/support-centers/electronic-business-center/kind-codes-included-uspto-patent
 
*Series codes: http://www.uspto.gov/web/offices/ac/ido/oeip/taf/filingyr.htm
 
 
 
== Parsing and Processing the XML files ==
 
 
 
The ParserSpliter.pl script will first split a large Patent Data XML file into smaller XML files, one for each patent data. And it will then parse and process each Patent Data XML file.
 
 
 
Some of the files are somehow mal-formatted, and will be moved to a ./failed_files directory If you add a character anywhere in these files, they somehow become fine to be processed by the script.
 
 
In order to use this script, you will need to have XML::Simple and Try::Tiny installed.
 
 
 
Open up CPAN shell:
 
leodu$ perl -e shell -MCPAN
 
 
 
Install:
 
cpan[0]> install XML::Simple
 
cpan[1]> install Try::Tiny
 
cpan[2]> install Switch
 
 
 
==Other Resources==
 
[http://www.uspto.gov/learning-and-resources/xml-resources Documentations for the xml files]
 
 
 
[http://www.uspto.gov/learning-and-resources/xml-resources/xml-resources-retrospective See Also]
 
 
 
[https://www.w3.org/2000/04/schema_hack/ tool to convert dtd to xsd]
 

Latest revision as of 13:44, 21 September 2020


Project
Patent Data
Project logo 02.png
Project Information
Has title Patent Data
Has owner Marcela Interiano
Has start date Spring 2016
Has deadline date
Has keywords Patent, Database, Data
Has project status Subsume
Subsumed by: Patent Assignment Data Restructure
Has sponsor McNair Center
Has project output Data, Tool, Content, Guide
Copyright © 2019 edegan.com. All Rights Reserved.

This project is concerned with maintaining and updating patent data, to enable the McNair Center staff to extract meaningful data for academic papers and reports. Currently, there are two primary sources for this data - the US Patent and Trademark Office as well as the Harvard Dataverse. Data from the LexMachina online database has been added to provide information on patent litigation. All the acquired data is stored in normalized tables to be accessed and modified using PostgreSQL. The patent data has been separated into multiple databases based on data source or subject matter. Each database consists of several tables for which the known issues have been recorded.

Data Sources

Data has been extracted from the USPTO Bulk Data, the Harvard Dataverse, and Lex Machina, an online patent litigation database. The USPTO provides bulk data recording patent transactions, applications, properties, reassignments, and history through XML files to the general public. These files have been downloaded and the data has been compiled in tables using PostgreSQL. The objective of processing the bulk data is to enhance the McNair Center's historical datasets (patent_2015 and patentdata) and track the entirety of US patent activity, specifically concerning utility patents.

The Harvard Dataverse provides clean versions of the U.S. utility patent datasets spanning 1975-2010. The data is post author disambiguation. The sources used were intended to follow the overall Data Model established by the McNair Center.

Database Specifics

The Patent Database contains the merged datasets from the USPTO bulk data and Harvard Dataverse using PostgreSQL. Specifics on how the datasets were merged are given in Patent Data Processing - SQL Steps. Patent Database focuses on patents, patent litigation, patent maintenance, patent assignment, and other details on patent owners. The USPTO Assignees Database (version 2) focuses on patent assignments, a transaction between one or more patent owners with one or more parties where ownership or interest in one or more patents is assigned or shared. The database consists of historical assignment data provided by the USPTO in XML files. Specifics on how the database are given on the USPTO Assignees Data Processing Page.

Academic Projects

'Little Guy' Academic Paper

The first application of the refined database will be the Little Guy Academic Paper.

Patent Trolls

Academic Paper: The patent database will also be used to explore the existence of patent trolls and characteristic litigation activity. An academic paper may be developed defining patent trolls and other entities often confused as patent trolls. The data from Lex Machina will be used to track troll behavior and associated outcomes as well as the impact of other patent intermediary and assertion bodies.

Issue Brief: Based on an analysis of the litigation data from Lex Machina, an issue brief, tentatively titled The Truth Behind Patent Trolls, on patent troll activity may be written to report on how best to curve abuses through innovation policy and reform.