Difference between revisions of "Reproducible Patent Data"
Jump to navigation
Jump to search
| Line 15: | Line 15: | ||
# <del>Parser</del> ''done'' | # <del>Parser</del> ''done'' | ||
# Data Source Merger | # Data Source Merger | ||
| − | # Database Insert | + | # Database Insert (modify <code>models/</code> files with some mapping to database fields) |
| − | # Data Cleanup | + | # Data Cleanup (reference [[Patent_Assignment_Data_Restructure|Marcela and Sonia's work]]) |
== Directory Layout == | == Directory Layout == | ||
| Line 24: | Line 24: | ||
There are three interesting directories: | There are three interesting directories: | ||
| − | * <code> | + | * <code>data/downloads/</code> is USPTO bulkdata, unmodified straight from the scraper |
| − | * <code>extracts/</code> is a directory of a strict subset of the information stored in <code> | + | * <code>data/extracts/</code> is a directory of a strict subset of the information stored in <code>data/downloads/</code>. It is the result of running a bulk 7-zip job on that directory to get everything unzipped in a flat data structure. Note that these files have the USPTO modified-by time since that metadata is stored in the zipfiles. To extract files in this nice format, select all of the zipfiles and setup an extraction job like in this [[media:7zip-params.png|screenshot]] |
* <code>src/</code> is the main code repository for the java project | * <code>src/</code> is the main code repository for the java project | ||
| − | + | === Input Files === | |
| − | + | All of the text-only Red Book files for '''granted patents''' from 1976 to 2016, inclusive. To find a specific year's XML file, find it in | |
| − | + | <code>E:\McNair\Projects\SimplerPatentData\data\extracts\granted\</code> | |
| − | |||
| − | |||
| − | + | '''To find assignment data''', look in | |
| − | |||
| − | + | <code>E:\McNair\Projects\SimplerPatentData\data\extracts\granted\</code> | |
| − | + | '''To find maintenance fee data''', look in | |
| − | <code>E:\McNair\Projects\SimplerPatentData\ | + | <code>E:\McNair\Projects\SimplerPatentData\data\downloads\maintenance</code> |
== Schema Reconciliation == | == Schema Reconciliation == | ||
| Line 54: | Line 51: | ||
|January 1976 to December 2001 | |January 1976 to December 2001 | ||
|APS | |APS | ||
| − | | | + | |Yes (syntactic parsing but little semantic knowledge) |
|- | |- | ||
|<del>January 2001 to December 2001</del> | |<del>January 2001 to December 2001</del> | ||
| Line 66: | Line 63: | ||
|January 2005 to December 2005 | |January 2005 to December 2005 | ||
|XML Version 4.0 ICE | |XML Version 4.0 ICE | ||
| − | | | + | |Maybe |
|- | |- | ||
|January 2006 to December 2006 | |January 2006 to December 2006 | ||
|XML Version 4.1 ICE | |XML Version 4.1 ICE | ||
| − | | | + | |Maybe |
|- | |- | ||
|January 2007 to December 2012 | |January 2007 to December 2012 | ||
|XML Version 4.2 ICE | |XML Version 4.2 ICE | ||
| − | | | + | |Maybe |
|- | |- | ||
|January 2013 to September 24, 2013 | |January 2013 to September 24, 2013 | ||
| Line 88: | Line 85: | ||
|style="background: green; color: white;" | Yes | |style="background: green; color: white;" | Yes | ||
|} | |} | ||
| − | |||
| − | |||
| − | |||
| − | |||
=== Attributes === | === Attributes === | ||
Revision as of 21:02, 30 May 2017
| Reproducible Patent Data | |
|---|---|
| Project Information | |
| Project Title | Reproducible Patent Data |
| Owner | Oliver Chang |
| Start Date | May 17 |
| Deadline | |
| Primary Billing | |
| Notes | |
| Has project status | Active |
| Subsumes: | Redesigning Patent Database, Patent Assignment Data Restructure |
| Copyright © 2016 edegan.com. All Rights Reserved. | |
A continuation of Redesigning Patent Database that aims to write faster, more centralized code to deal with the USPTO data. By having an end-to-end pipeline we can easily reproduce or update data without worrying about unintentional side effects or missing data.
Contents
Progress
DownloaderdoneSplitterdoneParserdone- Data Source Merger
- Database Insert (modify
models/files with some mapping to database fields) - Data Cleanup (reference Marcela and Sonia's work)
Directory Layout
All of the information for this project is located at E:\McNair\Projects\SimplerPatentData
There are three interesting directories:
data/downloads/is USPTO bulkdata, unmodified straight from the scraperdata/extracts/is a directory of a strict subset of the information stored indata/downloads/. It is the result of running a bulk 7-zip job on that directory to get everything unzipped in a flat data structure. Note that these files have the USPTO modified-by time since that metadata is stored in the zipfiles. To extract files in this nice format, select all of the zipfiles and setup an extraction job like in this screenshotsrc/is the main code repository for the java project
Input Files
All of the text-only Red Book files for granted patents from 1976 to 2016, inclusive. To find a specific year's XML file, find it in
E:\McNair\Projects\SimplerPatentData\data\extracts\granted\
To find assignment data, look in
E:\McNair\Projects\SimplerPatentData\data\extracts\granted\
To find maintenance fee data, look in
E:\McNair\Projects\SimplerPatentData\data\downloads\maintenance
Schema Reconciliation
| Dates Used | Format | Supported by Parser? |
|---|---|---|
| January 1976 to December 2001 | APS | Yes (syntactic parsing but little semantic knowledge) |
| January 2002 to December 2004 | XML Version 2.5 | No |
| January 2005 to December 2005 | XML Version 4.0 ICE | Maybe |
| January 2006 to December 2006 | XML Version 4.1 ICE | Maybe |
| January 2007 to December 2012 | XML Version 4.2 ICE | Maybe |
| January 2013 to September 24, 2013 | XML Version 4.3 ICE | Yes |
| October 8, 2013 to December 2014 | XML Version 4.4 ICE | Yes |
| January 2015 to December 2016 | XML Version 4.5 ICE | Yes |
Attributes
See E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\models\GrantedPatent.java. In the first couple of lines there is a public enum Fields which describes what attributes are parsed out. In addition, there are 5 lists of citations, scirefs, inventors, assignees, and lawyers associated with this patent--each of which has its own parsed attributes which you can find by looking in the same directory. These attributes (except for the number of claims) are stored as strings (text) rather than numbers.
