Difference between revisions of "Reproducible Patent Data"
(comment out long regions not used yet) |
|||
| Line 91: | Line 91: | ||
|style="background: green; color: white;" | Yes | |style="background: green; color: white;" | Yes | ||
|} | |} | ||
| + | |||
| + | <!-- | ||
== Attributes == | == Attributes == | ||
| Line 146: | Line 148: | ||
text fields: <code>CITATION_DESCRIPTION</code> | text fields: <code>CITATION_DESCRIPTION</code> | ||
| + | --> | ||
| + | |||
| + | <!-- | ||
== New Schema == | == New Schema == | ||
| Line 151: | Line 156: | ||
[[File:Oyc-patent-schema-1.PNG|640px]] | [[File:Oyc-patent-schema-1.PNG|640px]] | ||
| − | + | --> | |
== Related Projects == | == Related Projects == | ||
* [[Patent|Existing Database Schema]] | * [[Patent|Existing Database Schema]] | ||
Revision as of 00:32, 1 June 2017
| Reproducible Patent Data | |
|---|---|
| Project Information | |
| Project Title | Reproducible Patent Data |
| Owner | Oliver Chang |
| Start Date | May 17 |
| Deadline | |
| Primary Billing | |
| Notes | |
| Has project status | Active |
| Subsumes: | Redesigning Patent Database, Patent Assignment Data Restructure |
| Copyright © 2016 edegan.com. All Rights Reserved. | |
A continuation of Redesigning Patent Database that aims to write faster, more centralized code to deal with data from the United States Patent and Trademark Office (USPTO). By having an end-to-end pipeline we can easily reproduce or update data without worrying about unintentional side effects or missing data. Currently, it succeeds in bulk downloading from the USPTO; streaming file splitting, that is, splitting large concatenated files into their component parts in-memory; and parsing of XML to Java objects, APS to Java Maps, and maintenance fee data to Java objects.
Progress
DownloaderdoneSplitterdoneParserdone- Create tooling for minions
- Setup PostgreSQL JDBC
- Create naive schema based on previous approaches
- Create new data structures
- Database Insert (modify
models/files with some mapping to database fields) - Data Cleanup (reference Marcela and Sonia's work)
- Setup pipeline script to complete all of these steps in series
- Data Source Merger (only USPTO granted, maintfee, assignment not USPTO applications or Harvard Dataverse or Lex Machina currently)
Directory Layout
All of the information for this project is located at E:\McNair\Projects\SimplerPatentData
There are four interesting directories:
data/downloads/is USPTO bulkdata, unmodified straight from the scraperdata/extracts/is a directory of a strict subset of the information stored indata/downloads/. It is the result of running a bulk 7-zip job on that directory to get everything unzipped in a flat data structure. Note that these files have the USPTO modified-by time since that metadata is stored in the zipfiles. To extract files in this nice format, select all of the zipfiles and setup an extraction job like in this screenshotdata/backups/is a 7zip'd backup of the corresponding directory in extractssrc/is the main code repository for the java project
Input Files
All of the text-only Red Book files for granted patents from 1976 to 2016, inclusive. To find a specific year's XML file, find it in
E:\McNair\Projects\SimplerPatentData\data\extracts\granted\
To find assignment data, look in
E:\McNair\Projects\SimplerPatentData\data\extracts\granted\
To find maintenance fee data, look in
E:\McNair\Projects\SimplerPatentData\data\downloads\maintenance
Schema Reconciliation
| Dates Used | Format | Supported by Parser? |
|---|---|---|
| January 1976 to December 2001 | APS | Only syntax |
| January 2002 to December 2004 | XML Version 2.5 | Only syntax |
| January 2005 to December 2005 | XML Version 4.0 ICE | Maybe |
| January 2006 to December 2006 | XML Version 4.1 ICE | Maybe |
| January 2007 to December 2012 | XML Version 4.2 ICE | Maybe |
| January 2013 to September 24, 2013 | XML Version 4.3 ICE | Yes |
| October 8, 2013 to December 2014 | XML Version 4.4 ICE | Yes |
| January 2015 to December 2016 | XML Version 4.5 ICE | Yes |
