Difference between revisions of "Reproducible Patent Data"

From edegan.com
Jump to navigation Jump to search
Line 87: Line 87:
 
|}
 
|}
  
=== Attributes ===
+
== Attributes ==
 +
 
 +
'''Note''': these values are likely to change without warning. For the latest version of these see the actual files at <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\models</code>.
 +
 
 +
=== Assignee ===
 +
 
 +
text fields: <code>NAME, ADDR1, ADDR2, CITY, STATE, COUNTRY_NAME, POSTCODE</code>
 +
 
 +
=== Assignment ===
 +
 
 +
text fields: <code>REEL_NUMBER, FRAME_NUMBER, LAST_UPDATE_DATE, RECORDED_DATE, CONVEYANCE_TEXT</code>
 +
 
 +
lists: <code>correspondents, assignors, assignees</code>
 +
 
 +
=== Assignment Summary ===
 +
 
 +
text fields: <code>LAST_NAME, FIRST_NAME, ORG_NAME, CITY, COUNTRY, STATE, ADDRESS, POSTCODE</code>
 +
 
 +
=== Assignor ===
 +
=== Citation ===
 +
=== Correspondent ===
 +
=== GrantedPatent ===
 +
=== Inventor ===
 +
=== Lawyer ===
 +
=== MaintenanceFeeEvent ===
 +
=== Sciref ===
  
See <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\models\GrantedPatent.java</code>. In the first couple of lines there is a <code>public enum Fields</code> which describes what attributes are parsed out. In addition, there are 5 lists of citations, scirefs, inventors, assignees, and lawyers associated with this patent--each of which has its own parsed attributes which you can find by looking in the same directory. These attributes (except for the number of claims) are stored as strings (text) rather than numbers.
 
  
 
== New Schema ==
 
== New Schema ==

Revision as of 21:23, 30 May 2017


McNair Project
Reproducible Patent Data
Project logo 02.png
Project Information
Project Title Reproducible Patent Data
Owner Oliver Chang
Start Date May 17
Deadline
Primary Billing
Notes
Has project status Active
Subsumes: Redesigning Patent Database, Patent Assignment Data Restructure
Copyright © 2016 edegan.com. All Rights Reserved.


A continuation of Redesigning Patent Database that aims to write faster, more centralized code to deal with the USPTO data. By having an end-to-end pipeline we can easily reproduce or update data without worrying about unintentional side effects or missing data. Currently, it succeeds in bulk downloading from the USPTO; streaming file splitting, that is, splitting large concatenated files into their component parts in-memory; and parsing of XML to Java objects, APS to Java Maps, and maintenance fee data to Java objects.

Progress

  1. Downloader done
  2. Splitter done
  3. Parser done
  4. Data Source Merger (only USPTO not Harvard Dataverse or Lex Machina)
  5. Database Insert (modify models/ files with some mapping to database fields)
  6. Data Cleanup (reference Marcela and Sonia's work)

Directory Layout

All of the information for this project is located at E:\McNair\Projects\SimplerPatentData

There are three interesting directories:

  • data/downloads/ is USPTO bulkdata, unmodified straight from the scraper
  • data/extracts/ is a directory of a strict subset of the information stored in data/downloads/. It is the result of running a bulk 7-zip job on that directory to get everything unzipped in a flat data structure. Note that these files have the USPTO modified-by time since that metadata is stored in the zipfiles. To extract files in this nice format, select all of the zipfiles and setup an extraction job like in this screenshot
  • data/backups/ is a 7zip'd backup of the corresponding directory in extracts
  • src/ is the main code repository for the java project

Input Files

All of the text-only Red Book files for granted patents from 1976 to 2016, inclusive. To find a specific year's XML file, find it in

E:\McNair\Projects\SimplerPatentData\data\extracts\granted\

To find assignment data, look in

E:\McNair\Projects\SimplerPatentData\data\extracts\granted\

To find maintenance fee data, look in

E:\McNair\Projects\SimplerPatentData\data\downloads\maintenance

Schema Reconciliation

Data Formats
Dates Used Format Supported by Parser?
January 1976 to December 2001 APS Yes (syntactic parsing but little semantic knowledge)
January 2001 to December 2001 SGML No
January 2002 to December 2004 XML Version 2.5 No
January 2005 to December 2005 XML Version 4.0 ICE Maybe
January 2006 to December 2006 XML Version 4.1 ICE Maybe
January 2007 to December 2012 XML Version 4.2 ICE Maybe
January 2013 to September 24, 2013 XML Version 4.3 ICE Yes
October 8, 2013 to December 2014 XML Version 4.4 ICE Yes
January 2015 to December 2016 XML Version 4.5 ICE Yes

Attributes

Note: these values are likely to change without warning. For the latest version of these see the actual files at E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\models.

Assignee

text fields: NAME, ADDR1, ADDR2, CITY, STATE, COUNTRY_NAME, POSTCODE

Assignment

text fields: REEL_NUMBER, FRAME_NUMBER, LAST_UPDATE_DATE, RECORDED_DATE, CONVEYANCE_TEXT

lists: correspondents, assignors, assignees

Assignment Summary

text fields: LAST_NAME, FIRST_NAME, ORG_NAME, CITY, COUNTRY, STATE, ADDRESS, POSTCODE

Assignor

Citation

Correspondent

GrantedPatent

Inventor

Lawyer

MaintenanceFeeEvent

Sciref

New Schema

Rough sketch: https://app.quickdatabasediagrams.com/#/schema/Huo3bW9jK065GlXoTitReQ

Oyc-patent-schema-1.PNG


Related Projects