Reproducible Patent Data

A continuation of Redesigning Patent Database that aims to write faster, more centralized code to deal with data from the United States Patent and Trademark Office (USPTO). By having an end-to-end pipeline we can easily reproduce or update data without worrying about unintentional side effects or missing data. Currently, it succeeds in bulk downloading from the USPTO; streaming file splitting, that is, splitting large concatenated files into their component parts in-memory; and parsing of XML to Java objects, APS to Java Maps, and maintenance fee data to Java objects.

McNair Project

Reproducible Patent Data
Project Information
Project Title	Reproducible Patent Data
Owner	Oliver Chang
Start Date	May 17
Deadline
Primary Billing
Notes
Has project status	Active
Subsumes:	Redesigning Patent Database, Patent Assignment Data Restructure
Copyright © 2016 edegan.com. All Rights Reserved.

Progress

1. Downloader done
2. Splitter done
3. Parser done
4. Create tooling for minions
5. Setup PostgreSQL JDBC
6. Create naive schema based on previous approaches
7. Create new data structures
8. Database Insert (modify models/ files with some mapping to database fields)
9. Data Cleanup (reference Marcela and Sonia's work)
10. Setup pipeline script to complete all of these steps in series
11. Investigate parallel speedup (e.g. multithread, mmap)
12. Data Source Merger (only USPTO granted, maintfee, assignment not USPTO applications or Harvard Dataverse or Lex Machina currently)

Directory Layout

All of the information for this project is located at E:\McNair\Projects\SimplerPatentData

There are four interesting directories:

• data/downloads/ is USPTO bulkdata, unmodified straight from the scraper
• data/extracts/ is a directory of a strict subset of the information stored in data/downloads/. It is the result of running a bulk 7-zip job on that directory to get everything unzipped in a flat data structure. Note that these files have the USPTO modified-by time since that metadata is stored in the zipfiles. To extract files in this nice format, select all of the zipfiles and setup an extraction job like in this screenshot
• data/backups/ is a 7zip'd backup of the corresponding directory in extracts
• src/ is the main code repository for the java project

Input Files

All of the text-only Red Book files for granted patents from 1976 to 2016, inclusive. To find a specific year's XML file, find it in

E:\McNair\Projects\SimplerPatentData\data\extracts\granted\

To find assignment data, look in

E:\McNair\Projects\SimplerPatentData\data\extracts\granted\

To find maintenance fee data, look in

E:\McNair\Projects\SimplerPatentData\data\downloads\maintenance

Schema Reconciliation

Data Formats

Dates Used	Format	Supported by Parser?
January 1976 to December 2001	APS	Only syntax
January 2001 to December 2001	SGML	No
January 2002 to December 2004	XML Version 2.5	Only syntax
January 2005 to December 2005	XML Version 4.0 ICE	Maybe
January 2006 to December 2006	XML Version 4.1 ICE	Maybe
January 2007 to December 2012	XML Version 4.2 ICE	Maybe
January 2013 to September 24, 2013	XML Version 4.3 ICE	Yes
October 8, 2013 to December 2014	XML Version 4.4 ICE	Yes
January 2015 to December 2016	XML Version 4.5 ICE	Yes

Related Projects

• Existing Database Schema

External Links

• USPTO Bulk Data Storage System (BDSS)
• Builder Pattern in Object-Oriented Programming