7,612
edits
Changes
Jump to navigation
Jump to search
A <onlyinclude>The [[Reproducible Patent Data]] project is a continuation of the [[Redesigning Patent Database]] that project. It aims to write faster, more centralized code to deal with data from the United States Patent and Trademark Office (USPTO). By having an end-to-end pipeline we can easily reproduce or update data without worrying about unintentional side effects or missing data.See also the [[Patent Data]] umbrella project. </onlyinclude>
TODOTo get the most granular address data (street level, or at least postcode level) about who owns patents, the path is not so straightforward because off the complicated mapping of ownership to a granted patent.This is the final part of this project that I am working on and it is all at the level of SQL.See <code>E:\McNair\Projects\SimplerPatentData\src\db\joins</code> for my attempts to create a clean mapping.Optimistically speaking, the data generated here should be superset of the data present in the Patent Assignment Data Restructure project. Note that as of the beginning of August 2017, this part '''has not been completed.'''
no edit summary
{{Project|Has project output=Data,Tool,Content,How-to,Guide|Has sponsor=McNair ProjectsCenter
|Has Image=Uspto web logo.jpg
|Has title=Reproducible Patent Data
}}
== Quickstart ==
# Run the table creation scripts in <code>src/db/schemas/</code> in your new database
# Modify the constant <code>DATABASE_NAME</code> in <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\postgres\DatabaseHelper.java</code>
# Run the Driver scripts in IntelliJ with the correct value for <code>DATA_DIRECTORY</code>(or run <code>RunInitialImport.java</code> which will do all of the data directories for that patent item type)
# [Take a really, really long lunch...in total should take no more than five hours to load data on RDP]
# Run scripts in <code>src/db/constraints</code> to check data assumptions
# That's it!
===Troubleshooting===
If you're new to IntelliJ (and even if you're not) you might run into problems with importing the project.
'''Setting Up Project as a Maven project'''
It should be clear if the project is not set up as a Maven project - when you right click on RunInitialImport.java, for example, you won't see an option with
a green triangle next to it that says "Run 'RunInitialImport.java'", and the green triangle in the top toolbar will be grayed out. If the project is not set up as a Maven project, you will not be able to run any of the code. To set up the project as a Maven project, when you import the project, follow the instructions at the following [https://www.jetbrains.com/help/idea/maven.html#maven_import_project_start link].
* Note that when you click "Import Project", you should select the "Simpler Patent Data" folder, not the "src" folder within Simpler Patent Data, otherwise you won't get the pom.xml file that you need to let IntelliJ know that this is a Maven project.
* On the second window (there will be several options with check boxes next to them) make sure "import Maven projects automatically" is selected
* On the "Please select project SDK" window, make sure it says "1.8" in the "Name" slot.
* On the next window, enter a name for the project and enter a folder location.
This should ensure that the project is set up as a Maven project.
'''Setting Up Your Data Source'''
If you run into a message across the top that says something along the lines of "Configure Data Source", then you have not connected IntelliJ to a database. You will not be able to run the code located under src/db until you configure one. Start by clicking on the link in the message, or if it doesn't appear, follow the instructions [https://www.jetbrains.com/help/idea/connecting-to-a-database.html here] to open up the "Data Sources and Drivers" pop-up to add a PostgreSQL database. When you get to the dialogue asking about the host, database, user, and password, do the following to connect to the database on the RDP:
host: localhost
database: whatever the constant DATABASE_NAME is set to - the default is patentsj
user: postgres
password: tabspaceenter
Make sure to test the connection by clicking "Test Connection". Now you should be able to run the scripts under src/db.
If you're seeing issues such as "column [something] of relation [something] doesn't exist" but you've run the schema scripts, you probably have different database name under the data source than the one the constant DATABASE_NAME is set to. To change this, right click on the data source in the Database tab and select "Properties".
== Directory Layout ==
The development environment is Java 8 JDK, IntelliJ Ultimate IDE, Maven build tools, and git VCS.
The git repository can be found at httpshttp://rdp.mcnaircenter.org/codebase/Repository/ReproduciblePatent
==== Prior Art ====
! scope="col" | Dates Used !! scope="col" | Format !! scope="col" | Location !! scope="col" | Supported by Parser?
|-
|January March 15, 2001 to December 2001
|XML Version 1.5
|<code>data/extracts/applications/vintage</code>
|style="background: red; color: whiteyellow;" | NoYes, for basic information, inventors, and correspondents
|-
|January 2002 to December 2004
|XML Version 1.6
|<code>data/extracts/applications/vintage</code>
|style="background: red; color: whiteyellow;" | NoDitto
|-
|January 2005 to December 2005
|XML Version 4.0 ICE
|<code>data/extracts/applications/modern</code>
|style="background: yellow;" | MaybeDitto
|-
|January 2006 to December 2006
|XML Version 4.1 ICE
|<code>data/extracts/applications/modern</code>
|style="background: yellow;" | MaybeDitto
|-
|January 2007 to December 2012
|XML Version 4.2 ICE
|<code>data/extracts/applications/modern</code>
|style="background: yellow;" | MaybeDitto
|-
|January 2013 to September 24, 2013December 2014
|XML Version 4.3 ICE
|<code>data/extracts/applications/modern</code>
|style="background: yellow;" | MaybeDitto
|-
|October 8, 2013 January 2015 to December 2014''Present''
|XML Version 4.4 ICE
|<code>data/extracts/applications/modern</code>
|style="background: yellow;" | Maybe|-|January 2015 to December 2016|XML Version 4.5 ICE|<code>data/extracts/applications/modern</code>|style="background: yellow;" | MaybeDitto
|}
Since writing raw SQL is a bit cumbersome and error-prone, I have added some abstraction layers that make it much easier to quickly add bulk data. By using Postgres's <code>CopyManager</code> class, we buffer SQL copy commands in memory (as many as possible) and then flush these rows. To understand how the abstraction layers work, see the code in <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\postgres</code>. See <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\models\GrantedPatent.java</code> for '''an example of how to extend''' the abstraction layer to deal with more complex scenarios.
=== New Table Checklist ===
* Create schema DDL SQL code for the new table in <code>E:\McNair\Projects\SimplerPatentData\src\db</code>
* Run the schema creation
* Create an enum with the same names for attributes as in the DDL (case-insensitive! prefer all-caps screaming snake case)
* Create a class which subclasses <code>AbstractInsertableData</code>
* Inside that class, create a static class which subclasses <code>AbstractTableMetadata</code> and has the proper values for getTableName(), getStringColumns(), getIntColumns()
* (Optional) Implement builder pattern
* (Optional) Create a custom databasehelper for complex extras (see PatentApplication and GrantedPatent for examples)
* Write the data to the table (see DatabaseHelper for the pattern I use)
== Address Data ==
=== Intuition ===
== Related Pages ==
* [[Redesign_Assignment_and_Patent_Database|Redesign Assignment and Patent Database, Fall 2017 by Shelby]]* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries, Summer 2017 by this guyOliver & Joe]]
* [[US_Address_Verification|US Address Verification, Summer 2017 based on tables from Assignment Data Restructure]]
* [[Patent_Assignment_Data_Restructure|Assignment Data Restructure, Spring 2017 by Marcela and Sonia]]
* [[Lex_Machina|Lex Machina]]
* [[USPTO_Patent_Litigation_Data|USPTO Patent Litigation Research Dataset by Ed]]
* [[Patent_Litigation_and_Review|Patent Litigation and Review by Marcela]]* [[Bag_of_Words_Analysis_of_Patent_Data|Bag of Words Analysis]]* [[Patent|Existing Database Schema]]
* [[Oliver_Chang_(Work_Log)|My Work Log]]
* [https://bulkdata.uspto.gov/ USPTO Bulk Data Storage System (BDSS)]
* [https://en.wikipedia.org/wiki/Builder_pattern Builder Pattern in Object-Oriented Programming]
* [httpshttp://rdp.mcnaircenter.org/codebase/Repository/ReproduciblePatent Git Repository]
