http://www.edegan.com/mediawiki/api.php?action=feedcontributions&feedformat=atom&user=OliverCedegan.com - User contributions [en]2026-06-08T19:39:50ZUser contributionsMediaWiki 1.34.2http://www.edegan.com/mediawiki/index.php?title=Reproducible_Patent_Data&diff=22277Reproducible Patent Data2017-12-05T05:34:30Z<p>OliverC: Remove dead link</p>
<hr />
<div>{{McNair Projects<br />
|Has Image=Uspto web logo.jpg<br />
|Has title=Reproducible Patent Data<br />
|Has owner=Oliver Chang<br />
|Has start date=May 17<br />
|Has project status=Active<br />
|Does subsume=Redesigning Patent Database, Patent Assignment Data Restructure,<br />
}}<br />
<br />
A continuation of [[Redesigning Patent Database]] that aims to write faster, more centralized code to deal with data from the United States Patent and Trademark Office (USPTO). By having an end-to-end pipeline we can easily reproduce or update data without worrying about unintentional side effects or missing data.<br />
<br />
== Quickstart ==<br />
<br />
To get up and running with the code, do the following:<br />
<br />
# Clone the git project (link at end of page) to your user directory<br />
# Launch IntelliJ with >= Java 8 and Maven configured (default version installed on the RDP is setup to do this)<br />
# Open project in IntelliJ<br />
# Create an empty database (see [[#Database]])<br />
# Run the table creation scripts in <code>src/db/schemas/</code> in your new database<br />
# Modify the constant <code>DATABASE_NAME</code> in <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\postgres\DatabaseHelper.java</code><br />
# Run the Driver scripts in IntelliJ with the correct value for <code>DATA_DIRECTORY</code> (or run <code>RunInitialImport.java</code> which will do all of the data directories for that patent item type)<br />
# [Take a really, really long lunch...in total should take no more than five hours to load data on RDP]<br />
# Run scripts in <code>src/db/constraints</code> to check data assumptions<br />
# That's it!<br />
<br />
== Directory Layout ==<br />
<br />
=== Where is the Data? ===<br />
<br />
==== Directories ====<br />
<br />
All of the information for this project is located at <code>E:\McNair\Projects\SimplerPatentData</code><br />
<br />
There are several interesting directories:<br />
<br />
* <code>data/downloads/</code> is USPTO bulkdata, unmodified straight from the scraper<br />
* <code>data/extracts/</code> is a directory of a strict subset of the information stored in <code>data/downloads/</code>. It is the result of running a bulk 7-zip job on that directory to get everything unzipped in a flat data structure. Note that these files have the USPTO modified-by time since that metadata is stored in the zipfiles. To extract files in this nice format, select all of the zipfiles and setup an extraction job like in this [[media:7zip-params.png|screenshot]]<br />
* <code>data/backups/</code> is a 7zip'd backup of the corresponding directory in extracts<br />
* <code>src/</code> is the main code repository for the java project<br />
<br />
==== Input Files ====<br />
<br />
All of the text-only Red Book files for '''granted patents''' from 1976 to 2016, inclusive. To find a specific year's XML file, find it in<br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\granted\</code><br />
<br />
'''To find application data''' from 2001 to 2016, inclusive, look in<br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\applications\</code><br />
<br />
'''To find assignment data''', look in <br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\granted\</code><br />
<br />
'''To find maintenance fee data''', look in <br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\downloads\maintenance\</code><br />
<br />
=== Where is the Code? ===<br />
<br />
The code has the same parent directory as the data, so it is at <code>E:\McNair\Projects\SimplerPatentData\src</code>. You might notice a lot of single-entry directories; this is an idiomatic Java pattern that is used for package separation. If using IntelliJ or some other IDE, these directories are a bit less annoying.<br />
<br />
The development environment is Java 8 JDK, IntelliJ Ultimate IDE, Maven build tools, and git VCS.<br />
<br />
The git repository can be found at http://rdp.mcnaircenter.org/codebase/Repository/ReproduciblePatent<br />
<br />
==== Prior Art ====<br />
<br />
This tool is not so concerned with adding new functionality; rather, it aims to take a bunch of spread out Perl scripts and create a faster system that is easier to work with. As such, its functionality is largely stolen from those scripts:<br />
<br />
* Downloader: <code>E:\McNair\Software\Scripts\Patent\USPTO_Parser.pl</code><br />
* XML Splitter: <code>E:\McNair\PatentData\splitter.pl</code><br />
* XML Parsing: <code>E:\McNair\PatentData\Processed\xmlparser_4.5_4.4_4.3.pl</code> and <code>E:\McNair\PatentData\Processed\*.pm</code><br />
<br />
In addition, I used several non-standard Java libraries listed below:<br />
<br />
* [https://github.com/mashape/unirest-java/ Unirest] for easy HTTP requests (MIT License)<br />
* [https://github.com/google/guava Google Guava] for immutable collections and Stream utilities (Apache v2.0 License)<br />
* [https://github.com/jhy/jsoup/ jsoup] for HTML parsing (MIT License)<br />
* [http://commons.apache.org/proper/commons-codec/ Apache Commons Codec] (Apache v2.0 License)<br />
* [http://commons.apache.org/proper/commons-lang/ Apache Commons Lang v3] (Apache v2.0 License)<br />
* [https://mvnrepository.com/artifact/org.jetbrains/annotations/15.0 Jetbrains Annotations] for enhanced null checks (Apache v2.0 License)<br />
* [http://search.maven.org/#artifactdetails%7Corg.postgresql%7Cpostgresql%7C42.1.1.jre7%7Cbundle PostgreSQL JDBC] (BSD 3-clause per https://github.com/pgjdbc/pgjdbc-jre7/blob/master/LICENSE)<br />
<br />
If using maven, these dependencies are listed and should automatically be setup.<br />
<br />
==== Using Code ====<br />
<br />
Any file with a line that says <code>public static void main(String[] args) {</code> can be run as a standalone file. The easiest way to do this is to load the project and then the file in IntelliJ and click the little green play arrow next to this bit of code.<br />
<br />
The code can also be run via the standard <code>javac</code> and <code>java</code> commands but since this project has a complicated structure you end up having to run commands like <br />
<br />
<code>"C:\Program Files\Java\jdk1.8.0_131\bin\java" "-javaagent:C:\Users\OliverC\IntelliJ IDEA 2017.1.3\lib\idea_rt.jar=62364:C:\Users\OliverC\IntelliJ IDEA 2017.1.3\bin" -Dfile.encoding=UTF-8 -classpath "[...contents truncated...];C:\Users\OliverC\.m2\repository\org\postgresql\postgresql\42.1.1\postgresql-42.1.1.jar" org.bakerinstitute.mcnair.uspto_assignments.XmlDriver</code><br />
<br />
to include all of the runtime dependencies and it's just not worth it.<br />
<br />
==== Altering Code ====<br />
<br />
* Use the IntelliJ command Reformat code (found in the menus at <code>Code > Reformat Code</code><br />
* Use the optimize imports function found under the same menu<br />
* Use spaces for indentation<br />
* Loosely try to keep lines below 120 characters<br />
* Commit changes to the Git remote repository "bonobo"<br />
<br />
== Schema Reconciliation ==<br />
<br />
For the work by Joe, see the [[Patent Schema Reconciliation]] page <br />
<br />
=== Patents (Granted) ===<br />
<br />
See <code>E:\McNair\Projects\SimplerPatentData\data\examples\granted</code> for extracted examples of what specific data is available for a sample of the data.<br />
<br />
{| class="wikitable"<br />
|+Granted Patent Data Formats<br />
|-<br />
! scope="col" | Dates Used !! scope="col" | Format !! scope="col" | Location !! scope="col" | Supported? !! scope="col" | Utility !! scope="col" | Reissue !! scope="col" | Design !! scope="col" | Plant<br />
|-<br />
|January 1976 to December 2001<br />
|APS<br />
|<code>data/extracts/granted/vintage</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|<del>January 2001 to December 2001</del><br />
|<del>SGML</del><br />
|Ignored; use concurrently recorded APS data<br />
|No<br />
|N/A<br />
|N/A<br />
|N/A<br />
|N/A<br />
|-<br />
|January 2002 to December 2004<br />
|XML Version 2.5<br />
|<code>data/extracts/granted/blunderyears</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2005 to December 2005<br />
|XML Version 4.0 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2006 to December 2006<br />
|XML Version 4.1 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2007 to December 2012<br />
|XML Version 4.2 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2013 to September 24, 2013<br />
|XML Version 4.3 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|October 8, 2013 to December 2014<br />
|XML Version 4.4 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2015 to December 2016<br />
|XML Version 4.5 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|}<br />
<br />
=== APS Rosetta Stone ===<br />
<br />
The Advanced Patent System (APS) is a fixed-width text format used to store historical patent grant data. The documentation for this sucks; there are pages missing at random. Luckily, we only care about the content contained here: [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]].<br />
<br />
It's worth mentioning that the APS contains an advanced text markup system for chemical formulae, basic text markup, tables, etc. that can lead to seemingly garbled text that is perfectly well-formed.<br />
<br />
==== APS Gotchas ====<br />
<br />
* PATN.WKU is the granted patent number. It is 7 digits while the spec promises 6 digits. The rightmost digit is a check digit modulus 11. See [[File:Aps-wku-modulus11.pdf]] for the words from the horse's mouth.<br />
<br />
=== Patents (Applications) ===<br />
<br />
{| class="wikitable"<br />
|+Patent Application Data Formats<br />
|-<br />
! scope="col" | Dates Used !! scope="col" | Format !! scope="col" | Location !! scope="col" | Supported by Parser?<br />
|-<br />
|March 15, 2001 to December 2001<br />
|XML Version 1.5<br />
|<code>data/extracts/applications/vintage</code><br />
|style="background: yellow;" | Yes, for basic information, inventors, and correspondents<br />
|-<br />
|January 2002 to December 2004<br />
|XML Version 1.6<br />
|<code>data/extracts/applications/vintage</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2005 to December 2005<br />
|XML Version 4.0 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2006 to December 2006<br />
|XML Version 4.1 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2007 to December 2012<br />
|XML Version 4.2 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2013 to December 2014<br />
|XML Version 4.3 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2015 to ''Present''<br />
|XML Version 4.4 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|}<br />
<br />
== Database ==<br />
<br />
Because there isn't a compelling reason not to, I used the existing PostgreSQL infrastructure on the RDP.<br />
The "Java Way" of interacting with databases is the Java Database Connectivity API (JDBC), an implementation-agnostic API for interacting with databases.<br />
This project uses the stock [https://jdbc.postgresql.org/ Postgres JDBC], version 42.1.1<br />
<br />
=== Create an empty database on RDP ===<br />
<br />
To create an empty database, run this command: <code>$ createdb --username=postgres database-name-goes-here # password is tabspaceenter</code><br />
<br />
=== Abstraction Layer ===<br />
<br />
Since writing raw SQL is a bit cumbersome and error-prone, I have added some abstraction layers that make it much easier to quickly add bulk data. By using Postgres's <code>CopyManager</code> class, we buffer SQL copy commands in memory (as many as possible) and then flush these rows. To understand how the abstraction layers work, see the code in <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\postgres</code>. See <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\models\GrantedPatent.java</code> for '''an example of how to extend''' the abstraction layer to deal with more complex scenarios.<br />
<br />
=== New Table Checklist ===<br />
<br />
* Create schema DDL SQL code for the new table in <code>E:\McNair\Projects\SimplerPatentData\src\db</code><br />
* Run the schema creation<br />
* Create an enum with the same names for attributes as in the DDL (case-insensitive! prefer all-caps screaming snake case)<br />
* Create a class which subclasses <code>AbstractInsertableData</code><br />
* Inside that class, create a static class which subclasses <code>AbstractTableMetadata</code> and has the proper values for getTableName(), getStringColumns(), getIntColumns()<br />
* (Optional) Implement builder pattern<br />
* (Optional) Create a custom databasehelper for complex extras (see PatentApplication and GrantedPatent for examples)<br />
* Write the data to the table (see DatabaseHelper for the pattern I use)<br />
<br />
== Address Data ==<br />
<br />
To get the most granular address data (street level, or at least postcode level) about who owns patents, the path is not so straightforward because off the complicated mapping of ownership to a granted patent.<br />
This is the final part of this project that I am working on and it is all at the level of SQL.<br />
See <code>E:\McNair\Projects\SimplerPatentData\src\db\joins</code> for my attempts to create a clean mapping.<br />
Optimistically speaking, the data generated here should be superset of the data present in the Patent Assignment Data Restructure project.<br />
<br />
Note that as of the beginning of August 2017, this part '''has not been completed.'''<br />
<br />
<br />
=== Intuition ===<br />
<br />
Use <code>assignments_longform.last_update_date</code> to find current/latest (or first/earliest) date of assignment. Then match with <code>properties.docid</code> on <code>reelno, frameno</code> to find patent application id. With this mapping to granted patents, we can discover the details of the original granted patent. And with the right date and reelno and frameno, we can match to the <code>assignees</code> table and get fine granularity addresses.<br />
<br />
== Related Pages ==<br />
<br />
* [[Redesign_Assignment_and_Patent_Database|Redesign Assignment and Patent Database, Fall 2017 by Shelby]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries, Summer 2017 by Oliver & Joe]]<br />
* [[US_Address_Verification|US Address Verification, Summer 2017 based on tables from Assignment Data Restructure]]<br />
* [[Patent_Assignment_Data_Restructure|Assignment Data Restructure, Spring 2017 by Marcela and Sonia]]<br />
* [[Redesigning_Patent_Database|Redesigning Patent Database, Spring 2017 by Shelby]]<br />
* [[Patent_Data_Cleanup_(June_2016)|Patent Data Cleanup, June 2016 by Marcela]]<br />
* [[Patent_Data|Patent Data, Spring 2016 by Marcela]] <br />
* [[Lex_Machina|Lex Machina]]<br />
* [[USPTO_Patent_Litigation_Data|USPTO Patent Litigation Research Dataset by Ed]]<br />
* [[Patent_Litigation_and_Review|Patent Litigation and Review by Marcela]]* [[Patent|Existing Database Schema]]<br />
* [[Oliver_Chang_(Work_Log)|My Work Log]]<br />
<br />
== External Links ==<br />
<br />
* Understanding Assignment Data: [https://www.uspto.gov/sites/default/files/documents/USPTO_Patents_Assignment_Dataset_WP.pdf USPTO Documentation on their cleanup of this data]<br />
* [https://bulkdata.uspto.gov/data2/patent/grant/redbook/fulltext/1976/PatentFullTextAPSDoc_GreenBook.pdf USPTO Green Book (APS) Documentation]<br />
* [https://bulkdata.uspto.gov/ USPTO Bulk Data Storage System (BDSS)]<br />
* [https://en.wikipedia.org/wiki/Builder_pattern Builder Pattern in Object-Oriented Programming]<br />
* [http://rdp.mcnaircenter.org/codebase/Repository/ReproduciblePatent Git Repository]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=22269Oliver Chang (Work Log)2017-12-03T03:19:15Z<p>OliverC: </p>
<hr />
<div>''Projects'':<br />
<br />
* [[PostGIS_Installation|PostGIS Installation]]<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
* [[Python_on_the_RDP|Python on the RDP]]<br />
* [[Hierarchical_Clustering|Hierarchical Clustering]]<br />
<br />
''Uploads'':<br />
<br />
* [http://mcnair.bakerinstitute.org/wiki/File:Path-example.PNG File:Path-example.PNG]<br />
** Screenshot of how to set the path environment variable<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
===Fall 2017=== <br />
<onlyinclude><br />
<br />
[[Oliver Chang]] [[Work Logs]] [[Oliver Chang (Work Log)|(log page)]]<br />
<br />
2017-12-02: communicated results of running ECA on 2 stddev table (not the error source); update db and web server software; check if scipy elision is ECA bug (it is not)<br />
<br />
2017-12-01: re-tasked Kyran with the hierarchical clustering algorithm implementation; create 2 standard deviations tables; freed up DB space<br />
<br />
2017-11-30: stub out implementation, add parsing code and mapping code...just need the meat of the algorithm now<br />
<br />
2017-11-27: documentation & bug finding on the parallel enclosing circle project; research hierarchical linkage approach<br />
<br />
2017-11-14: hand off work on xpathing to Shelby; walked through some program design decisions<br />
<br />
2017-11-13: finish javadoc of common/ and some trickier parts about downloading; added descriptions, results to one-off java/python scripts so that they actually make sense in context<br />
<br />
2017-11-10: add javadoc documentation to patent reproducibility project after forgetting half of the stuff myself<br />
<br />
2017-10-25: start ingestion of application xml files and deal with all the bugs which accompany that<br />
<br />
2017-10-21: create xml explorer script to mass-inspect xpaths (can be found at <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\xml_schema_explorer</code>)<br />
<br />
2017-10-20: re-run assignment import, review xpaths<br />
<br />
2017-10-19: repopulate patents database and create spreadsheet of assignment equivalencies<br />
<br />
2017-10-11: add issues to work on to the PECA wiki page; cleanup PECA git repo; start patent application code from granted patent code and start customizing to new domain<br />
<br />
2017-10-03: troubleshoot vc_circles.py and make command line interface a little nicer<br />
<br />
2017-10-02: discuss mapping strategies & investigate missing eca data<br />
<br />
2017-09-23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
<br />
2017-09-22: goal setting & server debugging & meet with Yang<br />
</onlyinclude><br />
<br />
===Summer 2017=== <br />
<br />
2017-08-04: setup parallel instance python framework for job reporting; begin test run<br />
<br />
2017-08-02: finish up some documentation of the code and for the wiki<br />
<br />
2017-08-01: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
2017-07-31: sketch out parallel enclosing circle algorithm<br />
<br />
2017-07-18: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
<br />
2017-07-19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
<br />
2017-07-18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
<br />
2017-07-17: redo db operations after cleaning up granted patent number bugs<br />
<br />
2017-07-13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
<br />
2017-07-12: generate some example data illustrating the difficulty of joining different tables<br />
<br />
2017-07-11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
<br />
2017-07-07: catch up on documentation<br />
<br />
2017-07-06: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
<br />
2017-07-05: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
2017-06-30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
<br />
2017-06-29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
<br />
2017-06-28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
<br />
2017-06-27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
<br />
2017-06-26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
<br />
2017-06-25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
<br />
2017-06-23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
<br />
2017-06-22: create postcode<->patent table<br />
<br />
2017-06-29: document granted patent queries and equivalencies<br />
<br />
2017-06-20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
<br />
2017-06-19: skim address regular expressions; cursory investigation of patent table<br />
<br />
2017-06-16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
<br />
2017-06-15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
<br />
2017-06-14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
<br />
2017-06-13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
<br />
2017-06-12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
<br />
2017-06-08: add foreign key inserts; create pretty printer for XML analysis<br />
<br />
2017-06-07: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
<br />
2017-06-06: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
<br />
2017-06-05: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
<br />
2017-06-01: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
2017-05-31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
<br />
2017-05-30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
<br />
2017-05-29: expand to APS; expand to raw assignment data<br />
<br />
2017-05-27: expand to maintenance fee data<br />
<br />
2017-05-26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
<br />
2017-05-25: sketch out OO design of project; download bulk data<br />
<br />
2017-05-24: move wiki pages around; start git repository for project<br />
<br />
2017-05-21: discuss technical details of previous work with Ed<br />
<br />
2017-05-08: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
<br />
2017-05-04: setup wiki account, rdp account, database training<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=22255Oliver Chang (Work Log)2017-12-01T18:40:18Z<p>OliverC: /* Fall 2017 */</p>
<hr />
<div>''Projects'':<br />
<br />
* [[PostGIS_Installation|PostGIS Installation]]<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
* [[Python_on_the_RDP|Python on the RDP]]<br />
<br />
''Uploads'':<br />
<br />
* [http://mcnair.bakerinstitute.org/wiki/File:Path-example.PNG File:Path-example.PNG]<br />
** Screenshot of how to set the path environment variable<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
===Fall 2017=== <br />
<onlyinclude><br />
<br />
[[Oliver Chang]] [[Work Logs]] [[Oliver Chang (Work Log)|(log page)]]<br />
<br />
2017-12-01: TBD<br />
<br />
2017-11-30: stub out implementation, add parsing code and mapping code...just need the meat of the algorithm now<br />
<br />
2017-11-27: documentation & bug finding on the parallel enclosing circle project; research hierarchical linkage approach<br />
<br />
2017-11-14: hand off work on xpathing to Shelby; walked through some program design decisions<br />
<br />
2017-11-13: finish javadoc of common/ and some trickier parts about downloading; added descriptions, results to one-off java/python scripts so that they actually make sense in context<br />
<br />
2017-11-10: add javadoc documentation to patent reproducibility project after forgetting half of the stuff myself<br />
<br />
2017-10-25: start ingestion of application xml files and deal with all the bugs which accompany that<br />
<br />
2017-10-21: create xml explorer script to mass-inspect xpaths (can be found at <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\xml_schema_explorer</code>)<br />
<br />
2017-10-20: re-run assignment import, review xpaths<br />
<br />
2017-10-19: repopulate patents database and create spreadsheet of assignment equivalencies<br />
<br />
2017-10-11: add issues to work on to the PECA wiki page; cleanup PECA git repo; start patent application code from granted patent code and start customizing to new domain<br />
<br />
2017-10-03: troubleshoot vc_circles.py and make command line interface a little nicer<br />
<br />
2017-10-02: discuss mapping strategies & investigate missing eca data<br />
<br />
2017-09-23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
<br />
2017-09-22: goal setting & server debugging & meet with Yang<br />
</onlyinclude><br />
<br />
===Summer 2017=== <br />
<br />
2017-08-04: setup parallel instance python framework for job reporting; begin test run<br />
<br />
2017-08-02: finish up some documentation of the code and for the wiki<br />
<br />
2017-08-01: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
2017-07-31: sketch out parallel enclosing circle algorithm<br />
<br />
2017-07-18: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
<br />
2017-07-19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
<br />
2017-07-18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
<br />
2017-07-17: redo db operations after cleaning up granted patent number bugs<br />
<br />
2017-07-13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
<br />
2017-07-12: generate some example data illustrating the difficulty of joining different tables<br />
<br />
2017-07-11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
<br />
2017-07-07: catch up on documentation<br />
<br />
2017-07-06: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
<br />
2017-07-05: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
2017-06-30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
<br />
2017-06-29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
<br />
2017-06-28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
<br />
2017-06-27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
<br />
2017-06-26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
<br />
2017-06-25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
<br />
2017-06-23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
<br />
2017-06-22: create postcode<->patent table<br />
<br />
2017-06-29: document granted patent queries and equivalencies<br />
<br />
2017-06-20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
<br />
2017-06-19: skim address regular expressions; cursory investigation of patent table<br />
<br />
2017-06-16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
<br />
2017-06-15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
<br />
2017-06-14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
<br />
2017-06-13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
<br />
2017-06-12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
<br />
2017-06-08: add foreign key inserts; create pretty printer for XML analysis<br />
<br />
2017-06-07: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
<br />
2017-06-06: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
<br />
2017-06-05: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
<br />
2017-06-01: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
2017-05-31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
<br />
2017-05-30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
<br />
2017-05-29: expand to APS; expand to raw assignment data<br />
<br />
2017-05-27: expand to maintenance fee data<br />
<br />
2017-05-26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
<br />
2017-05-25: sketch out OO design of project; download bulk data<br />
<br />
2017-05-24: move wiki pages around; start git repository for project<br />
<br />
2017-05-21: discuss technical details of previous work with Ed<br />
<br />
2017-05-08: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
<br />
2017-05-04: setup wiki account, rdp account, database training<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Installing_TensorFlow&diff=22030Installing TensorFlow2017-11-22T00:40:25Z<p>OliverC: </p>
<hr />
<div>Currently installed with Anaconda Python 3.<br />
<br />
https://stackoverflow.com/questions/36355073/upgrading-numpy-fails-with-permission-denied-error<br />
<br />
https://www.tensorflow.org/install/install_windows<br />
<br />
with cpu support only<br />
<br />
https://www.tensorflow.org/install/install_linux<br />
<br />
need to logoff other users via server manager<br />
<br />
https://stackoverflow.com/questions/46499808/pip-throws-typeerror-parse-got-an-unexpected-keyword-argument-transport-enco#_=_</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Installing_TensorFlow&diff=22029Installing TensorFlow2017-11-22T00:39:32Z<p>OliverC: </p>
<hr />
<div>https://stackoverflow.com/questions/36355073/upgrading-numpy-fails-with-permission-denied-error<br />
<br />
https://www.tensorflow.org/install/install_windows<br />
<br />
with cpu support only<br />
<br />
https://www.tensorflow.org/install/install_linux<br />
<br />
https://stackoverflow.com/questions/46499808/pip-throws-typeerror-parse-got-an-unexpected-keyword-argument-transport-enco#_=_</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Installing_TensorFlow&diff=22016Installing TensorFlow2017-11-21T20:21:22Z<p>OliverC: Created page with "https://stackoverflow.com/questions/36355073/upgrading-numpy-fails-with-permission-denied-error https://www.tensorflow.org/install/install_windows with cpu support only htt..."</p>
<hr />
<div>https://stackoverflow.com/questions/36355073/upgrading-numpy-fails-with-permission-denied-error<br />
<br />
https://www.tensorflow.org/install/install_windows<br />
<br />
with cpu support only<br />
<br />
https://www.tensorflow.org/install/install_linux</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=22013Oliver Chang (Work Log)2017-11-21T19:08:58Z<p>OliverC: </p>
<hr />
<div>''Projects'':<br />
<br />
* [[PostGIS_Installation|PostGIS Installation]]<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
* [[Python_on_the_RDP|Python on the RDP]]<br />
<br />
''Uploads'':<br />
<br />
* [http://mcnair.bakerinstitute.org/wiki/File:Path-example.PNG File:Path-example.PNG]<br />
** Screenshot of how to set the path environment variable<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
===Fall 2017=== <br />
<onlyinclude><br />
<br />
[[Oliver Chang]] [[Work Logs]] [[Oliver Chang (Work Log)|(log page)]]<br />
<br />
2017-11-14: hand off work on xpathing to Shelby; walked through some program design decisions<br />
<br />
2017-11-13: finish javadoc of common/ and some trickier parts about downloading; added descriptions, results to one-off java/python scripts so that they actually make sense in context<br />
<br />
2017-11-10: add javadoc documentation to patent reproducibility project after forgetting half of the stuff myself<br />
<br />
2017-10-25: start ingestion of application xml files and deal with all the bugs which accompany that<br />
<br />
2017-10-21: create xml explorer script to mass-inspect xpaths (can be found at <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\xml_schema_explorer</code>)<br />
<br />
2017-10-20: re-run assignment import, review xpaths<br />
<br />
2017-10-19: repopulate patents database and create spreadsheet of assignment equivalencies<br />
<br />
2017-10-11: add issues to work on to the PECA wiki page; cleanup PECA git repo; start patent application code from granted patent code and start customizing to new domain<br />
<br />
2017-10-03: troubleshoot vc_circles.py and make command line interface a little nicer<br />
<br />
2017-10-02: discuss mapping strategies & investigate missing eca data<br />
<br />
2017-09-23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
<br />
2017-09-22: goal setting & server debugging & meet with Yang<br />
</onlyinclude><br />
<br />
===Summer 2017=== <br />
<br />
2017-08-04: setup parallel instance python framework for job reporting; begin test run<br />
<br />
2017-08-02: finish up some documentation of the code and for the wiki<br />
<br />
2017-08-01: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
2017-07-31: sketch out parallel enclosing circle algorithm<br />
<br />
2017-07-18: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
<br />
2017-07-19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
<br />
2017-07-18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
<br />
2017-07-17: redo db operations after cleaning up granted patent number bugs<br />
<br />
2017-07-13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
<br />
2017-07-12: generate some example data illustrating the difficulty of joining different tables<br />
<br />
2017-07-11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
<br />
2017-07-07: catch up on documentation<br />
<br />
2017-07-06: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
<br />
2017-07-05: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
2017-06-30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
<br />
2017-06-29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
<br />
2017-06-28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
<br />
2017-06-27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
<br />
2017-06-26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
<br />
2017-06-25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
<br />
2017-06-23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
<br />
2017-06-22: create postcode<->patent table<br />
<br />
2017-06-29: document granted patent queries and equivalencies<br />
<br />
2017-06-20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
<br />
2017-06-19: skim address regular expressions; cursory investigation of patent table<br />
<br />
2017-06-16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
<br />
2017-06-15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
<br />
2017-06-14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
<br />
2017-06-13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
<br />
2017-06-12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
<br />
2017-06-08: add foreign key inserts; create pretty printer for XML analysis<br />
<br />
2017-06-07: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
<br />
2017-06-06: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
<br />
2017-06-05: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
<br />
2017-06-01: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
2017-05-31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
<br />
2017-05-30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
<br />
2017-05-29: expand to APS; expand to raw assignment data<br />
<br />
2017-05-27: expand to maintenance fee data<br />
<br />
2017-05-26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
<br />
2017-05-25: sketch out OO design of project; download bulk data<br />
<br />
2017-05-24: move wiki pages around; start git repository for project<br />
<br />
2017-05-21: discuss technical details of previous work with Ed<br />
<br />
2017-05-08: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
<br />
2017-05-04: setup wiki account, rdp account, database training<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=22012Oliver Chang (Work Log)2017-11-21T19:07:46Z<p>OliverC: </p>
<hr />
<div>''Projects'':<br />
<br />
* [[PostGIS_Installation|PostGIS Installation]]<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
<br />
''Uploads'':<br />
<br />
* [http://mcnair.bakerinstitute.org/wiki/File:Path-example.PNG File:Path-example.PNG]<br />
** Screenshot of how to set the path environment variable<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
===Fall 2017=== <br />
<onlyinclude><br />
<br />
[[Oliver Chang]] [[Work Logs]] [[Oliver Chang (Work Log)|(log page)]]<br />
<br />
2017-11-14: hand off work on xpathing to Shelby; walked through some program design decisions<br />
<br />
2017-11-13: finish javadoc of common/ and some trickier parts about downloading; added descriptions, results to one-off java/python scripts so that they actually make sense in context<br />
<br />
2017-11-10: add javadoc documentation to patent reproducibility project after forgetting half of the stuff myself<br />
<br />
2017-10-25: start ingestion of application xml files and deal with all the bugs which accompany that<br />
<br />
2017-10-21: create xml explorer script to mass-inspect xpaths (can be found at <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\xml_schema_explorer</code>)<br />
<br />
2017-10-20: re-run assignment import, review xpaths<br />
<br />
2017-10-19: repopulate patents database and create spreadsheet of assignment equivalencies<br />
<br />
2017-10-11: add issues to work on to the PECA wiki page; cleanup PECA git repo; start patent application code from granted patent code and start customizing to new domain<br />
<br />
2017-10-03: troubleshoot vc_circles.py and make command line interface a little nicer<br />
<br />
2017-10-02: discuss mapping strategies & investigate missing eca data<br />
<br />
2017-09-23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
<br />
2017-09-22: goal setting & server debugging & meet with Yang<br />
</onlyinclude><br />
<br />
===Summer 2017=== <br />
<br />
2017-08-04: setup parallel instance python framework for job reporting; begin test run<br />
<br />
2017-08-02: finish up some documentation of the code and for the wiki<br />
<br />
2017-08-01: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
2017-07-31: sketch out parallel enclosing circle algorithm<br />
<br />
2017-07-18: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
<br />
2017-07-19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
<br />
2017-07-18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
<br />
2017-07-17: redo db operations after cleaning up granted patent number bugs<br />
<br />
2017-07-13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
<br />
2017-07-12: generate some example data illustrating the difficulty of joining different tables<br />
<br />
2017-07-11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
<br />
2017-07-07: catch up on documentation<br />
<br />
2017-07-06: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
<br />
2017-07-05: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
2017-06-30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
<br />
2017-06-29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
<br />
2017-06-28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
<br />
2017-06-27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
<br />
2017-06-26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
<br />
2017-06-25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
<br />
2017-06-23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
<br />
2017-06-22: create postcode<->patent table<br />
<br />
2017-06-29: document granted patent queries and equivalencies<br />
<br />
2017-06-20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
<br />
2017-06-19: skim address regular expressions; cursory investigation of patent table<br />
<br />
2017-06-16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
<br />
2017-06-15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
<br />
2017-06-14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
<br />
2017-06-13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
<br />
2017-06-12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
<br />
2017-06-08: add foreign key inserts; create pretty printer for XML analysis<br />
<br />
2017-06-07: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
<br />
2017-06-06: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
<br />
2017-06-05: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
<br />
2017-06-01: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
2017-05-31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
<br />
2017-05-30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
<br />
2017-05-29: expand to APS; expand to raw assignment data<br />
<br />
2017-05-27: expand to maintenance fee data<br />
<br />
2017-05-26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
<br />
2017-05-25: sketch out OO design of project; download bulk data<br />
<br />
2017-05-24: move wiki pages around; start git repository for project<br />
<br />
2017-05-21: discuss technical details of previous work with Ed<br />
<br />
2017-05-08: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
<br />
2017-05-04: setup wiki account, rdp account, database training<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Python_on_the_RDP&diff=22011Python on the RDP2017-11-21T19:07:21Z<p>OliverC: /* Simultaneously Registering Python 2 and 3 */</p>
<hr />
<div>There are two production versions of Python: Python 2 and Python 3.<br />
<br />
* You can run scripts written for 3 with 2 if you import the special package [https://docs.python.org/2/library/__future__.html <code>__future__</code>]<br />
* You can run scripts written for 2 with 3 if you run the script [https://docs.python.org/3.1/library/2to3.html <code>2to3</code>]<br />
* '''All new code should be written in Python 3''' for library support and maintainability reasons. If for no other reason know that it uses UTF-8 as the default encoding and so it makes your life easier in every possible way.<br />
<br />
== Installation ==<br />
<br />
Many Python modules are actually 90% low-level C code with the remaining 10% a python wrapper.<br />
That means to install these modules (e.g. numpy, scipy, hdf) you need to have all of the development libraries and compiler tools to compile the C stuff.<br />
Enter [https://anaconda.org/ Anaconda], a widely used repackaging of Python for Windows that comes with most of the libraries you'll ever need.<br />
In addition, they've made it easy to have two installs of Python running at once.<br />
<br />
* Python2<br />
** Python 2.7.14 via Anaconda2 5.0.1 for 64-bit<br />
** Located at <code>C:\ProgramData\Anaconda2</code>*<br />
<br />
* Python3<br />
** Python 3.6.3 via Anaconda3 5.0.1 for 64-bit<br />
** Located at <code>C:\ProgramData\Anaconda3</code>*<br />
<br />
<nowiki>*</nowiki> Note that Anaconda is not installed in <code>C:\Program Files\AnacondaN</code> intentionally. Spaces in the path lead to a bunch of unpredictable errors.<br />
<br />
=== Manually Defining Explicit Python Binary Executables ===<br />
<br />
A common practice is to have a bunch of specific Python commands to force explicit behavior. For example on Ubuntu, you can choose from running python, python2, python 2.6, python2.7, python3, python3.5, etc in the shell. To emulate this behavior, copy and paste the <code>python.exe</code> and rename it to <code>pythonN.exe</code> where N is whatever specific version you copied. The same applies to scripts you want to use like <code>pip</code>: copy-and-paste the file to a more explicit name.<br />
<br />
Path is evaluated with the first entries taking precedence. Thus, whichever version of Python you want to be the default should be listed at the leftmost position on the string. You can also just delete the unregistered python.exe to achieve the same result.<br />
<br />
=== ArcMap Aside ===<br />
<br />
Note that by default, ESRI ArcMap maintains a separate Python 2 installation at <code>C:\Python27</code>. Just ignore this...it is not to be used for development or as the environment for running programs other than ArcMap.<br />
<br />
== Simultaneously Registering Python 2 and 3 ==<br />
<br />
Anaconda automatically "registers" the separate installations for use with some development environments. It does not (by default) add executables to the Path system environment. We will add the base <code>AnacondaN/*</code> files and <code>AnacondaN/Scripts/*</code> to the path as so:<br />
<br />
# Right-click on the Windows icon in the bottom left<br />
# Select System<br />
# Select Advanced System Settings (must be administrator)<br />
# Select the Advanced Tab<br />
# Select "Environment Variables..." at the bottom right<br />
# Add semicolon-separated directories to Path in the lower box labeled System Variables<br />
<br />
See [http://mcnair.bakerinstitute.org/wiki/File:Path-example.PNG File:Path-example.PNG] for an example.<br />
<br />
== Explicitly Running a Specific Python Version ==<br />
<br />
All of these are valid ways to run Python on the RDP:<br />
<br />
* <code>python</code><br />
* <code>python2</code><br />
* <code>python3</code><br />
<br />
By default, running <code>python</code> will run Python 2. '''This behavior should not be relied on.''' On most other systems, Python 3 is the default instead.<br />
<br />
It is the same case with <code>pip</code> which can be run via <code>pip</code>, <code>pip2</code>, <code>pip3</code></div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=22010Oliver Chang (Work Log)2017-11-21T19:02:13Z<p>OliverC: </p>
<hr />
<div>''Projects'':<br />
<br />
* [[PostGIS_Installation|PostGIS Installation]]<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
<br />
''Uploads'':<br />
<br />
* [[File:Path-example.PNG]]<br />
** Screenshot of how to set the path environment variable<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
===Fall 2017=== <br />
<onlyinclude><br />
<br />
[[Oliver Chang]] [[Work Logs]] [[Oliver Chang (Work Log)|(log page)]]<br />
<br />
2017-11-14: hand off work on xpathing to Shelby; walked through some program design decisions<br />
<br />
2017-11-13: finish javadoc of common/ and some trickier parts about downloading; added descriptions, results to one-off java/python scripts so that they actually make sense in context<br />
<br />
2017-11-10: add javadoc documentation to patent reproducibility project after forgetting half of the stuff myself<br />
<br />
2017-10-25: start ingestion of application xml files and deal with all the bugs which accompany that<br />
<br />
2017-10-21: create xml explorer script to mass-inspect xpaths (can be found at <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\xml_schema_explorer</code>)<br />
<br />
2017-10-20: re-run assignment import, review xpaths<br />
<br />
2017-10-19: repopulate patents database and create spreadsheet of assignment equivalencies<br />
<br />
2017-10-11: add issues to work on to the PECA wiki page; cleanup PECA git repo; start patent application code from granted patent code and start customizing to new domain<br />
<br />
2017-10-03: troubleshoot vc_circles.py and make command line interface a little nicer<br />
<br />
2017-10-02: discuss mapping strategies & investigate missing eca data<br />
<br />
2017-09-23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
<br />
2017-09-22: goal setting & server debugging & meet with Yang<br />
</onlyinclude><br />
<br />
===Summer 2017=== <br />
<br />
2017-08-04: setup parallel instance python framework for job reporting; begin test run<br />
<br />
2017-08-02: finish up some documentation of the code and for the wiki<br />
<br />
2017-08-01: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
2017-07-31: sketch out parallel enclosing circle algorithm<br />
<br />
2017-07-18: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
<br />
2017-07-19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
<br />
2017-07-18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
<br />
2017-07-17: redo db operations after cleaning up granted patent number bugs<br />
<br />
2017-07-13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
<br />
2017-07-12: generate some example data illustrating the difficulty of joining different tables<br />
<br />
2017-07-11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
<br />
2017-07-07: catch up on documentation<br />
<br />
2017-07-06: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
<br />
2017-07-05: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
2017-06-30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
<br />
2017-06-29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
<br />
2017-06-28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
<br />
2017-06-27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
<br />
2017-06-26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
<br />
2017-06-25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
<br />
2017-06-23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
<br />
2017-06-22: create postcode<->patent table<br />
<br />
2017-06-29: document granted patent queries and equivalencies<br />
<br />
2017-06-20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
<br />
2017-06-19: skim address regular expressions; cursory investigation of patent table<br />
<br />
2017-06-16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
<br />
2017-06-15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
<br />
2017-06-14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
<br />
2017-06-13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
<br />
2017-06-12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
<br />
2017-06-08: add foreign key inserts; create pretty printer for XML analysis<br />
<br />
2017-06-07: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
<br />
2017-06-06: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
<br />
2017-06-05: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
<br />
2017-06-01: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
2017-05-31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
<br />
2017-05-30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
<br />
2017-05-29: expand to APS; expand to raw assignment data<br />
<br />
2017-05-27: expand to maintenance fee data<br />
<br />
2017-05-26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
<br />
2017-05-25: sketch out OO design of project; download bulk data<br />
<br />
2017-05-24: move wiki pages around; start git repository for project<br />
<br />
2017-05-21: discuss technical details of previous work with Ed<br />
<br />
2017-05-08: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
<br />
2017-05-04: setup wiki account, rdp account, database training<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Python_on_the_RDP&diff=22009Python on the RDP2017-11-21T19:01:18Z<p>OliverC: /* Simultaneously Registering Python 2 and 3 */</p>
<hr />
<div>There are two production versions of Python: Python 2 and Python 3.<br />
<br />
* You can run scripts written for 3 with 2 if you import the special package [https://docs.python.org/2/library/__future__.html <code>__future__</code>]<br />
* You can run scripts written for 2 with 3 if you run the script [https://docs.python.org/3.1/library/2to3.html <code>2to3</code>]<br />
* '''All new code should be written in Python 3''' for library support and maintainability reasons. If for no other reason know that it uses UTF-8 as the default encoding and so it makes your life easier in every possible way.<br />
<br />
== Installation ==<br />
<br />
Many Python modules are actually 90% low-level C code with the remaining 10% a python wrapper.<br />
That means to install these modules (e.g. numpy, scipy, hdf) you need to have all of the development libraries and compiler tools to compile the C stuff.<br />
Enter [https://anaconda.org/ Anaconda], a widely used repackaging of Python for Windows that comes with most of the libraries you'll ever need.<br />
In addition, they've made it easy to have two installs of Python running at once.<br />
<br />
* Python2<br />
** Python 2.7.14 via Anaconda2 5.0.1 for 64-bit<br />
** Located at <code>C:\ProgramData\Anaconda2</code>*<br />
<br />
* Python3<br />
** Python 3.6.3 via Anaconda3 5.0.1 for 64-bit<br />
** Located at <code>C:\ProgramData\Anaconda3</code>*<br />
<br />
<nowiki>*</nowiki> Note that Anaconda is not installed in <code>C:\Program Files\AnacondaN</code> intentionally. Spaces in the path lead to a bunch of unpredictable errors.<br />
<br />
=== Manually Defining Explicit Python Binary Executables ===<br />
<br />
A common practice is to have a bunch of specific Python commands to force explicit behavior. For example on Ubuntu, you can choose from running python, python2, python 2.6, python2.7, python3, python3.5, etc in the shell. To emulate this behavior, copy and paste the <code>python.exe</code> and rename it to <code>pythonN.exe</code> where N is whatever specific version you copied. The same applies to scripts you want to use like <code>pip</code>: copy-and-paste the file to a more explicit name.<br />
<br />
Path is evaluated with the first entries taking precedence. Thus, whichever version of Python you want to be the default should be listed at the leftmost position on the string. You can also just delete the unregistered python.exe to achieve the same result.<br />
<br />
=== ArcMap Aside ===<br />
<br />
Note that by default, ESRI ArcMap maintains a separate Python 2 installation at <code>C:\Python27</code>. Just ignore this...it is not to be used for development or as the environment for running programs other than ArcMap.<br />
<br />
== Simultaneously Registering Python 2 and 3 ==<br />
<br />
Anaconda automatically "registers" the separate installations for use with some development environments. It does not (by default) add executables to the Path system environment. We will add the base <code>AnacondaN/*</code> files and <code>AnacondaN/Scripts/*</code> to the path as so:<br />
<br />
# Right-click on the Windows icon in the bottom left<br />
# Select System<br />
# Select Advanced System Settings (must be administrator)<br />
# Select the Advanced Tab<br />
# Select "Environment Variables..." at the bottom right<br />
# Add semicolon-separated directories to Path in the lower box labeled System Variables<br />
<br />
[[File:Path-example.PNG]]<br />
<br />
== Explicitly Running a Specific Python Version ==<br />
<br />
All of these are valid ways to run Python on the RDP:<br />
<br />
* <code>python</code><br />
* <code>python2</code><br />
* <code>python3</code><br />
<br />
By default, running <code>python</code> will run Python 2. '''This behavior should not be relied on.''' On most other systems, Python 3 is the default instead.<br />
<br />
It is the same case with <code>pip</code> which can be run via <code>pip</code>, <code>pip2</code>, <code>pip3</code></div>OliverChttp://www.edegan.com/mediawiki/index.php?title=File:Path-example.PNG&diff=22008File:Path-example.PNG2017-11-21T19:00:06Z<p>OliverC: </p>
<hr />
<div></div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Python_on_the_RDP&diff=22007Python on the RDP2017-11-21T18:59:40Z<p>OliverC: /* Manually Defining Explicit Python Binary Executables */</p>
<hr />
<div>There are two production versions of Python: Python 2 and Python 3.<br />
<br />
* You can run scripts written for 3 with 2 if you import the special package [https://docs.python.org/2/library/__future__.html <code>__future__</code>]<br />
* You can run scripts written for 2 with 3 if you run the script [https://docs.python.org/3.1/library/2to3.html <code>2to3</code>]<br />
* '''All new code should be written in Python 3''' for library support and maintainability reasons. If for no other reason know that it uses UTF-8 as the default encoding and so it makes your life easier in every possible way.<br />
<br />
== Installation ==<br />
<br />
Many Python modules are actually 90% low-level C code with the remaining 10% a python wrapper.<br />
That means to install these modules (e.g. numpy, scipy, hdf) you need to have all of the development libraries and compiler tools to compile the C stuff.<br />
Enter [https://anaconda.org/ Anaconda], a widely used repackaging of Python for Windows that comes with most of the libraries you'll ever need.<br />
In addition, they've made it easy to have two installs of Python running at once.<br />
<br />
* Python2<br />
** Python 2.7.14 via Anaconda2 5.0.1 for 64-bit<br />
** Located at <code>C:\ProgramData\Anaconda2</code>*<br />
<br />
* Python3<br />
** Python 3.6.3 via Anaconda3 5.0.1 for 64-bit<br />
** Located at <code>C:\ProgramData\Anaconda3</code>*<br />
<br />
<nowiki>*</nowiki> Note that Anaconda is not installed in <code>C:\Program Files\AnacondaN</code> intentionally. Spaces in the path lead to a bunch of unpredictable errors.<br />
<br />
=== Manually Defining Explicit Python Binary Executables ===<br />
<br />
A common practice is to have a bunch of specific Python commands to force explicit behavior. For example on Ubuntu, you can choose from running python, python2, python 2.6, python2.7, python3, python3.5, etc in the shell. To emulate this behavior, copy and paste the <code>python.exe</code> and rename it to <code>pythonN.exe</code> where N is whatever specific version you copied. The same applies to scripts you want to use like <code>pip</code>: copy-and-paste the file to a more explicit name.<br />
<br />
Path is evaluated with the first entries taking precedence. Thus, whichever version of Python you want to be the default should be listed at the leftmost position on the string. You can also just delete the unregistered python.exe to achieve the same result.<br />
<br />
=== ArcMap Aside ===<br />
<br />
Note that by default, ESRI ArcMap maintains a separate Python 2 installation at <code>C:\Python27</code>. Just ignore this...it is not to be used for development or as the environment for running programs other than ArcMap.<br />
<br />
== Simultaneously Registering Python 2 and 3 ==<br />
<br />
Anaconda automatically "registers" the separate installations for use with some development environments. It does not (by default) add executables to the Path system environment. We will add the base <code>AnacondaN/*</code> files and <code>AnacondaN/Scripts/*</code> to the path as so:<br />
<br />
# Right-click on the Windows icon in the bottom left<br />
# Select System<br />
# Select Advanced System Settings (must be administrator)<br />
# Select the Advanced Tab<br />
# Select "Environment Variables..." at the bottom right<br />
# Add semicolon-separated directories to Path in the lower box labeled System Variables <br />
<br />
== Explicitly Running a Specific Python Version ==<br />
<br />
All of these are valid ways to run Python on the RDP:<br />
<br />
* <code>python</code><br />
* <code>python2</code><br />
* <code>python3</code><br />
<br />
By default, running <code>python</code> will run Python 2. '''This behavior should not be relied on.''' On most other systems, Python 3 is the default instead.<br />
<br />
It is the same case with <code>pip</code> which can be run via <code>pip</code>, <code>pip2</code>, <code>pip3</code></div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Python_on_the_RDP&diff=22006Python on the RDP2017-11-21T18:58:47Z<p>OliverC: </p>
<hr />
<div>There are two production versions of Python: Python 2 and Python 3.<br />
<br />
* You can run scripts written for 3 with 2 if you import the special package [https://docs.python.org/2/library/__future__.html <code>__future__</code>]<br />
* You can run scripts written for 2 with 3 if you run the script [https://docs.python.org/3.1/library/2to3.html <code>2to3</code>]<br />
* '''All new code should be written in Python 3''' for library support and maintainability reasons. If for no other reason know that it uses UTF-8 as the default encoding and so it makes your life easier in every possible way.<br />
<br />
== Installation ==<br />
<br />
Many Python modules are actually 90% low-level C code with the remaining 10% a python wrapper.<br />
That means to install these modules (e.g. numpy, scipy, hdf) you need to have all of the development libraries and compiler tools to compile the C stuff.<br />
Enter [https://anaconda.org/ Anaconda], a widely used repackaging of Python for Windows that comes with most of the libraries you'll ever need.<br />
In addition, they've made it easy to have two installs of Python running at once.<br />
<br />
* Python2<br />
** Python 2.7.14 via Anaconda2 5.0.1 for 64-bit<br />
** Located at <code>C:\ProgramData\Anaconda2</code>*<br />
<br />
* Python3<br />
** Python 3.6.3 via Anaconda3 5.0.1 for 64-bit<br />
** Located at <code>C:\ProgramData\Anaconda3</code>*<br />
<br />
<nowiki>*</nowiki> Note that Anaconda is not installed in <code>C:\Program Files\AnacondaN</code> intentionally. Spaces in the path lead to a bunch of unpredictable errors.<br />
<br />
=== Manually Defining Explicit Python Binary Executables ===<br />
<br />
A common practice is to have a bunch of specific Python commands to force explicit behavior. For example on Ubuntu, you can choose from running python, python2, python 2.6, python2.7, python3, python3.5, etc in the shell. To emulate this behavior, copy and paste the <code>python.exe</code> and rename it to <code>pythonN.exe</code> where N is whatever specific version you copied.<br />
<br />
Path is evaluated with the first entries taking precedence. Thus, whichever version of Python you want to be the default should be listed at the leftmost position on the string. You can also just delete the unregistered python.exe to achieve the same result.<br />
<br />
=== ArcMap Aside ===<br />
<br />
Note that by default, ESRI ArcMap maintains a separate Python 2 installation at <code>C:\Python27</code>. Just ignore this...it is not to be used for development or as the environment for running programs other than ArcMap.<br />
<br />
== Simultaneously Registering Python 2 and 3 ==<br />
<br />
Anaconda automatically "registers" the separate installations for use with some development environments. It does not (by default) add executables to the Path system environment. We will add the base <code>AnacondaN/*</code> files and <code>AnacondaN/Scripts/*</code> to the path as so:<br />
<br />
# Right-click on the Windows icon in the bottom left<br />
# Select System<br />
# Select Advanced System Settings (must be administrator)<br />
# Select the Advanced Tab<br />
# Select "Environment Variables..." at the bottom right<br />
# Add semicolon-separated directories to Path in the lower box labeled System Variables <br />
<br />
== Explicitly Running a Specific Python Version ==<br />
<br />
All of these are valid ways to run Python on the RDP:<br />
<br />
* <code>python</code><br />
* <code>python2</code><br />
* <code>python3</code><br />
<br />
By default, running <code>python</code> will run Python 2. '''This behavior should not be relied on.''' On most other systems, Python 3 is the default instead.<br />
<br />
It is the same case with <code>pip</code> which can be run via <code>pip</code>, <code>pip2</code>, <code>pip3</code></div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Python_on_the_RDP&diff=22005Python on the RDP2017-11-21T18:42:57Z<p>OliverC: Created page with "There are two production versions of Python: Python 2 and Python 3. * You can run scripts written for 3 with 2 if you import the special package [https://docs.python.org/2/li..."</p>
<hr />
<div>There are two production versions of Python: Python 2 and Python 3.<br />
<br />
* You can run scripts written for 3 with 2 if you import the special package [https://docs.python.org/2/library/__future__.html <code>__future__</code>]<br />
* You can run scripts written for 2 with 3 if you run the script [https://docs.python.org/3.1/library/2to3.html <code>2to3</code>]<br />
* '''All new code should be written in Python 3''' for library support and maintainability reasons. If for no other reason know that it uses UTF-8 as the default encoding and so it makes your life easier in every possible way.<br />
<br />
== Installation ==<br />
<br />
Many Python modules are actually 90% low-level C code with the remaining 10% a python wrapper.<br />
That means to install these modules (e.g. numpy, scipy, hdf) you need to have all of the development libraries and compiler tools to compile the C stuff.<br />
Enter [https://anaconda.org/ Anaconda], a widely used repackaging of Python for Windows that comes with most of the libraries you'll ever need.<br />
In addition, they've made it easy to have two installs of Python running at once.<br />
<br />
* Python2<br />
** Python 2.7.14 via Anaconda2 5.0.1 for 64-bit<br />
** Located at <code>C:\ProgramData\Anaconda2</code>*<br />
<br />
* Python3<br />
** Python 3.6.3 via Anaconda3 5.0.1 for 64-bit<br />
** Located at <code>C:\ProgramData\Anaconda3</code>*<br />
<br />
<nowiki>*</nowiki> Note that Anaconda is not installed in <code>C:\Program Files\AnacondaN</code> intentionally. Spaces in the path lead to a bunch of unpredictable errors.</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=21884Oliver Chang (Work Log)2017-11-15T03:24:19Z<p>OliverC: /* Fall 2017 */</p>
<hr />
<div>''Projects'':<br />
<br />
* [[PostGIS_Installation|PostGIS Installation]]<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
<br />
''Uploads'':<br />
<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
===Fall 2017=== <br />
<onlyinclude><br />
<br />
[[Oliver Chang]] [[Work Logs]] [[Oliver Chang (Work Log)|(log page)]]<br />
<br />
2017-11-14: hand off work on xpathing to Shelby; walked through some program design decisions<br />
<br />
2017-11-13: finish javadoc of common/ and some trickier parts about downloading; added descriptions, results to one-off java/python scripts so that they actually make sense in context<br />
<br />
2017-11-10: add javadoc documentation to patent reproducibility project after forgetting half of the stuff myself<br />
<br />
2017-10-25: start ingestion of application xml files and deal with all the bugs which accompany that<br />
<br />
2017-10-21: create xml explorer script to mass-inspect xpaths (can be found at <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\xml_schema_explorer</code>)<br />
<br />
2017-10-20: re-run assignment import, review xpaths<br />
<br />
2017-10-19: repopulate patents database and create spreadsheet of assignment equivalencies<br />
<br />
2017-10-11: add issues to work on to the PECA wiki page; cleanup PECA git repo; start patent application code from granted patent code and start customizing to new domain<br />
<br />
2017-10-03: troubleshoot vc_circles.py and make command line interface a little nicer<br />
<br />
2017-10-02: discuss mapping strategies & investigate missing eca data<br />
<br />
2017-09-23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
<br />
2017-09-22: goal setting & server debugging & meet with Yang<br />
</onlyinclude><br />
<br />
===Summer 2017=== <br />
<br />
2017-08-04: setup parallel instance python framework for job reporting; begin test run<br />
<br />
2017-08-02: finish up some documentation of the code and for the wiki<br />
<br />
2017-08-01: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
2017-07-31: sketch out parallel enclosing circle algorithm<br />
<br />
2017-07-18: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
<br />
2017-07-19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
<br />
2017-07-18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
<br />
2017-07-17: redo db operations after cleaning up granted patent number bugs<br />
<br />
2017-07-13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
<br />
2017-07-12: generate some example data illustrating the difficulty of joining different tables<br />
<br />
2017-07-11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
<br />
2017-07-07: catch up on documentation<br />
<br />
2017-07-06: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
<br />
2017-07-05: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
2017-06-30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
<br />
2017-06-29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
<br />
2017-06-28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
<br />
2017-06-27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
<br />
2017-06-26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
<br />
2017-06-25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
<br />
2017-06-23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
<br />
2017-06-22: create postcode<->patent table<br />
<br />
2017-06-29: document granted patent queries and equivalencies<br />
<br />
2017-06-20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
<br />
2017-06-19: skim address regular expressions; cursory investigation of patent table<br />
<br />
2017-06-16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
<br />
2017-06-15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
<br />
2017-06-14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
<br />
2017-06-13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
<br />
2017-06-12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
<br />
2017-06-08: add foreign key inserts; create pretty printer for XML analysis<br />
<br />
2017-06-07: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
<br />
2017-06-06: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
<br />
2017-06-05: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
<br />
2017-06-01: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
2017-05-31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
<br />
2017-05-30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
<br />
2017-05-29: expand to APS; expand to raw assignment data<br />
<br />
2017-05-27: expand to maintenance fee data<br />
<br />
2017-05-26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
<br />
2017-05-25: sketch out OO design of project; download bulk data<br />
<br />
2017-05-24: move wiki pages around; start git repository for project<br />
<br />
2017-05-21: discuss technical details of previous work with Ed<br />
<br />
2017-05-08: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
<br />
2017-05-04: setup wiki account, rdp account, database training<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Reproducible_Patent_Data&diff=21832Reproducible Patent Data2017-11-14T16:47:20Z<p>OliverC: /* New Table Checklist */</p>
<hr />
<div>{{McNair Projects<br />
|Has Image=Uspto web logo.jpg<br />
|Has title=Reproducible Patent Data<br />
|Has owner=Oliver Chang<br />
|Has start date=May 17<br />
|Has project status=Active<br />
|Does subsume=Redesigning Patent Database, Patent Assignment Data Restructure,<br />
}}<br />
<br />
A continuation of [[Redesigning Patent Database]] that aims to write faster, more centralized code to deal with data from the United States Patent and Trademark Office (USPTO). By having an end-to-end pipeline we can easily reproduce or update data without worrying about unintentional side effects or missing data.<br />
<br />
== Quickstart ==<br />
<br />
To get up and running with the code, do the following:<br />
<br />
# Clone the git project (link at end of page) to your user directory<br />
# Launch IntelliJ with >= Java 8 and Maven configured (default version installed on the RDP is setup to do this)<br />
# Open project in IntelliJ<br />
# Create an empty database (see [[#Database]])<br />
# Run the table creation scripts in <code>src/db/schemas/</code> in your new database<br />
# Modify the constant <code>DATABASE_NAME</code> in <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\postgres\DatabaseHelper.java</code><br />
# Run the Driver scripts in IntelliJ with the correct value for <code>DATA_DIRECTORY</code> (or run <code>RunInitialImport.java</code> which will do all of the data directories for that patent item type)<br />
# [Take a really, really long lunch...in total should take no more than five hours to load data on RDP]<br />
# Run scripts in <code>src/db/constraints</code> to check data assumptions<br />
# That's it!<br />
<br />
== Directory Layout ==<br />
<br />
=== Where is the Data? ===<br />
<br />
==== Directories ====<br />
<br />
All of the information for this project is located at <code>E:\McNair\Projects\SimplerPatentData</code><br />
<br />
There are several interesting directories:<br />
<br />
* <code>data/downloads/</code> is USPTO bulkdata, unmodified straight from the scraper<br />
* <code>data/extracts/</code> is a directory of a strict subset of the information stored in <code>data/downloads/</code>. It is the result of running a bulk 7-zip job on that directory to get everything unzipped in a flat data structure. Note that these files have the USPTO modified-by time since that metadata is stored in the zipfiles. To extract files in this nice format, select all of the zipfiles and setup an extraction job like in this [[media:7zip-params.png|screenshot]]<br />
* <code>data/backups/</code> is a 7zip'd backup of the corresponding directory in extracts<br />
* <code>src/</code> is the main code repository for the java project<br />
<br />
==== Input Files ====<br />
<br />
All of the text-only Red Book files for '''granted patents''' from 1976 to 2016, inclusive. To find a specific year's XML file, find it in<br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\granted\</code><br />
<br />
'''To find application data''' from 2001 to 2016, inclusive, look in<br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\applications\</code><br />
<br />
'''To find assignment data''', look in <br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\granted\</code><br />
<br />
'''To find maintenance fee data''', look in <br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\downloads\maintenance\</code><br />
<br />
=== Where is the Code? ===<br />
<br />
The code has the same parent directory as the data, so it is at <code>E:\McNair\Projects\SimplerPatentData\src</code>. You might notice a lot of single-entry directories; this is an idiomatic Java pattern that is used for package separation. If using IntelliJ or some other IDE, these directories are a bit less annoying.<br />
<br />
The development environment is Java 8 JDK, IntelliJ Ultimate IDE, Maven build tools, and git VCS.<br />
<br />
The git repository can be found at http://rdp.mcnaircenter.org/codebase/Repository/ReproduciblePatent<br />
<br />
==== Prior Art ====<br />
<br />
This tool is not so concerned with adding new functionality; rather, it aims to take a bunch of spread out Perl scripts and create a faster system that is easier to work with. As such, its functionality is largely stolen from those scripts:<br />
<br />
* Downloader: <code>E:\McNair\Software\Scripts\Patent\USPTO_Parser.pl</code><br />
* XML Splitter: <code>E:\McNair\PatentData\splitter.pl</code><br />
* XML Parsing: <code>E:\McNair\PatentData\Processed\xmlparser_4.5_4.4_4.3.pl</code> and <code>E:\McNair\PatentData\Processed\*.pm</code><br />
<br />
In addition, I used several non-standard Java libraries listed below:<br />
<br />
* [https://github.com/mashape/unirest-java/ Unirest] for easy HTTP requests (MIT License)<br />
* [https://github.com/google/guava Google Guava] for immutable collections and Stream utilities (Apache v2.0 License)<br />
* [https://github.com/jhy/jsoup/ jsoup] for HTML parsing (MIT License)<br />
* [http://commons.apache.org/proper/commons-codec/ Apache Commons Codec] (Apache v2.0 License)<br />
* [http://commons.apache.org/proper/commons-lang/ Apache Commons Lang v3] (Apache v2.0 License)<br />
* [https://mvnrepository.com/artifact/org.jetbrains/annotations/15.0 Jetbrains Annotations] for enhanced null checks (Apache v2.0 License)<br />
* [http://search.maven.org/#artifactdetails%7Corg.postgresql%7Cpostgresql%7C42.1.1.jre7%7Cbundle PostgreSQL JDBC] (BSD 3-clause per https://github.com/pgjdbc/pgjdbc-jre7/blob/master/LICENSE)<br />
<br />
If using maven, these dependencies are listed and should automatically be setup.<br />
<br />
==== Using Code ====<br />
<br />
Any file with a line that says <code>public static void main(String[] args) {</code> can be run as a standalone file. The easiest way to do this is to load the project and then the file in IntelliJ and click the little green play arrow next to this bit of code.<br />
<br />
The code can also be run via the standard <code>javac</code> and <code>java</code> commands but since this project has a complicated structure you end up having to run commands like <br />
<br />
<code>"C:\Program Files\Java\jdk1.8.0_131\bin\java" "-javaagent:C:\Users\OliverC\IntelliJ IDEA 2017.1.3\lib\idea_rt.jar=62364:C:\Users\OliverC\IntelliJ IDEA 2017.1.3\bin" -Dfile.encoding=UTF-8 -classpath "[...contents truncated...];C:\Users\OliverC\.m2\repository\org\postgresql\postgresql\42.1.1\postgresql-42.1.1.jar" org.bakerinstitute.mcnair.uspto_assignments.XmlDriver</code><br />
<br />
to include all of the runtime dependencies and it's just not worth it.<br />
<br />
==== Altering Code ====<br />
<br />
* Use the IntelliJ command Reformat code (found in the menus at <code>Code > Reformat Code</code><br />
* Use the optimize imports function found under the same menu<br />
* Use spaces for indentation<br />
* Loosely try to keep lines below 120 characters<br />
* Commit changes to the Git remote repository "bonobo"<br />
<br />
== Schema Reconciliation ==<br />
<br />
For the work by Joe, see the [[Patent Schema Reconciliation]] page <br />
<br />
=== Patents (Granted) ===<br />
<br />
See <code>E:\McNair\Projects\SimplerPatentData\data\examples\granted</code> for extracted examples of what specific data is available for a sample of the data.<br />
<br />
{| class="wikitable"<br />
|+Granted Patent Data Formats<br />
|-<br />
! scope="col" | Dates Used !! scope="col" | Format !! scope="col" | Location !! scope="col" | Supported? !! scope="col" | Utility !! scope="col" | Reissue !! scope="col" | Design !! scope="col" | Plant<br />
|-<br />
|January 1976 to December 2001<br />
|APS<br />
|<code>data/extracts/granted/vintage</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|<del>January 2001 to December 2001</del><br />
|<del>SGML</del><br />
|Ignored; use concurrently recorded APS data<br />
|No<br />
|N/A<br />
|N/A<br />
|N/A<br />
|N/A<br />
|-<br />
|January 2002 to December 2004<br />
|XML Version 2.5<br />
|<code>data/extracts/granted/blunderyears</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2005 to December 2005<br />
|XML Version 4.0 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2006 to December 2006<br />
|XML Version 4.1 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2007 to December 2012<br />
|XML Version 4.2 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2013 to September 24, 2013<br />
|XML Version 4.3 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|October 8, 2013 to December 2014<br />
|XML Version 4.4 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2015 to December 2016<br />
|XML Version 4.5 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|}<br />
<br />
=== APS Rosetta Stone ===<br />
<br />
The Advanced Patent System (APS) is a fixed-width text format used to store historical patent grant data. The documentation for this sucks; there are pages missing at random. Luckily, we only care about the content contained here: [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]].<br />
<br />
It's worth mentioning that the APS contains an advanced text markup system for chemical formulae, basic text markup, tables, etc. that can lead to seemingly garbled text that is perfectly well-formed.<br />
<br />
==== APS Gotchas ====<br />
<br />
* PATN.WKU is the granted patent number. It is 7 digits while the spec promises 6 digits. The rightmost digit is a check digit modulus 11. See [[File:Aps-wku-modulus11.pdf]] for the words from the horse's mouth.<br />
<br />
=== Patents (Applications) ===<br />
<br />
{| class="wikitable"<br />
|+Patent Application Data Formats<br />
|-<br />
! scope="col" | Dates Used !! scope="col" | Format !! scope="col" | Location !! scope="col" | Supported by Parser?<br />
|-<br />
|March 15, 2001 to December 2001<br />
|XML Version 1.5<br />
|<code>data/extracts/applications/vintage</code><br />
|style="background: yellow;" | Yes, for basic information, inventors, and correspondents<br />
|-<br />
|January 2002 to December 2004<br />
|XML Version 1.6<br />
|<code>data/extracts/applications/vintage</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2005 to December 2005<br />
|XML Version 4.0 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2006 to December 2006<br />
|XML Version 4.1 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2007 to December 2012<br />
|XML Version 4.2 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2013 to December 2014<br />
|XML Version 4.3 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2015 to ''Present''<br />
|XML Version 4.4 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|}<br />
<br />
== Database ==<br />
<br />
Because there isn't a compelling reason not to, I used the existing PostgreSQL infrastructure on the RDP.<br />
The "Java Way" of interacting with databases is the Java Database Connectivity API (JDBC), an implementation-agnostic API for interacting with databases.<br />
This project uses the stock [https://jdbc.postgresql.org/ Postgres JDBC], version 42.1.1<br />
<br />
=== Create an empty database on RDP ===<br />
<br />
To create an empty database, run this command: <code>$ createdb --username=postgres database-name-goes-here # password is tabspaceenter</code><br />
<br />
=== Abstraction Layer ===<br />
<br />
Since writing raw SQL is a bit cumbersome and error-prone, I have added some abstraction layers that make it much easier to quickly add bulk data. By using Postgres's <code>CopyManager</code> class, we buffer SQL copy commands in memory (as many as possible) and then flush these rows. To understand how the abstraction layers work, see the code in <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\postgres</code>. See <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\models\GrantedPatent.java</code> for '''an example of how to extend''' the abstraction layer to deal with more complex scenarios.<br />
<br />
=== New Table Checklist ===<br />
<br />
* Create schema DDL SQL code for the new table in <code>E:\McNair\Projects\SimplerPatentData\src\db</code><br />
* Run the schema creation<br />
* Create an enum with the same names for attributes as in the DDL (case-insensitive! prefer all-caps screaming snake case)<br />
* Create a class which subclasses <code>AbstractInsertableData</code><br />
* Inside that class, create a static class which subclasses <code>AbstractTableMetadata</code> and has the proper values for getTableName(), getStringColumns(), getIntColumns()<br />
* (Optional) Implement builder pattern<br />
* (Optional) Create a custom databasehelper for complex extras (see PatentApplication and GrantedPatent for examples)<br />
* Write the data to the table (see DatabaseHelper for the pattern I use)<br />
<br />
== Address Data ==<br />
<br />
To get the most granular address data (street level, or at least postcode level) about who owns patents, the path is not so straightforward because off the complicated mapping of ownership to a granted patent.<br />
This is the final part of this project that I am working on and it is all at the level of SQL.<br />
See <code>E:\McNair\Projects\SimplerPatentData\src\db\joins</code> for my attempts to create a clean mapping.<br />
Optimistically speaking, the data generated here should be superset of the data present in the Patent Assignment Data Restructure project.<br />
<br />
Note that as of the beginning of August 2017, this part '''has not been completed.'''<br />
<br />
<br />
=== Intuition ===<br />
<br />
Use <code>assignments_longform.last_update_date</code> to find current/latest (or first/earliest) date of assignment. Then match with <code>properties.docid</code> on <code>reelno, frameno</code> to find patent application id. With this mapping to granted patents, we can discover the details of the original granted patent. And with the right date and reelno and frameno, we can match to the <code>assignees</code> table and get fine granularity addresses.<br />
<br />
== Related Pages ==<br />
<br />
* [[Redesign_Assignment_and_Patent_Database|Redesign Assignment and Patent Database, Fall 2017 by Shelby]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries, Summer 2017 by Oliver & Joe]]<br />
* [[US_Address_Verification|US Address Verification, Summer 2017 based on tables from Assignment Data Restructure]]<br />
* [[Patent_Assignment_Data_Restructure|Assignment Data Restructure, Spring 2017 by Marcela and Sonia]]<br />
* [[Redesigning_Patent_Database|Redesigning Patent Database, Spring 2017 by Shelby]]<br />
* [[Patent_Data_Cleanup_(June_2016)|Patent Data Cleanup, June 2016 by Marcela]]<br />
* [[Patent_Data|Patent Data, Spring 2016 by Marcela]] <br />
* [[Lex_Machina|Lex Machina]]<br />
* [[USPTO_Patent_Litigation_Data|USPTO Patent Litigation Research Dataset by Ed]]<br />
* [[Patent_Litigation_and_Review|Patent Litigation and Review by Marcela]]<br />
* [[Bag_of_Words_Analysis_of_Patent_Data|Bag of Words Analysis]]<br />
* [[Patent|Existing Database Schema]]<br />
* [[Oliver_Chang_(Work_Log)|My Work Log]]<br />
<br />
== External Links ==<br />
<br />
* Understanding Assignment Data: [https://www.uspto.gov/sites/default/files/documents/USPTO_Patents_Assignment_Dataset_WP.pdf USPTO Documentation on their cleanup of this data]<br />
* [https://bulkdata.uspto.gov/data2/patent/grant/redbook/fulltext/1976/PatentFullTextAPSDoc_GreenBook.pdf USPTO Green Book (APS) Documentation]<br />
* [https://bulkdata.uspto.gov/ USPTO Bulk Data Storage System (BDSS)]<br />
* [https://en.wikipedia.org/wiki/Builder_pattern Builder Pattern in Object-Oriented Programming]<br />
* [http://rdp.mcnaircenter.org/codebase/Repository/ReproduciblePatent Git Repository]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Reproducible_Patent_Data&diff=21830Reproducible Patent Data2017-11-14T16:44:35Z<p>OliverC: </p>
<hr />
<div>{{McNair Projects<br />
|Has Image=Uspto web logo.jpg<br />
|Has title=Reproducible Patent Data<br />
|Has owner=Oliver Chang<br />
|Has start date=May 17<br />
|Has project status=Active<br />
|Does subsume=Redesigning Patent Database, Patent Assignment Data Restructure,<br />
}}<br />
<br />
A continuation of [[Redesigning Patent Database]] that aims to write faster, more centralized code to deal with data from the United States Patent and Trademark Office (USPTO). By having an end-to-end pipeline we can easily reproduce or update data without worrying about unintentional side effects or missing data.<br />
<br />
== Quickstart ==<br />
<br />
To get up and running with the code, do the following:<br />
<br />
# Clone the git project (link at end of page) to your user directory<br />
# Launch IntelliJ with >= Java 8 and Maven configured (default version installed on the RDP is setup to do this)<br />
# Open project in IntelliJ<br />
# Create an empty database (see [[#Database]])<br />
# Run the table creation scripts in <code>src/db/schemas/</code> in your new database<br />
# Modify the constant <code>DATABASE_NAME</code> in <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\postgres\DatabaseHelper.java</code><br />
# Run the Driver scripts in IntelliJ with the correct value for <code>DATA_DIRECTORY</code> (or run <code>RunInitialImport.java</code> which will do all of the data directories for that patent item type)<br />
# [Take a really, really long lunch...in total should take no more than five hours to load data on RDP]<br />
# Run scripts in <code>src/db/constraints</code> to check data assumptions<br />
# That's it!<br />
<br />
== Directory Layout ==<br />
<br />
=== Where is the Data? ===<br />
<br />
==== Directories ====<br />
<br />
All of the information for this project is located at <code>E:\McNair\Projects\SimplerPatentData</code><br />
<br />
There are several interesting directories:<br />
<br />
* <code>data/downloads/</code> is USPTO bulkdata, unmodified straight from the scraper<br />
* <code>data/extracts/</code> is a directory of a strict subset of the information stored in <code>data/downloads/</code>. It is the result of running a bulk 7-zip job on that directory to get everything unzipped in a flat data structure. Note that these files have the USPTO modified-by time since that metadata is stored in the zipfiles. To extract files in this nice format, select all of the zipfiles and setup an extraction job like in this [[media:7zip-params.png|screenshot]]<br />
* <code>data/backups/</code> is a 7zip'd backup of the corresponding directory in extracts<br />
* <code>src/</code> is the main code repository for the java project<br />
<br />
==== Input Files ====<br />
<br />
All of the text-only Red Book files for '''granted patents''' from 1976 to 2016, inclusive. To find a specific year's XML file, find it in<br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\granted\</code><br />
<br />
'''To find application data''' from 2001 to 2016, inclusive, look in<br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\applications\</code><br />
<br />
'''To find assignment data''', look in <br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\granted\</code><br />
<br />
'''To find maintenance fee data''', look in <br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\downloads\maintenance\</code><br />
<br />
=== Where is the Code? ===<br />
<br />
The code has the same parent directory as the data, so it is at <code>E:\McNair\Projects\SimplerPatentData\src</code>. You might notice a lot of single-entry directories; this is an idiomatic Java pattern that is used for package separation. If using IntelliJ or some other IDE, these directories are a bit less annoying.<br />
<br />
The development environment is Java 8 JDK, IntelliJ Ultimate IDE, Maven build tools, and git VCS.<br />
<br />
The git repository can be found at http://rdp.mcnaircenter.org/codebase/Repository/ReproduciblePatent<br />
<br />
==== Prior Art ====<br />
<br />
This tool is not so concerned with adding new functionality; rather, it aims to take a bunch of spread out Perl scripts and create a faster system that is easier to work with. As such, its functionality is largely stolen from those scripts:<br />
<br />
* Downloader: <code>E:\McNair\Software\Scripts\Patent\USPTO_Parser.pl</code><br />
* XML Splitter: <code>E:\McNair\PatentData\splitter.pl</code><br />
* XML Parsing: <code>E:\McNair\PatentData\Processed\xmlparser_4.5_4.4_4.3.pl</code> and <code>E:\McNair\PatentData\Processed\*.pm</code><br />
<br />
In addition, I used several non-standard Java libraries listed below:<br />
<br />
* [https://github.com/mashape/unirest-java/ Unirest] for easy HTTP requests (MIT License)<br />
* [https://github.com/google/guava Google Guava] for immutable collections and Stream utilities (Apache v2.0 License)<br />
* [https://github.com/jhy/jsoup/ jsoup] for HTML parsing (MIT License)<br />
* [http://commons.apache.org/proper/commons-codec/ Apache Commons Codec] (Apache v2.0 License)<br />
* [http://commons.apache.org/proper/commons-lang/ Apache Commons Lang v3] (Apache v2.0 License)<br />
* [https://mvnrepository.com/artifact/org.jetbrains/annotations/15.0 Jetbrains Annotations] for enhanced null checks (Apache v2.0 License)<br />
* [http://search.maven.org/#artifactdetails%7Corg.postgresql%7Cpostgresql%7C42.1.1.jre7%7Cbundle PostgreSQL JDBC] (BSD 3-clause per https://github.com/pgjdbc/pgjdbc-jre7/blob/master/LICENSE)<br />
<br />
If using maven, these dependencies are listed and should automatically be setup.<br />
<br />
==== Using Code ====<br />
<br />
Any file with a line that says <code>public static void main(String[] args) {</code> can be run as a standalone file. The easiest way to do this is to load the project and then the file in IntelliJ and click the little green play arrow next to this bit of code.<br />
<br />
The code can also be run via the standard <code>javac</code> and <code>java</code> commands but since this project has a complicated structure you end up having to run commands like <br />
<br />
<code>"C:\Program Files\Java\jdk1.8.0_131\bin\java" "-javaagent:C:\Users\OliverC\IntelliJ IDEA 2017.1.3\lib\idea_rt.jar=62364:C:\Users\OliverC\IntelliJ IDEA 2017.1.3\bin" -Dfile.encoding=UTF-8 -classpath "[...contents truncated...];C:\Users\OliverC\.m2\repository\org\postgresql\postgresql\42.1.1\postgresql-42.1.1.jar" org.bakerinstitute.mcnair.uspto_assignments.XmlDriver</code><br />
<br />
to include all of the runtime dependencies and it's just not worth it.<br />
<br />
==== Altering Code ====<br />
<br />
* Use the IntelliJ command Reformat code (found in the menus at <code>Code > Reformat Code</code><br />
* Use the optimize imports function found under the same menu<br />
* Use spaces for indentation<br />
* Loosely try to keep lines below 120 characters<br />
* Commit changes to the Git remote repository "bonobo"<br />
<br />
== Schema Reconciliation ==<br />
<br />
For the work by Joe, see the [[Patent Schema Reconciliation]] page <br />
<br />
=== Patents (Granted) ===<br />
<br />
See <code>E:\McNair\Projects\SimplerPatentData\data\examples\granted</code> for extracted examples of what specific data is available for a sample of the data.<br />
<br />
{| class="wikitable"<br />
|+Granted Patent Data Formats<br />
|-<br />
! scope="col" | Dates Used !! scope="col" | Format !! scope="col" | Location !! scope="col" | Supported? !! scope="col" | Utility !! scope="col" | Reissue !! scope="col" | Design !! scope="col" | Plant<br />
|-<br />
|January 1976 to December 2001<br />
|APS<br />
|<code>data/extracts/granted/vintage</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|<del>January 2001 to December 2001</del><br />
|<del>SGML</del><br />
|Ignored; use concurrently recorded APS data<br />
|No<br />
|N/A<br />
|N/A<br />
|N/A<br />
|N/A<br />
|-<br />
|January 2002 to December 2004<br />
|XML Version 2.5<br />
|<code>data/extracts/granted/blunderyears</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2005 to December 2005<br />
|XML Version 4.0 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2006 to December 2006<br />
|XML Version 4.1 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2007 to December 2012<br />
|XML Version 4.2 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2013 to September 24, 2013<br />
|XML Version 4.3 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|October 8, 2013 to December 2014<br />
|XML Version 4.4 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2015 to December 2016<br />
|XML Version 4.5 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|}<br />
<br />
=== APS Rosetta Stone ===<br />
<br />
The Advanced Patent System (APS) is a fixed-width text format used to store historical patent grant data. The documentation for this sucks; there are pages missing at random. Luckily, we only care about the content contained here: [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]].<br />
<br />
It's worth mentioning that the APS contains an advanced text markup system for chemical formulae, basic text markup, tables, etc. that can lead to seemingly garbled text that is perfectly well-formed.<br />
<br />
==== APS Gotchas ====<br />
<br />
* PATN.WKU is the granted patent number. It is 7 digits while the spec promises 6 digits. The rightmost digit is a check digit modulus 11. See [[File:Aps-wku-modulus11.pdf]] for the words from the horse's mouth.<br />
<br />
=== Patents (Applications) ===<br />
<br />
{| class="wikitable"<br />
|+Patent Application Data Formats<br />
|-<br />
! scope="col" | Dates Used !! scope="col" | Format !! scope="col" | Location !! scope="col" | Supported by Parser?<br />
|-<br />
|March 15, 2001 to December 2001<br />
|XML Version 1.5<br />
|<code>data/extracts/applications/vintage</code><br />
|style="background: yellow;" | Yes, for basic information, inventors, and correspondents<br />
|-<br />
|January 2002 to December 2004<br />
|XML Version 1.6<br />
|<code>data/extracts/applications/vintage</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2005 to December 2005<br />
|XML Version 4.0 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2006 to December 2006<br />
|XML Version 4.1 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2007 to December 2012<br />
|XML Version 4.2 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2013 to December 2014<br />
|XML Version 4.3 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2015 to ''Present''<br />
|XML Version 4.4 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|}<br />
<br />
== Database ==<br />
<br />
Because there isn't a compelling reason not to, I used the existing PostgreSQL infrastructure on the RDP.<br />
The "Java Way" of interacting with databases is the Java Database Connectivity API (JDBC), an implementation-agnostic API for interacting with databases.<br />
This project uses the stock [https://jdbc.postgresql.org/ Postgres JDBC], version 42.1.1<br />
<br />
=== Create an empty database on RDP ===<br />
<br />
To create an empty database, run this command: <code>$ createdb --username=postgres database-name-goes-here # password is tabspaceenter</code><br />
<br />
=== Abstraction Layer ===<br />
<br />
Since writing raw SQL is a bit cumbersome and error-prone, I have added some abstraction layers that make it much easier to quickly add bulk data. By using Postgres's <code>CopyManager</code> class, we buffer SQL copy commands in memory (as many as possible) and then flush these rows. To understand how the abstraction layers work, see the code in <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\postgres</code>. See <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\models\GrantedPatent.java</code> for '''an example of how to extend''' the abstraction layer to deal with more complex scenarios.<br />
<br />
=== New Table Checklist ===<br />
<br />
* Create schema DDL SQL code for the new table in <code>E:\McNair\Projects\SimplerPatentData\src\db</code><br />
* Run the schema creation<br />
* Create an enum with the same names for attributes as in the DDL (case-insensitive! prefer all-caps screaming snake case)<br />
* Create a class which subclasses <code>AbstractIntertableData</code><br />
* Inside that class, create a static class which subclasses <code>AbstractTableMetadata</code> and has the proper values for getTableName(), getStringColumns(), getIntColumns()<br />
* (Optional) Implement builder pattern<br />
* (Optional) Create a custom databasehelper for complex extras (see PatentApplication and GrantedPatent for examples)<br />
* Write the data to the table (see DatabaseHelper for the pattern I use)<br />
<br />
== Address Data ==<br />
<br />
To get the most granular address data (street level, or at least postcode level) about who owns patents, the path is not so straightforward because off the complicated mapping of ownership to a granted patent.<br />
This is the final part of this project that I am working on and it is all at the level of SQL.<br />
See <code>E:\McNair\Projects\SimplerPatentData\src\db\joins</code> for my attempts to create a clean mapping.<br />
Optimistically speaking, the data generated here should be superset of the data present in the Patent Assignment Data Restructure project.<br />
<br />
Note that as of the beginning of August 2017, this part '''has not been completed.'''<br />
<br />
<br />
=== Intuition ===<br />
<br />
Use <code>assignments_longform.last_update_date</code> to find current/latest (or first/earliest) date of assignment. Then match with <code>properties.docid</code> on <code>reelno, frameno</code> to find patent application id. With this mapping to granted patents, we can discover the details of the original granted patent. And with the right date and reelno and frameno, we can match to the <code>assignees</code> table and get fine granularity addresses.<br />
<br />
== Related Pages ==<br />
<br />
* [[Redesign_Assignment_and_Patent_Database|Redesign Assignment and Patent Database, Fall 2017 by Shelby]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries, Summer 2017 by Oliver & Joe]]<br />
* [[US_Address_Verification|US Address Verification, Summer 2017 based on tables from Assignment Data Restructure]]<br />
* [[Patent_Assignment_Data_Restructure|Assignment Data Restructure, Spring 2017 by Marcela and Sonia]]<br />
* [[Redesigning_Patent_Database|Redesigning Patent Database, Spring 2017 by Shelby]]<br />
* [[Patent_Data_Cleanup_(June_2016)|Patent Data Cleanup, June 2016 by Marcela]]<br />
* [[Patent_Data|Patent Data, Spring 2016 by Marcela]] <br />
* [[Lex_Machina|Lex Machina]]<br />
* [[USPTO_Patent_Litigation_Data|USPTO Patent Litigation Research Dataset by Ed]]<br />
* [[Patent_Litigation_and_Review|Patent Litigation and Review by Marcela]]<br />
* [[Bag_of_Words_Analysis_of_Patent_Data|Bag of Words Analysis]]<br />
* [[Patent|Existing Database Schema]]<br />
* [[Oliver_Chang_(Work_Log)|My Work Log]]<br />
<br />
== External Links ==<br />
<br />
* Understanding Assignment Data: [https://www.uspto.gov/sites/default/files/documents/USPTO_Patents_Assignment_Dataset_WP.pdf USPTO Documentation on their cleanup of this data]<br />
* [https://bulkdata.uspto.gov/data2/patent/grant/redbook/fulltext/1976/PatentFullTextAPSDoc_GreenBook.pdf USPTO Green Book (APS) Documentation]<br />
* [https://bulkdata.uspto.gov/ USPTO Bulk Data Storage System (BDSS)]<br />
* [https://en.wikipedia.org/wiki/Builder_pattern Builder Pattern in Object-Oriented Programming]<br />
* [http://rdp.mcnaircenter.org/codebase/Repository/ReproduciblePatent Git Repository]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=21814Oliver Chang (Work Log)2017-11-14T04:23:28Z<p>OliverC: /* Fall 2017 */</p>
<hr />
<div>''Projects'':<br />
<br />
* [[PostGIS_Installation|PostGIS Installation]]<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
<br />
''Uploads'':<br />
<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
===Fall 2017=== <br />
<onlyinclude><br />
<br />
[[Oliver Chang]] [[Work Logs]] [[Oliver Chang (Work Log)|(log page)]]<br />
<br />
2017-11-13: finish javadoc of common/ and some trickier parts about downloading; added descriptions, results to one-off java/python scripts so that they actually make sense in context<br />
<br />
2017-11-10: add javadoc documentation to patent reproducibility project after forgetting half of the stuff myself<br />
<br />
2017-10-25: start ingestion of application xml files and deal with all the bugs which accompany that<br />
<br />
2017-10-21: create xml explorer script to mass-inspect xpaths (can be found at <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\xml_schema_explorer</code>)<br />
<br />
2017-10-20: re-run assignment import, review xpaths<br />
<br />
2017-10-19: repopulate patents database and create spreadsheet of assignment equivalencies<br />
<br />
2017-10-11: add issues to work on to the PECA wiki page; cleanup PECA git repo; start patent application code from granted patent code and start customizing to new domain<br />
<br />
2017-10-03: troubleshoot vc_circles.py and make command line interface a little nicer<br />
<br />
2017-10-02: discuss mapping strategies & investigate missing eca data<br />
<br />
2017-09-23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
<br />
2017-09-22: goal setting & server debugging & meet with Yang<br />
</onlyinclude><br />
<br />
===Summer 2017=== <br />
<br />
2017-08-04: setup parallel instance python framework for job reporting; begin test run<br />
<br />
2017-08-02: finish up some documentation of the code and for the wiki<br />
<br />
2017-08-01: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
2017-07-31: sketch out parallel enclosing circle algorithm<br />
<br />
2017-07-18: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
<br />
2017-07-19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
<br />
2017-07-18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
<br />
2017-07-17: redo db operations after cleaning up granted patent number bugs<br />
<br />
2017-07-13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
<br />
2017-07-12: generate some example data illustrating the difficulty of joining different tables<br />
<br />
2017-07-11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
<br />
2017-07-07: catch up on documentation<br />
<br />
2017-07-06: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
<br />
2017-07-05: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
2017-06-30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
<br />
2017-06-29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
<br />
2017-06-28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
<br />
2017-06-27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
<br />
2017-06-26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
<br />
2017-06-25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
<br />
2017-06-23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
<br />
2017-06-22: create postcode<->patent table<br />
<br />
2017-06-29: document granted patent queries and equivalencies<br />
<br />
2017-06-20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
<br />
2017-06-19: skim address regular expressions; cursory investigation of patent table<br />
<br />
2017-06-16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
<br />
2017-06-15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
<br />
2017-06-14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
<br />
2017-06-13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
<br />
2017-06-12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
<br />
2017-06-08: add foreign key inserts; create pretty printer for XML analysis<br />
<br />
2017-06-07: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
<br />
2017-06-06: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
<br />
2017-06-05: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
<br />
2017-06-01: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
2017-05-31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
<br />
2017-05-30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
<br />
2017-05-29: expand to APS; expand to raw assignment data<br />
<br />
2017-05-27: expand to maintenance fee data<br />
<br />
2017-05-26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
<br />
2017-05-25: sketch out OO design of project; download bulk data<br />
<br />
2017-05-24: move wiki pages around; start git repository for project<br />
<br />
2017-05-21: discuss technical details of previous work with Ed<br />
<br />
2017-05-08: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
<br />
2017-05-04: setup wiki account, rdp account, database training<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang&diff=21813Oliver Chang2017-11-14T04:22:21Z<p>OliverC: </p>
<hr />
<div>{{McNair Staff<br />
|position=Tech Team<br />
|name=Oliver Chang,<br />
|user_image=Oliver-face.jpg<br />
|degree=Ph.D<br />
|major=Computer Science<br />
|class=2020<br />
|join_date=05/04/2017<br />
|skills=Coding,<br />
|interests=Cooking, Crosswords, Bicycling, Music, Sustainability, Trivia,<br />
|fun_fact=I own a tandem bicycle,<br />
|email=oyc@rice.edu<br />
|skype_name=oychang1<br />
|status=Active<br />
}}<br />
<br />
Second-year computer science PhD Student working on deep learning and program synthesis by examples. Advised by Swarat Chaudhuri. I have an office at Duncan Hall 3006.<br />
<br />
==Time at McNair==<br />
<br />
[[Oliver_Chang_(Work_Log)|Oliver Chang (Work Log)]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=21784Oliver Chang (Work Log)2017-11-10T22:26:53Z<p>OliverC: /* Fall 2017 */</p>
<hr />
<div>''Projects'':<br />
<br />
* [[PostGIS_Installation|PostGIS Installation]]<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
<br />
''Uploads'':<br />
<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
===Fall 2017=== <br />
<onlyinclude><br />
<br />
[[Oliver Chang]] [[Work Logs]] [[Oliver Chang (Work Log)|(log page)]]<br />
<br />
2017-11-10: add javadoc documentation to patent reproducibility project after forgetting half of the stuff myself<br />
<br />
2017-10-25: start ingestion of application xml files and deal with all the bugs which accompany that<br />
<br />
2017-10-21: create xml explorer script to mass-inspect xpaths (can be found at <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\xml_schema_explorer</code>)<br />
<br />
2017-10-20: re-run assignment import, review xpaths<br />
<br />
2017-10-19: repopulate patents database and create spreadsheet of assignment equivalencies<br />
<br />
2017-10-11: add issues to work on to the PECA wiki page; cleanup PECA git repo; start patent application code from granted patent code and start customizing to new domain<br />
<br />
2017-10-03: troubleshoot vc_circles.py and make command line interface a little nicer<br />
<br />
2017-10-02: discuss mapping strategies & investigate missing eca data<br />
<br />
2017-09-23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
<br />
2017-09-22: goal setting & server debugging & meet with Yang<br />
</onlyinclude><br />
<br />
===Summer 2017=== <br />
<br />
2017-08-04: setup parallel instance python framework for job reporting; begin test run<br />
<br />
2017-08-02: finish up some documentation of the code and for the wiki<br />
<br />
2017-08-01: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
2017-07-31: sketch out parallel enclosing circle algorithm<br />
<br />
2017-07-18: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
<br />
2017-07-19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
<br />
2017-07-18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
<br />
2017-07-17: redo db operations after cleaning up granted patent number bugs<br />
<br />
2017-07-13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
<br />
2017-07-12: generate some example data illustrating the difficulty of joining different tables<br />
<br />
2017-07-11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
<br />
2017-07-07: catch up on documentation<br />
<br />
2017-07-06: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
<br />
2017-07-05: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
2017-06-30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
<br />
2017-06-29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
<br />
2017-06-28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
<br />
2017-06-27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
<br />
2017-06-26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
<br />
2017-06-25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
<br />
2017-06-23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
<br />
2017-06-22: create postcode<->patent table<br />
<br />
2017-06-29: document granted patent queries and equivalencies<br />
<br />
2017-06-20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
<br />
2017-06-19: skim address regular expressions; cursory investigation of patent table<br />
<br />
2017-06-16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
<br />
2017-06-15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
<br />
2017-06-14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
<br />
2017-06-13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
<br />
2017-06-12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
<br />
2017-06-08: add foreign key inserts; create pretty printer for XML analysis<br />
<br />
2017-06-07: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
<br />
2017-06-06: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
<br />
2017-06-05: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
<br />
2017-06-01: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
2017-05-31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
<br />
2017-05-30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
<br />
2017-05-29: expand to APS; expand to raw assignment data<br />
<br />
2017-05-27: expand to maintenance fee data<br />
<br />
2017-05-26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
<br />
2017-05-25: sketch out OO design of project; download bulk data<br />
<br />
2017-05-24: move wiki pages around; start git repository for project<br />
<br />
2017-05-21: discuss technical details of previous work with Ed<br />
<br />
2017-05-08: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
<br />
2017-05-04: setup wiki account, rdp account, database training<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang&diff=21715Oliver Chang2017-11-09T18:35:15Z<p>OliverC: </p>
<hr />
<div>{{McNair Staff<br />
|position=Tech Team<br />
|name=Oliver Chang,<br />
|user_image=Oliver-face.jpg<br />
|degree=Ph.D<br />
|major=Computer Science<br />
|class=2020,<br />
|join_date=05/04/2017,<br />
|skills=Coding,<br />
|interests=Cooking, Crosswords, Bicycling, Music, Sustainability, Trivia,<br />
|fun_fact=I own a tandem bicycle,<br />
|email=oyc@rice.edu<br />
|skype_name=oychang1,<br />
|status=Active<br />
}}<br />
<br />
First-year computer science PhD Student working on deep learning and program synthesis by examples. Advised by Swarat Chaudhuri. I have an office at Duncan Hall 3006.<br />
<br />
==Time at McNair==<br />
<br />
[[Oliver_Chang_(Work_Log)|Oliver Chang (Work Log)]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=21714Oliver Chang (Work Log)2017-11-09T18:33:14Z<p>OliverC: /* Summer 2017 */</p>
<hr />
<div>''Projects'':<br />
<br />
* [[PostGIS_Installation|PostGIS Installation]]<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
<br />
''Uploads'':<br />
<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
===Fall 2017=== <br />
<onlyinclude><br />
<br />
[[Oliver Chang]] [[Work Logs]] [[Oliver Chang (Work Log)|(log page)]]<br />
<br />
2017-10-25: start ingestion of application xml files and deal with all the bugs which accompany that<br />
<br />
2017-10-21: create xml explorer script to mass-inspect xpaths (can be found at <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\xml_schema_explorer</code>)<br />
<br />
2017-10-20: re-run assignment import, review xpaths<br />
<br />
2017-10-19: repopulate patents database and create spreadsheet of assignment equivalencies<br />
<br />
2017-10-11: add issues to work on to the PECA wiki page; cleanup PECA git repo; start patent application code from granted patent code and start customizing to new domain<br />
<br />
2017-10-03: troubleshoot vc_circles.py and make command line interface a little nicer<br />
<br />
2017-10-02: discuss mapping strategies & investigate missing eca data<br />
<br />
2017-09-23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
<br />
2017-09-22: goal setting & server debugging & meet with Yang<br />
</onlyinclude><br />
<br />
===Summer 2017=== <br />
<br />
2017-08-04: setup parallel instance python framework for job reporting; begin test run<br />
<br />
2017-08-02: finish up some documentation of the code and for the wiki<br />
<br />
2017-08-01: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
2017-07-31: sketch out parallel enclosing circle algorithm<br />
<br />
2017-07-18: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
<br />
2017-07-19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
<br />
2017-07-18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
<br />
2017-07-17: redo db operations after cleaning up granted patent number bugs<br />
<br />
2017-07-13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
<br />
2017-07-12: generate some example data illustrating the difficulty of joining different tables<br />
<br />
2017-07-11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
<br />
2017-07-07: catch up on documentation<br />
<br />
2017-07-06: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
<br />
2017-07-05: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
2017-06-30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
<br />
2017-06-29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
<br />
2017-06-28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
<br />
2017-06-27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
<br />
2017-06-26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
<br />
2017-06-25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
<br />
2017-06-23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
<br />
2017-06-22: create postcode<->patent table<br />
<br />
2017-06-29: document granted patent queries and equivalencies<br />
<br />
2017-06-20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
<br />
2017-06-19: skim address regular expressions; cursory investigation of patent table<br />
<br />
2017-06-16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
<br />
2017-06-15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
<br />
2017-06-14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
<br />
2017-06-13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
<br />
2017-06-12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
<br />
2017-06-08: add foreign key inserts; create pretty printer for XML analysis<br />
<br />
2017-06-07: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
<br />
2017-06-06: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
<br />
2017-06-05: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
<br />
2017-06-01: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
2017-05-31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
<br />
2017-05-30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
<br />
2017-05-29: expand to APS; expand to raw assignment data<br />
<br />
2017-05-27: expand to maintenance fee data<br />
<br />
2017-05-26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
<br />
2017-05-25: sketch out OO design of project; download bulk data<br />
<br />
2017-05-24: move wiki pages around; start git repository for project<br />
<br />
2017-05-21: discuss technical details of previous work with Ed<br />
<br />
2017-05-08: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
<br />
2017-05-04: setup wiki account, rdp account, database training<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=21713Oliver Chang (Work Log)2017-11-09T18:32:55Z<p>OliverC: transition to new format</p>
<hr />
<div>''Projects'':<br />
<br />
* [[PostGIS_Installation|PostGIS Installation]]<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
<br />
''Uploads'':<br />
<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
===Fall 2017=== <br />
<onlyinclude><br />
<br />
[[Oliver Chang]] [[Work Logs]] [[Oliver Chang (Work Log)|(log page)]]<br />
<br />
2017-10-25: start ingestion of application xml files and deal with all the bugs which accompany that<br />
<br />
2017-10-21: create xml explorer script to mass-inspect xpaths (can be found at <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\xml_schema_explorer</code>)<br />
<br />
2017-10-20: re-run assignment import, review xpaths<br />
<br />
2017-10-19: repopulate patents database and create spreadsheet of assignment equivalencies<br />
<br />
2017-10-11: add issues to work on to the PECA wiki page; cleanup PECA git repo; start patent application code from granted patent code and start customizing to new domain<br />
<br />
2017-10-03: troubleshoot vc_circles.py and make command line interface a little nicer<br />
<br />
2017-10-02: discuss mapping strategies & investigate missing eca data<br />
<br />
2017-09-23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
<br />
2017-09-22: goal setting & server debugging & meet with Yang<br />
</onlyinclude><br />
<br />
===Summer 2017=== <br />
<br />
2017-08-04: setup parallel instance python framework for job reporting; begin test run<br />
<br />
2017-08-02: finish up some documentation of the code and for the wiki<br />
<br />
2017-08-01: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
2017-07-31: sketch out parallel enclosing circle algorithm<br />
<br />
2017-07-18: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
<br />
2017-07-19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
<br />
2017-07-18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
<br />
2017-07-17: redo db operations after cleaning up granted patent number bugs<br />
<br />
2017-07-13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
<br />
2017-07-12: generate some example data illustrating the difficulty of joining different tables<br />
<br />
2017-07-11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
<br />
2017-07-07: catch up on documentation<br />
<br />
2017-07-06: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
<br />
2017-07-05: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
2017-06-30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
<br />
2017-06-29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
<br />
2017-06-28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
<br />
2017-06-27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
<br />
2017-06-26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
<br />
2017-06-25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
<br />
2017-06-23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
<br />
2017-06-22: create postcode<->patent table<br />
<br />
2017-06-29: document granted patent queries and equivalencies<br />
<br />
2017-06-20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
<br />
2017-06-19: skim address regular expressions; cursory investigation of patent table<br />
<br />
2017-06-16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
<br />
<br />
2017-06-15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
<br />
2017-06-14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
<br />
2017-06-13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
<br />
2017-06-12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
<br />
2017-06-08: add foreign key inserts; create pretty printer for XML analysis<br />
<br />
2017-06-07: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
<br />
2017-06-06: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
<br />
2017-06-05: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
<br />
2017-06-01: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
2017-05-31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
<br />
2017-05-30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
<br />
2017-05-29: expand to APS; expand to raw assignment data<br />
<br />
2017-05-27: expand to maintenance fee data<br />
<br />
2017-05-26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
<br />
2017-05-25: sketch out OO design of project; download bulk data<br />
<br />
2017-05-24: move wiki pages around; start git repository for project<br />
<br />
2017-05-21: discuss technical details of previous work with Ed<br />
<br />
2017-05-08: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
<br />
2017-05-04: setup wiki account, rdp account, database training<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=21712Oliver Chang (Work Log)2017-11-09T18:26:22Z<p>OliverC: remove old todo list</p>
<hr />
<div>''Projects'':<br />
<br />
* [[PostGIS_Installation|PostGIS Installation]]<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
<br />
''Uploads'':<br />
<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
===Fall 2017=== <br />
<onlyinclude><br />
<br />
[[Oliver Chang]] [[Work Logs]] [[Oliver Chang (Work Log)|(log page)]]<br />
<br />
2017-10-25: start ingestion of application xml files and deal with all the bugs which accompany that<br />
<br />
2017-10-21: create xml explorer script to mass-inspect xpaths (can be found at <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\xml_schema_explorer</code><br />
<br />
2017-10-20: re-run assignment import, review xpaths<br />
<br />
2017-10-19: repopulate patents database and create spreadsheet of assignment equivalencies<br />
<br />
2017-10-11: add issues to work on to the PECA wiki page; cleanup PECA git repo; start patent application code from granted patent code and start customizing to new domain<br />
<br />
2017-10-03: troubleshoot vc_circles.py and make command line interface a little nicer<br />
<br />
2017-10-02: discuss mapping strategies & investigate missing eca data<br />
<br />
2017-09-23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
<br />
2017-09-22: goal setting & server debugging & meet with Yang<br />
</onlyinclude><br />
<br />
===Summer 2017=== <br />
<br />
* Aug 4: setup parallel instance python framework for job reporting; begin test run<br />
* Aug 2: finish up some documentation of the code and for the wiki<br />
* Aug 1: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
* July 31: sketch out parallel enclosing circle algorithm<br />
* July 28: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
* ''travelling''<br />
* July 19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
* July 18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
* July 17: redo db operations after cleaning up granted patent number bugs<br />
* July 13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
* July 12: generate some example data illustrating the difficulty of joining different tables<br />
* July 11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
* July 7: catch up on documentation<br />
* July 6: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
* July 5: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
* June 30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
* June 29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
* June 28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
* June 27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
* June 26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
* June 25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
* June 23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
* June 22: create postcode<->patent table<br />
* June 21: document granted patent queries and equivalencies<br />
* June 20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
* June 19: skim address regular expressions; cursory investigation of patent table<br />
* June 16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
* June 15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
* June 14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
* June 13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
* June 12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
* June 8: add foreign key inserts; create pretty printer for XML analysis<br />
* June 7: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
* June 6: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
* June 5: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
* June 1: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
* May 31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
* May 30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
* May 29: expand to APS; expand to raw assignment data<br />
* May 27: expand to maintenance fee data<br />
* May 26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
* May 25: sketch out OO design of project; download bulk data<br />
* May 24: move wiki pages around; start git repository for project<br />
* May 21: discuss technical details of previous work with Ed<br />
* May 8: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
* May 4: setup wiki account, rdp account, database training<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Reproducible_Patent_Data&diff=21421Reproducible Patent Data2017-10-31T22:51:15Z<p>OliverC: /* Related Pages */</p>
<hr />
<div>{{McNair Projects<br />
|Has Image=Uspto web logo.jpg<br />
|Has title=Reproducible Patent Data<br />
|Has owner=Oliver Chang<br />
|Has start date=May 17<br />
|Has project status=Active<br />
|Does subsume=Redesigning Patent Database, Patent Assignment Data Restructure,<br />
}}<br />
<br />
A continuation of [[Redesigning Patent Database]] that aims to write faster, more centralized code to deal with data from the United States Patent and Trademark Office (USPTO). By having an end-to-end pipeline we can easily reproduce or update data without worrying about unintentional side effects or missing data.<br />
<br />
== Quickstart ==<br />
<br />
To get up and running with the code, do the following:<br />
<br />
# Clone the git project (link at end of page) to your user directory<br />
# Launch IntelliJ with >= Java 8 and Maven configured (default version installed on the RDP is setup to do this)<br />
# Open project in IntelliJ<br />
# Create an empty database (see [[#Database]])<br />
# Run the table creation scripts in <code>src/db/schemas/</code> in your new database<br />
# Modify the constant <code>DATABASE_NAME</code> in <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\postgres\DatabaseHelper.java</code><br />
# Run the Driver scripts in IntelliJ with the correct value for <code>DATA_DIRECTORY</code> (or run <code>RunInitialImport.java</code> which will do all of the data directories for that patent item type)<br />
# [Take a really, really long lunch...in total should take no more than five hours to load data on RDP]<br />
# Run scripts in <code>src/db/constraints</code> to check data assumptions<br />
# That's it!<br />
<br />
== Directory Layout ==<br />
<br />
=== Where is the Data? ===<br />
<br />
==== Directories ====<br />
<br />
All of the information for this project is located at <code>E:\McNair\Projects\SimplerPatentData</code><br />
<br />
There are several interesting directories:<br />
<br />
* <code>data/downloads/</code> is USPTO bulkdata, unmodified straight from the scraper<br />
* <code>data/extracts/</code> is a directory of a strict subset of the information stored in <code>data/downloads/</code>. It is the result of running a bulk 7-zip job on that directory to get everything unzipped in a flat data structure. Note that these files have the USPTO modified-by time since that metadata is stored in the zipfiles. To extract files in this nice format, select all of the zipfiles and setup an extraction job like in this [[media:7zip-params.png|screenshot]]<br />
* <code>data/backups/</code> is a 7zip'd backup of the corresponding directory in extracts<br />
* <code>src/</code> is the main code repository for the java project<br />
<br />
==== Input Files ====<br />
<br />
All of the text-only Red Book files for '''granted patents''' from 1976 to 2016, inclusive. To find a specific year's XML file, find it in<br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\granted\</code><br />
<br />
'''To find application data''' from 2001 to 2016, inclusive, look in<br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\applications\</code><br />
<br />
'''To find assignment data''', look in <br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\granted\</code><br />
<br />
'''To find maintenance fee data''', look in <br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\downloads\maintenance\</code><br />
<br />
=== Where is the Code? ===<br />
<br />
The code has the same parent directory as the data, so it is at <code>E:\McNair\Projects\SimplerPatentData\src</code>. You might notice a lot of single-entry directories; this is an idiomatic Java pattern that is used for package separation. If using IntelliJ or some other IDE, these directories are a bit less annoying.<br />
<br />
The development environment is Java 8 JDK, IntelliJ Ultimate IDE, Maven build tools, and git VCS.<br />
<br />
The git repository can be found at https://rdp.mcnaircenter.org/codebase/Repository/ReproduciblePatent<br />
<br />
==== Prior Art ====<br />
<br />
This tool is not so concerned with adding new functionality; rather, it aims to take a bunch of spread out Perl scripts and create a faster system that is easier to work with. As such, its functionality is largely stolen from those scripts:<br />
<br />
* Downloader: <code>E:\McNair\Software\Scripts\Patent\USPTO_Parser.pl</code><br />
* XML Splitter: <code>E:\McNair\PatentData\splitter.pl</code><br />
* XML Parsing: <code>E:\McNair\PatentData\Processed\xmlparser_4.5_4.4_4.3.pl</code> and <code>E:\McNair\PatentData\Processed\*.pm</code><br />
<br />
In addition, I used several non-standard Java libraries listed below:<br />
<br />
* [https://github.com/mashape/unirest-java/ Unirest] for easy HTTP requests (MIT License)<br />
* [https://github.com/google/guava Google Guava] for immutable collections and Stream utilities (Apache v2.0 License)<br />
* [https://github.com/jhy/jsoup/ jsoup] for HTML parsing (MIT License)<br />
* [http://commons.apache.org/proper/commons-codec/ Apache Commons Codec] (Apache v2.0 License)<br />
* [http://commons.apache.org/proper/commons-lang/ Apache Commons Lang v3] (Apache v2.0 License)<br />
* [https://mvnrepository.com/artifact/org.jetbrains/annotations/15.0 Jetbrains Annotations] for enhanced null checks (Apache v2.0 License)<br />
* [http://search.maven.org/#artifactdetails%7Corg.postgresql%7Cpostgresql%7C42.1.1.jre7%7Cbundle PostgreSQL JDBC] (BSD 3-clause per https://github.com/pgjdbc/pgjdbc-jre7/blob/master/LICENSE)<br />
<br />
If using maven, these dependencies are listed and should automatically be setup.<br />
<br />
==== Using Code ====<br />
<br />
Any file with a line that says <code>public static void main(String[] args) {</code> can be run as a standalone file. The easiest way to do this is to load the project and then the file in IntelliJ and click the little green play arrow next to this bit of code.<br />
<br />
The code can also be run via the standard <code>javac</code> and <code>java</code> commands but since this project has a complicated structure you end up having to run commands like <br />
<br />
<code>"C:\Program Files\Java\jdk1.8.0_131\bin\java" "-javaagent:C:\Users\OliverC\IntelliJ IDEA 2017.1.3\lib\idea_rt.jar=62364:C:\Users\OliverC\IntelliJ IDEA 2017.1.3\bin" -Dfile.encoding=UTF-8 -classpath "[...contents truncated...];C:\Users\OliverC\.m2\repository\org\postgresql\postgresql\42.1.1\postgresql-42.1.1.jar" org.bakerinstitute.mcnair.uspto_assignments.XmlDriver</code><br />
<br />
to include all of the runtime dependencies and it's just not worth it.<br />
<br />
==== Altering Code ====<br />
<br />
* Use the IntelliJ command Reformat code (found in the menus at <code>Code > Reformat Code</code><br />
* Use the optimize imports function found under the same menu<br />
* Use spaces for indentation<br />
* Loosely try to keep lines below 120 characters<br />
* Commit changes to the Git remote repository "bonobo"<br />
<br />
== Schema Reconciliation ==<br />
<br />
For the work by Joe, see the [[Patent Schema Reconciliation]] page <br />
<br />
=== Patents (Granted) ===<br />
<br />
See <code>E:\McNair\Projects\SimplerPatentData\data\examples\granted</code> for extracted examples of what specific data is available for a sample of the data.<br />
<br />
{| class="wikitable"<br />
|+Granted Patent Data Formats<br />
|-<br />
! scope="col" | Dates Used !! scope="col" | Format !! scope="col" | Location !! scope="col" | Supported? !! scope="col" | Utility !! scope="col" | Reissue !! scope="col" | Design !! scope="col" | Plant<br />
|-<br />
|January 1976 to December 2001<br />
|APS<br />
|<code>data/extracts/granted/vintage</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|<del>January 2001 to December 2001</del><br />
|<del>SGML</del><br />
|Ignored; use concurrently recorded APS data<br />
|No<br />
|N/A<br />
|N/A<br />
|N/A<br />
|N/A<br />
|-<br />
|January 2002 to December 2004<br />
|XML Version 2.5<br />
|<code>data/extracts/granted/blunderyears</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2005 to December 2005<br />
|XML Version 4.0 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2006 to December 2006<br />
|XML Version 4.1 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2007 to December 2012<br />
|XML Version 4.2 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2013 to September 24, 2013<br />
|XML Version 4.3 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|October 8, 2013 to December 2014<br />
|XML Version 4.4 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2015 to December 2016<br />
|XML Version 4.5 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|}<br />
<br />
=== APS Rosetta Stone ===<br />
<br />
The Advanced Patent System (APS) is a fixed-width text format used to store historical patent grant data. The documentation for this sucks; there are pages missing at random. Luckily, we only care about the content contained here: [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]].<br />
<br />
It's worth mentioning that the APS contains an advanced text markup system for chemical formulae, basic text markup, tables, etc. that can lead to seemingly garbled text that is perfectly well-formed.<br />
<br />
==== APS Gotchas ====<br />
<br />
* PATN.WKU is the granted patent number. It is 7 digits while the spec promises 6 digits. The rightmost digit is a check digit modulus 11. See [[File:Aps-wku-modulus11.pdf]] for the words from the horse's mouth.<br />
<br />
=== Patents (Applications) ===<br />
<br />
{| class="wikitable"<br />
|+Patent Application Data Formats<br />
|-<br />
! scope="col" | Dates Used !! scope="col" | Format !! scope="col" | Location !! scope="col" | Supported by Parser?<br />
|-<br />
|March 15, 2001 to December 2001<br />
|XML Version 1.5<br />
|<code>data/extracts/applications/vintage</code><br />
|style="background: yellow;" | Yes, for basic information, inventors, and correspondents<br />
|-<br />
|January 2002 to December 2004<br />
|XML Version 1.6<br />
|<code>data/extracts/applications/vintage</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2005 to December 2005<br />
|XML Version 4.0 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2006 to December 2006<br />
|XML Version 4.1 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2007 to December 2012<br />
|XML Version 4.2 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2013 to December 2014<br />
|XML Version 4.3 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2015 to ''Present''<br />
|XML Version 4.4 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|}<br />
<br />
== Database ==<br />
<br />
Because there isn't a compelling reason not to, I used the existing PostgreSQL infrastructure on the RDP.<br />
The "Java Way" of interacting with databases is the Java Database Connectivity API (JDBC), an implementation-agnostic API for interacting with databases.<br />
This project uses the stock [https://jdbc.postgresql.org/ Postgres JDBC], version 42.1.1<br />
<br />
=== Create an empty database on RDP ===<br />
<br />
To create an empty database, run this command: <code>$ createdb --username=postgres database-name-goes-here # password is tabspaceenter</code><br />
<br />
=== Abstraction Layer ===<br />
<br />
Since writing raw SQL is a bit cumbersome and error-prone, I have added some abstraction layers that make it much easier to quickly add bulk data. By using Postgres's <code>CopyManager</code> class, we buffer SQL copy commands in memory (as many as possible) and then flush these rows. To understand how the abstraction layers work, see the code in <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\postgres</code>. See <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\models\GrantedPatent.java</code> for '''an example of how to extend''' the abstraction layer to deal with more complex scenarios.<br />
<br />
=== New Table Checklist ===<br />
<br />
* Create schema DDL SQL code for the new table in <code>E:\McNair\Projects\SimplerPatentData\src\db</code><br />
* Run the schema creation<br />
* Create an enum with the same names for attributes as in the DDL (case-insensitive! prefer all-caps screaming snake case)<br />
* Create a class which subclasses <code>AbstractIntertableData</code><br />
* Inside that class, create a static class which subclasses <code>AbstractTableMetadata</code> and has the proper values for getTableName(), getStringColumns(), getIntColumns()<br />
* (Optional) Implement builder pattern<br />
* (Optional) Create a custom databasehelper for complex extras (see PatentApplication and GrantedPatent for examples)<br />
* Write the data to the table (see DatabaseHelper for the pattern I use)<br />
<br />
== Address Data ==<br />
<br />
To get the most granular address data (street level, or at least postcode level) about who owns patents, the path is not so straightforward because off the complicated mapping of ownership to a granted patent.<br />
This is the final part of this project that I am working on and it is all at the level of SQL.<br />
See <code>E:\McNair\Projects\SimplerPatentData\src\db\joins</code> for my attempts to create a clean mapping.<br />
Optimistically speaking, the data generated here should be superset of the data present in the Patent Assignment Data Restructure project.<br />
<br />
Note that as of the beginning of August 2017, this part '''has not been completed.'''<br />
<br />
<br />
=== Intuition ===<br />
<br />
Use <code>assignments_longform.last_update_date</code> to find current/latest (or first/earliest) date of assignment. Then match with <code>properties.docid</code> on <code>reelno, frameno</code> to find patent application id. With this mapping to granted patents, we can discover the details of the original granted patent. And with the right date and reelno and frameno, we can match to the <code>assignees</code> table and get fine granularity addresses.<br />
<br />
== Related Pages ==<br />
<br />
* [[Redesign_Assignment_and_Patent_Database|Redesign Assignment and Patent Database, Fall 2017 by Shelby]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries, Summer 2017 by Oliver & Joe]]<br />
* [[US_Address_Verification|US Address Verification, Summer 2017 based on tables from Assignment Data Restructure]]<br />
* [[Patent_Assignment_Data_Restructure|Assignment Data Restructure, Spring 2017 by Marcela and Sonia]]<br />
* [[Redesigning_Patent_Database|Redesigning Patent Database, Spring 2017 by Shelby]]<br />
* [[Patent_Data_Cleanup_(June_2016)|Patent Data Cleanup, June 2016 by Marcela]]<br />
* [[Patent_Data|Patent Data, Spring 2016 by Marcela]] <br />
* [[Lex_Machina|Lex Machina]]<br />
* [[USPTO_Patent_Litigation_Data|USPTO Patent Litigation Research Dataset by Ed]]<br />
* [[Patent_Litigation_and_Review|Patent Litigation and Review by Marcela]]<br />
* [[Bag_of_Words_Analysis_of_Patent_Data|Bag of Words Analysis]]<br />
* [[Patent|Existing Database Schema]]<br />
* [[Oliver_Chang_(Work_Log)|My Work Log]]<br />
<br />
== External Links ==<br />
<br />
* Understanding Assignment Data: [https://www.uspto.gov/sites/default/files/documents/USPTO_Patents_Assignment_Dataset_WP.pdf USPTO Documentation on their cleanup of this data]<br />
* [https://bulkdata.uspto.gov/data2/patent/grant/redbook/fulltext/1976/PatentFullTextAPSDoc_GreenBook.pdf USPTO Green Book (APS) Documentation]<br />
* [https://bulkdata.uspto.gov/ USPTO Bulk Data Storage System (BDSS)]<br />
* [https://en.wikipedia.org/wiki/Builder_pattern Builder Pattern in Object-Oriented Programming]<br />
* [https://rdp.mcnaircenter.org/codebase/Repository/ReproduciblePatent Git Repository]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Reproducible_Patent_Data&diff=21420Reproducible Patent Data2017-10-31T22:43:38Z<p>OliverC: </p>
<hr />
<div>{{McNair Projects<br />
|Has Image=Uspto web logo.jpg<br />
|Has title=Reproducible Patent Data<br />
|Has owner=Oliver Chang<br />
|Has start date=May 17<br />
|Has project status=Active<br />
|Does subsume=Redesigning Patent Database, Patent Assignment Data Restructure,<br />
}}<br />
<br />
A continuation of [[Redesigning Patent Database]] that aims to write faster, more centralized code to deal with data from the United States Patent and Trademark Office (USPTO). By having an end-to-end pipeline we can easily reproduce or update data without worrying about unintentional side effects or missing data.<br />
<br />
== Quickstart ==<br />
<br />
To get up and running with the code, do the following:<br />
<br />
# Clone the git project (link at end of page) to your user directory<br />
# Launch IntelliJ with >= Java 8 and Maven configured (default version installed on the RDP is setup to do this)<br />
# Open project in IntelliJ<br />
# Create an empty database (see [[#Database]])<br />
# Run the table creation scripts in <code>src/db/schemas/</code> in your new database<br />
# Modify the constant <code>DATABASE_NAME</code> in <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\postgres\DatabaseHelper.java</code><br />
# Run the Driver scripts in IntelliJ with the correct value for <code>DATA_DIRECTORY</code> (or run <code>RunInitialImport.java</code> which will do all of the data directories for that patent item type)<br />
# [Take a really, really long lunch...in total should take no more than five hours to load data on RDP]<br />
# Run scripts in <code>src/db/constraints</code> to check data assumptions<br />
# That's it!<br />
<br />
== Directory Layout ==<br />
<br />
=== Where is the Data? ===<br />
<br />
==== Directories ====<br />
<br />
All of the information for this project is located at <code>E:\McNair\Projects\SimplerPatentData</code><br />
<br />
There are several interesting directories:<br />
<br />
* <code>data/downloads/</code> is USPTO bulkdata, unmodified straight from the scraper<br />
* <code>data/extracts/</code> is a directory of a strict subset of the information stored in <code>data/downloads/</code>. It is the result of running a bulk 7-zip job on that directory to get everything unzipped in a flat data structure. Note that these files have the USPTO modified-by time since that metadata is stored in the zipfiles. To extract files in this nice format, select all of the zipfiles and setup an extraction job like in this [[media:7zip-params.png|screenshot]]<br />
* <code>data/backups/</code> is a 7zip'd backup of the corresponding directory in extracts<br />
* <code>src/</code> is the main code repository for the java project<br />
<br />
==== Input Files ====<br />
<br />
All of the text-only Red Book files for '''granted patents''' from 1976 to 2016, inclusive. To find a specific year's XML file, find it in<br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\granted\</code><br />
<br />
'''To find application data''' from 2001 to 2016, inclusive, look in<br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\applications\</code><br />
<br />
'''To find assignment data''', look in <br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\granted\</code><br />
<br />
'''To find maintenance fee data''', look in <br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\downloads\maintenance\</code><br />
<br />
=== Where is the Code? ===<br />
<br />
The code has the same parent directory as the data, so it is at <code>E:\McNair\Projects\SimplerPatentData\src</code>. You might notice a lot of single-entry directories; this is an idiomatic Java pattern that is used for package separation. If using IntelliJ or some other IDE, these directories are a bit less annoying.<br />
<br />
The development environment is Java 8 JDK, IntelliJ Ultimate IDE, Maven build tools, and git VCS.<br />
<br />
The git repository can be found at https://rdp.mcnaircenter.org/codebase/Repository/ReproduciblePatent<br />
<br />
==== Prior Art ====<br />
<br />
This tool is not so concerned with adding new functionality; rather, it aims to take a bunch of spread out Perl scripts and create a faster system that is easier to work with. As such, its functionality is largely stolen from those scripts:<br />
<br />
* Downloader: <code>E:\McNair\Software\Scripts\Patent\USPTO_Parser.pl</code><br />
* XML Splitter: <code>E:\McNair\PatentData\splitter.pl</code><br />
* XML Parsing: <code>E:\McNair\PatentData\Processed\xmlparser_4.5_4.4_4.3.pl</code> and <code>E:\McNair\PatentData\Processed\*.pm</code><br />
<br />
In addition, I used several non-standard Java libraries listed below:<br />
<br />
* [https://github.com/mashape/unirest-java/ Unirest] for easy HTTP requests (MIT License)<br />
* [https://github.com/google/guava Google Guava] for immutable collections and Stream utilities (Apache v2.0 License)<br />
* [https://github.com/jhy/jsoup/ jsoup] for HTML parsing (MIT License)<br />
* [http://commons.apache.org/proper/commons-codec/ Apache Commons Codec] (Apache v2.0 License)<br />
* [http://commons.apache.org/proper/commons-lang/ Apache Commons Lang v3] (Apache v2.0 License)<br />
* [https://mvnrepository.com/artifact/org.jetbrains/annotations/15.0 Jetbrains Annotations] for enhanced null checks (Apache v2.0 License)<br />
* [http://search.maven.org/#artifactdetails%7Corg.postgresql%7Cpostgresql%7C42.1.1.jre7%7Cbundle PostgreSQL JDBC] (BSD 3-clause per https://github.com/pgjdbc/pgjdbc-jre7/blob/master/LICENSE)<br />
<br />
If using maven, these dependencies are listed and should automatically be setup.<br />
<br />
==== Using Code ====<br />
<br />
Any file with a line that says <code>public static void main(String[] args) {</code> can be run as a standalone file. The easiest way to do this is to load the project and then the file in IntelliJ and click the little green play arrow next to this bit of code.<br />
<br />
The code can also be run via the standard <code>javac</code> and <code>java</code> commands but since this project has a complicated structure you end up having to run commands like <br />
<br />
<code>"C:\Program Files\Java\jdk1.8.0_131\bin\java" "-javaagent:C:\Users\OliverC\IntelliJ IDEA 2017.1.3\lib\idea_rt.jar=62364:C:\Users\OliverC\IntelliJ IDEA 2017.1.3\bin" -Dfile.encoding=UTF-8 -classpath "[...contents truncated...];C:\Users\OliverC\.m2\repository\org\postgresql\postgresql\42.1.1\postgresql-42.1.1.jar" org.bakerinstitute.mcnair.uspto_assignments.XmlDriver</code><br />
<br />
to include all of the runtime dependencies and it's just not worth it.<br />
<br />
==== Altering Code ====<br />
<br />
* Use the IntelliJ command Reformat code (found in the menus at <code>Code > Reformat Code</code><br />
* Use the optimize imports function found under the same menu<br />
* Use spaces for indentation<br />
* Loosely try to keep lines below 120 characters<br />
* Commit changes to the Git remote repository "bonobo"<br />
<br />
== Schema Reconciliation ==<br />
<br />
For the work by Joe, see the [[Patent Schema Reconciliation]] page <br />
<br />
=== Patents (Granted) ===<br />
<br />
See <code>E:\McNair\Projects\SimplerPatentData\data\examples\granted</code> for extracted examples of what specific data is available for a sample of the data.<br />
<br />
{| class="wikitable"<br />
|+Granted Patent Data Formats<br />
|-<br />
! scope="col" | Dates Used !! scope="col" | Format !! scope="col" | Location !! scope="col" | Supported? !! scope="col" | Utility !! scope="col" | Reissue !! scope="col" | Design !! scope="col" | Plant<br />
|-<br />
|January 1976 to December 2001<br />
|APS<br />
|<code>data/extracts/granted/vintage</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|<del>January 2001 to December 2001</del><br />
|<del>SGML</del><br />
|Ignored; use concurrently recorded APS data<br />
|No<br />
|N/A<br />
|N/A<br />
|N/A<br />
|N/A<br />
|-<br />
|January 2002 to December 2004<br />
|XML Version 2.5<br />
|<code>data/extracts/granted/blunderyears</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2005 to December 2005<br />
|XML Version 4.0 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2006 to December 2006<br />
|XML Version 4.1 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2007 to December 2012<br />
|XML Version 4.2 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2013 to September 24, 2013<br />
|XML Version 4.3 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|October 8, 2013 to December 2014<br />
|XML Version 4.4 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2015 to December 2016<br />
|XML Version 4.5 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|}<br />
<br />
=== APS Rosetta Stone ===<br />
<br />
The Advanced Patent System (APS) is a fixed-width text format used to store historical patent grant data. The documentation for this sucks; there are pages missing at random. Luckily, we only care about the content contained here: [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]].<br />
<br />
It's worth mentioning that the APS contains an advanced text markup system for chemical formulae, basic text markup, tables, etc. that can lead to seemingly garbled text that is perfectly well-formed.<br />
<br />
==== APS Gotchas ====<br />
<br />
* PATN.WKU is the granted patent number. It is 7 digits while the spec promises 6 digits. The rightmost digit is a check digit modulus 11. See [[File:Aps-wku-modulus11.pdf]] for the words from the horse's mouth.<br />
<br />
=== Patents (Applications) ===<br />
<br />
{| class="wikitable"<br />
|+Patent Application Data Formats<br />
|-<br />
! scope="col" | Dates Used !! scope="col" | Format !! scope="col" | Location !! scope="col" | Supported by Parser?<br />
|-<br />
|March 15, 2001 to December 2001<br />
|XML Version 1.5<br />
|<code>data/extracts/applications/vintage</code><br />
|style="background: yellow;" | Yes, for basic information, inventors, and correspondents<br />
|-<br />
|January 2002 to December 2004<br />
|XML Version 1.6<br />
|<code>data/extracts/applications/vintage</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2005 to December 2005<br />
|XML Version 4.0 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2006 to December 2006<br />
|XML Version 4.1 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2007 to December 2012<br />
|XML Version 4.2 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2013 to December 2014<br />
|XML Version 4.3 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2015 to ''Present''<br />
|XML Version 4.4 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|}<br />
<br />
== Database ==<br />
<br />
Because there isn't a compelling reason not to, I used the existing PostgreSQL infrastructure on the RDP.<br />
The "Java Way" of interacting with databases is the Java Database Connectivity API (JDBC), an implementation-agnostic API for interacting with databases.<br />
This project uses the stock [https://jdbc.postgresql.org/ Postgres JDBC], version 42.1.1<br />
<br />
=== Create an empty database on RDP ===<br />
<br />
To create an empty database, run this command: <code>$ createdb --username=postgres database-name-goes-here # password is tabspaceenter</code><br />
<br />
=== Abstraction Layer ===<br />
<br />
Since writing raw SQL is a bit cumbersome and error-prone, I have added some abstraction layers that make it much easier to quickly add bulk data. By using Postgres's <code>CopyManager</code> class, we buffer SQL copy commands in memory (as many as possible) and then flush these rows. To understand how the abstraction layers work, see the code in <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\postgres</code>. See <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\models\GrantedPatent.java</code> for '''an example of how to extend''' the abstraction layer to deal with more complex scenarios.<br />
<br />
=== New Table Checklist ===<br />
<br />
* Create schema DDL SQL code for the new table in <code>E:\McNair\Projects\SimplerPatentData\src\db</code><br />
* Run the schema creation<br />
* Create an enum with the same names for attributes as in the DDL (case-insensitive! prefer all-caps screaming snake case)<br />
* Create a class which subclasses <code>AbstractIntertableData</code><br />
* Inside that class, create a static class which subclasses <code>AbstractTableMetadata</code> and has the proper values for getTableName(), getStringColumns(), getIntColumns()<br />
* (Optional) Implement builder pattern<br />
* (Optional) Create a custom databasehelper for complex extras (see PatentApplication and GrantedPatent for examples)<br />
* Write the data to the table (see DatabaseHelper for the pattern I use)<br />
<br />
== Address Data ==<br />
<br />
To get the most granular address data (street level, or at least postcode level) about who owns patents, the path is not so straightforward because off the complicated mapping of ownership to a granted patent.<br />
This is the final part of this project that I am working on and it is all at the level of SQL.<br />
See <code>E:\McNair\Projects\SimplerPatentData\src\db\joins</code> for my attempts to create a clean mapping.<br />
Optimistically speaking, the data generated here should be superset of the data present in the Patent Assignment Data Restructure project.<br />
<br />
Note that as of the beginning of August 2017, this part '''has not been completed.'''<br />
<br />
<br />
=== Intuition ===<br />
<br />
Use <code>assignments_longform.last_update_date</code> to find current/latest (or first/earliest) date of assignment. Then match with <code>properties.docid</code> on <code>reelno, frameno</code> to find patent application id. With this mapping to granted patents, we can discover the details of the original granted patent. And with the right date and reelno and frameno, we can match to the <code>assignees</code> table and get fine granularity addresses.<br />
<br />
== Related Pages ==<br />
<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries, Summer 2017 by this guy]]<br />
* [[US_Address_Verification|US Address Verification, Summer 2017 based on tables from Assignment Data Restructure]]<br />
* [[Patent_Assignment_Data_Restructure|Assignment Data Restructure, Spring 2017 by Marcela and Sonia]]<br />
* [[Redesigning_Patent_Database|Redesigning Patent Database, Spring 2017 by Shelby]]<br />
* [[Patent_Data_Cleanup_(June_2016)|Patent Data Cleanup, June 2016 by Marcela]]<br />
* [[Patent_Data|Patent Data, Spring 2016 by Marcela]] <br />
* [[Lex_Machina|Lex Machina]]<br />
* [[USPTO_Patent_Litigation_Data|USPTO Patent Litigation Research Dataset by Ed]]<br />
* [[Patent_Litigation_and_Review|Patent Litigation and Review by Marcela]]<br />
* [[Bag_of_Words_Analysis_of_Patent_Data|Bag of Words Analysis]]<br />
* [[Patent|Existing Database Schema]]<br />
* [[Oliver_Chang_(Work_Log)|My Work Log]]<br />
<br />
== External Links ==<br />
<br />
* Understanding Assignment Data: [https://www.uspto.gov/sites/default/files/documents/USPTO_Patents_Assignment_Dataset_WP.pdf USPTO Documentation on their cleanup of this data]<br />
* [https://bulkdata.uspto.gov/data2/patent/grant/redbook/fulltext/1976/PatentFullTextAPSDoc_GreenBook.pdf USPTO Green Book (APS) Documentation]<br />
* [https://bulkdata.uspto.gov/ USPTO Bulk Data Storage System (BDSS)]<br />
* [https://en.wikipedia.org/wiki/Builder_pattern Builder Pattern in Object-Oriented Programming]<br />
* [https://rdp.mcnaircenter.org/codebase/Repository/ReproduciblePatent Git Repository]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Reproducible_Patent_Data&diff=21419Reproducible Patent Data2017-10-31T22:43:00Z<p>OliverC: /* Database */</p>
<hr />
<div>{{McNair Projects<br />
|Has Image=Uspto web logo.jpg<br />
|Has title=Reproducible Patent Data<br />
|Has owner=Oliver Chang<br />
|Has start date=May 17<br />
|Has project status=Active<br />
|Does subsume=Redesigning Patent Database, Patent Assignment Data Restructure,<br />
}}<br />
<br />
A continuation of [[Redesigning Patent Database]] that aims to write faster, more centralized code to deal with data from the United States Patent and Trademark Office (USPTO). By having an end-to-end pipeline we can easily reproduce or update data without worrying about unintentional side effects or missing data.<br />
<br />
== Quickstart ==<br />
<br />
To get up and running with the code, do the following:<br />
<br />
# Clone the git project (link at end of page) to your user directory<br />
# Launch IntelliJ with >= Java 8 and Maven configured (default version installed on the RDP is setup to do this)<br />
# Open project in IntelliJ<br />
# Create an empty database (see [[#Database]])<br />
# Run the table creation scripts in <code>src/db/schemas/</code> in your new database<br />
# Modify the constant <code>DATABASE_NAME</code> in <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\postgres\DatabaseHelper.java</code><br />
# Run the Driver scripts in IntelliJ with the correct value for <code>DATA_DIRECTORY</code> (or run <code>RunInitialImport.java</code> which will do all of the data directories for that patent item type)<br />
# [Take a really, really long lunch...in total should take no more than five hours to load data on RDP]<br />
# Run scripts in <code>src/db/constraints</code> to check data assumptions<br />
# That's it!<br />
<br />
== Directory Layout ==<br />
<br />
=== Where is the Data? ===<br />
<br />
==== Directories ====<br />
<br />
All of the information for this project is located at <code>E:\McNair\Projects\SimplerPatentData</code><br />
<br />
There are several interesting directories:<br />
<br />
* <code>data/downloads/</code> is USPTO bulkdata, unmodified straight from the scraper<br />
* <code>data/extracts/</code> is a directory of a strict subset of the information stored in <code>data/downloads/</code>. It is the result of running a bulk 7-zip job on that directory to get everything unzipped in a flat data structure. Note that these files have the USPTO modified-by time since that metadata is stored in the zipfiles. To extract files in this nice format, select all of the zipfiles and setup an extraction job like in this [[media:7zip-params.png|screenshot]]<br />
* <code>data/backups/</code> is a 7zip'd backup of the corresponding directory in extracts<br />
* <code>src/</code> is the main code repository for the java project<br />
<br />
==== Input Files ====<br />
<br />
All of the text-only Red Book files for '''granted patents''' from 1976 to 2016, inclusive. To find a specific year's XML file, find it in<br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\granted\</code><br />
<br />
'''To find application data''' from 2001 to 2016, inclusive, look in<br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\applications\</code><br />
<br />
'''To find assignment data''', look in <br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\granted\</code><br />
<br />
'''To find maintenance fee data''', look in <br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\downloads\maintenance\</code><br />
<br />
=== Where is the Code? ===<br />
<br />
The code has the same parent directory as the data, so it is at <code>E:\McNair\Projects\SimplerPatentData\src</code>. You might notice a lot of single-entry directories; this is an idiomatic Java pattern that is used for package separation. If using IntelliJ or some other IDE, these directories are a bit less annoying.<br />
<br />
The development environment is Java 8 JDK, IntelliJ Ultimate IDE, Maven build tools, and git VCS.<br />
<br />
The git repository can be found at https://rdp.mcnaircenter.org/codebase/Repository/ReproduciblePatent<br />
<br />
==== Prior Art ====<br />
<br />
This tool is not so concerned with adding new functionality; rather, it aims to take a bunch of spread out Perl scripts and create a faster system that is easier to work with. As such, its functionality is largely stolen from those scripts:<br />
<br />
* Downloader: <code>E:\McNair\Software\Scripts\Patent\USPTO_Parser.pl</code><br />
* XML Splitter: <code>E:\McNair\PatentData\splitter.pl</code><br />
* XML Parsing: <code>E:\McNair\PatentData\Processed\xmlparser_4.5_4.4_4.3.pl</code> and <code>E:\McNair\PatentData\Processed\*.pm</code><br />
<br />
In addition, I used several non-standard Java libraries listed below:<br />
<br />
* [https://github.com/mashape/unirest-java/ Unirest] for easy HTTP requests (MIT License)<br />
* [https://github.com/google/guava Google Guava] for immutable collections and Stream utilities (Apache v2.0 License)<br />
* [https://github.com/jhy/jsoup/ jsoup] for HTML parsing (MIT License)<br />
* [http://commons.apache.org/proper/commons-codec/ Apache Commons Codec] (Apache v2.0 License)<br />
* [http://commons.apache.org/proper/commons-lang/ Apache Commons Lang v3] (Apache v2.0 License)<br />
* [https://mvnrepository.com/artifact/org.jetbrains/annotations/15.0 Jetbrains Annotations] for enhanced null checks (Apache v2.0 License)<br />
* [http://search.maven.org/#artifactdetails%7Corg.postgresql%7Cpostgresql%7C42.1.1.jre7%7Cbundle PostgreSQL JDBC] (BSD 3-clause per https://github.com/pgjdbc/pgjdbc-jre7/blob/master/LICENSE)<br />
<br />
If using maven, these dependencies are listed and should automatically be setup.<br />
<br />
==== Using Code ====<br />
<br />
Any file with a line that says <code>public static void main(String[] args) {</code> can be run as a standalone file. The easiest way to do this is to load the project and then the file in IntelliJ and click the little green play arrow next to this bit of code.<br />
<br />
The code can also be run via the standard <code>javac</code> and <code>java</code> commands but since this project has a complicated structure you end up having to run commands like <br />
<br />
<code>"C:\Program Files\Java\jdk1.8.0_131\bin\java" "-javaagent:C:\Users\OliverC\IntelliJ IDEA 2017.1.3\lib\idea_rt.jar=62364:C:\Users\OliverC\IntelliJ IDEA 2017.1.3\bin" -Dfile.encoding=UTF-8 -classpath "[...contents truncated...];C:\Users\OliverC\.m2\repository\org\postgresql\postgresql\42.1.1\postgresql-42.1.1.jar" org.bakerinstitute.mcnair.uspto_assignments.XmlDriver</code><br />
<br />
to include all of the runtime dependencies and it's just not worth it.<br />
<br />
==== Altering Code ====<br />
<br />
* Use the IntelliJ command Reformat code (found in the menus at <code>Code > Reformat Code</code><br />
* Use the optimize imports function found under the same menu<br />
* Use spaces for indentation<br />
* Loosely try to keep lines below 120 characters<br />
* Commit changes to the Git remote repository "bonobo"<br />
<br />
== Schema Reconciliation ==<br />
<br />
For the work by Joe, see the [[Patent Schema Reconciliation]] page <br />
<br />
=== Patents (Granted) ===<br />
<br />
See <code>E:\McNair\Projects\SimplerPatentData\data\examples\granted</code> for extracted examples of what specific data is available for a sample of the data.<br />
<br />
{| class="wikitable"<br />
|+Granted Patent Data Formats<br />
|-<br />
! scope="col" | Dates Used !! scope="col" | Format !! scope="col" | Location !! scope="col" | Supported? !! scope="col" | Utility !! scope="col" | Reissue !! scope="col" | Design !! scope="col" | Plant<br />
|-<br />
|January 1976 to December 2001<br />
|APS<br />
|<code>data/extracts/granted/vintage</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|<del>January 2001 to December 2001</del><br />
|<del>SGML</del><br />
|Ignored; use concurrently recorded APS data<br />
|No<br />
|N/A<br />
|N/A<br />
|N/A<br />
|N/A<br />
|-<br />
|January 2002 to December 2004<br />
|XML Version 2.5<br />
|<code>data/extracts/granted/blunderyears</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2005 to December 2005<br />
|XML Version 4.0 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2006 to December 2006<br />
|XML Version 4.1 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2007 to December 2012<br />
|XML Version 4.2 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2013 to September 24, 2013<br />
|XML Version 4.3 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|October 8, 2013 to December 2014<br />
|XML Version 4.4 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2015 to December 2016<br />
|XML Version 4.5 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|}<br />
<br />
=== APS Rosetta Stone ===<br />
<br />
The Advanced Patent System (APS) is a fixed-width text format used to store historical patent grant data. The documentation for this sucks; there are pages missing at random. Luckily, we only care about the content contained here: [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]].<br />
<br />
It's worth mentioning that the APS contains an advanced text markup system for chemical formulae, basic text markup, tables, etc. that can lead to seemingly garbled text that is perfectly well-formed.<br />
<br />
==== APS Gotchas ====<br />
<br />
* PATN.WKU is the granted patent number. It is 7 digits while the spec promises 6 digits. The rightmost digit is a check digit modulus 11. See [[File:Aps-wku-modulus11.pdf]] for the words from the horse's mouth.<br />
<br />
=== Patents (Applications) ===<br />
<br />
{| class="wikitable"<br />
|+Patent Application Data Formats<br />
|-<br />
! scope="col" | Dates Used !! scope="col" | Format !! scope="col" | Location !! scope="col" | Supported by Parser?<br />
|-<br />
|March 15, 2001 to December 2001<br />
|XML Version 1.5<br />
|<code>data/extracts/applications/vintage</code><br />
|style="background: yellow;" | Yes, for basic information, inventors, and correspondents<br />
|-<br />
|January 2002 to December 2004<br />
|XML Version 1.6<br />
|<code>data/extracts/applications/vintage</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2005 to December 2005<br />
|XML Version 4.0 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2006 to December 2006<br />
|XML Version 4.1 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2007 to December 2012<br />
|XML Version 4.2 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2013 to December 2014<br />
|XML Version 4.3 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2015 to ''Present''<br />
|XML Version 4.4 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|}<br />
<br />
== Database ==<br />
<br />
Because there isn't a compelling reason not to, I used the existing PostgreSQL infrastructure on the RDP.<br />
The "Java Way" of interacting with databases is the Java Database Connectivity API (JDBC), an implementation-agnostic API for interacting with databases.<br />
This project uses the stock [https://jdbc.postgresql.org/ Postgres JDBC], version 42.1.1<br />
<br />
=== Create an empty database on RDP ===<br />
<br />
To create an empty database, run this command: <code>$ createdb --username=postgres database-name-goes-here # password is tabspaceenter</code><br />
<br />
=== Abstraction Layer ===<br />
<br />
Since writing raw SQL is a bit cumbersome and error-prone, I have added some abstraction layers that make it much easier to quickly add bulk data. By using Postgres's <code>CopyManager</code> class, we buffer SQL copy commands in memory (as many as possible) and then flush these rows. To understand how the abstraction layers work, see the code in <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\postgres</code>. See <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\models\GrantedPatent.java</code> for '''an example of how to extend''' the abstraction layer to deal with more complex scenarios.<br />
<br />
=== New Table Checklist ==<br />
<br />
* Create schema DDL SQL code for the new table in <code>E:\McNair\Projects\SimplerPatentData\src\db</code><br />
* Run the schema creation<br />
* Create an enum with the same names for attributes as in the DDL (case-insensitive! prefer all-caps screaming snake case)<br />
* Create a class which subclasses <code>AbstractIntertableData</code><br />
* Inside that class, create a static class which subclasses <code>AbstractTableMetadata</code> and has the proper values for getTableName(), getStringColumns(), getIntColumns()<br />
* (Optional) Implement builder pattern<br />
* (Optional) Create a custom databasehelper for complex extras (see PatentApplication and GrantedPatent for examples)<br />
* Write the data to the table (see DatabaseHelper for the pattern I use)<br />
<br />
== Address Data ==<br />
<br />
To get the most granular address data (street level, or at least postcode level) about who owns patents, the path is not so straightforward because off the complicated mapping of ownership to a granted patent.<br />
This is the final part of this project that I am working on and it is all at the level of SQL.<br />
See <code>E:\McNair\Projects\SimplerPatentData\src\db\joins</code> for my attempts to create a clean mapping.<br />
Optimistically speaking, the data generated here should be superset of the data present in the Patent Assignment Data Restructure project.<br />
<br />
Note that as of the beginning of August 2017, this part '''has not been completed.'''<br />
<br />
<br />
=== Intuition ===<br />
<br />
Use <code>assignments_longform.last_update_date</code> to find current/latest (or first/earliest) date of assignment. Then match with <code>properties.docid</code> on <code>reelno, frameno</code> to find patent application id. With this mapping to granted patents, we can discover the details of the original granted patent. And with the right date and reelno and frameno, we can match to the <code>assignees</code> table and get fine granularity addresses.<br />
<br />
== Related Pages ==<br />
<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries, Summer 2017 by this guy]]<br />
* [[US_Address_Verification|US Address Verification, Summer 2017 based on tables from Assignment Data Restructure]]<br />
* [[Patent_Assignment_Data_Restructure|Assignment Data Restructure, Spring 2017 by Marcela and Sonia]]<br />
* [[Redesigning_Patent_Database|Redesigning Patent Database, Spring 2017 by Shelby]]<br />
* [[Patent_Data_Cleanup_(June_2016)|Patent Data Cleanup, June 2016 by Marcela]]<br />
* [[Patent_Data|Patent Data, Spring 2016 by Marcela]] <br />
* [[Lex_Machina|Lex Machina]]<br />
* [[USPTO_Patent_Litigation_Data|USPTO Patent Litigation Research Dataset by Ed]]<br />
* [[Patent_Litigation_and_Review|Patent Litigation and Review by Marcela]]<br />
* [[Bag_of_Words_Analysis_of_Patent_Data|Bag of Words Analysis]]<br />
* [[Patent|Existing Database Schema]]<br />
* [[Oliver_Chang_(Work_Log)|My Work Log]]<br />
<br />
== External Links ==<br />
<br />
* Understanding Assignment Data: [https://www.uspto.gov/sites/default/files/documents/USPTO_Patents_Assignment_Dataset_WP.pdf USPTO Documentation on their cleanup of this data]<br />
* [https://bulkdata.uspto.gov/data2/patent/grant/redbook/fulltext/1976/PatentFullTextAPSDoc_GreenBook.pdf USPTO Green Book (APS) Documentation]<br />
* [https://bulkdata.uspto.gov/ USPTO Bulk Data Storage System (BDSS)]<br />
* [https://en.wikipedia.org/wiki/Builder_pattern Builder Pattern in Object-Oriented Programming]<br />
* [https://rdp.mcnaircenter.org/codebase/Repository/ReproduciblePatent Git Repository]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Reproducible_Patent_Data&diff=21418Reproducible Patent Data2017-10-31T22:33:45Z<p>OliverC: /* Patents (Applications) */</p>
<hr />
<div>{{McNair Projects<br />
|Has Image=Uspto web logo.jpg<br />
|Has title=Reproducible Patent Data<br />
|Has owner=Oliver Chang<br />
|Has start date=May 17<br />
|Has project status=Active<br />
|Does subsume=Redesigning Patent Database, Patent Assignment Data Restructure,<br />
}}<br />
<br />
A continuation of [[Redesigning Patent Database]] that aims to write faster, more centralized code to deal with data from the United States Patent and Trademark Office (USPTO). By having an end-to-end pipeline we can easily reproduce or update data without worrying about unintentional side effects or missing data.<br />
<br />
== Quickstart ==<br />
<br />
To get up and running with the code, do the following:<br />
<br />
# Clone the git project (link at end of page) to your user directory<br />
# Launch IntelliJ with >= Java 8 and Maven configured (default version installed on the RDP is setup to do this)<br />
# Open project in IntelliJ<br />
# Create an empty database (see [[#Database]])<br />
# Run the table creation scripts in <code>src/db/schemas/</code> in your new database<br />
# Modify the constant <code>DATABASE_NAME</code> in <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\postgres\DatabaseHelper.java</code><br />
# Run the Driver scripts in IntelliJ with the correct value for <code>DATA_DIRECTORY</code> (or run <code>RunInitialImport.java</code> which will do all of the data directories for that patent item type)<br />
# [Take a really, really long lunch...in total should take no more than five hours to load data on RDP]<br />
# Run scripts in <code>src/db/constraints</code> to check data assumptions<br />
# That's it!<br />
<br />
== Directory Layout ==<br />
<br />
=== Where is the Data? ===<br />
<br />
==== Directories ====<br />
<br />
All of the information for this project is located at <code>E:\McNair\Projects\SimplerPatentData</code><br />
<br />
There are several interesting directories:<br />
<br />
* <code>data/downloads/</code> is USPTO bulkdata, unmodified straight from the scraper<br />
* <code>data/extracts/</code> is a directory of a strict subset of the information stored in <code>data/downloads/</code>. It is the result of running a bulk 7-zip job on that directory to get everything unzipped in a flat data structure. Note that these files have the USPTO modified-by time since that metadata is stored in the zipfiles. To extract files in this nice format, select all of the zipfiles and setup an extraction job like in this [[media:7zip-params.png|screenshot]]<br />
* <code>data/backups/</code> is a 7zip'd backup of the corresponding directory in extracts<br />
* <code>src/</code> is the main code repository for the java project<br />
<br />
==== Input Files ====<br />
<br />
All of the text-only Red Book files for '''granted patents''' from 1976 to 2016, inclusive. To find a specific year's XML file, find it in<br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\granted\</code><br />
<br />
'''To find application data''' from 2001 to 2016, inclusive, look in<br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\applications\</code><br />
<br />
'''To find assignment data''', look in <br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\granted\</code><br />
<br />
'''To find maintenance fee data''', look in <br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\downloads\maintenance\</code><br />
<br />
=== Where is the Code? ===<br />
<br />
The code has the same parent directory as the data, so it is at <code>E:\McNair\Projects\SimplerPatentData\src</code>. You might notice a lot of single-entry directories; this is an idiomatic Java pattern that is used for package separation. If using IntelliJ or some other IDE, these directories are a bit less annoying.<br />
<br />
The development environment is Java 8 JDK, IntelliJ Ultimate IDE, Maven build tools, and git VCS.<br />
<br />
The git repository can be found at https://rdp.mcnaircenter.org/codebase/Repository/ReproduciblePatent<br />
<br />
==== Prior Art ====<br />
<br />
This tool is not so concerned with adding new functionality; rather, it aims to take a bunch of spread out Perl scripts and create a faster system that is easier to work with. As such, its functionality is largely stolen from those scripts:<br />
<br />
* Downloader: <code>E:\McNair\Software\Scripts\Patent\USPTO_Parser.pl</code><br />
* XML Splitter: <code>E:\McNair\PatentData\splitter.pl</code><br />
* XML Parsing: <code>E:\McNair\PatentData\Processed\xmlparser_4.5_4.4_4.3.pl</code> and <code>E:\McNair\PatentData\Processed\*.pm</code><br />
<br />
In addition, I used several non-standard Java libraries listed below:<br />
<br />
* [https://github.com/mashape/unirest-java/ Unirest] for easy HTTP requests (MIT License)<br />
* [https://github.com/google/guava Google Guava] for immutable collections and Stream utilities (Apache v2.0 License)<br />
* [https://github.com/jhy/jsoup/ jsoup] for HTML parsing (MIT License)<br />
* [http://commons.apache.org/proper/commons-codec/ Apache Commons Codec] (Apache v2.0 License)<br />
* [http://commons.apache.org/proper/commons-lang/ Apache Commons Lang v3] (Apache v2.0 License)<br />
* [https://mvnrepository.com/artifact/org.jetbrains/annotations/15.0 Jetbrains Annotations] for enhanced null checks (Apache v2.0 License)<br />
* [http://search.maven.org/#artifactdetails%7Corg.postgresql%7Cpostgresql%7C42.1.1.jre7%7Cbundle PostgreSQL JDBC] (BSD 3-clause per https://github.com/pgjdbc/pgjdbc-jre7/blob/master/LICENSE)<br />
<br />
If using maven, these dependencies are listed and should automatically be setup.<br />
<br />
==== Using Code ====<br />
<br />
Any file with a line that says <code>public static void main(String[] args) {</code> can be run as a standalone file. The easiest way to do this is to load the project and then the file in IntelliJ and click the little green play arrow next to this bit of code.<br />
<br />
The code can also be run via the standard <code>javac</code> and <code>java</code> commands but since this project has a complicated structure you end up having to run commands like <br />
<br />
<code>"C:\Program Files\Java\jdk1.8.0_131\bin\java" "-javaagent:C:\Users\OliverC\IntelliJ IDEA 2017.1.3\lib\idea_rt.jar=62364:C:\Users\OliverC\IntelliJ IDEA 2017.1.3\bin" -Dfile.encoding=UTF-8 -classpath "[...contents truncated...];C:\Users\OliverC\.m2\repository\org\postgresql\postgresql\42.1.1\postgresql-42.1.1.jar" org.bakerinstitute.mcnair.uspto_assignments.XmlDriver</code><br />
<br />
to include all of the runtime dependencies and it's just not worth it.<br />
<br />
==== Altering Code ====<br />
<br />
* Use the IntelliJ command Reformat code (found in the menus at <code>Code > Reformat Code</code><br />
* Use the optimize imports function found under the same menu<br />
* Use spaces for indentation<br />
* Loosely try to keep lines below 120 characters<br />
* Commit changes to the Git remote repository "bonobo"<br />
<br />
== Schema Reconciliation ==<br />
<br />
For the work by Joe, see the [[Patent Schema Reconciliation]] page <br />
<br />
=== Patents (Granted) ===<br />
<br />
See <code>E:\McNair\Projects\SimplerPatentData\data\examples\granted</code> for extracted examples of what specific data is available for a sample of the data.<br />
<br />
{| class="wikitable"<br />
|+Granted Patent Data Formats<br />
|-<br />
! scope="col" | Dates Used !! scope="col" | Format !! scope="col" | Location !! scope="col" | Supported? !! scope="col" | Utility !! scope="col" | Reissue !! scope="col" | Design !! scope="col" | Plant<br />
|-<br />
|January 1976 to December 2001<br />
|APS<br />
|<code>data/extracts/granted/vintage</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|<del>January 2001 to December 2001</del><br />
|<del>SGML</del><br />
|Ignored; use concurrently recorded APS data<br />
|No<br />
|N/A<br />
|N/A<br />
|N/A<br />
|N/A<br />
|-<br />
|January 2002 to December 2004<br />
|XML Version 2.5<br />
|<code>data/extracts/granted/blunderyears</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2005 to December 2005<br />
|XML Version 4.0 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2006 to December 2006<br />
|XML Version 4.1 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2007 to December 2012<br />
|XML Version 4.2 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2013 to September 24, 2013<br />
|XML Version 4.3 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|October 8, 2013 to December 2014<br />
|XML Version 4.4 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2015 to December 2016<br />
|XML Version 4.5 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|}<br />
<br />
=== APS Rosetta Stone ===<br />
<br />
The Advanced Patent System (APS) is a fixed-width text format used to store historical patent grant data. The documentation for this sucks; there are pages missing at random. Luckily, we only care about the content contained here: [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]].<br />
<br />
It's worth mentioning that the APS contains an advanced text markup system for chemical formulae, basic text markup, tables, etc. that can lead to seemingly garbled text that is perfectly well-formed.<br />
<br />
==== APS Gotchas ====<br />
<br />
* PATN.WKU is the granted patent number. It is 7 digits while the spec promises 6 digits. The rightmost digit is a check digit modulus 11. See [[File:Aps-wku-modulus11.pdf]] for the words from the horse's mouth.<br />
<br />
=== Patents (Applications) ===<br />
<br />
{| class="wikitable"<br />
|+Patent Application Data Formats<br />
|-<br />
! scope="col" | Dates Used !! scope="col" | Format !! scope="col" | Location !! scope="col" | Supported by Parser?<br />
|-<br />
|March 15, 2001 to December 2001<br />
|XML Version 1.5<br />
|<code>data/extracts/applications/vintage</code><br />
|style="background: yellow;" | Yes, for basic information, inventors, and correspondents<br />
|-<br />
|January 2002 to December 2004<br />
|XML Version 1.6<br />
|<code>data/extracts/applications/vintage</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2005 to December 2005<br />
|XML Version 4.0 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2006 to December 2006<br />
|XML Version 4.1 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2007 to December 2012<br />
|XML Version 4.2 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2013 to December 2014<br />
|XML Version 4.3 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|-<br />
|January 2015 to ''Present''<br />
|XML Version 4.4 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Ditto<br />
|}<br />
<br />
== Database ==<br />
<br />
Because there isn't a compelling reason not to, I used the existing PostgreSQL infrastructure on the RDP.<br />
The "Java Way" of interacting with databases is the Java Database Connectivity API (JDBC), an implementation-agnostic API for interacting with databases.<br />
This project uses the stock [https://jdbc.postgresql.org/ Postgres JDBC], version 42.1.1<br />
<br />
=== Create an empty database on RDP ===<br />
<br />
To create an empty database, run this command: <code>$ createdb --username=postgres database-name-goes-here # password is tabspaceenter</code><br />
<br />
=== Abstraction Layer ===<br />
<br />
Since writing raw SQL is a bit cumbersome and error-prone, I have added some abstraction layers that make it much easier to quickly add bulk data. By using Postgres's <code>CopyManager</code> class, we buffer SQL copy commands in memory (as many as possible) and then flush these rows. To understand how the abstraction layers work, see the code in <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\postgres</code>. See <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\models\GrantedPatent.java</code> for '''an example of how to extend''' the abstraction layer to deal with more complex scenarios.<br />
<br />
== Address Data ==<br />
<br />
To get the most granular address data (street level, or at least postcode level) about who owns patents, the path is not so straightforward because off the complicated mapping of ownership to a granted patent.<br />
This is the final part of this project that I am working on and it is all at the level of SQL.<br />
See <code>E:\McNair\Projects\SimplerPatentData\src\db\joins</code> for my attempts to create a clean mapping.<br />
Optimistically speaking, the data generated here should be superset of the data present in the Patent Assignment Data Restructure project.<br />
<br />
Note that as of the beginning of August 2017, this part '''has not been completed.'''<br />
<br />
<br />
=== Intuition ===<br />
<br />
Use <code>assignments_longform.last_update_date</code> to find current/latest (or first/earliest) date of assignment. Then match with <code>properties.docid</code> on <code>reelno, frameno</code> to find patent application id. With this mapping to granted patents, we can discover the details of the original granted patent. And with the right date and reelno and frameno, we can match to the <code>assignees</code> table and get fine granularity addresses.<br />
<br />
== Related Pages ==<br />
<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries, Summer 2017 by this guy]]<br />
* [[US_Address_Verification|US Address Verification, Summer 2017 based on tables from Assignment Data Restructure]]<br />
* [[Patent_Assignment_Data_Restructure|Assignment Data Restructure, Spring 2017 by Marcela and Sonia]]<br />
* [[Redesigning_Patent_Database|Redesigning Patent Database, Spring 2017 by Shelby]]<br />
* [[Patent_Data_Cleanup_(June_2016)|Patent Data Cleanup, June 2016 by Marcela]]<br />
* [[Patent_Data|Patent Data, Spring 2016 by Marcela]] <br />
* [[Lex_Machina|Lex Machina]]<br />
* [[USPTO_Patent_Litigation_Data|USPTO Patent Litigation Research Dataset by Ed]]<br />
* [[Patent_Litigation_and_Review|Patent Litigation and Review by Marcela]]<br />
* [[Bag_of_Words_Analysis_of_Patent_Data|Bag of Words Analysis]]<br />
* [[Patent|Existing Database Schema]]<br />
* [[Oliver_Chang_(Work_Log)|My Work Log]]<br />
<br />
== External Links ==<br />
<br />
* Understanding Assignment Data: [https://www.uspto.gov/sites/default/files/documents/USPTO_Patents_Assignment_Dataset_WP.pdf USPTO Documentation on their cleanup of this data]<br />
* [https://bulkdata.uspto.gov/data2/patent/grant/redbook/fulltext/1976/PatentFullTextAPSDoc_GreenBook.pdf USPTO Green Book (APS) Documentation]<br />
* [https://bulkdata.uspto.gov/ USPTO Bulk Data Storage System (BDSS)]<br />
* [https://en.wikipedia.org/wiki/Builder_pattern Builder Pattern in Object-Oriented Programming]<br />
* [https://rdp.mcnaircenter.org/codebase/Repository/ReproduciblePatent Git Repository]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Reproducible_Patent_Data&diff=21417Reproducible Patent Data2017-10-31T22:31:08Z<p>OliverC: /* Quickstart */</p>
<hr />
<div>{{McNair Projects<br />
|Has Image=Uspto web logo.jpg<br />
|Has title=Reproducible Patent Data<br />
|Has owner=Oliver Chang<br />
|Has start date=May 17<br />
|Has project status=Active<br />
|Does subsume=Redesigning Patent Database, Patent Assignment Data Restructure,<br />
}}<br />
<br />
A continuation of [[Redesigning Patent Database]] that aims to write faster, more centralized code to deal with data from the United States Patent and Trademark Office (USPTO). By having an end-to-end pipeline we can easily reproduce or update data without worrying about unintentional side effects or missing data.<br />
<br />
== Quickstart ==<br />
<br />
To get up and running with the code, do the following:<br />
<br />
# Clone the git project (link at end of page) to your user directory<br />
# Launch IntelliJ with >= Java 8 and Maven configured (default version installed on the RDP is setup to do this)<br />
# Open project in IntelliJ<br />
# Create an empty database (see [[#Database]])<br />
# Run the table creation scripts in <code>src/db/schemas/</code> in your new database<br />
# Modify the constant <code>DATABASE_NAME</code> in <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\postgres\DatabaseHelper.java</code><br />
# Run the Driver scripts in IntelliJ with the correct value for <code>DATA_DIRECTORY</code> (or run <code>RunInitialImport.java</code> which will do all of the data directories for that patent item type)<br />
# [Take a really, really long lunch...in total should take no more than five hours to load data on RDP]<br />
# Run scripts in <code>src/db/constraints</code> to check data assumptions<br />
# That's it!<br />
<br />
== Directory Layout ==<br />
<br />
=== Where is the Data? ===<br />
<br />
==== Directories ====<br />
<br />
All of the information for this project is located at <code>E:\McNair\Projects\SimplerPatentData</code><br />
<br />
There are several interesting directories:<br />
<br />
* <code>data/downloads/</code> is USPTO bulkdata, unmodified straight from the scraper<br />
* <code>data/extracts/</code> is a directory of a strict subset of the information stored in <code>data/downloads/</code>. It is the result of running a bulk 7-zip job on that directory to get everything unzipped in a flat data structure. Note that these files have the USPTO modified-by time since that metadata is stored in the zipfiles. To extract files in this nice format, select all of the zipfiles and setup an extraction job like in this [[media:7zip-params.png|screenshot]]<br />
* <code>data/backups/</code> is a 7zip'd backup of the corresponding directory in extracts<br />
* <code>src/</code> is the main code repository for the java project<br />
<br />
==== Input Files ====<br />
<br />
All of the text-only Red Book files for '''granted patents''' from 1976 to 2016, inclusive. To find a specific year's XML file, find it in<br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\granted\</code><br />
<br />
'''To find application data''' from 2001 to 2016, inclusive, look in<br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\applications\</code><br />
<br />
'''To find assignment data''', look in <br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\granted\</code><br />
<br />
'''To find maintenance fee data''', look in <br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\downloads\maintenance\</code><br />
<br />
=== Where is the Code? ===<br />
<br />
The code has the same parent directory as the data, so it is at <code>E:\McNair\Projects\SimplerPatentData\src</code>. You might notice a lot of single-entry directories; this is an idiomatic Java pattern that is used for package separation. If using IntelliJ or some other IDE, these directories are a bit less annoying.<br />
<br />
The development environment is Java 8 JDK, IntelliJ Ultimate IDE, Maven build tools, and git VCS.<br />
<br />
The git repository can be found at https://rdp.mcnaircenter.org/codebase/Repository/ReproduciblePatent<br />
<br />
==== Prior Art ====<br />
<br />
This tool is not so concerned with adding new functionality; rather, it aims to take a bunch of spread out Perl scripts and create a faster system that is easier to work with. As such, its functionality is largely stolen from those scripts:<br />
<br />
* Downloader: <code>E:\McNair\Software\Scripts\Patent\USPTO_Parser.pl</code><br />
* XML Splitter: <code>E:\McNair\PatentData\splitter.pl</code><br />
* XML Parsing: <code>E:\McNair\PatentData\Processed\xmlparser_4.5_4.4_4.3.pl</code> and <code>E:\McNair\PatentData\Processed\*.pm</code><br />
<br />
In addition, I used several non-standard Java libraries listed below:<br />
<br />
* [https://github.com/mashape/unirest-java/ Unirest] for easy HTTP requests (MIT License)<br />
* [https://github.com/google/guava Google Guava] for immutable collections and Stream utilities (Apache v2.0 License)<br />
* [https://github.com/jhy/jsoup/ jsoup] for HTML parsing (MIT License)<br />
* [http://commons.apache.org/proper/commons-codec/ Apache Commons Codec] (Apache v2.0 License)<br />
* [http://commons.apache.org/proper/commons-lang/ Apache Commons Lang v3] (Apache v2.0 License)<br />
* [https://mvnrepository.com/artifact/org.jetbrains/annotations/15.0 Jetbrains Annotations] for enhanced null checks (Apache v2.0 License)<br />
* [http://search.maven.org/#artifactdetails%7Corg.postgresql%7Cpostgresql%7C42.1.1.jre7%7Cbundle PostgreSQL JDBC] (BSD 3-clause per https://github.com/pgjdbc/pgjdbc-jre7/blob/master/LICENSE)<br />
<br />
If using maven, these dependencies are listed and should automatically be setup.<br />
<br />
==== Using Code ====<br />
<br />
Any file with a line that says <code>public static void main(String[] args) {</code> can be run as a standalone file. The easiest way to do this is to load the project and then the file in IntelliJ and click the little green play arrow next to this bit of code.<br />
<br />
The code can also be run via the standard <code>javac</code> and <code>java</code> commands but since this project has a complicated structure you end up having to run commands like <br />
<br />
<code>"C:\Program Files\Java\jdk1.8.0_131\bin\java" "-javaagent:C:\Users\OliverC\IntelliJ IDEA 2017.1.3\lib\idea_rt.jar=62364:C:\Users\OliverC\IntelliJ IDEA 2017.1.3\bin" -Dfile.encoding=UTF-8 -classpath "[...contents truncated...];C:\Users\OliverC\.m2\repository\org\postgresql\postgresql\42.1.1\postgresql-42.1.1.jar" org.bakerinstitute.mcnair.uspto_assignments.XmlDriver</code><br />
<br />
to include all of the runtime dependencies and it's just not worth it.<br />
<br />
==== Altering Code ====<br />
<br />
* Use the IntelliJ command Reformat code (found in the menus at <code>Code > Reformat Code</code><br />
* Use the optimize imports function found under the same menu<br />
* Use spaces for indentation<br />
* Loosely try to keep lines below 120 characters<br />
* Commit changes to the Git remote repository "bonobo"<br />
<br />
== Schema Reconciliation ==<br />
<br />
For the work by Joe, see the [[Patent Schema Reconciliation]] page <br />
<br />
=== Patents (Granted) ===<br />
<br />
See <code>E:\McNair\Projects\SimplerPatentData\data\examples\granted</code> for extracted examples of what specific data is available for a sample of the data.<br />
<br />
{| class="wikitable"<br />
|+Granted Patent Data Formats<br />
|-<br />
! scope="col" | Dates Used !! scope="col" | Format !! scope="col" | Location !! scope="col" | Supported? !! scope="col" | Utility !! scope="col" | Reissue !! scope="col" | Design !! scope="col" | Plant<br />
|-<br />
|January 1976 to December 2001<br />
|APS<br />
|<code>data/extracts/granted/vintage</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|<del>January 2001 to December 2001</del><br />
|<del>SGML</del><br />
|Ignored; use concurrently recorded APS data<br />
|No<br />
|N/A<br />
|N/A<br />
|N/A<br />
|N/A<br />
|-<br />
|January 2002 to December 2004<br />
|XML Version 2.5<br />
|<code>data/extracts/granted/blunderyears</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2005 to December 2005<br />
|XML Version 4.0 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2006 to December 2006<br />
|XML Version 4.1 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2007 to December 2012<br />
|XML Version 4.2 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2013 to September 24, 2013<br />
|XML Version 4.3 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|October 8, 2013 to December 2014<br />
|XML Version 4.4 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2015 to December 2016<br />
|XML Version 4.5 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|}<br />
<br />
=== APS Rosetta Stone ===<br />
<br />
The Advanced Patent System (APS) is a fixed-width text format used to store historical patent grant data. The documentation for this sucks; there are pages missing at random. Luckily, we only care about the content contained here: [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]].<br />
<br />
It's worth mentioning that the APS contains an advanced text markup system for chemical formulae, basic text markup, tables, etc. that can lead to seemingly garbled text that is perfectly well-formed.<br />
<br />
==== APS Gotchas ====<br />
<br />
* PATN.WKU is the granted patent number. It is 7 digits while the spec promises 6 digits. The rightmost digit is a check digit modulus 11. See [[File:Aps-wku-modulus11.pdf]] for the words from the horse's mouth.<br />
<br />
=== Patents (Applications) ===<br />
<br />
{| class="wikitable"<br />
|+Patent Application Data Formats<br />
|-<br />
! scope="col" | Dates Used !! scope="col" | Format !! scope="col" | Location !! scope="col" | Supported by Parser?<br />
|-<br />
|March 15, 2001 to December 2001<br />
|XML Version 1.5<br />
|<code>data/extracts/applications/vintage</code><br />
|style="background: red; color: white;" | No<br />
|-<br />
|January 2002 to December 2004<br />
|XML Version 1.6<br />
|<code>data/extracts/applications/vintage</code><br />
|style="background: red; color: white;" | No<br />
|-<br />
|January 2005 to December 2005<br />
|XML Version 4.0 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Maybe<br />
|-<br />
|January 2006 to December 2006<br />
|XML Version 4.1 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Maybe<br />
|-<br />
|January 2007 to December 2012<br />
|XML Version 4.2 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Maybe<br />
|-<br />
|January 2013 to December 2014<br />
|XML Version 4.3 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Maybe<br />
|-<br />
|January 2015 to ''Present''<br />
|XML Version 4.4 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Maybe<br />
|}<br />
<br />
== Database ==<br />
<br />
Because there isn't a compelling reason not to, I used the existing PostgreSQL infrastructure on the RDP.<br />
The "Java Way" of interacting with databases is the Java Database Connectivity API (JDBC), an implementation-agnostic API for interacting with databases.<br />
This project uses the stock [https://jdbc.postgresql.org/ Postgres JDBC], version 42.1.1<br />
<br />
=== Create an empty database on RDP ===<br />
<br />
To create an empty database, run this command: <code>$ createdb --username=postgres database-name-goes-here # password is tabspaceenter</code><br />
<br />
=== Abstraction Layer ===<br />
<br />
Since writing raw SQL is a bit cumbersome and error-prone, I have added some abstraction layers that make it much easier to quickly add bulk data. By using Postgres's <code>CopyManager</code> class, we buffer SQL copy commands in memory (as many as possible) and then flush these rows. To understand how the abstraction layers work, see the code in <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\postgres</code>. See <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\models\GrantedPatent.java</code> for '''an example of how to extend''' the abstraction layer to deal with more complex scenarios.<br />
<br />
== Address Data ==<br />
<br />
To get the most granular address data (street level, or at least postcode level) about who owns patents, the path is not so straightforward because off the complicated mapping of ownership to a granted patent.<br />
This is the final part of this project that I am working on and it is all at the level of SQL.<br />
See <code>E:\McNair\Projects\SimplerPatentData\src\db\joins</code> for my attempts to create a clean mapping.<br />
Optimistically speaking, the data generated here should be superset of the data present in the Patent Assignment Data Restructure project.<br />
<br />
Note that as of the beginning of August 2017, this part '''has not been completed.'''<br />
<br />
<br />
=== Intuition ===<br />
<br />
Use <code>assignments_longform.last_update_date</code> to find current/latest (or first/earliest) date of assignment. Then match with <code>properties.docid</code> on <code>reelno, frameno</code> to find patent application id. With this mapping to granted patents, we can discover the details of the original granted patent. And with the right date and reelno and frameno, we can match to the <code>assignees</code> table and get fine granularity addresses.<br />
<br />
== Related Pages ==<br />
<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries, Summer 2017 by this guy]]<br />
* [[US_Address_Verification|US Address Verification, Summer 2017 based on tables from Assignment Data Restructure]]<br />
* [[Patent_Assignment_Data_Restructure|Assignment Data Restructure, Spring 2017 by Marcela and Sonia]]<br />
* [[Redesigning_Patent_Database|Redesigning Patent Database, Spring 2017 by Shelby]]<br />
* [[Patent_Data_Cleanup_(June_2016)|Patent Data Cleanup, June 2016 by Marcela]]<br />
* [[Patent_Data|Patent Data, Spring 2016 by Marcela]] <br />
* [[Lex_Machina|Lex Machina]]<br />
* [[USPTO_Patent_Litigation_Data|USPTO Patent Litigation Research Dataset by Ed]]<br />
* [[Patent_Litigation_and_Review|Patent Litigation and Review by Marcela]]<br />
* [[Bag_of_Words_Analysis_of_Patent_Data|Bag of Words Analysis]]<br />
* [[Patent|Existing Database Schema]]<br />
* [[Oliver_Chang_(Work_Log)|My Work Log]]<br />
<br />
== External Links ==<br />
<br />
* Understanding Assignment Data: [https://www.uspto.gov/sites/default/files/documents/USPTO_Patents_Assignment_Dataset_WP.pdf USPTO Documentation on their cleanup of this data]<br />
* [https://bulkdata.uspto.gov/data2/patent/grant/redbook/fulltext/1976/PatentFullTextAPSDoc_GreenBook.pdf USPTO Green Book (APS) Documentation]<br />
* [https://bulkdata.uspto.gov/ USPTO Bulk Data Storage System (BDSS)]<br />
* [https://en.wikipedia.org/wiki/Builder_pattern Builder Pattern in Object-Oriented Programming]<br />
* [https://rdp.mcnaircenter.org/codebase/Repository/ReproduciblePatent Git Repository]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=PostGIS_Installation&diff=21124PostGIS Installation2017-10-25T21:59:32Z<p>OliverC: /* Installation Commands */</p>
<hr />
<div>[http://postgis.net/install/ PostGIS] is a spatial database extender for PostgreSQL object-relational database. It adds support for geographic objects allowing location queries to be run in SQL.<br />
<br />
<br />
== Installation Commands ==<br />
<br />
Going off of http://trac.osgeo.org/postgis/wiki/UsersWikiPostGIS23UbuntuPGSQL96Apt<br />
<br />
<nowiki><br />
$ lsb_release --codename<br />
Codename: trusty<br />
$ sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt trusty-pgdg main" >> /etc/apt/sources.list'<br />
$ wget --quiet -O - http://apt.postgresql.org/pub/repos/apt/ACCC4CF8.asc | sudo apt-key add -<br />
$ sudo apt-get update<br />
$ psql -V<br />
psql (PostgreSQL) 9.5.4<br />
$ sudo apt install postgresql-9.5-postgis-2.3<br />
$ sudo apt-get install --no-install-recommends postgis</nowiki><br />
<br />
Note the last line. Without the <code>--no-install-recommends</code> line postgres assumes you're using the default configuration (in this case, postgis for postgresql 9.6). Since we're using a different flavor of postgresql, we do not want this.<br />
<br />
The <code>postgis</code> apt package is not the extension to postgres. It is a collection of command line utilities for importing and exporting data to a postgis db.<br />
<br />
== Creating a Postgis DB ==<br />
<br />
=== Initial Setup ===<br />
<br />
<nowiki><br />
$ createdb --username researcher tigertest<br />
$ psql --username researcher --dbname tigertest<br />
> create extension postgis;<br />
> select postgis_full_version(); -- sanity test and make sure installed and enabled<br />
> \q</nowiki><br />
<br />
=== Bulk Download TIGER Shapefiles ===<br />
<br />
For example, say we want all of the state-level place data. The first step is to find a programmatic URL that we can use. You can inspect the HTML on the HTML interface to place data to get the correct mapping of states/territories to two digit integer. Note that they are not necessarily sequential and there is not strictly fifty.<br />
<br />
<nowiki><br />
$ for i in $(seq -f "%02g" 1 80); do<br />
wget "https://www2.census.gov/geo/tiger/TIGER2016/PLACE/tl_2016_${i}_place.zip";<br />
sleep 3 # not necessary but a good guy scraper move<br />
done;<br />
$ for f in *.zip; do unzip "$f"; done<br />
$ for f in *.shp; do shp2pgsql -I "$f" | psql -U researcher -d tigertest; done<br />
$ psql --username researcher --dbname tigertest<br />
> select count(*) from tl_2016_01_place; -- 585</nowiki><br />
<br />
Also say we want to separate New York City, New York into a more granular by-borough set of polygons. Using [https://geo.nyu.edu/catalog/nyu_2451_34505 City University of New York data], we will import this shapefile.<br />
<br />
<nowiki><br />
$ curl --insecure https://archive.nyu.edu/retrieve/74704/nyu_2451_34505.zip > /tmp/boroughs.zip<br />
$ cd /tmp<br />
$ unzip boroughs.zip<br />
$ shp2pgsql -I nyu_2451_34505/nyu_2451_34505.shp | psql -U researcher -d tigertest<br />
$ psql --username researcher --dbname tigertest<br />
> select count(*) from nyu_2451_34505;<br />
> \q</nowiki><br />
<br />
== To get into tigertest database ==<br />
ssh researcher@128.42.44.181<br />
cd /bulk<br />
psql tigertest<br />
<br />
== Translating Table names to corresponding States ==<br />
<br />
See: https://www.census.gov/geo/reference/ansi_statetables.html<br />
(note that the numbers are FIPS state numeric codes)<br />
<br />
{| class="wikitable"<br />
|-<br />
! Table Name<br />
! Corresponding State<br />
|-<br />
| tl_2016_01_place<br />
| Alabama<br />
|-<br />
| tl_2016_02_place<br />
| Alaska<br />
|-<br />
| tl_2016_03_place<br />
| <br />
|-<br />
| tl_2016_04_place<br />
| Arizona<br />
|-<br />
| tl_2016_05_place<br />
| Arkansas<br />
|-<br />
| tl_2016_06_place<br />
| California<br />
|-<br />
| tl_2016_07_place<br />
| <br />
|-<br />
| tl_2016_08_place<br />
| Colorado<br />
|-<br />
| tl_2016_09_place<br />
| Missouri<br />
|-<br />
| tl_2016_10_place<br />
| Delaware<br />
|-<br />
| tl_2016_11_place<br />
| District of Columbia<br />
|-<br />
| tl_2016_12_place<br />
| Florida<br />
|-<br />
| tl_2016_13_place<br />
| Georgia<br />
|-<br />
| tl_2016_14_place<br />
|<br />
|-<br />
| tl_2016_15_place<br />
| Hawaii<br />
|-<br />
| tl_2016_16_place<br />
| Idaho<br />
|-<br />
| tl_2016_17_place<br />
| Illinois<br />
|-<br />
| tl_2016_18_place<br />
| Indiana<br />
|-<br />
| tl_2016_19_place<br />
| Iowa<br />
|-<br />
| tl_2016_20_place<br />
| Kansas<br />
|-<br />
| tl_2016_21_place<br />
| Kentucky<br />
|-<br />
| tl_2016_22_place<br />
| Louisiana<br />
|-<br />
| tl_2016_23_place<br />
| Maine<br />
|-<br />
| tl_2016_24_place<br />
| Maryland<br />
|-<br />
| tl_2016_25_place<br />
| Massachusetts<br />
|-<br />
| tl_2016_26_place<br />
| Michigan<br />
|-<br />
| tl_2016_27_place<br />
| Minnesota<br />
|-<br />
| tl_2016_28_place<br />
| Mississippi<br />
|-<br />
| tl_2016_29_place<br />
| Missouri<br />
|-<br />
| tl_2016_30_place<br />
| Motana<br />
|-<br />
| tl_2016_31_place<br />
| Nebraska<br />
|-<br />
| tl_2016_32_place<br />
| Nevada<br />
|-<br />
| tl_2016_33_place<br />
| New Hampshire<br />
|-<br />
| tl_2016_34_place<br />
| New Jersey<br />
|-<br />
| tl_2016_35_place<br />
| New Mexico<br />
|-<br />
| tl_2016_36_place<br />
| New York<br />
|-<br />
| tl_2016_37_place<br />
| North Carolina<br />
|-<br />
| tl_2016_38_place<br />
| North Dakota<br />
|-<br />
| tl_2016_39_place<br />
| Ohio<br />
|-<br />
| tl_2016_40_place<br />
| Oklahoma<br />
|-<br />
| tl_2016_41_place<br />
| Oregon<br />
|-<br />
| tl_2016_42_place<br />
| Pennsylvania<br />
|-<br />
| tl_2016_43_place<br />
| <br />
|-<br />
| tl_2016_44_place<br />
| Rhode Island<br />
|-<br />
| tl_2016_45_place<br />
| South Carolina<br />
|-<br />
| tl_2016_46_place<br />
| South Dakota<br />
|-<br />
| tl_2016_47_place<br />
| Tennessee<br />
|-<br />
| tl_2016_48_place<br />
| Texas<br />
|-<br />
| tl_2016_49_place<br />
| Utah<br />
|-<br />
| tl_2016_50_place<br />
| Vermont<br />
|-<br />
| tl_2016_51_place<br />
| Virginia<br />
|-<br />
| tl_2016_52_place<br />
| <br />
|-<br />
| tl_2016_53_place<br />
| Washington<br />
|-<br />
| tl_2016_54_place<br />
| West Virginia<br />
|-<br />
| tl_2016_55_place<br />
| Wisconsin<br />
|-<br />
| tl_2016_56_place<br />
| Wyoming<br />
|-<br />
| tl_2016_57_place<br />
| <br />
|-<br />
| tl_2016_58_place<br />
| <br />
|-<br />
| tl_2016_59_place<br />
| <br />
|-<br />
| tl_2016_60_place<br />
| American Samoa<br />
|-<br />
| tl_2016_61_place<br />
| <br />
|-<br />
| tl_2016_62_place<br />
| <br />
|-<br />
| tl_2016_63_place<br />
| <br />
|-<br />
| tl_2016_64_place<br />
| <br />
|-<br />
| tl_2016_65_place<br />
| <br />
|-<br />
| tl_2016_66_place<br />
| Guam<br />
|-<br />
| tl_2016_67_place<br />
| <br />
|-<br />
| tl_2016_68_place<br />
| <br />
|-<br />
| tl_2016_69_place<br />
| Northern Marinas Islands<br />
|-<br />
| tl_2016_70_place<br />
| <br />
|-<br />
| tl_2016_71_place<br />
| <br />
|-<br />
| tl_2016_72_place<br />
| Puerto Rico<br />
|-<br />
| tl_2016_73_place<br />
| <br />
|-<br />
| tl_2016_74_place<br />
| <br />
|-<br />
| tl_2016_75_place<br />
| <br />
|-<br />
| tl_2016_76_place<br />
| <br />
|-<br />
| tl_2016_77_place<br />
| <br />
|-<br />
| tl_2016_78_place<br />
| Virgin Islands<br />
|}<br />
<br />
==Tiger Geocoder Extension==<br />
This section details the process to install and use the Tiger Geocoder Extension of PostGIS. The official docmentation can be found [https://postgis.net/docs/Extras.html here].<br />
<br />
[https://postgis.net/docs/postgis_installation.html#install_tiger_geocoder_extension This link] outlines the process to enable our Postgres Database to support Tiger functionality.<br />
<br />
I began by adding the extension listed above. First, enter into Postgres by using the psql command. Then:<br />
--Add Extensions to database<br />
CREATE EXTENSION postgis;<br />
CREATE EXTENSION fuzzystrmatch;<br />
CREATE EXTENSION postgis_tiger_geocoder;<br />
CREATE EXTENSION address_standardizer;<br />
<br />
You can test that the installation worked by running the following query: <br />
SELECT na.address, na.streetname,na.streettypeabbrev, na.zip<br />
FROM normalize_address('1 Devonshire Place, Boston, MA 02109') AS na;<br />
<br />
This should return the following:<br />
address | streetname | streettypeabbrev | zip<br />
---------+------------+------------------+-------<br />
1 | Devonshire | Pl | 02109<br />
<br />
Next, a new profile needs to be created by using the following command.<br />
INSERT INTO tiger.loader_platform(os, declare_sect, pgbin, wget, unzip_command, psql, path_sep, <br />
loader, environ_set_command, county_process_command)<br />
SELECT 'newuser', declare_sect, pgbin, wget, unzip_command, psql, path_sep,<br />
loader, environ_set_command, county_process_command<br />
FROM tiger.loader_platform<br />
WHERE os = 'sh';<br />
<br />
The installation instructions also provide the following note:<br />
<br />
As of PostGIS 2.4.1 the Zip code-5 digit tabulation area zcta5 load step was revised to load current zcta5 data and is part of the Loader_Generate_Nation_Script when enabled. It is turned off by default because it takes quite a bit of time to load (20 to 60 minutes), takes up quite a bit of disk space, and is not used that often.<br />
<br />
If you would like this feature, you can enable it by using the following command. This should be done before loading the script.<br />
<br />
UPDATE tiger.loader_lookuptables SET load = true WHERE table_name = 'zcta510';<br />
<br />
The paths in declare_sect need to be edited so they match our server locations. One option is to edit the declare_sect column in the tiger.loader_platform table. If so, the declare_sect looks like the following:<br />
export PGHOST=localhost +<br />
export PGUSER=postgres +<br />
export PGPASSWORD=yourpasswordhere +<br />
export PGDATABASE=geocoder +<br />
PSQL=${PGBIN}/psql +<br />
SHP2PGSQL=shp2pgsql +<br />
cd ${staging_fold} +<br />
<br />
TMPDIR="${staging_fold}/temp/" +<br />
UNZIPTOOL=unzip +<br />
WGETTOOL="/usr/bin/wget" +<br />
export PGBIN=/usr/lib/postgresql/9.6/bin +<br />
export PGPORT=5432 +<br />
export PGHOST=localhost +<br />
export PGUSER=postgres +<br />
export PGPASSWORD=yourpasswordhere +<br />
export PGDATABASE=geocoder +<br />
PSQL=${PGBIN}/psql +<br />
SHP2PGSQL=shp2pgsql +<br />
cd ${staging_fold}<br />
<br />
Another option is to edit the sh file before running the script. The downloaded script is located in the following directory:<br />
/gisdata<br />
<br />
There needs to be a directory called "temp" in the gisdata directory. To make the script, use the following from the command line:<br />
psql -c "SELECT Loader_Generate_Nation_Script('debbie')" -d geocoder -tA > /gisdata/nation_script_load.sh<br />
<br />
This will create a script in the gisdata directory. Change to that directory. If you did not edit the paths in the declare_sect table in psql, then you will need to edit this file to contain the correct paths. Run the script by using: <br />
sh nation_script_load.sh</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=21123Oliver Chang (Work Log)2017-10-25T21:58:45Z<p>OliverC: </p>
<hr />
<div>[[Oliver Chang]] [[Work Logs]]<br />
[[Oliver Chang (Work Log)|(logpage)]]<br />
<br />
''To-do List'':<br />
<br />
* Fix St. Louis problem<br />
* Query xml_explorer data<br />
* Create modern application xml schema (version > 1.6)<br />
* Re-run import of data Friday afternoon<br />
* Finish ID joining<br />
<br />
''Projects'':<br />
<br />
* [[PostGIS_Installation|PostGIS Installation]]<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
<br />
''Uploads'':<br />
<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
<br />
== Day-by-Day (in reverse chronological order) ==<br />
<br />
=== October 2017 ===<br />
<br />
* Oct 25: start ingestion of application xml files and deal with all the bugs which accompany that<br />
* Oct 21: create xml explorer script to mass-inspect xpaths (can be found at <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\xml_schema_explorer</code><br />
* Oct 20: re-run assignment import, review xpaths<br />
* Oct 19: repopulate patents database and create spreadsheet of assignment equivalencies<br />
* Oct 11: add issues to work on to the PECA wiki page; cleanup PECA git repo; start patent application code from granted patent code and start customizing to new domain<br />
* Oct 3: troubleshoot vc_circles.py and make command line interface a little nicer<br />
* Oct 2: discuss mapping strategies & investigate missing eca data<br />
<br />
=== September 2017 ===<br />
<br />
* Sept 23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
* Sept 22: goal setting & server debugging & meet with Yang<br />
<br />
=== August 2017 ===<br />
<br />
* Aug 4: setup parallel instance python framework for job reporting; begin test run<br />
* Aug 2: finish up some documentation of the code and for the wiki<br />
* Aug 1: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
=== July 2017 ===<br />
<br />
* July 31: sketch out parallel enclosing circle algorithm<br />
* July 28: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
* ''travelling''<br />
* July 19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
* July 18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
* July 17: redo db operations after cleaning up granted patent number bugs<br />
* July 13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
* July 12: generate some example data illustrating the difficulty of joining different tables<br />
* July 11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
* July 7: catch up on documentation<br />
* July 6: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
* July 5: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
=== June 2017 ===<br />
<br />
* June 30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
* June 29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
* June 28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
* June 27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
* June 26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
* June 25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
* June 23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
* June 22: create postcode<->patent table<br />
* June 21: document granted patent queries and equivalencies<br />
* June 20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
* June 19: skim address regular expressions; cursory investigation of patent table<br />
* June 16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
* June 15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
* June 14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
* June 13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
* June 12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
* June 8: add foreign key inserts; create pretty printer for XML analysis<br />
* June 7: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
* June 6: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
* June 5: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
* June 1: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
=== May 2017 ===<br />
<br />
* May 31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
* May 30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
* May 29: expand to APS; expand to raw assignment data<br />
* May 27: expand to maintenance fee data<br />
* May 26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
* May 25: sketch out OO design of project; download bulk data<br />
* May 24: move wiki pages around; start git repository for project<br />
* May 21: discuss technical details of previous work with Ed<br />
* May 8: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
* May 4: setup wiki account, rdp account, database training<br />
<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=21122Oliver Chang (Work Log)2017-10-25T21:53:12Z<p>OliverC: /* October 2017 */</p>
<hr />
<div>[[Oliver Chang]] [[Work Logs]]<br />
[[Oliver Chang (Work Log)|(logpage)]]<br />
<br />
''To-do List'':<br />
<br />
* Expand XPath use in the patent data<br />
* Edit to include Application data<br />
* Finish ID joining<br />
<br />
''Projects'':<br />
<br />
* [[PostGIS_Installation|PostGIS Installation]]<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
<br />
''Uploads'':<br />
<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
<br />
== Day-by-Day (in reverse chronological order) ==<br />
<br />
=== October 2017 ===<br />
<br />
* Oct 25: start ingestion of application xml files and deal with all the bugs which accompany that<br />
* Oct 21: create xml explorer script to mass-inspect xpaths (can be found at <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\xml_schema_explorer</code><br />
* Oct 20: re-run assignment import, review xpaths<br />
* Oct 19: repopulate patents database and create spreadsheet of assignment equivalencies<br />
* Oct 11: add issues to work on to the PECA wiki page; cleanup PECA git repo; start patent application code from granted patent code and start customizing to new domain<br />
* Oct 3: troubleshoot vc_circles.py and make command line interface a little nicer<br />
* Oct 2: discuss mapping strategies & investigate missing eca data<br />
<br />
=== September 2017 ===<br />
<br />
* Sept 23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
* Sept 22: goal setting & server debugging & meet with Yang<br />
<br />
=== August 2017 ===<br />
<br />
* Aug 4: setup parallel instance python framework for job reporting; begin test run<br />
* Aug 2: finish up some documentation of the code and for the wiki<br />
* Aug 1: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
=== July 2017 ===<br />
<br />
* July 31: sketch out parallel enclosing circle algorithm<br />
* July 28: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
* ''travelling''<br />
* July 19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
* July 18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
* July 17: redo db operations after cleaning up granted patent number bugs<br />
* July 13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
* July 12: generate some example data illustrating the difficulty of joining different tables<br />
* July 11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
* July 7: catch up on documentation<br />
* July 6: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
* July 5: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
=== June 2017 ===<br />
<br />
* June 30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
* June 29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
* June 28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
* June 27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
* June 26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
* June 25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
* June 23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
* June 22: create postcode<->patent table<br />
* June 21: document granted patent queries and equivalencies<br />
* June 20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
* June 19: skim address regular expressions; cursory investigation of patent table<br />
* June 16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
* June 15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
* June 14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
* June 13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
* June 12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
* June 8: add foreign key inserts; create pretty printer for XML analysis<br />
* June 7: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
* June 6: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
* June 5: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
* June 1: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
=== May 2017 ===<br />
<br />
* May 31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
* May 30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
* May 29: expand to APS; expand to raw assignment data<br />
* May 27: expand to maintenance fee data<br />
* May 26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
* May 25: sketch out OO design of project; download bulk data<br />
* May 24: move wiki pages around; start git repository for project<br />
* May 21: discuss technical details of previous work with Ed<br />
* May 8: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
* May 4: setup wiki account, rdp account, database training<br />
<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=21015Oliver Chang (Work Log)2017-10-23T20:51:23Z<p>OliverC: /* October 2017 */</p>
<hr />
<div>[[Oliver Chang]] [[Work Logs]]<br />
[[Oliver Chang (Work Log)|(logpage)]]<br />
<br />
''To-do List'':<br />
<br />
* Expand XPath use in the patent data<br />
* Edit to include Application data<br />
* Finish ID joining<br />
<br />
''Projects'':<br />
<br />
* [[PostGIS_Installation|PostGIS Installation]]<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
<br />
''Uploads'':<br />
<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
<br />
== Day-by-Day (in reverse chronological order) ==<br />
<br />
=== October 2017 ===<br />
<br />
* Oct 21: create xml explorer script to mass-inspect xpaths (can be found at <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\xml_schema_explorer</code><br />
* Oct 20: re-run assignment import, review xpaths<br />
* Oct 19: repopulate patents database and create spreadsheet of assignment equivalencies<br />
* Oct 11: add issues to work on to the PECA wiki page; cleanup PECA git repo; start patent application code from granted patent code and start customizing to new domain<br />
* Oct 3: troubleshoot vc_circles.py and make command line interface a little nicer<br />
* Oct 2: discuss mapping strategies & investigate missing eca data<br />
<br />
=== September 2017 ===<br />
<br />
* Sept 23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
* Sept 22: goal setting & server debugging & meet with Yang<br />
<br />
=== August 2017 ===<br />
<br />
* Aug 4: setup parallel instance python framework for job reporting; begin test run<br />
* Aug 2: finish up some documentation of the code and for the wiki<br />
* Aug 1: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
=== July 2017 ===<br />
<br />
* July 31: sketch out parallel enclosing circle algorithm<br />
* July 28: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
* ''travelling''<br />
* July 19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
* July 18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
* July 17: redo db operations after cleaning up granted patent number bugs<br />
* July 13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
* July 12: generate some example data illustrating the difficulty of joining different tables<br />
* July 11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
* July 7: catch up on documentation<br />
* July 6: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
* July 5: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
=== June 2017 ===<br />
<br />
* June 30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
* June 29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
* June 28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
* June 27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
* June 26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
* June 25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
* June 23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
* June 22: create postcode<->patent table<br />
* June 21: document granted patent queries and equivalencies<br />
* June 20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
* June 19: skim address regular expressions; cursory investigation of patent table<br />
* June 16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
* June 15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
* June 14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
* June 13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
* June 12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
* June 8: add foreign key inserts; create pretty printer for XML analysis<br />
* June 7: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
* June 6: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
* June 5: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
* June 1: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
=== May 2017 ===<br />
<br />
* May 31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
* May 30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
* May 29: expand to APS; expand to raw assignment data<br />
* May 27: expand to maintenance fee data<br />
* May 26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
* May 25: sketch out OO design of project; download bulk data<br />
* May 24: move wiki pages around; start git repository for project<br />
* May 21: discuss technical details of previous work with Ed<br />
* May 8: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
* May 4: setup wiki account, rdp account, database training<br />
<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=20986Oliver Chang (Work Log)2017-10-21T16:08:21Z<p>OliverC: /* October 2017 */</p>
<hr />
<div>[[Oliver Chang]] [[Work Logs]]<br />
[[Oliver Chang (Work Log)|(logpage)]]<br />
<br />
''To-do List'':<br />
<br />
* Expand XPath use in the patent data<br />
* Edit to include Application data<br />
* Finish ID joining<br />
<br />
''Projects'':<br />
<br />
* [[PostGIS_Installation|PostGIS Installation]]<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
<br />
''Uploads'':<br />
<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
<br />
== Day-by-Day (in reverse chronological order) ==<br />
<br />
=== October 2017 ===<br />
<br />
* Oct 20: re-run assignment import, review xpaths<br />
* Oct 19: repopulate patents database and create spreadsheet of assignment equivalencies<br />
* Oct 11: add issues to work on to the PECA wiki page; cleanup PECA git repo; start patent application code from granted patent code and start customizing to new domain<br />
* Oct 3: troubleshoot vc_circles.py and make command line interface a little nicer<br />
* Oct 2: discuss mapping strategies & investigate missing eca data<br />
<br />
=== September 2017 ===<br />
<br />
* Sept 23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
* Sept 22: goal setting & server debugging & meet with Yang<br />
<br />
=== August 2017 ===<br />
<br />
* Aug 4: setup parallel instance python framework for job reporting; begin test run<br />
* Aug 2: finish up some documentation of the code and for the wiki<br />
* Aug 1: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
=== July 2017 ===<br />
<br />
* July 31: sketch out parallel enclosing circle algorithm<br />
* July 28: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
* ''travelling''<br />
* July 19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
* July 18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
* July 17: redo db operations after cleaning up granted patent number bugs<br />
* July 13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
* July 12: generate some example data illustrating the difficulty of joining different tables<br />
* July 11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
* July 7: catch up on documentation<br />
* July 6: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
* July 5: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
=== June 2017 ===<br />
<br />
* June 30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
* June 29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
* June 28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
* June 27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
* June 26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
* June 25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
* June 23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
* June 22: create postcode<->patent table<br />
* June 21: document granted patent queries and equivalencies<br />
* June 20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
* June 19: skim address regular expressions; cursory investigation of patent table<br />
* June 16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
* June 15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
* June 14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
* June 13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
* June 12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
* June 8: add foreign key inserts; create pretty printer for XML analysis<br />
* June 7: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
* June 6: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
* June 5: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
* June 1: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
=== May 2017 ===<br />
<br />
* May 31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
* May 30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
* May 29: expand to APS; expand to raw assignment data<br />
* May 27: expand to maintenance fee data<br />
* May 26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
* May 25: sketch out OO design of project; download bulk data<br />
* May 24: move wiki pages around; start git repository for project<br />
* May 21: discuss technical details of previous work with Ed<br />
* May 8: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
* May 4: setup wiki account, rdp account, database training<br />
<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Reproducible_Patent_Data&diff=20961Reproducible Patent Data2017-10-20T18:30:30Z<p>OliverC: update application use dates and remove imaginary xml 4.5</p>
<hr />
<div>{{McNair Projects<br />
|Has Image=Uspto web logo.jpg<br />
|Has title=Reproducible Patent Data<br />
|Has owner=Oliver Chang<br />
|Has start date=May 17<br />
|Has project status=Active<br />
|Does subsume=Redesigning Patent Database, Patent Assignment Data Restructure,<br />
}}<br />
<br />
A continuation of [[Redesigning Patent Database]] that aims to write faster, more centralized code to deal with data from the United States Patent and Trademark Office (USPTO). By having an end-to-end pipeline we can easily reproduce or update data without worrying about unintentional side effects or missing data.<br />
<br />
== Quickstart ==<br />
<br />
To get up and running with the code, do the following:<br />
<br />
# Clone the git project (link at end of page) to your user directory<br />
# Launch IntelliJ with >= Java 8 and Maven configured (default version installed on the RDP is setup to do this)<br />
# Open project in IntelliJ<br />
# Create an empty database (see [[#Database]])<br />
# Run the table creation scripts in <code>src/db/schemas/</code> in your new database<br />
# Modify the constant <code>DATABASE_NAME</code> in <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\postgres\DatabaseHelper.java</code><br />
# Run the Driver scripts in IntelliJ with the correct value for <code>DATA_DIRECTORY</code><br />
# [Take a really, really long lunch...in total should take no more than five hours to load data on RDP]<br />
# Run scripts in <code>src/db/constraints</code> to check data assumptions<br />
# That's it!<br />
<br />
== Directory Layout ==<br />
<br />
=== Where is the Data? ===<br />
<br />
==== Directories ====<br />
<br />
All of the information for this project is located at <code>E:\McNair\Projects\SimplerPatentData</code><br />
<br />
There are several interesting directories:<br />
<br />
* <code>data/downloads/</code> is USPTO bulkdata, unmodified straight from the scraper<br />
* <code>data/extracts/</code> is a directory of a strict subset of the information stored in <code>data/downloads/</code>. It is the result of running a bulk 7-zip job on that directory to get everything unzipped in a flat data structure. Note that these files have the USPTO modified-by time since that metadata is stored in the zipfiles. To extract files in this nice format, select all of the zipfiles and setup an extraction job like in this [[media:7zip-params.png|screenshot]]<br />
* <code>data/backups/</code> is a 7zip'd backup of the corresponding directory in extracts<br />
* <code>src/</code> is the main code repository for the java project<br />
<br />
==== Input Files ====<br />
<br />
All of the text-only Red Book files for '''granted patents''' from 1976 to 2016, inclusive. To find a specific year's XML file, find it in<br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\granted\</code><br />
<br />
'''To find application data''' from 2001 to 2016, inclusive, look in<br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\applications\</code><br />
<br />
'''To find assignment data''', look in <br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\extracts\granted\</code><br />
<br />
'''To find maintenance fee data''', look in <br />
<br />
<code>E:\McNair\Projects\SimplerPatentData\data\downloads\maintenance\</code><br />
<br />
=== Where is the Code? ===<br />
<br />
The code has the same parent directory as the data, so it is at <code>E:\McNair\Projects\SimplerPatentData\src</code>. You might notice a lot of single-entry directories; this is an idiomatic Java pattern that is used for package separation. If using IntelliJ or some other IDE, these directories are a bit less annoying.<br />
<br />
The development environment is Java 8 JDK, IntelliJ Ultimate IDE, Maven build tools, and git VCS.<br />
<br />
The git repository can be found at https://rdp.mcnaircenter.org/codebase/Repository/ReproduciblePatent<br />
<br />
==== Prior Art ====<br />
<br />
This tool is not so concerned with adding new functionality; rather, it aims to take a bunch of spread out Perl scripts and create a faster system that is easier to work with. As such, its functionality is largely stolen from those scripts:<br />
<br />
* Downloader: <code>E:\McNair\Software\Scripts\Patent\USPTO_Parser.pl</code><br />
* XML Splitter: <code>E:\McNair\PatentData\splitter.pl</code><br />
* XML Parsing: <code>E:\McNair\PatentData\Processed\xmlparser_4.5_4.4_4.3.pl</code> and <code>E:\McNair\PatentData\Processed\*.pm</code><br />
<br />
In addition, I used several non-standard Java libraries listed below:<br />
<br />
* [https://github.com/mashape/unirest-java/ Unirest] for easy HTTP requests (MIT License)<br />
* [https://github.com/google/guava Google Guava] for immutable collections and Stream utilities (Apache v2.0 License)<br />
* [https://github.com/jhy/jsoup/ jsoup] for HTML parsing (MIT License)<br />
* [http://commons.apache.org/proper/commons-codec/ Apache Commons Codec] (Apache v2.0 License)<br />
* [http://commons.apache.org/proper/commons-lang/ Apache Commons Lang v3] (Apache v2.0 License)<br />
* [https://mvnrepository.com/artifact/org.jetbrains/annotations/15.0 Jetbrains Annotations] for enhanced null checks (Apache v2.0 License)<br />
* [http://search.maven.org/#artifactdetails%7Corg.postgresql%7Cpostgresql%7C42.1.1.jre7%7Cbundle PostgreSQL JDBC] (BSD 3-clause per https://github.com/pgjdbc/pgjdbc-jre7/blob/master/LICENSE)<br />
<br />
If using maven, these dependencies are listed and should automatically be setup.<br />
<br />
==== Using Code ====<br />
<br />
Any file with a line that says <code>public static void main(String[] args) {</code> can be run as a standalone file. The easiest way to do this is to load the project and then the file in IntelliJ and click the little green play arrow next to this bit of code.<br />
<br />
The code can also be run via the standard <code>javac</code> and <code>java</code> commands but since this project has a complicated structure you end up having to run commands like <br />
<br />
<code>"C:\Program Files\Java\jdk1.8.0_131\bin\java" "-javaagent:C:\Users\OliverC\IntelliJ IDEA 2017.1.3\lib\idea_rt.jar=62364:C:\Users\OliverC\IntelliJ IDEA 2017.1.3\bin" -Dfile.encoding=UTF-8 -classpath "[...contents truncated...];C:\Users\OliverC\.m2\repository\org\postgresql\postgresql\42.1.1\postgresql-42.1.1.jar" org.bakerinstitute.mcnair.uspto_assignments.XmlDriver</code><br />
<br />
to include all of the runtime dependencies and it's just not worth it.<br />
<br />
==== Altering Code ====<br />
<br />
* Use the IntelliJ command Reformat code (found in the menus at <code>Code > Reformat Code</code><br />
* Use the optimize imports function found under the same menu<br />
* Use spaces for indentation<br />
* Loosely try to keep lines below 120 characters<br />
* Commit changes to the Git remote repository "bonobo"<br />
<br />
== Schema Reconciliation ==<br />
<br />
For the work by Joe, see the [[Patent Schema Reconciliation]] page <br />
<br />
=== Patents (Granted) ===<br />
<br />
See <code>E:\McNair\Projects\SimplerPatentData\data\examples\granted</code> for extracted examples of what specific data is available for a sample of the data.<br />
<br />
{| class="wikitable"<br />
|+Granted Patent Data Formats<br />
|-<br />
! scope="col" | Dates Used !! scope="col" | Format !! scope="col" | Location !! scope="col" | Supported? !! scope="col" | Utility !! scope="col" | Reissue !! scope="col" | Design !! scope="col" | Plant<br />
|-<br />
|January 1976 to December 2001<br />
|APS<br />
|<code>data/extracts/granted/vintage</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|<del>January 2001 to December 2001</del><br />
|<del>SGML</del><br />
|Ignored; use concurrently recorded APS data<br />
|No<br />
|N/A<br />
|N/A<br />
|N/A<br />
|N/A<br />
|-<br />
|January 2002 to December 2004<br />
|XML Version 2.5<br />
|<code>data/extracts/granted/blunderyears</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2005 to December 2005<br />
|XML Version 4.0 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2006 to December 2006<br />
|XML Version 4.1 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2007 to December 2012<br />
|XML Version 4.2 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2013 to September 24, 2013<br />
|XML Version 4.3 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|October 8, 2013 to December 2014<br />
|XML Version 4.4 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|-<br />
|January 2015 to December 2016<br />
|XML Version 4.5 ICE<br />
|<code>data/extracts/granted/modern</code><br />
|style="background: green; color: white;" | Yes<br />
|✓<br />
|~<br />
|~<br />
|~<br />
|}<br />
<br />
=== APS Rosetta Stone ===<br />
<br />
The Advanced Patent System (APS) is a fixed-width text format used to store historical patent grant data. The documentation for this sucks; there are pages missing at random. Luckily, we only care about the content contained here: [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]].<br />
<br />
It's worth mentioning that the APS contains an advanced text markup system for chemical formulae, basic text markup, tables, etc. that can lead to seemingly garbled text that is perfectly well-formed.<br />
<br />
==== APS Gotchas ====<br />
<br />
* PATN.WKU is the granted patent number. It is 7 digits while the spec promises 6 digits. The rightmost digit is a check digit modulus 11. See [[File:Aps-wku-modulus11.pdf]] for the words from the horse's mouth.<br />
<br />
=== Patents (Applications) ===<br />
<br />
{| class="wikitable"<br />
|+Patent Application Data Formats<br />
|-<br />
! scope="col" | Dates Used !! scope="col" | Format !! scope="col" | Location !! scope="col" | Supported by Parser?<br />
|-<br />
|March 15, 2001 to December 2001<br />
|XML Version 1.5<br />
|<code>data/extracts/applications/vintage</code><br />
|style="background: red; color: white;" | No<br />
|-<br />
|January 2002 to December 2004<br />
|XML Version 1.6<br />
|<code>data/extracts/applications/vintage</code><br />
|style="background: red; color: white;" | No<br />
|-<br />
|January 2005 to December 2005<br />
|XML Version 4.0 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Maybe<br />
|-<br />
|January 2006 to December 2006<br />
|XML Version 4.1 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Maybe<br />
|-<br />
|January 2007 to December 2012<br />
|XML Version 4.2 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Maybe<br />
|-<br />
|January 2013 to December 2014<br />
|XML Version 4.3 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Maybe<br />
|-<br />
|January 2015 to ''Present''<br />
|XML Version 4.4 ICE<br />
|<code>data/extracts/applications/modern</code><br />
|style="background: yellow;" | Maybe<br />
|}<br />
<br />
== Database ==<br />
<br />
Because there isn't a compelling reason not to, I used the existing PostgreSQL infrastructure on the RDP.<br />
The "Java Way" of interacting with databases is the Java Database Connectivity API (JDBC), an implementation-agnostic API for interacting with databases.<br />
This project uses the stock [https://jdbc.postgresql.org/ Postgres JDBC], version 42.1.1<br />
<br />
=== Create an empty database on RDP ===<br />
<br />
To create an empty database, run this command: <code>$ createdb --username=postgres database-name-goes-here # password is tabspaceenter</code><br />
<br />
=== Abstraction Layer ===<br />
<br />
Since writing raw SQL is a bit cumbersome and error-prone, I have added some abstraction layers that make it much easier to quickly add bulk data. By using Postgres's <code>CopyManager</code> class, we buffer SQL copy commands in memory (as many as possible) and then flush these rows. To understand how the abstraction layers work, see the code in <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\postgres</code>. See <code>E:\McNair\Projects\SimplerPatentData\src\main\java\org\bakerinstitute\mcnair\models\GrantedPatent.java</code> for '''an example of how to extend''' the abstraction layer to deal with more complex scenarios.<br />
<br />
== Address Data ==<br />
<br />
To get the most granular address data (street level, or at least postcode level) about who owns patents, the path is not so straightforward because off the complicated mapping of ownership to a granted patent.<br />
This is the final part of this project that I am working on and it is all at the level of SQL.<br />
See <code>E:\McNair\Projects\SimplerPatentData\src\db\joins</code> for my attempts to create a clean mapping.<br />
Optimistically speaking, the data generated here should be superset of the data present in the Patent Assignment Data Restructure project.<br />
<br />
Note that as of the beginning of August 2017, this part '''has not been completed.'''<br />
<br />
<br />
=== Intuition ===<br />
<br />
Use <code>assignments_longform.last_update_date</code> to find current/latest (or first/earliest) date of assignment. Then match with <code>properties.docid</code> on <code>reelno, frameno</code> to find patent application id. With this mapping to granted patents, we can discover the details of the original granted patent. And with the right date and reelno and frameno, we can match to the <code>assignees</code> table and get fine granularity addresses.<br />
<br />
== Related Pages ==<br />
<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries, Summer 2017 by this guy]]<br />
* [[US_Address_Verification|US Address Verification, Summer 2017 based on tables from Assignment Data Restructure]]<br />
* [[Patent_Assignment_Data_Restructure|Assignment Data Restructure, Spring 2017 by Marcela and Sonia]]<br />
* [[Redesigning_Patent_Database|Redesigning Patent Database, Spring 2017 by Shelby]]<br />
* [[Patent_Data_Cleanup_(June_2016)|Patent Data Cleanup, June 2016 by Marcela]]<br />
* [[Patent_Data|Patent Data, Spring 2016 by Marcela]] <br />
* [[Lex_Machina|Lex Machina]]<br />
* [[USPTO_Patent_Litigation_Data|USPTO Patent Litigation Research Dataset by Ed]]<br />
* [[Patent_Litigation_and_Review|Patent Litigation and Review by Marcela]]<br />
* [[Bag_of_Words_Analysis_of_Patent_Data|Bag of Words Analysis]]<br />
* [[Patent|Existing Database Schema]]<br />
* [[Oliver_Chang_(Work_Log)|My Work Log]]<br />
<br />
== External Links ==<br />
<br />
* Understanding Assignment Data: [https://www.uspto.gov/sites/default/files/documents/USPTO_Patents_Assignment_Dataset_WP.pdf USPTO Documentation on their cleanup of this data]<br />
* [https://bulkdata.uspto.gov/data2/patent/grant/redbook/fulltext/1976/PatentFullTextAPSDoc_GreenBook.pdf USPTO Green Book (APS) Documentation]<br />
* [https://bulkdata.uspto.gov/ USPTO Bulk Data Storage System (BDSS)]<br />
* [https://en.wikipedia.org/wiki/Builder_pattern Builder Pattern in Object-Oriented Programming]<br />
* [https://rdp.mcnaircenter.org/codebase/Repository/ReproduciblePatent Git Repository]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=20960Oliver Chang (Work Log)2017-10-20T17:01:57Z<p>OliverC: /* Day-by-Day (in reverse chronological order) */</p>
<hr />
<div>[[Oliver Chang]] [[Work Logs]]<br />
[[Oliver Chang (Work Log)|(logpage)]]<br />
<br />
''To-do List'':<br />
<br />
* Expand XPath use in the patent data<br />
* Edit to include Application data<br />
* Finish ID joining<br />
<br />
''Projects'':<br />
<br />
* [[PostGIS_Installation|PostGIS Installation]]<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
<br />
''Uploads'':<br />
<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
<br />
== Day-by-Day (in reverse chronological order) ==<br />
<br />
=== October 2017 ===<br />
<br />
* Oct 19: repopulate patents database and create spreadsheet of assignment equivalencies<br />
* Oct 11: add issues to work on to the PECA wiki page; cleanup PECA git repo; start patent application code from granted patent code and start customizing to new domain<br />
* Oct 3: troubleshoot vc_circles.py and make command line interface a little nicer<br />
* Oct 2: discuss mapping strategies & investigate missing eca data <br />
<br />
=== September 2017 ===<br />
<br />
* Sept 23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
* Sept 22: goal setting & server debugging & meet with Yang<br />
<br />
=== August 2017 ===<br />
<br />
* Aug 4: setup parallel instance python framework for job reporting; begin test run<br />
* Aug 2: finish up some documentation of the code and for the wiki<br />
* Aug 1: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
=== July 2017 ===<br />
<br />
* July 31: sketch out parallel enclosing circle algorithm<br />
* July 28: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
* ''travelling''<br />
* July 19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
* July 18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
* July 17: redo db operations after cleaning up granted patent number bugs<br />
* July 13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
* July 12: generate some example data illustrating the difficulty of joining different tables<br />
* July 11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
* July 7: catch up on documentation<br />
* July 6: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
* July 5: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
=== June 2017 ===<br />
<br />
* June 30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
* June 29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
* June 28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
* June 27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
* June 26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
* June 25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
* June 23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
* June 22: create postcode<->patent table<br />
* June 21: document granted patent queries and equivalencies<br />
* June 20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
* June 19: skim address regular expressions; cursory investigation of patent table<br />
* June 16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
* June 15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
* June 14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
* June 13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
* June 12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
* June 8: add foreign key inserts; create pretty printer for XML analysis<br />
* June 7: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
* June 6: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
* June 5: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
* June 1: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
=== May 2017 ===<br />
<br />
* May 31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
* May 30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
* May 29: expand to APS; expand to raw assignment data<br />
* May 27: expand to maintenance fee data<br />
* May 26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
* May 25: sketch out OO design of project; download bulk data<br />
* May 24: move wiki pages around; start git repository for project<br />
* May 21: discuss technical details of previous work with Ed<br />
* May 8: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
* May 4: setup wiki account, rdp account, database training<br />
<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=20736Oliver Chang (Work Log)2017-10-11T19:19:38Z<p>OliverC: </p>
<hr />
<div>[[Oliver Chang]] [[Work Logs]]<br />
[[Oliver Chang (Work Log)|(logpage)]]<br />
<br />
''To-do List'':<br />
<br />
* Expand XPath use in the patent data<br />
* Edit to include Application data<br />
* Finish ID joining<br />
<br />
''Projects'':<br />
<br />
* [[PostGIS_Installation|PostGIS Installation]]<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
<br />
''Uploads'':<br />
<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
<br />
== Day-by-Day (in reverse chronological order) ==<br />
<br />
=== October 2017 ===<br />
<br />
* Oct 11: add issues to work on to the PECA wiki page; cleanup PECA git repo; start patent application code from granted patent code and start customizing to new domain<br />
* Oct 3: troubleshoot vc_circles.py and make command line interface a little nicer<br />
* Oct 2: discuss mapping strategies & investigate missing eca data <br />
<br />
=== September 2017 ===<br />
<br />
* Sept 23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
* Sept 22: goal setting & server debugging & meet with Yang<br />
<br />
=== August 2017 ===<br />
<br />
* Aug 4: setup parallel instance python framework for job reporting; begin test run<br />
* Aug 2: finish up some documentation of the code and for the wiki<br />
* Aug 1: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
=== July 2017 ===<br />
<br />
* July 31: sketch out parallel enclosing circle algorithm<br />
* July 28: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
* ''travelling''<br />
* July 19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
* July 18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
* July 17: redo db operations after cleaning up granted patent number bugs<br />
* July 13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
* July 12: generate some example data illustrating the difficulty of joining different tables<br />
* July 11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
* July 7: catch up on documentation<br />
* July 6: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
* July 5: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
=== June 2017 ===<br />
<br />
* June 30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
* June 29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
* June 28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
* June 27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
* June 26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
* June 25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
* June 23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
* June 22: create postcode<->patent table<br />
* June 21: document granted patent queries and equivalencies<br />
* June 20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
* June 19: skim address regular expressions; cursory investigation of patent table<br />
* June 16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
* June 15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
* June 14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
* June 13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
* June 12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
* June 8: add foreign key inserts; create pretty printer for XML analysis<br />
* June 7: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
* June 6: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
* June 5: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
* June 1: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
=== May 2017 ===<br />
<br />
* May 31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
* May 30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
* May 29: expand to APS; expand to raw assignment data<br />
* May 27: expand to maintenance fee data<br />
* May 26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
* May 25: sketch out OO design of project; download bulk data<br />
* May 24: move wiki pages around; start git repository for project<br />
* May 21: discuss technical details of previous work with Ed<br />
* May 8: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
* May 4: setup wiki account, rdp account, database training<br />
<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Parallel_Enclosing_Circle_Algorithm&diff=20735Parallel Enclosing Circle Algorithm2017-10-11T18:52:33Z<p>OliverC: add notes about the current issues faced by the program that were mentioned on Monday</p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallel Enclosing Circle Algorithm<br />
|Has owner=Oliver Chang,<br />
|Has start date=July 31, 2017<br />
|Has deadline=October 4, 2017<br />
|Has project status=Complete<br />
|Is dependent on=Enclosing Circle Algorithm,<br />
}}<br />
<br />
A thin-wrapper around [[Enclosing_Circle_Algorithm_(Rework)|the enclosing circle algorithm]] which allows for instance-level parallelization.<br />
This project consists of the python files in <code>E:\McNair\Projects\OliverLovesCircles\src\python</code>.<br />
<br />
Parallelization is implemented via Python2's [https://docs.python.org/2/library/subprocess.html#subprocess.Popen <code>subprocess.open()</code>] which is non-blocking and available in the standard library.<br />
<br />
== The Problem ==<br />
<br />
Note that this '''is not the classical enclosing circle algorithm'''.<br />
Rather, we seek to minimize the sum of enclosing circles containing at least <code>n</code> points.<br />
Thus, multiple circles are allowed and inclusion in multiple circles is possible.<br />
<br />
This algorithm has terrible time-performance characteristics, so we make the assumption that we can divide a large number of points with k-means and then solve those subproblems.<br />
In other words, we make the simplifying assumption that the Enclosing Circle Algorithm has [https://en.wikipedia.org/wiki/Optimal_substructure Optimal Substructure].<br />
<br />
== Parameters ==<br />
<br />
* in <code>circles.py</code>:<br />
** <code>ITERATIONS</code>: the number of iterations to attempt for each <code>k</code> to find minimum for that <code>k</code><br />
** <code>MIN_POINTS_PER_CIRCLE</code> (AKA <code>n</code>): the minimum number of data points that must be included in a circle<br />
* in <code>vc_circles.py</code><br />
** <code>NUMBER_INSTANCES</code>: number of parallel instances to run; assume no data-races between instances<br />
** <code>SWEEP_CYCLE_SECONDS</code>: amount of time before removing completed jobs from the current job and adding new jobs if any files are left to process<br />
** <code>TIMEOUT_MINUTES</code>: maximum running time of a parallel instance of the algorithm<br />
** <code>SPLIT_THRESHOLD</code>: if a dataset has more than this threshold of data points, it will be split via k-means<br />
<br />
== Example Usage ==<br />
<br />
<nowiki><br />
$ python vc_circles.py --infile E:/McNair/Projects/OliverLovesCircles/CoLevelForCirclesNotRunGTE200.txt<br />
$ python outjoiner.py</nowiki><br />
where <code>CoLevelForCirclesNotRunGTE200.txt</code> is a tab-separated values file with the columns<br />
<code>placestate, place, statecode, year, latitude, longitude, coname, datefirstinv, placens, geoid, city</code><br />
<br />
This command will populate (and overwrite) any files in <code>data/</code>. The format of the filenames in this directory are <code>{city}{sep}{state}{sep}{year}{sep}{num}.tsv</code> where <code>num</code> is a 0-indexed integer of a split city/state/year <code>infile</code> that has greater than <code>SPLIT_THRESHOLD</code><br />
<br />
== Bugs/Issues ==<br />
<br />
# "St. Paul" and "St. Louis" have un-enclosed points--speculate because of weird file path issues<br />
# Some place/state/year combinations do not run to completion regardless of how tractable the number of points<br />
# How to merge small enclosing circles? This is a better measure of agglomeration regardless<br />
# How to separate outliers? <br />
<br />
== Related Pages ==<br />
<br />
* [[Enclosing_Circle_Algorithm_(Rework)|Abhi's Rework (Summer 2017)]]<br />
* [[Enclosing_Circle_Algorithm|Original Implementation (Spring 2017)]]<br />
<br />
== External Links ==<br />
<br />
* [https://rdp.mcnaircenter.org/codebase/Repository/OliverEnclosingCircle Git Repository]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=20652Oliver Chang (Work Log)2017-10-06T16:55:28Z<p>OliverC: </p>
<hr />
<div>[[Oliver Chang]] [[Work Logs]]<br />
[[Oliver Chang (Work Log)|(logpage)]]<br />
<br />
''To-do List'':<br />
<br />
* Expand XPath use in the patent data<br />
* Edit to include Application data<br />
* Finish ID joining<br />
<br />
''Projects'':<br />
<br />
* [[PostGIS_Installation|PostGIS Installation]]<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
<br />
''Uploads'':<br />
<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
<br />
== Day-by-Day (in reverse chronological order) ==<br />
<br />
=== October 2017 ===<br />
<br />
* Oct 3: troubleshoot vc_circles.py and make command line interface a little nicer<br />
* Oct 2: discuss mapping strategies & investigate missing eca data <br />
<br />
=== September 2017 ===<br />
<br />
* Sept 23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
* Sept 22: goal setting & server debugging & meet with Yang<br />
<br />
=== August 2017 ===<br />
<br />
* Aug 4: setup parallel instance python framework for job reporting; begin test run<br />
* Aug 2: finish up some documentation of the code and for the wiki<br />
* Aug 1: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
=== July 2017 ===<br />
<br />
* July 31: sketch out parallel enclosing circle algorithm<br />
* July 28: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
* ''travelling''<br />
* July 19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
* July 18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
* July 17: redo db operations after cleaning up granted patent number bugs<br />
* July 13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
* July 12: generate some example data illustrating the difficulty of joining different tables<br />
* July 11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
* July 7: catch up on documentation<br />
* July 6: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
* July 5: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
=== June 2017 ===<br />
<br />
* June 30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
* June 29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
* June 28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
* June 27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
* June 26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
* June 25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
* June 23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
* June 22: create postcode<->patent table<br />
* June 21: document granted patent queries and equivalencies<br />
* June 20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
* June 19: skim address regular expressions; cursory investigation of patent table<br />
* June 16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
* June 15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
* June 14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
* June 13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
* June 12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
* June 8: add foreign key inserts; create pretty printer for XML analysis<br />
* June 7: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
* June 6: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
* June 5: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
* June 1: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
=== May 2017 ===<br />
<br />
* May 31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
* May 30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
* May 29: expand to APS; expand to raw assignment data<br />
* May 27: expand to maintenance fee data<br />
* May 26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
* May 25: sketch out OO design of project; download bulk data<br />
* May 24: move wiki pages around; start git repository for project<br />
* May 21: discuss technical details of previous work with Ed<br />
* May 8: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
* May 4: setup wiki account, rdp account, database training<br />
<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Parallel_Enclosing_Circle_Algorithm&diff=20651Parallel Enclosing Circle Algorithm2017-10-06T16:52:10Z<p>OliverC: /* Example Usage */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallel Enclosing Circle Algorithm<br />
|Has owner=Oliver Chang,<br />
|Has start date=July 31, 2017<br />
|Has deadline=October 4, 2017<br />
|Has project status=Complete<br />
|Is dependent on=Enclosing Circle Algorithm,<br />
}}<br />
<br />
A thin-wrapper around [[Enclosing_Circle_Algorithm_(Rework)|the enclosing circle algorithm]] which allows for instance-level parallelization.<br />
This project consists of the python files in <code>E:\McNair\Projects\OliverLovesCircles\src\python</code>.<br />
<br />
Parallelization is implemented via Python2's [https://docs.python.org/2/library/subprocess.html#subprocess.Popen <code>subprocess.open()</code>] which is non-blocking and available in the standard library.<br />
<br />
== The Problem ==<br />
<br />
Note that this '''is not the classical enclosing circle algorithm'''.<br />
Rather, we seek to minimize the sum of enclosing circles containing at least <code>n</code> points.<br />
Thus, multiple circles are allowed and inclusion in multiple circles is possible.<br />
<br />
This algorithm has terrible time-performance characteristics, so we make the assumption that we can divide a large number of points with k-means and then solve those subproblems.<br />
In other words, we make the simplifying assumption that the Enclosing Circle Algorithm has [https://en.wikipedia.org/wiki/Optimal_substructure Optimal Substructure].<br />
<br />
== Parameters ==<br />
<br />
* in <code>circles.py</code>:<br />
** <code>ITERATIONS</code>: the number of iterations to attempt for each <code>k</code> to find minimum for that <code>k</code><br />
** <code>MIN_POINTS_PER_CIRCLE</code> (AKA <code>n</code>): the minimum number of data points that must be included in a circle<br />
* in <code>vc_circles.py</code><br />
** <code>NUMBER_INSTANCES</code>: number of parallel instances to run; assume no data-races between instances<br />
** <code>SWEEP_CYCLE_SECONDS</code>: amount of time before removing completed jobs from the current job and adding new jobs if any files are left to process<br />
** <code>TIMEOUT_MINUTES</code>: maximum running time of a parallel instance of the algorithm<br />
** <code>SPLIT_THRESHOLD</code>: if a dataset has more than this threshold of data points, it will be split via k-means<br />
<br />
== Example Usage ==<br />
<br />
<nowiki><br />
$ python vc_circles.py --infile E:/McNair/Projects/OliverLovesCircles/CoLevelForCirclesNotRunGTE200.txt<br />
$ python outjoiner.py</nowiki><br />
where <code>CoLevelForCirclesNotRunGTE200.txt</code> is a tab-separated values file with the columns<br />
<code>placestate, place, statecode, year, latitude, longitude, coname, datefirstinv, placens, geoid, city</code><br />
<br />
This command will populate (and overwrite) any files in <code>data/</code>. The format of the filenames in this directory are <code>{city}{sep}{state}{sep}{year}{sep}{num}.tsv</code> where <code>num</code> is a 0-indexed integer of a split city/state/year <code>infile</code> that has greater than <code>SPLIT_THRESHOLD</code><br />
<br />
== Related Pages ==<br />
<br />
* [[Enclosing_Circle_Algorithm_(Rework)|Abhi's Rework (Summer 2017)]]<br />
* [[Enclosing_Circle_Algorithm|Original Implementation (Spring 2017)]]<br />
<br />
== External Links ==<br />
<br />
* [https://rdp.mcnaircenter.org/codebase/Repository/OliverEnclosingCircle Git Repository]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Parallel_Enclosing_Circle_Algorithm&diff=20650Parallel Enclosing Circle Algorithm2017-10-06T16:50:32Z<p>OliverC: </p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallel Enclosing Circle Algorithm<br />
|Has owner=Oliver Chang,<br />
|Has start date=July 31, 2017<br />
|Has deadline=October 4, 2017<br />
|Has project status=Complete<br />
|Is dependent on=Enclosing Circle Algorithm,<br />
}}<br />
<br />
A thin-wrapper around [[Enclosing_Circle_Algorithm_(Rework)|the enclosing circle algorithm]] which allows for instance-level parallelization.<br />
This project consists of the python files in <code>E:\McNair\Projects\OliverLovesCircles\src\python</code>.<br />
<br />
Parallelization is implemented via Python2's [https://docs.python.org/2/library/subprocess.html#subprocess.Popen <code>subprocess.open()</code>] which is non-blocking and available in the standard library.<br />
<br />
== The Problem ==<br />
<br />
Note that this '''is not the classical enclosing circle algorithm'''.<br />
Rather, we seek to minimize the sum of enclosing circles containing at least <code>n</code> points.<br />
Thus, multiple circles are allowed and inclusion in multiple circles is possible.<br />
<br />
This algorithm has terrible time-performance characteristics, so we make the assumption that we can divide a large number of points with k-means and then solve those subproblems.<br />
In other words, we make the simplifying assumption that the Enclosing Circle Algorithm has [https://en.wikipedia.org/wiki/Optimal_substructure Optimal Substructure].<br />
<br />
== Parameters ==<br />
<br />
* in <code>circles.py</code>:<br />
** <code>ITERATIONS</code>: the number of iterations to attempt for each <code>k</code> to find minimum for that <code>k</code><br />
** <code>MIN_POINTS_PER_CIRCLE</code> (AKA <code>n</code>): the minimum number of data points that must be included in a circle<br />
* in <code>vc_circles.py</code><br />
** <code>NUMBER_INSTANCES</code>: number of parallel instances to run; assume no data-races between instances<br />
** <code>SWEEP_CYCLE_SECONDS</code>: amount of time before removing completed jobs from the current job and adding new jobs if any files are left to process<br />
** <code>TIMEOUT_MINUTES</code>: maximum running time of a parallel instance of the algorithm<br />
** <code>SPLIT_THRESHOLD</code>: if a dataset has more than this threshold of data points, it will be split via k-means<br />
<br />
== Example Usage ==<br />
<br />
<nowiki><br />
$ python vc_circles.py --infile E:/McNair/Projects/OliverLovesCircles/CoLevelForCirclesNotRunGTE200.txt</nowiki><br />
where <code>CoLevelForCirclesNotRunGTE200.txt</code> is a tab-separated values file with the columns<br />
<code>placestate, place, statecode, year, latitude, longitude, coname, datefirstinv, placens, geoid, city</code><br />
<br />
This command will populate (and overwrite) any files in <code>data/</code>. The format of the filenames in this directory are <code>{city}{sep}{state}{sep}{year}{sep}{num}.tsv</code> where <code>num</code> is a 0-indexed integer of a split city/state/year <code>infile</code> that has greater than <code>SPLIT_THRESHOLD</code><br />
<br />
== Related Pages ==<br />
<br />
* [[Enclosing_Circle_Algorithm_(Rework)|Abhi's Rework (Summer 2017)]]<br />
* [[Enclosing_Circle_Algorithm|Original Implementation (Spring 2017)]]<br />
<br />
== External Links ==<br />
<br />
* [https://rdp.mcnaircenter.org/codebase/Repository/OliverEnclosingCircle Git Repository]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Parallel_Enclosing_Circle_Algorithm&diff=20649Parallel Enclosing Circle Algorithm2017-10-06T16:47:42Z<p>OliverC: </p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallel Enclosing Circle Algorithm<br />
|Has owner=Oliver Chang,<br />
|Has start date=July 31, 2017<br />
|Has deadline=October 4, 2017<br />
|Has project status=Complete<br />
|Is dependent on=Enclosing Circle Algorithm,<br />
}}<br />
<br />
A thin-wrapper around [[Enclosing_Circle_Algorithm_(Rework)|the enclosing circle algorithm]] which allows for instance-level parallelization.<br />
This project consists of the python files in <code>E:\McNair\Projects\OliverLovesCircles\src\python</code>.<br />
<br />
Parallelization is implemented via Python2's [https://docs.python.org/2/library/subprocess.html#subprocess.Popen <code>subprocess.open()</code>] which is non-blocking and available in the standard library.<br />
<br />
<br />
== The Problem ==<br />
<br />
Note that this '''is not the classical enclosing circle algorithm'''.<br />
Rather, we seek to minimize the sum of enclosing circles containing at least <code>n</code> points.<br />
Thus, multiple circles are allowed and inclusion in multiple circles is possible.<br />
<br />
== Parameters ==<br />
<br />
* in <code>circles.py</code>:<br />
** <code>ITERATIONS</code>: the number of iterations to attempt for each <code>k</code> to find minimum for that <code>k</code><br />
** <code>MIN_POINTS_PER_CIRCLE</code> (AKA <code>n</code>): the minimum number of data points that must be included in a circle<br />
* in <code>vc_circles.py</code><br />
** <code>NUMBER_INSTANCES</code>: number of parallel instances to run; assume no data-races between instances<br />
** <code>SWEEP_CYCLE_SECONDS</code>: amount of time before removing completed jobs from the current job and adding new jobs if any files are left to process<br />
** <code>TIMEOUT_MINUTES</code>: maximum running time of a parallel instance of the algorithm<br />
** <code>SPLIT_THRESHOLD</code>: if a dataset has more than this threshold of data points, it will be split via k-means<br />
<br />
== Example Usage ==<br />
<br />
<nowiki><br />
$ python vc_circles.py --infile E:/McNair/Projects/OliverLovesCircles/CoLevelForCirclesNotRunGTE200.txt</nowiki><br />
where <code>CoLevelForCirclesNotRunGTE200.txt</code> is a tab-separated values file with the columns<br />
<code>placestate, place, statecode, year, latitude, longitude, coname, datefirstinv, placens, geoid, city</code><br />
<br />
This command will populate (and overwrite) any files in <code>data/</code>. The format of the filenames in this directory are <code>{city}{sep}{state}{sep}{year}{sep}{num}.tsv</code> where <code>num</code> is a 0-indexed integer of a split city/state/year <code>infile</code> that has greater than <code>SPLIT_THRESHOLD</code><br />
<br />
<br />
== Related Pages ==<br />
<br />
* [[Enclosing_Circle_Algorithm_(Rework)|Abhi's Rework (Summer 2017)]]<br />
* [[Enclosing_Circle_Algorithm|Original Implementation (Spring 2017)]]<br />
<br />
== External Links ==<br />
<br />
* [https://rdp.mcnaircenter.org/codebase/Repository/OliverEnclosingCircle Git Repository]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=20587Oliver Chang (Work Log)2017-10-04T02:41:12Z<p>OliverC: </p>
<hr />
<div>[[Oliver Chang]] [[Work Logs]]<br />
[[Oliver Chang (Work Log)|(logpage)]]<br />
<br />
''To-do List'':<br />
<br />
* Expand XPath use in the patent data<br />
* Edit to include Application data<br />
* Finish ID joining<br />
* Look into NIH document similarity algorithm<br />
* Sysadminy stuff<br />
<br />
''Projects'':<br />
<br />
* [[PostGIS_Installation|PostGIS Installation]]<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
<br />
''Uploads'':<br />
<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
<br />
== Day-by-Day (in reverse chronological order) ==<br />
<br />
=== October 2017 ===<br />
<br />
* Oct 3: troubleshoot vc_circles.py and make command line interface a little nicer<br />
* Oct 2: discuss mapping strategies & investigate missing eca data <br />
<br />
=== September 2017 ===<br />
<br />
* Sept 23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
* Sept 22: goal setting & server debugging & meet with Yang<br />
<br />
=== August 2017 ===<br />
<br />
* Aug 4: setup parallel instance python framework for job reporting; begin test run<br />
* Aug 2: finish up some documentation of the code and for the wiki<br />
* Aug 1: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
=== July 2017 ===<br />
<br />
* July 31: sketch out parallel enclosing circle algorithm<br />
* July 28: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
* ''travelling''<br />
* July 19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
* July 18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
* July 17: redo db operations after cleaning up granted patent number bugs<br />
* July 13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
* July 12: generate some example data illustrating the difficulty of joining different tables<br />
* July 11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
* July 7: catch up on documentation<br />
* July 6: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
* July 5: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
=== June 2017 ===<br />
<br />
* June 30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
* June 29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
* June 28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
* June 27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
* June 26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
* June 25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
* June 23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
* June 22: create postcode<->patent table<br />
* June 21: document granted patent queries and equivalencies<br />
* June 20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
* June 19: skim address regular expressions; cursory investigation of patent table<br />
* June 16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
* June 15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
* June 14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
* June 13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
* June 12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
* June 8: add foreign key inserts; create pretty printer for XML analysis<br />
* June 7: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
* June 6: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
* June 5: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
* June 1: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
=== May 2017 ===<br />
<br />
* May 31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
* May 30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
* May 29: expand to APS; expand to raw assignment data<br />
* May 27: expand to maintenance fee data<br />
* May 26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
* May 25: sketch out OO design of project; download bulk data<br />
* May 24: move wiki pages around; start git repository for project<br />
* May 21: discuss technical details of previous work with Ed<br />
* May 8: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
* May 4: setup wiki account, rdp account, database training<br />
<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=20586Oliver Chang (Work Log)2017-10-04T02:40:57Z<p>OliverC: </p>
<hr />
<div>[[Oliver Chang]] [[Work Logs]]<br />
[[Oliver Chang (Work Log)|(logpage)]]<br />
<br />
''To-do List'':<br />
<br />
* Expand XPath use in the patent data<br />
* Edit to include Application data<br />
* Finish ID joining<br />
* Look into NIH document similarity algorithm<br />
* Sysadminy stuff<br />
<br />
''Projects'':<br />
<br />
* [[PostGIS_Installation|PostGIS Installation]]<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
<br />
<br />
''Uploads'':<br />
<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
<br />
== Day-by-Day (in reverse chronological order) ==<br />
<br />
=== October 2017 ===<br />
<br />
* Oct 3: troubleshoot vc_circles.py and make command line interface a little nicer<br />
* Oct 2: discuss mapping strategies & investigate missing eca data <br />
<br />
=== September 2017 ===<br />
<br />
* Sept 23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
* Sept 22: goal setting & server debugging & meet with Yang<br />
<br />
=== August 2017 ===<br />
<br />
* Aug 4: setup parallel instance python framework for job reporting; begin test run<br />
* Aug 2: finish up some documentation of the code and for the wiki<br />
* Aug 1: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
=== July 2017 ===<br />
<br />
* July 31: sketch out parallel enclosing circle algorithm<br />
* July 28: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
* ''travelling''<br />
* July 19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
* July 18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
* July 17: redo db operations after cleaning up granted patent number bugs<br />
* July 13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
* July 12: generate some example data illustrating the difficulty of joining different tables<br />
* July 11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
* July 7: catch up on documentation<br />
* July 6: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
* July 5: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
=== June 2017 ===<br />
<br />
* June 30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
* June 29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
* June 28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
* June 27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
* June 26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
* June 25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
* June 23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
* June 22: create postcode<->patent table<br />
* June 21: document granted patent queries and equivalencies<br />
* June 20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
* June 19: skim address regular expressions; cursory investigation of patent table<br />
* June 16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
* June 15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
* June 14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
* June 13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
* June 12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
* June 8: add foreign key inserts; create pretty printer for XML analysis<br />
* June 7: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
* June 6: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
* June 5: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
* June 1: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
=== May 2017 ===<br />
<br />
* May 31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
* May 30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
* May 29: expand to APS; expand to raw assignment data<br />
* May 27: expand to maintenance fee data<br />
* May 26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
* May 25: sketch out OO design of project; download bulk data<br />
* May 24: move wiki pages around; start git repository for project<br />
* May 21: discuss technical details of previous work with Ed<br />
* May 8: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
* May 4: setup wiki account, rdp account, database training<br />
<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=20585Oliver Chang (Work Log)2017-10-04T02:40:43Z<p>OliverC: </p>
<hr />
<div>[[Oliver Chang]] [[Work Logs]]<br />
[[Oliver Chang (Work Log)|(logpage)]]<br />
<br />
''To-do List'':<br />
<br />
* Expand XPath use in the patent data<br />
* Edit to include Application data<br />
* Finish ID joining<br />
* Look into NIH document similarity algorithm<br />
* Sysadminy stuff<br />
<br />
''Projects'':<br />
<br />
* [[http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation|PostGIS Installation]]<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
<br />
<br />
''Uploads'':<br />
<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
<br />
== Day-by-Day (in reverse chronological order) ==<br />
<br />
=== October 2017 ===<br />
<br />
* Oct 3: troubleshoot vc_circles.py and make command line interface a little nicer<br />
* Oct 2: discuss mapping strategies & investigate missing eca data <br />
<br />
=== September 2017 ===<br />
<br />
* Sept 23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
* Sept 22: goal setting & server debugging & meet with Yang<br />
<br />
=== August 2017 ===<br />
<br />
* Aug 4: setup parallel instance python framework for job reporting; begin test run<br />
* Aug 2: finish up some documentation of the code and for the wiki<br />
* Aug 1: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
=== July 2017 ===<br />
<br />
* July 31: sketch out parallel enclosing circle algorithm<br />
* July 28: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
* ''travelling''<br />
* July 19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
* July 18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
* July 17: redo db operations after cleaning up granted patent number bugs<br />
* July 13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
* July 12: generate some example data illustrating the difficulty of joining different tables<br />
* July 11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
* July 7: catch up on documentation<br />
* July 6: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
* July 5: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
=== June 2017 ===<br />
<br />
* June 30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
* June 29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
* June 28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
* June 27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
* June 26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
* June 25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
* June 23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
* June 22: create postcode<->patent table<br />
* June 21: document granted patent queries and equivalencies<br />
* June 20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
* June 19: skim address regular expressions; cursory investigation of patent table<br />
* June 16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
* June 15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
* June 14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
* June 13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
* June 12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
* June 8: add foreign key inserts; create pretty printer for XML analysis<br />
* June 7: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
* June 6: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
* June 5: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
* June 1: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
=== May 2017 ===<br />
<br />
* May 31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
* May 30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
* May 29: expand to APS; expand to raw assignment data<br />
* May 27: expand to maintenance fee data<br />
* May 26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
* May 25: sketch out OO design of project; download bulk data<br />
* May 24: move wiki pages around; start git repository for project<br />
* May 21: discuss technical details of previous work with Ed<br />
* May 8: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
* May 4: setup wiki account, rdp account, database training<br />
<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=20584Oliver Chang (Work Log)2017-10-04T02:38:10Z<p>OliverC: /* Day-by-Day (in reverse chronological order) */</p>
<hr />
<div>[[Oliver Chang]] [[Work Logs]]<br />
[[Oliver Chang (Work Log)|(logpage)]]<br />
<br />
''To-do List'':<br />
<br />
* Expand XPath use in the patent data<br />
* Edit to include Application data<br />
* Finish ID joining<br />
* Look into NIH document similarity algorithm<br />
* Sysadminy stuff<br />
<br />
''Projects'':<br />
<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
<br />
<br />
''Uploads'':<br />
<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
<br />
== Day-by-Day (in reverse chronological order) ==<br />
<br />
=== October 2017 ===<br />
<br />
* Oct 3: troubleshoot vc_circles.py and make command line interface a little nicer<br />
* Oct 2: discuss mapping strategies & investigate missing eca data <br />
<br />
=== September 2017 ===<br />
<br />
* Sept 23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
* Sept 22: goal setting & server debugging & meet with Yang<br />
<br />
=== August 2017 ===<br />
<br />
* Aug 4: setup parallel instance python framework for job reporting; begin test run<br />
* Aug 2: finish up some documentation of the code and for the wiki<br />
* Aug 1: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
=== July 2017 ===<br />
<br />
* July 31: sketch out parallel enclosing circle algorithm<br />
* July 28: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
* ''travelling''<br />
* July 19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
* July 18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
* July 17: redo db operations after cleaning up granted patent number bugs<br />
* July 13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
* July 12: generate some example data illustrating the difficulty of joining different tables<br />
* July 11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
* July 7: catch up on documentation<br />
* July 6: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
* July 5: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
=== June 2017 ===<br />
<br />
* June 30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
* June 29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
* June 28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
* June 27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
* June 26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
* June 25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
* June 23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
* June 22: create postcode<->patent table<br />
* June 21: document granted patent queries and equivalencies<br />
* June 20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
* June 19: skim address regular expressions; cursory investigation of patent table<br />
* June 16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
* June 15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
* June 14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
* June 13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
* June 12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
* June 8: add foreign key inserts; create pretty printer for XML analysis<br />
* June 7: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
* June 6: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
* June 5: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
* June 1: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
=== May 2017 ===<br />
<br />
* May 31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
* May 30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
* May 29: expand to APS; expand to raw assignment data<br />
* May 27: expand to maintenance fee data<br />
* May 26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
* May 25: sketch out OO design of project; download bulk data<br />
* May 24: move wiki pages around; start git repository for project<br />
* May 21: discuss technical details of previous work with Ed<br />
* May 8: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
* May 4: setup wiki account, rdp account, database training<br />
<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=Oliver_Chang_(Work_Log)&diff=20246Oliver Chang (Work Log)2017-09-24T04:49:37Z<p>OliverC: </p>
<hr />
<div>[[Oliver Chang]] [[Work Logs]]<br />
[[Oliver Chang (Work Log)|(logpage)]]<br />
<br />
''To-do List'':<br />
<br />
* Expand XPath use in the patent data<br />
* Edit to include Application data<br />
* Finish ID joining<br />
* Look into NIH document similarity algorithm<br />
* Sysadminy stuff<br />
<br />
''Projects'':<br />
<br />
* [[Reproducible_Patent_Data|Reproducible Patent Data]]<br />
* [[Patent_Validity_Ideas_for_ML|Predictive Patent Validity Machine Learning Ideas]]<br />
* [[Equivalent_XPath_and_APS_Queries|Equivalent XPath and APS Queries]]<br />
* [[US_Address_Verification|US Address Verification]]<br />
* [[GPU_Build|GPU Computer Build]]<br />
* [[Parallel_Enclosing_Circle_Algorithm|Parallel Enclosing Circle Algorithm]]<br />
<br />
<br />
''Uploads'':<br />
<br />
* [[File:PADX-File-Description-v2_Hague.pdf]]<br />
** Describes patent kind codes (notably, what the hell X0 represents)<br />
* [[File:PatentFullTextAPSDoc_GreenBook_pgs13-22.pdf]]<br />
** Describes the fields in APS, their supposed character lengths, and if they are required/optional<br />
* [[File:Aps-wku-modulus11.pdf]]<br />
** Describes the layout of the check digit on magnetic tape<br />
* [[File:Mod-11-algorithm.pdf]]<br />
** Describes the algorithm used to calculate the check digit<br />
<br />
<br />
== Day-by-Day (in reverse chronological order) ==<br />
<br />
=== September 2017 ===<br />
<br />
* Sept 23: make Project/OliverLovesCircles usable and add initial splitting ability<br />
* Sept 22: goal setting & server debugging & meet with Yang<br />
<br />
=== August 2017 ===<br />
<br />
* Aug 4: setup parallel instance python framework for job reporting; begin test run<br />
* Aug 2: finish up some documentation of the code and for the wiki<br />
* Aug 1: discuss with Abhi & Ed about alternatives to Java port because of algorithmic constants that would be hard to port; run test batches on python with addition of equality operators and convergence early stopping<br />
<br />
=== July 2017 ===<br />
<br />
* July 31: sketch out parallel enclosing circle algorithm<br />
* July 28: field questions and data cleanup questions from Kerda & Joe & Adrian<br />
* ''travelling''<br />
* July 19: try to remove duplicated records (esp. those with empty titles) which are preventing the addition of a unique constraint<br />
* July 18: run correspondent join on properties and correspondents table to match previous project; sync with Adrian and Abhi<br />
* July 17: redo db operations after cleaning up granted patent number bugs<br />
* July 13: powwow about parallelizing Enclosing Circle Algorithm; sketch out what to do for the rest of the summer; work more on joins<br />
* July 12: generate some example data illustrating the difficulty of joining different tables<br />
* July 11: track down some bugs that happen very rarely and were missed in the initial qa phase<br />
* July 7: catch up on documentation<br />
* July 6: try (unsuccessfully) to understand docid mapping...create exploration scripts<br />
* July 5: add invention title to proper grouping of assignment properties; optimize XML parsing<br />
<br />
=== June 2017 ===<br />
<br />
* June 30: powwow with James, Abhi, Ed about optimization issues; discuss document ids, X0 etc with Ed; pinpoint issues with APS doc numbers (see Repro Pat Dat#Gotchas) for more info<br />
* June 29: add logging of copy commands, more chattiness to scripts, debug assignment data failure<br />
* June 28: create examples for expansion to plant, reissue, design patent collection; start optimizing xml<br />
* June 27: write SQL to replicate assignees, extract postcodes for ongoing projects<br />
* June 26: speedup code, abstract in-memory file splitters to avoid repetition and some weird edge cases<br />
* June 25: create mappings for APS, assignment properties, XML 2.5 for data import; run data imports for granted data<br />
* June 23: cleanup hacky models with a better set of abstractions; cleanup IDE warnings; redefine patent-address mapping<br />
* June 22: create postcode<->patent table<br />
* June 21: document granted patent queries and equivalencies<br />
* June 20: sketch out APS driver; discuss patent id problem; further document with evidence the zipcode data validity<br />
* June 19: skim address regular expressions; cursory investigation of patent table<br />
* June 16: create method of getting all data into the database, whether it likes it or not; copy over assignments, granted data using new scheme<br />
* June 15: add more robust error reporting, fix race conditions; build out assignment driver; build out fee event driver; add error logging<br />
* June 14: migrate bulk inserts to copy command; refresh on address data and start in on that; convert processor to multi-threaded application<br />
* June 13: spot check SQL tables; fix broken final case endlessly looping; investigate smarter insert methods<br />
* June 12: add XML printer, use it to inspect applications; extend BaseScraper to fetch patent application data; add applications documentation to my project page; add CREATE of other tables<br />
* June 8: add foreign key inserts; create pretty printer for XML analysis<br />
* June 7: finalize DB abstraction layer; migrate code to bulk inserts; upgrade webserver software and do optimization on RDP postgres with Ed<br />
* June 6: add jdbc; create basic schema; add db interaction; schedule meeting for later in the week<br />
* June 5: look into postgresql; refersh on postgis; add some notes to the Enclosing Circle Algorithm page<br />
* June 1: add RDP git remote; add more documentation to wiki page; refactor downloader scripts; start creation of tooling for interacting with data<br />
<br />
=== May 2017 ===<br />
<br />
* May 31: finish copy-pasting attributes into the wiki page; retroactively fill out work log; meet with Ed to discuss next steps<br />
* May 30: update documentation on wiki, restructure large binary files to have more hierarchy instead of a flat listing at the root<br />
* May 29: expand to APS; expand to raw assignment data<br />
* May 27: expand to maintenance fee data<br />
* May 26: create models, translate <code>xmlparser*.pl</code> file into Java; start using builder pattern<br />
* May 25: sketch out OO design of project; download bulk data<br />
* May 24: move wiki pages around; start git repository for project<br />
* May 21: discuss technical details of previous work with Ed<br />
* May 8: cleanup dead links on wiki and start reading about previous work; discuss current project status with Ed<br />
* May 4: setup wiki account, rdp account, database training<br />
<br />
<br />
[[Category:Work Log]]</div>OliverChttp://www.edegan.com/mediawiki/index.php?title=PostGIS_Installation&diff=20238PostGIS Installation2017-09-22T22:39:10Z<p>OliverC: /* Bulk Download TIGER Shapefiles */</p>
<hr />
<div>[http://postgis.net/install/ PostGIS] is a spatial database extender for PostgreSQL object-relational database. It adds support for geographic objects allowing location queries to be run in SQL.<br />
<br />
<br />
== Installation Commands ==<br />
<br />
Going off of http://trac.osgeo.org/postgis/wiki/UsersWikiPostGIS23UbuntuPGSQL96Apt<br />
<br />
<nowiki><br />
$ lsb_release --codename<br />
Codename: trusty<br />
$ sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt trusty-pgdg main" >> /etc/apt/sources.list'<br />
$ wget --quiet -O - http://apt.postgresql.org/pub/repos/apt/ACCC4CF8.asc | sudo apt-key add -<br />
$ sudo apt-get update<br />
$ psql -V<br />
psql (PostgreSQL) 9.5.4<br />
$ sudo apt install postgresql-9.5-postgis-2.3<br />
$ sudo apt-get install --no-install-recommends postgis</nowiki><br />
<br />
Note the last line. Without the <code>--no-install-recommends</code> line postgres assumes you're using the default configuration (in this case, postgis for postgresql 9.6). Since we're using a different flavor of postgis, we do not want this.<br />
<br />
The <code>postgis</code> apt package is not the extension to postgres. It is a collection of command line utilities for importing and exporting data to a postgis db.<br />
<br />
== Creating a Postgis DB ==<br />
<br />
=== Initial Setup ===<br />
<br />
<nowiki><br />
$ createdb --username researcher tigertest<br />
$ psql --username researcher --dbname tigertest<br />
> create extension postgis;<br />
> select postgis_full_version(); -- sanity test and make sure installed and enabled<br />
> \q</nowiki><br />
<br />
=== Bulk Download TIGER Shapefiles ===<br />
<br />
For example, say we want all of the state-level place data. The first step is to find a programmatic URL that we can use. You can inspect the HTML on the HTML interface to place data to get the correct mapping of states/territories to two digit integer. Note that they are not necessarily sequential and there is not strictly fifty.<br />
<br />
<nowiki><br />
$ for i in $(seq -f "%02g" 1 80); do<br />
wget "https://www2.census.gov/geo/tiger/TIGER2016/PLACE/tl_2016_${i}_place.zip";<br />
sleep 3 # not necessary but a good guy scraper move<br />
done;<br />
$ for f in *.zip; do unzip "$f"; done<br />
$ for f in *.shp; do shp2pgsql -I "$f" | psql -U researcher -d tigertest; done<br />
$ psql --username researcher --dbname tigertest<br />
> select count(*) from tl_2016_01_place; -- 585</nowiki><br />
<br />
Also say we want to separate New York City, New York into a more granular by-borough set of polygons. Using [https://geo.nyu.edu/catalog/nyu_2451_34505 City University of New York data], we will import this shapefile.<br />
<br />
<nowiki><br />
$ curl --insecure https://archive.nyu.edu/retrieve/74704/nyu_2451_34505.zip > /tmp/boroughs.zip<br />
$ cd /tmp<br />
$ unzip boroughs.zip<br />
$ shp2pgsql -I nyu_2451_34505/nyu_2451_34505.shp | psql -U researcher -d tigertest<br />
$ psql --username researcher --dbname tigertest<br />
> select count(*) from nyu_2451_34505;<br />
> \q</nowiki></div>OliverChttp://www.edegan.com/mediawiki/index.php?title=PostGIS_Installation&diff=20237PostGIS Installation2017-09-22T22:33:59Z<p>OliverC: /* Bulk Download TIGER Shapefiles */</p>
<hr />
<div>[http://postgis.net/install/ PostGIS] is a spatial database extender for PostgreSQL object-relational database. It adds support for geographic objects allowing location queries to be run in SQL.<br />
<br />
<br />
== Installation Commands ==<br />
<br />
Going off of http://trac.osgeo.org/postgis/wiki/UsersWikiPostGIS23UbuntuPGSQL96Apt<br />
<br />
<nowiki><br />
$ lsb_release --codename<br />
Codename: trusty<br />
$ sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt trusty-pgdg main" >> /etc/apt/sources.list'<br />
$ wget --quiet -O - http://apt.postgresql.org/pub/repos/apt/ACCC4CF8.asc | sudo apt-key add -<br />
$ sudo apt-get update<br />
$ psql -V<br />
psql (PostgreSQL) 9.5.4<br />
$ sudo apt install postgresql-9.5-postgis-2.3<br />
$ sudo apt-get install --no-install-recommends postgis</nowiki><br />
<br />
Note the last line. Without the <code>--no-install-recommends</code> line postgres assumes you're using the default configuration (in this case, postgis for postgresql 9.6). Since we're using a different flavor of postgis, we do not want this.<br />
<br />
The <code>postgis</code> apt package is not the extension to postgres. It is a collection of command line utilities for importing and exporting data to a postgis db.<br />
<br />
== Creating a Postgis DB ==<br />
<br />
=== Initial Setup ===<br />
<br />
<nowiki><br />
$ createdb --username researcher tigertest<br />
$ psql --username researcher --dbname tigertest<br />
> create extension postgis;<br />
> select postgis_full_version(); -- sanity test and make sure installed and enabled<br />
> \q</nowiki><br />
<br />
=== Bulk Download TIGER Shapefiles ===<br />
<br />
For example, say we want all of the state-level place data. The first step is to find a programmatic URL that we can use. You can inspect the HTML on the HTML interface to place data to get the correct mapping of states/territories to two digit integer. Note that they are not necessarily sequential and there is not strictly fifty.<br />
<br />
<nowiki><br />
#!/bin/bash<br />
<br />
for i in $(seq -f "%02g" 1 80); do<br />
wget "https://www2.census.gov/geo/tiger/TIGER2016/PLACE/tl_2016_${i}_place.zip";<br />
sleep 3 # not necessary but a good guy scraper move<br />
done;</nowiki><br />
<br />
Also say we want to separate New York City, New York into a more granular by-borough set of polygons. Using [https://geo.nyu.edu/catalog/nyu_2451_34505 City University of New York data], we will import this shapefile.<br />
<br />
<nowiki><br />
$ curl --insecure https://archive.nyu.edu/retrieve/74704/nyu_2451_34505.zip > /tmp/boroughs.zip<br />
$ cd /tmp<br />
$ unzip boroughs.zip<br />
$ shp2pgsql -I nyu_2451_34505/nyu_2451_34505.shp | psql -U researcher -d tigertest<br />
$ psql --username researcher --dbname tigertest<br />
> select count(*) from nyu_2451_34505;<br />
> \q</nowiki></div>OliverC