http://www.edegan.com/mediawiki/api.php?action=feedcontributions&feedformat=atom&user=ShoebMohammededegan.com - User contributions [en]2026-06-08T09:53:22ZUser contributionsMediaWiki 1.34.2http://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Work_Log)&diff=8001Shoeb Mohammed (Work Log)2016-08-08T20:03:31Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Category: McNair Admin]]<br />
[[admin_classification::General Information| ]]<br />
[[Shoeb Mohammed]] [[Work Logs]] [[Shoeb Mohammed (Work Log) | Work Log page]]<br />
<br />
<br />
*07/15/2016 <br />
**Started work log and research plan.<br />
**Review and profile the Matcher perl code.<br />
**Read tutorials on using Selenium with perl. Below links are good to get started <br />
https://www.usenix.org/system/files/login/articles/login_apr15_09_blank-edelman.pdf<br />
https://www.usenix.org/system/files/login/articles/blank-edelman_0.pdf<br />
<br />
<br />
*07/18/2016<br />
**Completed listing page on the wiki for the software repository.<br />
**Finished installing linux for code development.<br />
<br />
*07/19/2016<br />
**Installed development environment on the linux box - python bindings, editor(emacs), git<br />
**Decided to use python in place of perl because it is officially supported.<br />
<br />
*07/20/2016<br />
**Created accounts<br />
**Ran selenium to load a sample website<br />
<br />
*07/21/2016<br />
**read more selenium docs and tutorials.<br />
**waiting for Dan's codebase<br />
<br />
*07/22/2016<br />
**initial design for the program<br />
**still waiting for Dan's codebase<br />
<br />
*07/25/2016<br />
**started implementation. need clarification on requirements.<br />
<br />
*07/27/2016<br />
**continue with the implementation. need clarification on search handler requirements.<br />
<br />
*07/28/2016, 07/29/2016, 07/30/2016, 08/01/2016<br />
**continue development. <br />
<br />
*08/05/2016<br />
**completed development. Future todo: implement advance facilities.<br />
<br />
*08/06/2016<br />
**added Eds suggestions.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Work_Log)&diff=8000Shoeb Mohammed (Work Log)2016-08-06T20:39:21Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Category: McNair Admin]]<br />
[[admin_classification::General Information| ]]<br />
[[Shoeb Mohammed]] [[Work Logs]] [[Shoeb Mohammed (Work Log) | Work Log page]]<br />
<br />
<br />
*07/15/2016 <br />
**Started work log and research plan.<br />
**Review and profile the Matcher perl code.<br />
**Read tutorials on using Selenium with perl. Below links are good to get started <br />
https://www.usenix.org/system/files/login/articles/login_apr15_09_blank-edelman.pdf<br />
https://www.usenix.org/system/files/login/articles/blank-edelman_0.pdf<br />
<br />
<br />
*07/18/2016<br />
**Completed listing page on the wiki for the software repository.<br />
**Finished installing linux for code development.<br />
<br />
*07/19/2016<br />
**Installed development environment on the linux box - python bindings, editor(emacs), git<br />
**Decided to use python in place of perl because it is officially supported.<br />
<br />
*07/20/2016<br />
**Created accounts<br />
**Ran selenium to load a sample website<br />
<br />
*07/21/2016<br />
**read more selenium docs and tutorials.<br />
**waiting for Dan's codebase<br />
<br />
*07/22/2016<br />
**initial design for the program<br />
**still waiting for Dan's codebase<br />
<br />
*07/25/2016<br />
**started implementation. need clarification on requirements.<br />
<br />
*07/27/2016<br />
**continue with the implementation. need clarification on search handler requirements.<br />
<br />
*07/28/2016, 07/29/2016, 07/30/2016, 08/01/2016<br />
**continue development. <br />
<br />
*08/05/2016<br />
**completed development. Future todo: implement advance facilities.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Research_Plans)&diff=7999Shoeb Mohammed (Research Plans)2016-08-06T20:38:24Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Category: McNair Admin]]<br />
[[admin_classification::General Information| ]]<br />
[[Shoeb Mohammed]] [[Research Plans]] [[Shoeb Mohammed (Research Plans) | Research Plan page]]<br />
<br />
====Short Term====<br />
* Create a listing on the wiki for all software developed at McNair center. - '''''Completed'''''<br />
* Build a Linux box to run the crawler. - '''''Completed'''''<br />
<br />
====Long Term====<br />
* Optimize/re-design the 'Matcher' software. In particular, speed-up fuzzy matching and possibly re-structure the code to make usage easier.<br />
* Develop the crawler. Try to begin with code that Dan has. - '''''Completed'''''<br />
<br />
====Side Tasks====<br />
* If possible, redo the patent parser (previous coded by Kranti) to also pull in Patent Citation data.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Research_Plans)&diff=7998Shoeb Mohammed (Research Plans)2016-08-06T20:36:12Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Internal]]<br />
[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Research Plans]] [[Shoeb Mohammed (Research Plans) | Research Plan page]]<br />
<br />
====Short Term====<br />
* Create a listing on the wiki for all software developed at McNair center. - '''''Completed'''''<br />
* Build a Linux box to run the crawler. - '''''Completed'''''<br />
<br />
====Long Term====<br />
* Optimize/re-design the 'Matcher' software. In particular, speed-up fuzzy matching and possibly re-structure the code to make usage easier.<br />
* Develop the crawler. Try to begin with code that Dan has. - '''''Completed'''''<br />
<br />
====Side Tasks====<br />
* If possible, redo the patent parser (previous coded by Kranti) to also pull in Patent Citation data.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Research_Plans)&diff=7997Shoeb Mohammed (Research Plans)2016-08-06T20:35:49Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Internal]]<br />
[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Research Plans]] [[Shoeb Mohammed (Research Plans) | Research Plan page]]<br />
<br />
<br />
<br />
<br />
====Short Term====<br />
* Create a listing on the wiki for all software developed at McNair center. - '''''Completed'''''<br />
* Build a Linux box to run the crawler. - '''''Completed'''''<br />
<br />
====Long Term====<br />
* Optimize/re-design the 'Matcher' software. In particular, speed-up fuzzy matching and possibly re-structure the code to make usage easier.<br />
* Develop the crawler. Try to begin with code that Dan has. - '''''Completed'''''<br />
<br />
====Side Tasks====<br />
* If possible, redo the patent parser (previous coded by Kranti) to also pull in Patent Citation data.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Research_Plans)&diff=7986Shoeb Mohammed (Research Plans)2016-08-05T19:50:32Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Research Plans]] [[Shoeb Mohammed (Research Plans) | Research Plan page]]<br />
<br />
====Short Term====<br />
* Create a listing on the wiki for all software developed at McNair center. - '''''Completed'''''<br />
* Build a Linux box to run the crawler. - '''''Completed'''''<br />
<br />
====Long Term====<br />
* Optimize/re-design the 'Matcher' software. In particular, speed-up fuzzy matching and possibly re-structure the code to make usage easier.<br />
* Develop the crawler. Try to begin with code that Dan has. - '''''Completed'''''<br />
<br />
====Side Tasks====<br />
* If possible, redo the patent parser (previous coded by Kranti) to also pull in Patent Citation data.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Research_Plans)&diff=7985Shoeb Mohammed (Research Plans)2016-08-05T19:48:59Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Research Plans]] [[Shoeb Mohammed (Research Plans) | Research Plan page]]<br />
<br />
====Short Term====<br />
* Create a listing on the wiki for all software developed at McNair center. - completed<br />
* Build a Linux box to run the crawler. - completed<br />
<br />
====Long Term====<br />
* Optimize/re-design the 'Matcher' software. In particular, speed-up fuzzy matching and possibly re-structure the code to make usage easier.<br />
* Develop the crawler. Try to begin with code that Dan has. - completed<br />
<br />
====Side Tasks====<br />
* If possible, redo the patent parser (previous coded by Kranti) to also pull in Patent Citation data.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Research_Plans)&diff=7984Shoeb Mohammed (Research Plans)2016-08-05T19:48:07Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Research Plans]] [[Shoeb Mohammed (Research Plans) | Research Plan page]]<br />
<br />
<br />
====Short Term====<br />
* Create a listing on the wiki for all software developed at McNair center.<br />
* Build a Linux box to run the crawler.<br />
<br />
====Long Term====<br />
* Optimize/re-design the 'Matcher' software. In particular, speed-up fuzzy matching and possibly re-structure the code to make usage easier.<br />
* Develop the crawler. Try to begin with code that Dan has.<br />
<br />
====Side Tasks====<br />
* If possible, redo the patent parser (previous coded by Kranti) to also pull in Patent Citation data.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Work_Log)&diff=7983Shoeb Mohammed (Work Log)2016-08-05T19:45:08Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Work Logs]] [[Shoeb Mohammed (Work Log) | Work Log page]]<br />
<br />
<br />
*07/15/2016 <br />
**Started work log and research plan.<br />
**Review and profile the Matcher perl code.<br />
**Read tutorials on using Selenium with perl. Below links are good to get started <br />
https://www.usenix.org/system/files/login/articles/login_apr15_09_blank-edelman.pdf<br />
https://www.usenix.org/system/files/login/articles/blank-edelman_0.pdf<br />
<br />
<br />
*07/18/2016<br />
**Completed listing page on the wiki for the software repository.<br />
**Finished installing linux for code development.<br />
<br />
*07/19/2016<br />
**Installed development environment on the linux box - python bindings, editor(emacs), git<br />
**Decided to use python in place of perl because it is officially supported.<br />
<br />
*07/20/2016<br />
**Created accounts<br />
**Ran selenium to load a sample website<br />
<br />
*07/21/2016<br />
**read more selenium docs and tutorials.<br />
**waiting for Dan's codebase<br />
<br />
*07/22/2016<br />
**initial design for the program<br />
**still waiting for Dan's codebase<br />
<br />
*07/25/2016<br />
**started implementation. need clarification on requirements.<br />
<br />
*07/27/2016<br />
**continue with the implementation. need clarification on search handler requirements.<br />
<br />
*07/28/2016, 07/29/2016, 07/30/2016, 08/01/2016<br />
**continue development. <br />
<br />
*08/05/2016<br />
**completed development. Future todo: implement advance facilities.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Work_Log)&diff=7586Shoeb Mohammed (Work Log)2016-07-27T20:19:03Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Work Logs]] [[Shoeb Mohammed (Work Log) | Work Log page]]<br />
<br />
<br />
*07/15/2016 <br />
**Started work log and research plan.<br />
**Review and profile the Matcher perl code.<br />
**Read tutorials on using Selenium with perl. Below links are good to get started <br />
https://www.usenix.org/system/files/login/articles/login_apr15_09_blank-edelman.pdf<br />
https://www.usenix.org/system/files/login/articles/blank-edelman_0.pdf<br />
<br />
<br />
*07/18/2016<br />
**Completed listing page on the wiki for the software repository.<br />
**Finished installing linux for code development.<br />
<br />
*07/19/2016<br />
**Installed development environment on the linux box - python bindings, editor(emacs), git<br />
**Decided to use python in place of perl because it is officially supported.<br />
<br />
*07/20/2016<br />
**Created accounts<br />
**Ran selenium to load a sample website<br />
<br />
*07/21/2016<br />
**read more selenium docs and tutorials.<br />
**waiting for Dan's codebase<br />
<br />
*07/22/2016<br />
**initial design for the program<br />
**still waiting for Dan's codebase<br />
<br />
*07/25/2016<br />
**started implementation. need clarification on requirements.<br />
<br />
*07/27/2016<br />
**continue with the implementation. need clarification on search handler requirements.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Work_Log)&diff=7446Shoeb Mohammed (Work Log)2016-07-25T21:59:35Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Work Logs]] [[Shoeb Mohammed (Work Log) | Work Log page]]<br />
<br />
<br />
*07/15/2016 <br />
**Started work log and research plan.<br />
**Review and profile the Matcher perl code.<br />
**Read tutorials on using Selenium with perl. Below links are good to get started <br />
https://www.usenix.org/system/files/login/articles/login_apr15_09_blank-edelman.pdf<br />
https://www.usenix.org/system/files/login/articles/blank-edelman_0.pdf<br />
<br />
<br />
*07/18/2016<br />
**Completed listing page on the wiki for the software repository.<br />
**Finished installing linux for code development.<br />
<br />
*07/19/2016<br />
**Installed development environment on the linux box - python bindings, editor(emacs), git<br />
**Decided to use python in place of perl because it is officially supported.<br />
<br />
*07/20/2016<br />
**Created accounts<br />
**Ran selenium to load a sample website<br />
<br />
*07/21/2016<br />
**read more selenium docs and tutorials.<br />
**waiting for Dan's codebase<br />
<br />
*07/22/2016<br />
**initial design for the program<br />
**still waiting for Dan's codebase<br />
<br />
*07/25/2016<br />
**started implementation. need clarification on requirements.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Work_Log)&diff=7364Shoeb Mohammed (Work Log)2016-07-21T21:29:34Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Work Logs]] [[Shoeb Mohammed (Work Log) | Work Log page]]<br />
<br />
<br />
*07/15/2016 <br />
**Started work log and research plan.<br />
**Review and profile the Matcher perl code.<br />
**Read tutorials on using Selenium with perl. Below links are good to get started <br />
https://www.usenix.org/system/files/login/articles/login_apr15_09_blank-edelman.pdf<br />
https://www.usenix.org/system/files/login/articles/blank-edelman_0.pdf<br />
<br />
<br />
*07/18/2016<br />
**Completed listing page on the wiki for the software repository.<br />
**Finished installing linux for code development.<br />
<br />
*07/19/2016<br />
**Installed development environment on the linux box - python bindings, editor(emacs), git<br />
**Decided to use python in place of perl because it is officially supported.<br />
<br />
*07/20/2016<br />
**Created accounts<br />
**Ran selenium to load a sample website<br />
<br />
*07/21/2016<br />
**read more selenium docs and tutorials.<br />
**waiting for Dan's codebase</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Work_Log)&diff=7363Shoeb Mohammed (Work Log)2016-07-21T21:23:25Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Work Logs]] [[Shoeb Mohammed (Work Log) | Work Log page]]<br />
<br />
<br />
*07/15/2016 <br />
**Started work log and research plan.<br />
**Review and profile the Matcher perl code.<br />
**Read tutorials on using Selenium with perl. Below links are good to get started <br />
https://www.usenix.org/system/files/login/articles/login_apr15_09_blank-edelman.pdf<br />
https://www.usenix.org/system/files/login/articles/blank-edelman_0.pdf<br />
<br />
<br />
*07/18/2016<br />
**Completed listing page on the wiki for the software repository.<br />
**Finished installing linux for code development.<br />
<br />
*07/19/2016<br />
**Installed development environment on the linux box - python bindings, editor(emacs), git<br />
<br />
*07/20/2016<br />
**Created accounts<br />
**Ran selenium to load a sample website<br />
<br />
*07/21/2016<br />
**read more selenium docs and tutorials.<br />
**waiting for Dan's codebase</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Work_Log)&diff=7284Shoeb Mohammed (Work Log)2016-07-20T20:05:21Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Work Logs]] [[Shoeb Mohammed (Work Log) | Work Log page]]<br />
<br />
<br />
*07/15/2016 <br />
**Started work log and research plan.<br />
**Review and profile the Matcher perl code.<br />
**Read tutorials on using Selenium with perl. Below links are good to get started <br />
https://www.usenix.org/system/files/login/articles/login_apr15_09_blank-edelman.pdf<br />
https://www.usenix.org/system/files/login/articles/blank-edelman_0.pdf<br />
<br />
<br />
*07/18/2016<br />
**Completed listing page on the wiki for the software repository.<br />
**Finished installing linux for code development.<br />
<br />
*07/19/2016<br />
**Installed development environment on the linux box - python bindings, editor(emacs), git<br />
<br />
*07/20/2016<br />
**Created accounts<br />
**Ran selenium to load a sample website</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Work_Log)&diff=7127Shoeb Mohammed (Work Log)2016-07-19T17:32:10Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Work Logs]] [[Shoeb Mohammed (Work Log) | Work Log page]]<br />
<br />
<br />
*07/15/2016 <br />
**Started work log and research plan.<br />
**Review and profile the Matcher perl code.<br />
**Read tutorials on using Selenium with perl. Below links are good to get started <br />
https://www.usenix.org/system/files/login/articles/login_apr15_09_blank-edelman.pdf<br />
https://www.usenix.org/system/files/login/articles/blank-edelman_0.pdf<br />
<br />
<br />
*07/18/2016<br />
**Completed listing page on the wiki for the software repository.<br />
**Finished installing linux for code development.<br />
<br />
*07/19/2016<br />
**Installed development environment on the linux box - python bindings, editor(emacs), git</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Work_Log)&diff=7125Shoeb Mohammed (Work Log)2016-07-19T17:04:51Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Work Logs]] [[Shoeb Mohammed (Work Log) | Work Log page]]<br />
<br />
<br />
*07/15/2016 <br />
**Started work log and research plan.<br />
**Review and profile the Matcher perl code.<br />
**Read tutorials on using Selenium with perl. Below links are good to get started <br />
https://www.usenix.org/system/files/login/articles/login_apr15_09_blank-edelman.pdf<br />
https://www.usenix.org/system/files/login/articles/blank-edelman_0.pdf<br />
<br />
<br />
*07/18/2016<br />
**Completed listing page on the wiki for the software repository.<br />
**Finished installing linux for code development.<br />
<br />
*07/19/2016<br />
**Installed development environment on the linux box - python bindings, IDE(emacs), git</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Work_Log)&diff=7055Shoeb Mohammed (Work Log)2016-07-18T20:03:01Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Work Logs]] [[Shoeb Mohammed (Work Log) | Work Log page]]<br />
<br />
<br />
*07/15/2016 <br />
**Started work log and research plan.<br />
**Review and profile the Matcher perl code.<br />
**Read tutorials on using Selenium with perl. Below links are good to get started <br />
https://www.usenix.org/system/files/login/articles/login_apr15_09_blank-edelman.pdf<br />
https://www.usenix.org/system/files/login/articles/blank-edelman_0.pdf<br />
<br />
<br />
*07/18/2016<br />
**Completed listing page on the wiki for the software repository.<br />
**Finished installing linux for code development.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Work_Log)&diff=7052Shoeb Mohammed (Work Log)2016-07-18T20:00:24Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Work Logs]]<br />
<br />
*07/15/2016 <br />
**Started work log and research plan.<br />
**Review and profile the Matcher perl code.<br />
**Read tutorials on using Selenium with perl. Below links are good to get started <br />
https://www.usenix.org/system/files/login/articles/login_apr15_09_blank-edelman.pdf<br />
https://www.usenix.org/system/files/login/articles/blank-edelman_0.pdf<br />
<br />
<br />
*07/18/2016<br />
**Completed listing page on the wiki for the software repository.<br />
**Finished installing linux for code development.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Work_Log)&diff=7051Shoeb Mohammed (Work Log)2016-07-18T20:00:06Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Work Logs]] [[Shoeb Mohammed (Work Log) | (Work Log)]]<br />
<br />
*07/15/2016 <br />
**Started work log and research plan.<br />
**Review and profile the Matcher perl code.<br />
**Read tutorials on using Selenium with perl. Below links are good to get started <br />
https://www.usenix.org/system/files/login/articles/login_apr15_09_blank-edelman.pdf<br />
https://www.usenix.org/system/files/login/articles/blank-edelman_0.pdf<br />
<br />
<br />
*07/18/2016<br />
**Completed listing page on the wiki for the software repository.<br />
**Finished installing linux for code development.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Work_Log)&diff=6959Shoeb Mohammed (Work Log)2016-07-18T19:36:50Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Work Logs]] [[Shoeb Mohammed (Work Log) | (Work Log)]]<br />
<br />
*07/15/2016 <br />
**Started work log and research plan.<br />
**Review and profile the Matcher perl code.<br />
**Read tutorials on using Selenium with perl. Below links are good to get started <br />
https://www.usenix.org/system/files/login/articles/login_apr15_09_blank-edelman.pdf<br />
https://www.usenix.org/system/files/login/articles/blank-edelman_0.pdf<br />
<br />
<br />
*07/18/2016<br />
**Completed listing page on the wiki for the software repository.<br />
**Began installing linux for development.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Work_Log)&diff=6958Shoeb Mohammed (Work Log)2016-07-18T19:36:14Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Work Logs]] [[Shoeb Mohammed (Work Log) | (Work Log)]]<br />
<br />
*07/15/2016 <br />
**Started work log and research plan.<br />
**Review and profile the Matcher perl code.<br />
**Read tutorials on using Selenium with perl. Below links are good to get started <br />
https://www.usenix.org/system/files/login/articles/login_apr15_09_blank-edelman.pdf<br />
https://www.usenix.org/system/files/login/articles/blank-edelman_0.pdf<br />
<br />
<br />
*07/18/2016<br />
**Completed listing page on the wiki for the software repository.<br />
**Begin building linux box</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Software_Repository_Listing&diff=6915Software Repository Listing2016-07-18T17:15:22Z<p>ShoebMohammed: /* Geocoding Inventor Locations */</p>
<hr />
<div>This page lists all software/tools available on our [[Software Repository]]. The documentation on using a particular tool will be on its separate wiki page.<br />
<br />
For information and tutorial on how to access McNair git server, see [[Software Repository]].<br />
Read the tutorial and instructions first before pushing anything to the git-server.<br />
<br />
=Repositories on McNair [[Software Repository|git server]]=<br />
<br />
==Center IT Sysadmin==<br />
This repository contains all tools and scripts meant for system administration (stuff like backup scripts..)<br />
*See the [[Center IT]] page for current documentation.<br />
<br />
<br />
==Harvard Dataverse==<br />
This repository contains all tools and scripts related to Harvard Dataverse.<br />
*The [[Harvard Dataverse]] page provides instruction on how to access data.<br />
<br />
<br />
==Geocoding Inventor Locations==<br />
This repository holds software for matching Inventor addresses to known locations.<br />
There are two programs/scripts that do same job. One is implemented in Perl(old) and other in Python(new). You should probably use the newer tool.<br />
*See [[Geocoding Inventor Locations (Tool)]] for documentation on the older version implemented in Perl.<br />
*See [[Geocode.py]] for the newer version in Python.<br />
<br />
==Matcher==<br />
This repository contains the matcher tool which is used to match firm names given two lists.<br />
*See [[The Matcher (Tool)]] for documentation.<br />
<br />
==Patent Data Parser==<br />
This repository contains all tools developed for patent data parsing.<br />
*[[Patent Data (Tool)]] and [[Patent Data Extraction Scripts (Tool)]] pages on the wiki describe our Patent Database schema and corresponding XML parsing tools.<br />
*Also, see [[USPTO Assignees Data]] which explains Patent Assignee Database schema and relevant XML parsing tools.<br />
<br />
==Utilities==<br />
This repository contains various utilities developed for text processing and other generally useful tools. See the wiki pages for each tool's documentation.<br />
*[[Fuzzy match names (Tool)]]<br />
*[[Godo (Tool)]]<br />
*[[Normalizer Documentation | Normalizer]]. On the [[Software Repository|git-server]] we have many different versions like normalize fixed width, normalize surnames.<br />
<br />
==Web Crawler==<br />
This repository contains all software for web crawlers.<br />
*[[Whois Parser]] pulls the Whois information given a list for URLs.<br />
*[[PhD Masterclass - How to Build a Web Crawler]]: Ed's class on building a web crawler.<br />
<br />
<br />
[[category:McNair Admin]]<br />
[[admin_classification::Software Repository| ]]</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Software_Repository_Listing&diff=6912Software Repository Listing2016-07-18T17:06:27Z<p>ShoebMohammed: Created page with "This page lists all software/tools available on our Software Repository. The documentation on using a particular tool will be on its separate wiki page. For information..."</p>
<hr />
<div>This page lists all software/tools available on our [[Software Repository]]. The documentation on using a particular tool will be on its separate wiki page.<br />
<br />
For information and tutorial on how to access McNair git server, see [[Software Repository]].<br />
Read the tutorial and instructions first before pushing anything to the git-server.<br />
<br />
=Repositories on McNair [[Software Repository|git server]]=<br />
<br />
==Center IT Sysadmin==<br />
This repository contains all tools and scripts meant for system administration (stuff like backup scripts..)<br />
*See the [[Center IT]] page for current documentation.<br />
<br />
<br />
==Harvard Dataverse==<br />
This repository contains all tools and scripts related to Harvard Dataverse.<br />
*The [[Harvard Dataverse]] page provides instruction on how to access data.<br />
<br />
<br />
==Geocoding Inventor Locations==<br />
This repository holds software for matching Inventor addresses to known locations.<br />
There are two programs/scripts that do same job. One is implemented in Perl(old) and other in Python(new). You should probably use the newer tool.<br />
*See [[Geocoding Inventor Locations (Tool)]] for documentation on the older version implemented in Perl.<br />
*See [[Geocoding Inventor Locations - New (Tool)]] for the newer version in Python.<br />
<br />
<br />
==Matcher==<br />
This repository contains the matcher tool which is used to match firm names given two lists.<br />
*See [[The Matcher (Tool)]] for documentation.<br />
<br />
==Patent Data Parser==<br />
This repository contains all tools developed for patent data parsing.<br />
*[[Patent Data (Tool)]] and [[Patent Data Extraction Scripts (Tool)]] pages on the wiki describe our Patent Database schema and corresponding XML parsing tools.<br />
*Also, see [[USPTO Assignees Data]] which explains Patent Assignee Database schema and relevant XML parsing tools.<br />
<br />
==Utilities==<br />
This repository contains various utilities developed for text processing and other generally useful tools. See the wiki pages for each tool's documentation.<br />
*[[Fuzzy match names (Tool)]]<br />
*[[Godo (Tool)]]<br />
*[[Normalizer Documentation | Normalizer]]. On the [[Software Repository|git-server]] we have many different versions like normalize fixed width, normalize surnames.<br />
<br />
==Web Crawler==<br />
This repository contains all software for web crawlers.<br />
*[[Whois Parser]] pulls the Whois information given a list for URLs.<br />
*[[PhD Masterclass - How to Build a Web Crawler]]: Ed's class on building a web crawler.<br />
<br />
<br />
[[category:McNair Admin]]<br />
[[admin_classification::Software Repository| ]]</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Geocoding_Inventor_Locations_(Tool)&diff=6911Geocoding Inventor Locations (Tool)2016-07-18T17:06:11Z<p>ShoebMohammed: Created page with "*This page is part of a series under the NBER Patent Data Project This page details the various matching techniques used to Geocode inventor locations i..."</p>
<hr />
<div>*This page is part of a series under the [[NBER Patent Data |NBER Patent Data Project]]<br />
<br />
This page details the various matching techniques used to Geocode inventor locations in the NBER patent data. Geocoding inventor locations entails matching the inventor addresses provided in the patent data to known locations through-out the world and recording their longitude and latitude. <br />
<br />
==Script Files==<br />
<br />
The scripts and modules that operationalize these matching techniques can be downloaded as a bundle with ([http://www.edegan.com/repository/MatchLocations.tar.gz MatchLocations.tar.gz v1.0.1] ~20Mb) or without ([http://www.edegan.com/repository/MatchLocations_Full.tar.gz MatchLocations_Full.tar.gz v1.0.1] ~20Mb) all supporting data files. Note that the current version is 1.0.3, which will be posted shortly. The bundles contain the default directory structure. Defaults can be changed in the MatchLocations.pl script. <br />
<br />
<br />
The directories are as follows:<br />
*Source - Source data should be placed here. See below for formatting.<br />
*Results - Results generated by the scripts, including logs will appear here.<br />
*GNS - contains GNS reference data named GNS-XX.txt<br />
*Match - contains the modules<br />
<br />
<br />
The bundle contains:<br />
*MatchLocations.pl - The main script that initializes and processes the matching requests<br />
*BatchMatch.pl - A script for running batches <br />
*Match::GNS.pm - Interface to the GNS reference data (see below)<br />
*Match::Patent.pm - Interface to the Patent Location data (see below)<br />
*Match::Common.pm - Provides common (string cleaning) routines for both the reference and source interface modules<br />
*Match::PostalCodes.pm - A module that extracts postcodes of various formats from (address) strings<br />
*Match::Gram.pm - Custom NGram Module<br />
*Match::LCS.pm - A standard LCS Module<br />
*PatentLocations-Stopwords.txt - A Stop Word file (tab delimited)<br />
*GNS Reference Files - The full bundle contains a full set of correctly named GNS reference files<br />
<br />
The MatchLocations.pl script can be run from any shell or command line with perl installed. Example commands are:<br />
<br />
<tt>perl MatchLocations.pl -co GB -u -human -r -wf </tt><br />
<br />
which will process ISO3166 <tt>country</tt> code GB (Great Britain), include <tt>unmatched</tt> inputs in the results file, produce a <tt>human</tt> choices file, write the <tt>report</tt> to a text file, and <tt>write fuzzy</tt> matches to additional seperate files as well as the main results file. Other options include <tt>over</tt> to override country designations and <tt>o</tt> to specify the results filename.<br />
<br />
<tt>perl MatchLocations.pl -h</tt><br />
<br />
produces a simple help output.<br />
<br />
==The Source Files==<br />
<br />
Per country source files are extracted from the NBER patent data. The format of the source file(s) is as follows (XX is an ISO3166 code): <br />
<br />
XX.txt - Tab delimited plain text with no (intentional) string quotation. <br />
Column(s): <tt>country</tt> <tt>str</tt> <tt>cty</tt> <tt>adm</tt> <tt>city</tt> <tt>postcode</tt> <tt>str</tt><br />
<br />
The column order is not important. <tt>country</tt>, <tt>str</tt>, and <tt>cty</tt> can not all be null. <tt>adm</tt> <tt>city</tt> <tt>postcode</tt> are optional 'exception' fields that are processed with priority. They provide hand corrections and other specifically generated information.<br />
<br />
The perl module Match::Patent.pm loads and provides an interface to this source data. The source code is the primary module documentation. The Match::PostalCodes.pm perl module provides a method to extract [[Postal Codes]] from a the addresses for a large number of ISO3166 codes, and implements 'standard' postal code identification for all other jurisdictions.<br />
<br />
==Reference Data==<br />
<br />
The reference data for the locations (which provides the longitude and latitudes) is taken from the (U.S.) National Geospatial-Intelligence Agency's [[GEOnet Names Server | GEOnet Names Server (GNS)]] which covers the world excluding the U.S. and Antartica. <br />
<br />
This project uses [[ISO3166]] two-character country codes to name source and reference files. GNS does not use ISO3166 country codes, and so users will need to translate accordingly (see the [[GEOnet Names Server | GNS page]] for details). A full bundle of correctly names GNS files is also available.<br />
<br />
The perl module Match::GNS.pm loads, indexes and provides an interface to key variables from this data. The source code is the primary module documentation. The load() method takes an ISO3166 code, and the index methods and most other methods take specific GNS FC codes (e.g. "P" for populated place, "L" for locality, and "A" for administrative area). Which GNS FC codes are used is specified in the @Letters global varible of MatchLocations.pl and inherited by all other modules. <br />
<br />
MatchLocations.pl also retrieves a list of all ISO3166 codes included in the data (from the MatchPatent.pm module) and in any specified override file, and calls Match::GNS.pm to load them. An override file can be specified with the <tt>-over</tt> option. Override files are tab-delimited and have the format:<br />
<br />
ListedISO3166 1stPreference 2ndPreference 3rdPreference ...<br />
<br />
The ISO3166 listed in the source data is then overridden and the alternatives are searched for matches in order of preference. The search is terminated when a match is found or the override set is exhausted.<br />
<br />
==The Matching Process==<br />
<br />
The matching process is carried out by [http://www.edegan.com/repository/MatchPatentLocations.pl MatchLocations.pl] script, and its dependent modules (detailed above), which has a standard pod based command line interface. The <tt>-co</tt> option specifies the ISO3166 country code to be matched. If the override option is used, then the <tt>-co</tt> option can be used to specify the source file. When an override option is set to 1, rather than to the filename containing the overrides, then the source files countries are used to determine which GNS lookups to perform, otherwise the <tt>-co</tt> option specifies the GNS reference set.<br />
<br />
Glossary of terms:<br />
*Units - isolated logical units from an address, such as the street number and name, the town, or the region. Postal codes are treated separately. <br />
*Tokens - Single words or sequences of words separated by a space (note that this is a specific usage)<br />
*n-grams - character sequences, such as bigrams (two letters from aa to zz), trigrams (aaa-zzz) and so forth<br />
*Exact Matching - Case insensitive of matching of the entire sequence of both the source and the reference strings<br />
*LCS - Longest Common Subsequence based matching (See below)<br />
*Administrative area, populated place, and locality - locations identified as a FC=A, FC=P or FC=L respectively in the GNS data. Unless otherwise specified, matches are performed for all GNS FC codes requested (default is A,P,L) separately and in series.<br />
<br />
The sequence of processing is as follows (matching only the remaining unmatched locations at each stage):<br />
#Load the source files, clean and parse (parsing identifies units)<br />
#Load the reference file, build indices<br />
#Exact match the exception units of records with exceptions<br />
#Exact match the units of well-formatted records<br />
#Exact match tokens (1-5 words)<br />
#N-gram and LCS match<br />
#Reconsile multiple matches<br />
<br />
===Exact Matching Units===<br />
<br />
The exact matching of units is performed for both the exception units and units of "well-formatted" records, that is records that have comma seperated logical units. Postcodes are extracted as a logical unit if possible first (to generate the PRS_POSTCODE field). Exact matching is case insensitive and units are trimmed of preceeding and subsequent spaces, but otherwise the match must be exact. Units are matched from the bottom to the top, in order of precedence. That is if the string is Unit1, Unit2, Unit3, Postcode; then Unit3 is matched with precedence over Units 2 and 1, and so forth. However, if multiple matches are made for a some FC code and one match is made for another, then preference is given to the different combination. For example if the string were "Chelsea, London" and both Chelsea and London were recorded in the GNS data as FC=P, but only London was recorded as a FC=A, then it would be most sensible to record P=Chelsea, A=London, and not P=London, A=London. This is differencing is done in the matching method and independent from the resolution of multiple matches at the end.<br />
<br />
===Token Matching===<br />
<br />
Each source string is cleaned of upper-asci code (i.e. reduced to alphanumerics plus spaces), removed of its postcode (to generate PRS_POSTCODE) and then seperated into token arrays on space characters. Thus the string "String1 String2 String3 String4 Postcode" would become:<br />
[0]String1 <br />
[1]String2 <br />
[2]String3 <br />
[3]String4 <br />
<br />
An arbitrary upper token set length limit of 5 is used if the length of the source token array (4 in the example above) is greater than or equal to 5. Then beginning at the upper length limit and decreasing by one after each set of this lenght has been tried, and starting from the right hand-side and moving one unit to the left each time, the token sets are joined with spaces and exact matched against the reference string. This process iterates all length one token sets have been tried and records the matches in the order that they were made. Thus continuing the example above the space-joined source token sets would be, in the order that they are tried:<br />
#String1 String2 String3 String4 (token set length=4, first and only set)<br />
#String2 String3 String4 (token set length=3, first set)<br />
#String1 String2 String3 (token set length=3, second set)<br />
#String3 String4 (token set length=2, first set)<br />
#String2 String3 (token set length=2, second set)<br />
#String1 String2 (token set length=2, third set)<br />
#String4 (token set length=1, first set)<br />
#String3 (token set length=1, second set)<br />
#String2 (token set length=1, third set)<br />
#String1 (token set length=1, fourth set)<br />
<br />
===NGram and LCS Matching===<br />
<br />
Longest Common Subsequence (LCS) is an abundantly used fuzzy matching technique. The [http://en.wikipedia.org/wiki/Longest_common_subsequence Longest Common Subsequence page on wikipedia] provides a very detailed background. However, LCS matching of two datasets is an NP-Hard problem and extremely processor intensive. To avoid long run-times, LCS matching is done on only a small sub-set of strings that have met the NGram criteria detailed below.<br />
<br />
NGrams are character-based token strings. Source and reference strings are transformed to include only characters from one of the following numbered sets:<br />
#ABCDEFGHIJKLMNOPQRSTUVWXYZ (i.e. uppercase Latin alphabet)<br />
#0123456789 (i.e. Standard numbers)<br />
#" " (i.e. the space character)<br />
#Alphanumeric (i.e. 1 and 2 above)<br />
#Alphanumeric plus space (1,2,3 above)<br />
#Alphabet only plus space(1 and 3 above)<br />
<br />
Gramsets of various lengths, currently 2,3 and 4 characters, are used. Reference strings are decomposed into grams and both forward and reverse indexed for speed on large datasets. Source strings are decomposed into grams. Candidate reference strings are found by compiling a list of all reference strings that contain at least one of the source grams (this uses the reverse index). Then each of the candidate reference strings is gram-matched against the source gram to compute a number of scores. Specically gram-matching involves computing:<br />
*The number of grams in the source string<br />
*The number of grams in the reference string<br />
*The number of grams in the source string that appear in the reference string<br />
*The number of grams in the reference string that appear in the source string<br />
*The above two numbers as a source percentage and a reference percentage<br />
<br />
Then an optional constraint that the first letter of the source and reference strings must match can be employed. If a source and reference pair meets a variety of criteria on the above variables (for example the source string is greater than 10 characters long, the source percentage of grams is greater then 80% and less than 110%, the reference percentage of grams is greater than 90% and less than 105%, and the two strings start with the same letter), then the LCS is computed. An LCS percentage, using the maximum of the lengths of the source and reference strings as a denominator, is also calculated. <br />
<br />
The candidate reference string with the highest gram and LCS scores, assuming that these scores meet the decision threshold is then selected as the closest match. If no candidate reference string meets the decision threshold the source string is left unmatched. The decision thresholds are configured in the MatchLocations.pl script, and sets of Ngram/LCS matchings, using different character sets, gram lengths and decision thresholds, are performed sequentially, with the currently unmatched source strings used as input for each round.<br />
<br />
===Reconciling Multiple Matches===<br />
<br />
In a small number of cases it is possible that the source string will achieve more than one match for more than one FC code. For example suppose the string "Glouchester Street Cambridge Cambridgeshire" were considered. This could concievably produce two P matches and one A match with the token matching algorithm detailed above. <br />
<br />
To reconsile multiple matches the following process is undertaken:<br />
*If an FC code has only one match keep that one match<br />
*Aim for distinction in the set, giving priority in the order that the FC codes are specified in MatchLocations.pl. The default is to include A,P,L in order, so that precedence follows importance and size. This is important if multiple FC codes contain multiple overlapping matches. For example suppose A=1,2 P=2,3 and L=3. The algorithm will look forward and backwards to assign: A=1 P=2 L=3.<br />
*Determine the set of FC code matches with the shortest distance between them using a [http://en.wikipedia.org/wiki/Haversine_formula Haversine formula] distance calculation based on the GNS reported longitudes and latitudes. (Note that the Haversine formula is implemented in the Match::GNS.pm module and is the most accurate method over short distances, where other methods, like the great-circle method, suffer from compounded rounding error problems.) This is important when multiple FC codes have muliple matches but they do not overlap.<br />
*If one or more match is found for an FC code then one final 'best' match must be reported, even if it overlaps with another FC code or is distant.<br />
<br />
==Human Choices==<br />
<br />
It is generally preferrable to have a very high degree of confidence in the fuzzy matches, so that they can be treated as correct without individual inspection. However the script and modules are capable of matching to any degree of accuracy. To get further matches that can be inspected/validated/chosen by a human agent, a very weak criteria is set for two runs of fuzzy matching, and then in each run the best (in terms of parameter scores) options are recorded and written into a 'human choice' file.<br />
<br />
As a result a human choice file may contain:<br />
#No matches for a source string as none of the reference strings managed to reach even the very weak threshold criteria.<br />
#One match, as both runs of fuzzy matching produced the same recommendation.<br />
#Two matches, as both runs of fuzzy matching produced one best candidate and the candidates were unique.<br />
#More than two matches, as one or both of the fuzzy matching runs had multiple unique candidates with the same scores.<br />
<br />
It appears likely that blocks of matches will be able to be identified from the human choice files, by restricting the results sets to ranges for one or more of the provided match accuracy parameters.<br />
<br />
==Output Files==<br />
<br />
By default all files are outputted to the Results directory. Which files are outputted depends on the options selected, though the main results file is always outputted (with or without unmatched addresses) and includes fuzzy matches (unless the <tt>-e</tt> option is used to force just exact matching). The main results file outputs:<br />
*COUNTRY - From the source entry<br />
*STR - From the source entry<br />
*CTY - From the source entry<br />
*EXP_CITY - From the source entry<br />
*EXP_ADM - From the source entry<br />
*EXP_POSTCODE - From the source entry<br />
*CTY_STR - A compound entry, delimited by #, used as an internal key. It is the software's best estimate of an address structure.<br />
*EXP_STR - A compound entry, delimited by #, made from the exception data in a similar way to CTY_STR<br />
*PRS_POSTCODE - The software's best estimate of the postcode if any<br />
*MATCH_TYPE - The match type that was used to make the match<br />
*PLACE - The name of the most precise location<br />
*UNI - The GNS unique identifier of the most precise location<br />
*LAT - The latitude of the most precise location<br />
*LONG - The longitude of the most precise location<br />
*FC - The FC code of the most precise location<br />
<br />
The most precise location is taken to be the finest grained result. That is the match corresponding to the lowest level FC code. In the case of the default of FC=A,P,L preference is given to L then P then A. The following variables are then repeated for each FC code searched, and prefixed by the FC code (if no match was found for this FC code the entries will be blank):<br />
*NAME<br />
*UNI<br />
*LAT<br />
*LONG<br />
<br />
The fuzzy match file(s), if requested with <tt>-wf</tt>, have the same format (they are written by the same method). The report file is a copy of the output to the terminal, and can be enabled with the <tt>-r</tt> option. The human choice file (enabled with <tt>-human</tt>) has its own format as follows:<br />
*SOURCENAME - The word, token or string from the source entry that is being considered as relevant for a match<br />
*REFNAME - The name of a place in the GNS file<br />
*COUNTRY - From the source entry<br />
*STR - From the source entry<br />
*CTY - From the source entry<br />
*EXP_CITY - From the source entry<br />
*EXP_ADM - From the source entry<br />
*EXP_POSTCODE - From the source entry<br />
*REFTOTAL - The total number of grams in REFNAME<br />
*SOURCETOTAL - The total number of grams in SOURCENAME<br />
*REFPC - the percentage of the REFNAME grams that appear in the SOURCENAME gram set<br />
*SOURCEPC - the percentage of the SOURCENAME grams that appear in the REFNAME gram set<br />
*LEFTGRAMS - the number of the REFNAME grams that appear in the SOURCENAME gram set<br />
*RIGHTGRAMS - the number of the SOURCENAME grams that appear in the REFNAME gram set<br />
*LCSSCORE - The size of the longest common subsequence in characters<br />
*SOURCELENGTH - The length of SOURCENAME<br />
*REFLENGTH - The length of REFNAME<br />
*MAXLENGTH - The maximum of the lengths of SOURCENAME and REFNAME<br />
*LCSPC - The LCSSCORE divided by the MAXLENGTH<br />
*FIRSTLETTERBINDS - Whether the fuzzy matching algorithm required the same first letter in SOURCENAME and REFNAME<br />
*GRAMALPHABET - The gram alphabet used by the matching algorithm<br />
*GRAMLENGTH - The length of the n-grams used<br />
<br />
<br />
[[category:McNair Admin]]</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=PhD_Masterclass_-_How_to_Build_a_Web_Crawler&diff=6906PhD Masterclass - How to Build a Web Crawler2016-07-18T17:00:35Z<p>ShoebMohammed: Created page with "This page provides resources for the PhD Masterclass "How to Build a Web Crawler", which I gave on Friday 28th January 2011 to interested PhD students at Haas. ==Tools== *[h..."</p>
<hr />
<div>This page provides resources for the PhD Masterclass "How to Build a Web Crawler", which I gave on Friday 28th January 2011 to interested PhD students at Haas.<br />
<br />
==Tools==<br />
<br />
*[http://www.perl.org/ Perl] - Available with a large set of useful modules for Windows from ActiveState as [http://www.activestate.com/activeperl ActivePerl]<br />
*[http://www.activestate.com/komodo-ide Komodo] - An integrated development environment for Perl available from ActiveState<br />
*[http://www.textpad.com/ Textpad] - A powerful shareware text editor that supports [http://en.wikipedia.org/wiki/Regular_expression regular expressions]<br />
<br />
You should [http://www.activestate.com/komodo-ide/downloads download a trial of Komodo] to help you learn. The trial is valid for 21 days (longer if you keep changing your system clock). Komodo will let you step through your code, line by line, and see the values that your variables take on.<br />
<br />
Perl is a free and open language, with a rich history, so you will find a wealth of information on the web to help you learn and use it.<br />
<br />
==Sample Perl Code==<br />
<br />
We wrote a couple of simple scripts together to get to grips with Perl.<br />
<br />
===Running a Perl Script===<br />
<br />
The first was (save it in a file called Script1.pl in the root of your R drive):<br />
<br />
print "Hello World";<br />
<br />
To execute the script we can either open a command prompt and run the script:<br />
Start->Run->"cmd.exe"<br />
R:<br />
perl Script1.pl<br />
<br />
Or we can run it in Komodo by going:<br />
Debug->Go<br />
<br />
(Under Preferences->Debugger tick the box to avoid being prompted by the debug dialog each time)<br />
<br />
Or we can shell on to Bear and run it there:<br />
Use PuTTY to connect to bear.haas.berkeley.edu (see [[Research Computing At Haas|here]]).<br />
perl Script1.pl<br />
<br />
===Processing Text Data===<br />
<br />
Next we went to:<br />
<br />
http://www.contractormisconduct.org/index.cfm/1,73,222,html?CaseID=2<br />
<br />
And we created a file called Data.txt (saved next to the script) that contained the following:<br />
<br />
Accenture<br />
Potential Foreign Corrupt Practices Act Violation<br />
Date: 07/01/2003 (Date of Incident Report)<br />
<br />
Misconduct Type: Ethics<br />
<br />
Enforcement Agency: SEC<br />
<br />
Contracting Party: None<br />
<br />
Court Type: Administrative<br />
<br />
Amount: $0<br />
<br />
Disposition: Pending<br />
<br />
Synopsis: "As previously reported in July 2003, we became aware of an incident..."<br />
<br />
Document(s):<br />
•1. SEC 10-K (p. 34 of 137)<br />
<br />
We then wrote the following script to process the data:<br />
<br />
#!/usr/bin/perl -w<br />
#Lines that start with a # are comments that aren't read by the interpreter<br />
<br />
use strict;<br />
#The strict module forces us to declare variables before we use them<br />
<br />
my @Textfile;<br />
#Declare an array called TextFile<br />
<br />
open (DATA,"Data.txt");<br />
#Open a filehandle on our file<br />
<br />
while (<DATA>) {<br />
#Read the data from the filehandle, line by line<br />
<br />
chomp $_;<br />
#$_ is a special variable - it captures the line being read from the filehandle here<br />
<br />
if (!$_) {next;}<br />
#if the line is undefined (i.e. blank) move to the next loop iteration<br />
<br />
my $line = $_; <br />
#Set a local variable called line to $_<br />
<br />
push (@Textfile, $line);<br />
#Push the line onto the Textfile array<br />
}<br />
<br />
my $Doccell;<br />
#Declare the Doccell variable<br />
<br />
for (my $i=0; $i<=$#Textfile; $i++) {<br />
#Do a for loop, starting from i=0, going while i is less than the <br />
#last index of the Textfile array, and incrementing by one each time<br />
<br />
if ($Textfile[$i]=~/^Document\(s\):/) {$Doccell=$i;}<br />
#Test to see if the entry matches a regular expression, if it does record the index<br />
}<br />
<br />
my @docs = splice(@Textfile,$Doccell);<br />
#Create a next array by splicing out everything after the index we just found<br />
<br />
shift @docs;<br />
#Remove the first element of the docs array<br />
<br />
my $Firm = shift @Textfile;<br />
#Set Firm equal to the first element of Textfile (which we just removed)<br />
<br />
my $Violation =shift(@Textfile);<br />
#Set Violation equal to the (new) first element of Textfile (which we just removed)<br />
<br />
my $Offense={};<br />
#Create an anonymous hash<br />
<br />
foreach my $cell (@Textfile) {\<br />
#Iterative over Textfile, setting the current iteration to cell<br />
<br />
my ($name,@value)=split(":",$cell);<br />
#Spill the cell on :<br />
<br />
my $value=join(":",@value);<br />
#Join the Value array on :<br />
<br />
$Offense->{$name}=$value;<br />
#Set an entry in the Offense hash<br />
}<br />
<br />
$Offense->{"DocList"}=\@docs;<br />
#Set the doclist entry in the Offense hash to a reference to the docs array<br />
<br />
my $Master=[];<br />
#Define an anonymous array<br />
<br />
$Master->[0]={};<br />
#Define an anonymous hash in the zeroth cell of the anonymous array<br />
<br />
$Master->[0]->{FirmName}=$Firm;<br />
#Set a hash entry<br />
<br />
$Master->[0]->{Offense}=$Offense;<br />
#Set a hash entry<br />
<br />
$Master->[0]->{Violation}=$Violation;<br />
#Set a hash entry<br />
<br />
open(OUTPUT,">Result.txt");<br />
#Open a filehandle for writing (overwrite the file if it exists)<br />
<br />
print OUTPUT $Master->[0]->{FirmName};<br />
#Print the output file an entry from the anonymous hash in the anonymous array<br />
<br />
print OUTPUT "\t";<br />
#Print a tab<br />
<br />
print OUTPUT $Master->[0]->{Violation}."\t";<br />
#Print another entry with another tab on the end<br />
<br />
foreach my $key ( sort {$a cmp $b } (keys %{ $Master->[0]->{Offense} } )) {<br />
#Iterate through the hash's keys, in alphabetical order, setting the current key to $key<br />
<br />
print OUTPUT $Master->[0]->{Offense}->{$key}."\t";<br />
#Print an entry, with a tab<br />
}<br />
<br />
print OUTPUT "\n";<br />
#Print a new line<br />
<br />
close OUTPUT;<br />
#Close the output filehandle - this will flush the write buffer<br />
<br />
==Modules==<br />
<br />
One of the joys of Perl is [http://www.cpan.org/ CPAN - The Comprehensive Perl Archive Network] which acts as repository for perl modules (as well as scripts, distros and much else). There are modules written by people from all over the world for almost every conceivable purpose. There is usually no need to reinvent the wheel in Perl - just grab a module (e.g. Wheel::Base)!<br />
<br />
We tested some code using LWP::UserAgent and HTML::TreeBuilder. Useful documentation is here:<br />
<br />
*[http://search.cpan.org/~gaas/libwww-perl-5.837/lib/LWP/UserAgent.pm LWP::UserAgent]<br />
*[http://search.cpan.org/~petdance/WWW-Mechanize-1.66/lib/WWW/Mechanize.pm WWW::Mechanize]<br />
*[http://search.cpan.org/~gaas/libwww-perl-5.837/lib/HTTP/Response.pm HTTP::Response]<br />
*[http://search.cpan.org/~jfearn/HTML-Tree-4.1/lib/HTML/TreeBuilder.pm HTML::TreeBuilder]<br />
*[http://search.cpan.org/~jfearn/HTML-Tree-4.1/lib/HTML/Element.pm HTML::Element]<br />
*[http://annocpan.org/~GAAS/libwww-perl-5.837/lib/LWP/RobotUA.pm LWP::RobotUA]<br />
*[http://annocpan.org/~GRANTM/XML-Simple-2.18/lib/XML/Simple.pm XML::Simple]<br />
<br />
Below is a simple UserAgent example:<br />
<br />
use LWP::UserAgent;<br />
#Use the LWP::UserAgent modules<br />
<br />
my $ua = LWP::UserAgent->new;<br />
#Create a new UserAgent<br />
<br />
my $url="http://www.contractormisconduct.org/index.cfm/1,73,222,html?CaseID=2";<br />
#Set up a string containing a URL<br />
<br />
my $response = $ua->get($url);<br />
#Use the UA 'get' method to retrieve the webpage. This returns an HTTP Response object<br />
<br />
my $content=$response->decoded_content;<br />
#Get the response as one long text string, so we can work with it...<br />
<br />
And now for a TreeBuilder example:<br />
<br />
use HTML::TreeBuilder;<br />
#Use the HTML::TreeBuilder modules<br />
<br />
my $tree = HTML::TreeBuilder->new; # empty tree<br />
#Create a new tree object<br />
<br />
$tree->parse($content);<br />
#Load up the tree from the content string (that we got using UA)<br />
<br />
my $dump=$tree->as_text;<br />
#Dump the tree as text maybe<br />
<br />
my $incidentelement=$tree->look_down("id","primecontent");<br />
#Or use HTML::Element methods to look_down the tree for a tag with some properties<br />
<br />
==An Example Webcrawler==<br />
<br />
I wrote the following simple webcrawler for a fellow PhD student:<br />
<br />
#!/usr/bin/perl -w<br />
use strict;<br />
<br />
use LWP::UserAgent;<br />
#Use the LWP::UserAgent modules<br />
use HTML::TreeBuilder;<br />
#Use the HTML::TreeBuilder modules<br />
<br />
my $ua = LWP::UserAgent->new;<br />
#Create a new UserAgent<br />
<br />
my @Pkids;<br />
open (PKIDS,"Pkidfile.txt") || die "Can't open the PKID file to read $!";<br />
#Open the Pkid file to read - this file has a Pkid on each line. You can get some from here: http://myaccount.sdar.com/RealtorSrch.asp<br />
<br />
while (<PKIDS>) {<br />
#Read the pkid file line by line<br />
<br />
chomp $_;<br />
#Remove the \n (newline symbol) from each line<br />
<br />
push(@Pkids,$_);<br />
#Add the PKID to an array<br />
}<br />
<br />
open (RESULTS,">Results.txt") || die "Can't write the Results.txt file $!";<br />
#Open the Results file to write<br />
<br />
my $headerflag=0;<br />
#Set a flag to indicate whether we wrote the header line to the output file<br />
<br />
foreach my $Pkid (sort (@Pkids)) {<br />
#Go through the PKIDs in order<br />
<br />
my $url="http://myaccount.sdar.com/RealtorSrchDetail.asp?PKID=".$Pkid;<br />
#Set up a string containing a URL<br />
<br />
my $response = $ua->get($url);<br />
#Use the UA 'get' method to retrieve the webpage. This returns an HTTP Response object<br />
<br />
my $content=$response->decoded_content;<br />
#Get the response as one long text string, so we can work with it...<br />
<br />
my $tree = HTML::TreeBuilder->new; # empty tree<br />
#Create a new tree object<br />
<br />
$tree->parse($content);<br />
#Load up the tree from the content string (that we got using UA)<br />
<br />
my $name=$tree->look_down("width","520");<br />
#Find an element in the HTML that has width=520 (this is where names are stored)<br />
<br />
my $nametext=$name->as_text;<br />
#Convert it to text<br />
<br />
$nametext=~s/^\s{1,}//;<br />
#Remove leading spaces<br />
<br />
$nametext=~s/\s{1,}$//;<br />
#Remove trailing spaces<br />
<br />
$nametext=~s/^\s{2,}/ /g;<br />
#Replace double spaces with a single space, globally<br />
<br />
my @fieldstext;<br />
#Declare an array<br />
<br />
my @fields=$tree->look_down("class","field_labels");<br />
#Find all of the field elements<br />
<br />
foreach my $field (@fields) {<br />
#Go through them<br />
<br />
my $fieldparent=$field->parent;<br />
#Go to their parent<br />
<br />
my $fieldparenttext=$fieldparent->as_text;<br />
#Turn the parent into text<br />
<br />
$fieldparenttext=~s/^\s{1,}//; $fieldparenttext=~s/\s{1,}$//; $fieldparenttext=~s/^\s{2,}/ /g;<br />
#Deal with spaces again<br />
<br />
push @fieldstext,$fieldparenttext;<br />
#Add the fields to a list<br />
}<br />
<br />
&writeoutput($Pkid,$nametext,@fieldstext);<br />
#Call the write output subroutine<br />
<br />
$content=undef; $tree=undef; $name=undef; undef @fields;<br />
#Set a bunch of variables to undefined - this frees up memory<br />
<br />
sleep(2);<br />
#Pause for a second or two...<br />
}<br />
<br />
close (RESULTS);<br />
#Close the Results filehandle - this flushes the write buffer<br />
<br />
sub writeoutput {<br />
#Declare the writeoutput subroutine<br />
<br />
my $data={};<br />
#Set up an anonymous hash<br />
<br />
$data->{"A Pkid"}=shift @_;<br />
#Set the A PKID field to the first parameter passed to the subroutine<br />
<br />
$data->{"A Name"}=shift @_;<br />
#Set the A PKID field to the second parameter passed to the subroutine (the first has now gone)<br />
<br />
push(my @fields,@_);<br />
#Add the remaining parameters to an array<br />
<br />
foreach my $field (@fields) {<br />
#Go through the array<br />
<br />
my @fieldparts=split(":",$field);<br />
#Split the fields on semicolon<br />
<br />
my $key=shift(@fieldparts);<br />
#Set the key<br />
<br />
$data->{$key}=join(":",@fieldparts);<br />
#Write the hash entry<br />
}<br />
if (!$headerflag) {<br />
#If the headflag is 0 then do this<br />
<br />
foreach my $key (sort {$a cmp $b} (keys %{$data})) {<br />
#Go through the keys<br />
<br />
print RESULTS $key."\t";<br />
#Write the key followed by a tab<br />
}<br />
print RESULTS "\n";<br />
#Print a newline<br />
<br />
$headerflag=1;<br />
#Set the headflag to 1<br />
}<br />
<br />
foreach my $key (sort {$a cmp $b} (keys %{$data})) {<br />
#Go through the keys again<br />
<br />
print RESULTS $data->{$key}."\t";<br />
#This time print the data followed by tabs<br />
}<br />
print RESULTS "\n";<br />
#print a newline<br />
}<br />
<br />
print "Thanks to Ed";<br />
#Thank Ed.<br />
<br />
[[category:McNair Admin]]<br />
[[admin_classification::Software Tutorial| ]]</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Software_Repository&diff=6848Software Repository2016-07-18T16:06:20Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category: McNair Admin]]<br />
<br />
For a listing of all software tools and scripts see [[Software Repository Listing]].<br />
<br />
==Background==<br />
Given the amount of software that has been written by past computer science interns and more being written, we felt the need to have some kind of source code management system put into place so that developers can work without ever being in fear of breaking production and facing Ed's wrath (you do not want that dude angry! Wherever you go, he will find you! No escape.).<br />
<br />
To enforce efficient source control we(Ed) chose to host our own git server on the RDP machine using [https://bonobogitserver.com/ Bonobo Git Server] that makes use of the windows IIS platform and is open source.<br />
<br />
Installing Bonobo git server is pretty simple:<br />
* dowload the zip file from the Bonobo website.<br />
* extract its contents. It should be a single folder containing directories like App_Data, bin etc.<br />
* rename that folder to anything you want. I used the name "codebase"<br />
* copy the codebase folder to C:\inetpub\wwwroot\<br />
* Allow IIS User to modify C:\inetpub\wwwroot\codebase\App_Data folder. To do so:<br />
**select Properties of App_Data folder,<br />
**go to Security tab,<br />
**click edit,<br />
**select IIS user (in my case IIS_IUSRS) and add Modify and Write permission,<br />
**confirm these settings with Apply button.<br />
*Convert ''codebase'' to Application in IIS<br />
**Run IIS Manager and navigate to Sites -> Default Web Site. You should see Bonobo.Git.Server.<br />
**Right click on 'codebase' and convert to application.<br />
**Check if the selected application pool runs on .NET 4.0 and convert the site.<br />
*Enable Anonymous Authentication in IIS and disable the others. To do so, select the application in the left pane, double-click on the authentication icon in the right pane and set the value to of Anonymous Authentication to Enabled<br />
*Launch your browser and go to http://localhost/codebase. Now you can see the initial page of the Bonobo Git Server and everything should work.<br />
**default credentials are ''username'': '''admin''', ''password'': '''admin'''<br />
**[6-22-2016]: Can also use https://localhost/codebase which is preferable, otherwise username/passwords are transmitted plain text. The browser will show a security error because we have a self signed certificate. This is ok if we are restricted to intranet. If we want to allow public access, we probably need to get a certificate from a Certificate Authority like Verisign etc.<br />
<br />
==Our Git Server==<br />
We have already done the set up of the git server on the RDP machine. Here are the admin credentials:<br />
*Username: '''boss'''<br />
*Name: '''Ed'''<br />
*Surname: '''Egan'''<br />
*Email: '''Edward.Egan@rice.edu'''<br />
*Password: '''you_seriously_thought_Id_write_that_in_here??'''<br />
<br />
To access this from your computer and not the RDP you can go to http://128.42.44.182/codebase where it will prompt you for your username and password.<br />
**[6-22-2016]: Can also use https://128.42.44.182/codebase which is preferable. The browser will show a security error because we have a self signed certificate. This is ok if we are restricted to intranet. If we want to allow public access, we probably need to get a certificate from a Certificate Authority like Verisign etc.<br />
<br />
<br />
==Our Git workflow==<br />
We chose a simple git workflow.<br />
<br />
Our aim is not to break things in the master branch. All commits on the master should work.<br />
<br />
1.<br />
When adding a new feature or fixing a bug, ALWAYS check out a new feature branch from the master.<br />
NEVER checkout a feature branch from next (see below). The feature branch should be named user/feature_name. <br />
<br />
2.<br />
After feature development is complete merge your feature-branch into next.<br />
<br />
3.<br />
The next branch is intended for testing and confirming things do not break. So, after feature branches are merged into next and conflicts resolved, we merge into master.<br />
After this, you can end the feature branches if you want.<br />
<br />
==Quick and dirty github tutorial==<br />
For a cool interactive tutorial see http://learngitbranching.js.org/.<br />
<br />
***<br />
You can also use SourceTree which is a GUI interface for git-client. This is installed on the RDP.<br />
Like using git from CLI (see below), SourceTree constructs appropriate commands. But the good thing<br />
is it automatically generates all error check/logging options with each command that are difficult<br />
to recall from memory. SourceTree is freely available from Altassian at https://www.sourcetreeapp.com/<br />
***<br />
<br />
To use SourceTree you should have basic understanding of git (like branches,commits etc). The interactive tutorial above is very good for this purpose.<br />
<br />
<br />
<br />
*''Installing - '' Depending on your operating system you can install git in three different ways:<br />
** If you are a windows or a mac user user, you can simply download & install the latest release from [https://git-scm.com/ git scm website]<br />
** If you use ubuntu then all you need to do is type <code><big>sudo apt-get install git</big></code><br />
* Check your installation by typing <code><big>git</big></code> in terminal or windows powershell<br />
*Basic git operations:<br />
<div style="text-align: left; direction: ltr; margin-left: 2em;"><br />
* to checkout code from remote repository, use the <code><big>git clone</big></code> command. This will create a local repository on your disk as well as download the source code of the project you wish to work on. Here's an example:<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git clone http://128.42.44.182/codebase/Matcher.git</big></code></div><br />
<br />
* to update your repository to include others' work in your project use the <code><big>git update</big></code> command. Its always a good practice to update your code before you commit to ensure that others' code doesn't break yours. Also, you cannot push to remote unless your local repository is up to date. If you commit on a stale local repository that is fine, just that this would mean you are likely to have more trouble merging your code with others later on thanks to all the conflicts that you'll face when you actually try to update your repository later. See example:<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git update <optional folder path></big></code></div><br />
<br />
* to commit your changes to your local repository use the <code><big>git commit</big></code> command. Committing your changes is an essential step whether you are adding/removing items from the repository or changing existing items. See example :<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git commit -m "mandatory commit message"</big></code></div><br />
<br />
* to push your changes to remote repository use the <code><big>git push <optional file/folder name></big></code> command. Whatever you need to be pushed to the server must be committed to your local repository first. By default this command will push everything from current folder if no item is specified.<br />
<br />
* to add new files to your repository use the <code><big>git add</big></code> command. After that you must commit to ensure that your repository actually has the new file. See example:<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git add <filename/folder name></big></code></div><br />
<br />
* to remove items from your repository use the <code><big>git remove</big></code> command. After that you delete the file that you wanted removed from the repository and commit to ensure that your repository actually has the change persisted. Finally, you push to server to make sure the server has those items removed as well and that nobody in your team works under the assumption that those items are stills there. See example:<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git remove <filename></big></code></div><br />
<div style="text-align: left; direction: ltr; margin-left: 2em;"><br />
''Note'': if removing a non empty folder use the -r flag to recursively remove all contents of that folder as well :<br />
</div><br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git remove -r <foldername></big></code></div><br />
<br />
</div><br />
<br />
[[admin_classification::IT Build| ]]</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Software_Repository&diff=6840Software Repository2016-07-18T15:55:41Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category: McNair Admin]]<br />
<br />
For a listing of all software tools and scripts see [[Resources and Tools]].<br />
<br />
==Background==<br />
Given the amount of software that has been written by past computer science interns and more being written, we felt the need to have some kind of source code management system put into place so that developers can work without ever being in fear of breaking production and facing Ed's wrath (you do not want that dude angry! Wherever you go, he will find you! No escape.).<br />
<br />
To enforce efficient source control we(Ed) chose to host our own git server on the RDP machine using [https://bonobogitserver.com/ Bonobo Git Server] that makes use of the windows IIS platform and is open source.<br />
<br />
Installing Bonobo git server is pretty simple:<br />
* dowload the zip file from the Bonobo website.<br />
* extract its contents. It should be a single folder containing directories like App_Data, bin etc.<br />
* rename that folder to anything you want. I used the name "codebase"<br />
* copy the codebase folder to C:\inetpub\wwwroot\<br />
* Allow IIS User to modify C:\inetpub\wwwroot\codebase\App_Data folder. To do so:<br />
**select Properties of App_Data folder,<br />
**go to Security tab,<br />
**click edit,<br />
**select IIS user (in my case IIS_IUSRS) and add Modify and Write permission,<br />
**confirm these settings with Apply button.<br />
*Convert ''codebase'' to Application in IIS<br />
**Run IIS Manager and navigate to Sites -> Default Web Site. You should see Bonobo.Git.Server.<br />
**Right click on 'codebase' and convert to application.<br />
**Check if the selected application pool runs on .NET 4.0 and convert the site.<br />
*Enable Anonymous Authentication in IIS and disable the others. To do so, select the application in the left pane, double-click on the authentication icon in the right pane and set the value to of Anonymous Authentication to Enabled<br />
*Launch your browser and go to http://localhost/codebase. Now you can see the initial page of the Bonobo Git Server and everything should work.<br />
**default credentials are ''username'': '''admin''', ''password'': '''admin'''<br />
**[6-22-2016]: Can also use https://localhost/codebase which is preferable, otherwise username/passwords are transmitted plain text. The browser will show a security error because we have a self signed certificate. This is ok if we are restricted to intranet. If we want to allow public access, we probably need to get a certificate from a Certificate Authority like Verisign etc.<br />
<br />
==Our Git Server==<br />
We have already done the set up of the git server on the RDP machine. Here are the admin credentials:<br />
*Username: '''boss'''<br />
*Name: '''Ed'''<br />
*Surname: '''Egan'''<br />
*Email: '''Edward.Egan@rice.edu'''<br />
*Password: '''you_seriously_thought_Id_write_that_in_here??'''<br />
<br />
To access this from your computer and not the RDP you can go to http://128.42.44.182/codebase where it will prompt you for your username and password.<br />
**[6-22-2016]: Can also use https://128.42.44.182/codebase which is preferable. The browser will show a security error because we have a self signed certificate. This is ok if we are restricted to intranet. If we want to allow public access, we probably need to get a certificate from a Certificate Authority like Verisign etc.<br />
<br />
<br />
==Our Git workflow==<br />
We chose a simple git workflow.<br />
<br />
Our aim is not to break things in the master branch. All commits on the master should work.<br />
<br />
1.<br />
When adding a new feature or fixing a bug, ALWAYS check out a new feature branch from the master.<br />
NEVER checkout a feature branch from next (see below). The feature branch should be named user/feature_name. <br />
<br />
2.<br />
After feature development is complete merge your feature-branch into next.<br />
<br />
3.<br />
The next branch is intended for testing and confirming things do not break. So, after feature branches are merged into next and conflicts resolved, we merge into master.<br />
After this, you can end the feature branches if you want.<br />
<br />
==Quick and dirty github tutorial==<br />
For a cool interactive tutorial see http://learngitbranching.js.org/.<br />
<br />
***<br />
You can also use SourceTree which is a GUI interface for git-client. This is installed on the RDP.<br />
Like using git from CLI (see below), SourceTree constructs appropriate commands. But the good thing<br />
is it automatically generates all error check/logging options with each command that are difficult<br />
to recall from memory. SourceTree is freely available from Altassian at https://www.sourcetreeapp.com/<br />
***<br />
<br />
To use SourceTree you should have basic understanding of git (like branches,commits etc). The interactive tutorial above is very good for this purpose.<br />
<br />
<br />
<br />
*''Installing - '' Depending on your operating system you can install git in three different ways:<br />
** If you are a windows or a mac user user, you can simply download & install the latest release from [https://git-scm.com/ git scm website]<br />
** If you use ubuntu then all you need to do is type <code><big>sudo apt-get install git</big></code><br />
* Check your installation by typing <code><big>git</big></code> in terminal or windows powershell<br />
*Basic git operations:<br />
<div style="text-align: left; direction: ltr; margin-left: 2em;"><br />
* to checkout code from remote repository, use the <code><big>git clone</big></code> command. This will create a local repository on your disk as well as download the source code of the project you wish to work on. Here's an example:<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git clone http://128.42.44.182/codebase/Matcher.git</big></code></div><br />
<br />
* to update your repository to include others' work in your project use the <code><big>git update</big></code> command. Its always a good practice to update your code before you commit to ensure that others' code doesn't break yours. Also, you cannot push to remote unless your local repository is up to date. If you commit on a stale local repository that is fine, just that this would mean you are likely to have more trouble merging your code with others later on thanks to all the conflicts that you'll face when you actually try to update your repository later. See example:<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git update <optional folder path></big></code></div><br />
<br />
* to commit your changes to your local repository use the <code><big>git commit</big></code> command. Committing your changes is an essential step whether you are adding/removing items from the repository or changing existing items. See example :<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git commit -m "mandatory commit message"</big></code></div><br />
<br />
* to push your changes to remote repository use the <code><big>git push <optional file/folder name></big></code> command. Whatever you need to be pushed to the server must be committed to your local repository first. By default this command will push everything from current folder if no item is specified.<br />
<br />
* to add new files to your repository use the <code><big>git add</big></code> command. After that you must commit to ensure that your repository actually has the new file. See example:<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git add <filename/folder name></big></code></div><br />
<br />
* to remove items from your repository use the <code><big>git remove</big></code> command. After that you delete the file that you wanted removed from the repository and commit to ensure that your repository actually has the change persisted. Finally, you push to server to make sure the server has those items removed as well and that nobody in your team works under the assumption that those items are stills there. See example:<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git remove <filename></big></code></div><br />
<div style="text-align: left; direction: ltr; margin-left: 2em;"><br />
''Note'': if removing a non empty folder use the -r flag to recursively remove all contents of that folder as well :<br />
</div><br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git remove -r <foldername></big></code></div><br />
<br />
</div><br />
<br />
[[admin_classification::IT Build| ]]</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Software_Repository&diff=6838Software Repository2016-07-18T15:53:51Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category: McNair Admin]]<br />
<br />
For a listing of all software tools and scripts see [[Software Tools Listing]].<br />
<br />
==Background==<br />
Given the amount of software that has been written by past computer science interns and more being written, we felt the need to have some kind of source code management system put into place so that developers can work without ever being in fear of breaking production and facing Ed's wrath (you do not want that dude angry! Wherever you go, he will find you! No escape.).<br />
<br />
To enforce efficient source control we(Ed) chose to host our own git server on the RDP machine using [https://bonobogitserver.com/ Bonobo Git Server] that makes use of the windows IIS platform and is open source.<br />
<br />
Installing Bonobo git server is pretty simple:<br />
* dowload the zip file from the Bonobo website.<br />
* extract its contents. It should be a single folder containing directories like App_Data, bin etc.<br />
* rename that folder to anything you want. I used the name "codebase"<br />
* copy the codebase folder to C:\inetpub\wwwroot\<br />
* Allow IIS User to modify C:\inetpub\wwwroot\codebase\App_Data folder. To do so:<br />
**select Properties of App_Data folder,<br />
**go to Security tab,<br />
**click edit,<br />
**select IIS user (in my case IIS_IUSRS) and add Modify and Write permission,<br />
**confirm these settings with Apply button.<br />
*Convert ''codebase'' to Application in IIS<br />
**Run IIS Manager and navigate to Sites -> Default Web Site. You should see Bonobo.Git.Server.<br />
**Right click on 'codebase' and convert to application.<br />
**Check if the selected application pool runs on .NET 4.0 and convert the site.<br />
*Enable Anonymous Authentication in IIS and disable the others. To do so, select the application in the left pane, double-click on the authentication icon in the right pane and set the value to of Anonymous Authentication to Enabled<br />
*Launch your browser and go to http://localhost/codebase. Now you can see the initial page of the Bonobo Git Server and everything should work.<br />
**default credentials are ''username'': '''admin''', ''password'': '''admin'''<br />
**[6-22-2016]: Can also use https://localhost/codebase which is preferable, otherwise username/passwords are transmitted plain text. The browser will show a security error because we have a self signed certificate. This is ok if we are restricted to intranet. If we want to allow public access, we probably need to get a certificate from a Certificate Authority like Verisign etc.<br />
<br />
==Our Git Server==<br />
We have already done the set up of the git server on the RDP machine. Here are the admin credentials:<br />
*Username: '''boss'''<br />
*Name: '''Ed'''<br />
*Surname: '''Egan'''<br />
*Email: '''Edward.Egan@rice.edu'''<br />
*Password: '''you_seriously_thought_Id_write_that_in_here??'''<br />
<br />
To access this from your computer and not the RDP you can go to http://128.42.44.182/codebase where it will prompt you for your username and password.<br />
**[6-22-2016]: Can also use https://128.42.44.182/codebase which is preferable. The browser will show a security error because we have a self signed certificate. This is ok if we are restricted to intranet. If we want to allow public access, we probably need to get a certificate from a Certificate Authority like Verisign etc.<br />
<br />
<br />
==Our Git workflow==<br />
We chose a simple git workflow.<br />
<br />
Our aim is not to break things in the master branch. All commits on the master should work.<br />
<br />
1.<br />
When adding a new feature or fixing a bug, ALWAYS check out a new feature branch from the master.<br />
NEVER checkout a feature branch from next (see below). The feature branch should be named user/feature_name. <br />
<br />
2.<br />
After feature development is complete merge your feature-branch into next.<br />
<br />
3.<br />
The next branch is intended for testing and confirming things do not break. So, after feature branches are merged into next and conflicts resolved, we merge into master.<br />
After this, you can end the feature branches if you want.<br />
<br />
==Quick and dirty github tutorial==<br />
For a cool interactive tutorial see http://learngitbranching.js.org/.<br />
<br />
***<br />
You can also use SourceTree which is a GUI interface for git-client. This is installed on the RDP.<br />
Like using git from CLI (see below), SourceTree constructs appropriate commands. But the good thing<br />
is it automatically generates all error check/logging options with each command that are difficult<br />
to recall from memory. SourceTree is freely available from Altassian at https://www.sourcetreeapp.com/<br />
***<br />
<br />
To use SourceTree you should have basic understanding of git (like branches,commits etc). The interactive tutorial above is very good for this purpose.<br />
<br />
<br />
<br />
*''Installing - '' Depending on your operating system you can install git in three different ways:<br />
** If you are a windows or a mac user user, you can simply download & install the latest release from [https://git-scm.com/ git scm website]<br />
** If you use ubuntu then all you need to do is type <code><big>sudo apt-get install git</big></code><br />
* Check your installation by typing <code><big>git</big></code> in terminal or windows powershell<br />
*Basic git operations:<br />
<div style="text-align: left; direction: ltr; margin-left: 2em;"><br />
* to checkout code from remote repository, use the <code><big>git clone</big></code> command. This will create a local repository on your disk as well as download the source code of the project you wish to work on. Here's an example:<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git clone http://128.42.44.182/codebase/Matcher.git</big></code></div><br />
<br />
* to update your repository to include others' work in your project use the <code><big>git update</big></code> command. Its always a good practice to update your code before you commit to ensure that others' code doesn't break yours. Also, you cannot push to remote unless your local repository is up to date. If you commit on a stale local repository that is fine, just that this would mean you are likely to have more trouble merging your code with others later on thanks to all the conflicts that you'll face when you actually try to update your repository later. See example:<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git update <optional folder path></big></code></div><br />
<br />
* to commit your changes to your local repository use the <code><big>git commit</big></code> command. Committing your changes is an essential step whether you are adding/removing items from the repository or changing existing items. See example :<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git commit -m "mandatory commit message"</big></code></div><br />
<br />
* to push your changes to remote repository use the <code><big>git push <optional file/folder name></big></code> command. Whatever you need to be pushed to the server must be committed to your local repository first. By default this command will push everything from current folder if no item is specified.<br />
<br />
* to add new files to your repository use the <code><big>git add</big></code> command. After that you must commit to ensure that your repository actually has the new file. See example:<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git add <filename/folder name></big></code></div><br />
<br />
* to remove items from your repository use the <code><big>git remove</big></code> command. After that you delete the file that you wanted removed from the repository and commit to ensure that your repository actually has the change persisted. Finally, you push to server to make sure the server has those items removed as well and that nobody in your team works under the assumption that those items are stills there. See example:<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git remove <filename></big></code></div><br />
<div style="text-align: left; direction: ltr; margin-left: 2em;"><br />
''Note'': if removing a non empty folder use the -r flag to recursively remove all contents of that folder as well :<br />
</div><br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git remove -r <foldername></big></code></div><br />
<br />
</div><br />
<br />
[[admin_classification::IT Build| ]]</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Research_Plans)&diff=6837Shoeb Mohammed (Research Plans)2016-07-18T15:51:45Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Research Plans]]<br />
<br />
====Short Term====<br />
* Create a listing on the wiki for all software developed at McNair center.<br />
* Build a Linux box to run the crawler.<br />
<br />
====Long Term====<br />
* Optimize/re-design the 'Matcher' software. In particular, speed-up fuzzy matching and possibly re-structure the code to make usage easier.<br />
* Develop the crawler. Try to begin with code that Dan has.<br />
<br />
====Side Tasks====<br />
* If possible, redo the patent parser (previous coded by Kranti) to also pull in Patent Citation data.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Research_Plans)&diff=6784Shoeb Mohammed (Research Plans)2016-07-15T20:20:25Z<p>ShoebMohammed: /* Short Term */</p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Research Plans]] [[Shoeb Mohammed (Research Plans)|(Plan Page)]]<br />
<br />
====Short Term====<br />
* Create a listing on the wiki for all software developed at McNair center.<br />
* Build a Linux box to run the crawler.<br />
<br />
====Long Term====<br />
* Optimize/re-design the 'Matcher' software. In particular, speed-up fuzzy matching and possibly re-structure the code to make usage easier.<br />
* Develop the crawler. Try to begin with code that Dan has.<br />
<br />
====Side Tasks====<br />
* If possible, redo the patent parser (previous coded by Kranti) to also pull in Patent Citation data.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Research_Plans)&diff=6783Shoeb Mohammed (Research Plans)2016-07-15T20:19:17Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Research Plans]] [[Shoeb Mohammed (Research Plans)|(Plan Page)]]<br />
<br />
====Short Term====<br />
* Create a listing on the wiki for all software developed at McNair center.<br />
<br />
====Long Term====<br />
* Optimize/re-design the 'Matcher' software. In particular, speed-up fuzzy matching and possibly re-structure the code to make usage easier.<br />
* Develop the crawler. Try to begin with code that Dan has.<br />
<br />
====Side Tasks====<br />
* If possible, redo the patent parser (previous coded by Kranti) to also pull in Patent Citation data.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Research_Plans)&diff=6781Shoeb Mohammed (Research Plans)2016-07-15T20:15:21Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Research Plans]] [[Shoeb Mohammed (Research Plans)|(Plan Page)]]<br />
<br />
*07/15/2016 <br />
** Optimize/re-design the 'Matcher' software. In particular, speed-up fuzzy matching and possibly re-structure the code to make usage easier.<br />
** Develop the crawler. Try to begin with code that Dan has.<br />
** Create a listing on the wiki for all software developed at McNair center.<br />
** If possible, redo the patent parser (previous coded by Kranti) to also pull in Patent Citation data.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Research_Plans)&diff=6780Shoeb Mohammed (Research Plans)2016-07-15T20:14:35Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Research Plans]] [[Shoeb Mohammed (Research Plans)|(Plan Page)]]<br />
<br />
*07/15/2016 <br />
** Optimize/re-design the 'Matcher' software. In particular, speed-up fuzzy matching and possibly re-structure the code to make usage easier.<br />
** Develop the crawler starting with code from Dan.<br />
** Create a listing on the wiki for all software developed at McNair center.<br />
** If possible, redo the patent parser (previous coded by Kranti) to also pull in Patent Citation data.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Research_Plans&diff=6620Research Plans2016-07-15T16:18:10Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category: McNair Admin]]<br />
[[admin_classification::General Information| ]]<br />
==Dylan Dickens==<br />
{{:Dylan Dickens (Research Plan)}}<br />
<br />
==Ben Baldazo==<br />
{{:Ben Baldazo (Research Plans)}}<br />
<br />
==Jake Silberman==<br />
{{:Jake Silberman (Research Plan)}}<br />
<br />
<br />
==Shoeb Mohammed==<br />
{{:Shoeb Mohammed (Research Plans)}}</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Research_Plans)&diff=6618Shoeb Mohammed (Research Plans)2016-07-15T16:17:33Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Research Plans]] [[Shoeb Mohammed (Research Plans)|(Plan Page)]]<br />
<br />
*07/15/2016 <br />
**</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Research_Plans)&diff=6617Shoeb Mohammed (Research Plans)2016-07-15T16:14:24Z<p>ShoebMohammed: Created page with "Category:Work Log Shoeb Mohammed Research Plans (Plan Page) *07/15/2016 **"</p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Research Plans]] [[Shoeb Mohammed (Research Plan)|(Plan Page)]]<br />
<br />
*07/15/2016 <br />
**</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Work_Logs&diff=6615Work Logs2016-07-15T16:11:53Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category: McNair Admin]]<br />
==Dylan Dickens==<br />
{{:Dylan Dickens (Work Log)}}<br />
<br />
==Jake Silberman==<br />
{{:Jake Silberman (Work Log)}}<br />
<br />
==Marcela Interiano==<br />
{{:Marcela Interiano (Work Log)}}<br />
<br />
==Veeral Shah==<br />
{{:Veeral Shah (Work Log)}}<br />
<br />
==Ariel Sun==<br />
{{:Ariel Sun (Work Log)}}<br />
<br />
==Gunny Liu==<br />
{{:Gunny Liu (Work Log)}}<br />
<br />
[[admin_classification::General Information| ]]<br />
<br />
==Ben Baldazo==<br />
<br />
{{:Ben Baldazo (Work Log)}}<br />
<br />
==Shoeb Mohammed==<br />
{{:Shoeb Mohammed (Work Log)}}</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Work_Log)&diff=6594Shoeb Mohammed (Work Log)2016-07-15T16:06:14Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Work Logs]] [[Shoeb Mohammed (Work Log) | (Work Log)]]<br />
<br />
*07/15/2016 <br />
**Started work log and research plan.<br />
**Review and profile the Matcher perl code.<br />
**Read tutorials on using Selenium with perl. Below links are good to get started <br />
https://www.usenix.org/system/files/login/articles/login_apr15_09_blank-edelman.pdf<br />
https://www.usenix.org/system/files/login/articles/blank-edelman_0.pdf</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Work_Log)&diff=6591Shoeb Mohammed (Work Log)2016-07-15T16:05:56Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Work Logs]] [[Shoeb Mohammed (Work Log) | (Work Log)]]<br />
<br />
*07/15/2016 <br />
**Started work log and research plan.<br />
**Review and profile the Matcher perl code.<br />
**Read tutorials on using Selenium with perl. These links are good to get started <br />
https://www.usenix.org/system/files/login/articles/login_apr15_09_blank-edelman.pdf<br />
https://www.usenix.org/system/files/login/articles/blank-edelman_0.pdf</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Work_Log)&diff=6587Shoeb Mohammed (Work Log)2016-07-15T16:05:34Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Work Logs]] [[Shoeb Mohammed (Work Log) | (Work Log)]]<br />
<br />
*07/15/2016 <br />
**Started work log and research plan.<br />
**Review and profile the Matcher perl code.<br />
**Read using Selenium with perl. These links are good to get started <br />
https://www.usenix.org/system/files/login/articles/login_apr15_09_blank-edelman.pdf<br />
https://www.usenix.org/system/files/login/articles/blank-edelman_0.pdf</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Shoeb_Mohammed_(Work_Log)&diff=6495Shoeb Mohammed (Work Log)2016-07-15T15:56:26Z<p>ShoebMohammed: Created page with "Category:Work Log Shoeb Mohammed Work Logs (Work Log) *07/15/2016 **Started work log and research plan."</p>
<hr />
<div>[[Category:Work Log]]<br />
[[Shoeb Mohammed]] [[Work Logs]] [[Shoeb Mohammed (Work Log) | (Work Log)]]<br />
<br />
*07/15/2016 <br />
**Started work log and research plan.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Whois_Parser&diff=4790Whois Parser2016-07-11T19:08:50Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Internal]]<br />
[[Internal Classification::Legacy| ]]<br />
This wiki page is under Additional Links/WhoisParser<br />
<br />
The whoisParser was written by Kunal Shah on March 20, 2016 and is located <br />
repository: Web_Crawler<br />
branch: shoeb_patch/whoisParser<br />
directory: /WhoIsParser<br />
file: whoisParser.pl<br />
<br />
To use this parser, copy above perl program into a directory, make it current working directory (that is, use 'cd' command if needed) and run the following command. The directory should also have the input file(see below).<br />
<br />
perl WhoIsParser.pl -file=listofurls.txt -outfile=listofurls_processed.txt<br />
<br />
= NAME =<br />
<br />
WhoIs Parser - Retrieves and parses Whois information<br />
Specifically, takes a file with a column of domain names and populates the<br />
corresponding columns with information from the WhoIs API.<br />
<br />
= SYNOPSIS =<br />
<br />
perl whoisParser -file=<file> [-outfile=<file>] <br />
<br />
= OPTIONS =<br />
<br />
-file=<file>: Name of file of domain names. <br />
-outfile=<file>: The name of the outfile <br />
-h: Display help<br />
<br />
= USAGE & FEATURES =<br />
<br />
'''Arguments:''' <br />
<br />
A text file with a column of domain names<br />
<br />
'''Returns:''' <br />
<br />
A text file of the domain names with the next 12 columns populated with information pulled from the Whois API. A header specifying each column is inserted into the first row of the file. The columns of information outputed are:<br />
<br />
1. Domain Name<br />
<br />
2. Creation Date<br />
<br />
3. Expiration Date<br />
<br />
4. Update Date<br />
<br />
5. Registrant Name<br />
<br />
6. Registrant Street<br />
<br />
7. Registrant City<br />
<br />
8. Registrant Postal Code<br />
<br />
9. Registrant Country<br />
<br />
10. Admin Street<br />
<br />
11. Admin City<br />
<br />
12. Admin Postal Code<br />
<br />
13. Admin Country<br />
<br />
= BUGS & FEEDBACK =<br />
<br />
Worked as expected on all example files. Please report any discovered bugs to Kunal.<br />
<br />
Tested files:<br />
Input: example_file.txt<br />
<br />
Output: example_outfile.txt<br />
<br />
<br />
Input Text: <br />
<br />
<br />
[[Image:input.jpg|400px|thumb|right|WhoIs input file in Excel]]<br />
<br />
http://1986ventures.com<br />
<br />
http://2nd.md/<br />
<br />
http://www.2ndsquare.com<br />
<br />
http://www.32nddegree.com/<br />
<br />
http://www.80legs.com<br />
<br />
http://hotmailpasswordsupportnumber.info/<br />
<br />
http://www.MidtownDelivery.com<br />
<br />
http://accreu.com<br />
<br />
http://www.actionfigurelabs.com<br />
<br />
https://m.facebook.com/AddictivePerformance99<br />
<br />
http://www.additech.com/<br />
<br />
http://adknowledgents.wix.com/adknowledgents<br />
<br />
http://www.rmudata.com<br />
<br />
http://www.advancedcardiodr.com/<br />
<br />
http://alwii.org<br />
<br />
http://www.advancedseismic.com<br />
<br />
http://www.AdvoWire.com<br />
<br />
http://www.aggredyne.com<br />
<br />
http://www.akrostechlabs.com/<br />
<br />
http://www.aleedex.com<br />
<br />
http://www.alertlogic.com/<br />
<br />
http://www.aliceandlove.com<br />
<br />
https://www.alignedsigns.com/ppcregistration6.htm<br />
<br />
https://www.alliedwarranty.com/<br />
<br />
http://none yet<br />
<br />
http://www.alpheus.net<br />
<br />
Output Text:<br />
<br />
[[Image:output.jpg|400px|thumb|right|WhoIs output file in Excel]]<br />
<br />
Domain Name Creation Date Expiration Date Update Date Registrant Name Registrant Street Registrant City Registrant Postal Code Registrant Country Admin Street Admin City Admin Postal Code Admin Country<br />
<br />
http://1986ventures.com 2013-09-12T09:25:51Z 12-sep-2016 Domain Admin C/O ID#10760, PO Box 16 Note - Visit PrivacyProtect.org to contact the domain owner/operator Note - Visit PrivacyProtect.org to contact the domain owner/operator Nobby Beach QLD 4218 AU C/O ID#10760, PO Box 16 Note - Visit PrivacyProtect.org to contact the domain owner/operator Note - Visit PrivacyProtect.org to contact the domain owner/operator Nobby Beach QLD 4218 AU<br />
<br />
http://2nd.md/ 2010-11-17 2017-11-17 <br />
<br />
http://www.2ndsquare.com 2013-10-16T04:01:29Z 16-oct-2016 2015-10-16T20:38:12Z Sameer Khan 22215 Tower Terr San Antonio 78259 US 22215 Tower Terr San Antonio 78259 US<br />
<br />
http://www.32nddegree.com/ 2008-02-18T18:45:15Z 18-feb-2020 Cutshall, Wes 1321 Upland Dr. Houston 77043 US 1321 Upland Dr. Houston 77043 US<br />
<br />
http://www.80legs.com 2008-07-17T21:09:48Z 17-jul-2016 Shion Deysarkar 904 West Avenue Austin 78701 US 904 West Avenue Austin 78701 US<br />
<br />
http://hotmailpasswordsupportnumber.info/<br />
<br />
http://www.MidtownDelivery.com 2012-01-23T05:01:21Z 23-jan-2017 2015-01-05T05:24:56Z Jim Wiseheart 7655 S. Braeswood#21 Houston 77071 US 7655 S. Braeswood#21 Houston 77071 US<br />
<br />
http://accreu.com 2011-05-05T00:11:53.000Z 05-may-2016 Oneandone Private Registration 701 Lee Road Suite 300ATTN Chesterbrook 19087 US 701 Lee Road Suite 300ATTN Chesterbrook 19087 US<br />
<br />
http://www.actionfigurelabs.com 2011-02-18T17:40:24Z 18-feb-2017 Phillip Leech 2223 Willowby Dr Houston 77008 US 2223 Willowby Dr Houston 77008 US<br />
<br />
https://m.facebook.com/AddictivePerformance99<br />
<br />
http://www.additech.com/ 1997-01-24T05:00:00Z 25-jan-2018 Additech, Inc. 10925 Kinghurst Houston 77099 US 10925 Kinghurst Houston 77099 US<br />
<br />
http://adknowledgents.wix.com/adknowledgents<br />
<br />
http://www.rmudata.com 2000-04-13T17:09:54Z 13-apr-2017 PERFECT PRIVACY, LLC 12808 Gran Bay Parkway West Jacksonville 32258 US 12808 Gran Bay Parkway West Jacksonville 32258 US<br />
<br />
http://www.advancedcardiodr.com/ 2012-04-17T14:12:09Z 17-apr-2022 2015-01-08T22:09:14Z Sharafat Hussain Advanced Cardiovascular Care Center800 Peakwood Drive, Suite 8C Houston 77090 US Advanced Cardiovascular Care Center800 Peakwood Drive, Suite 8C Houston 77090 US<br />
<br />
http://alwii.org 2011-05-31T21:48:05Z Chi Mao 1917 Ashland St, 2nd FloorIn Select Specialty Hospital Houston 77008 US 1917 Ashland St, 2nd FloorIn Select Specialty Hospital Houston 77008 US<br />
<br />
http://www.advancedseismic.com 2009-10-30T19:00:47Z 30-oct-2016 2015-10-31T11:28:22Z na na na na 88888 US na na 88888 US<br />
<br />
http://www.AdvoWire.com 2013-07-13T08:43:39Z 13-jul-2018 2013-07-13T08:43:39Z Jason Pampell 6516 North Gessner Houston 77040 US 6516 North Gessner Houston 77040 US<br />
<br />
http://www.aggredyne.com 2011-04-01T21:03:52Z 01-apr-2018 Robert C. Hux 10530 Rockley Rd.,Suite 150 Houston 77099 US 10530 Rockley Rd.,Suite 150 Houston 77099 US<br />
<br />
http://www.akrostechlabs.com/ 2008-03-24T17:34:07Z 24-mar-2017 2015-03-24T01:54:15Z Registration Private DomainsByProxy.com14747 N Northsight Blvd Suite 111, PMB 309 Scottsdale 85260 US DomainsByProxy.com14747 N Northsight Blvd Suite 111, PMB 309 Scottsdale 85260 US<br />
<br />
http://www.aleedex.com 2012-12-27T20:15:55Z 10-jun-2019 2013-06-14T09:54:17Z Farid Premani 10500 Reserve at Fountain Lake Stafford 77477 US 10500 Reserve at Fountain Lake Stafford 77477 US<br />
<br />
http://www.alertlogic.com/ 2003-10-10T21:24:13Z 10-oct-2019 PERFECT PRIVACY, LLC 12808 Gran Bay Pkwy West Jacksonville 32258 US 12808 Gran Bay Pkwy West Jacksonville 32258 US<br />
<br />
http://www.aliceandlove.com 2014-08-07T01:42:29Z 07-aug-2016 c/o WHOIStrustee.com Limited Riverside View Thornes Lane WF1 5QW GB Riverside View Thornes Lane WF1 5QW GB<br />
<br />
https://www.alignedsigns.com/ppcregistration6.htm<br />
<br />
https://www.alliedwarranty.com/ 2004-03-31T20:07:28Z 31-mar-2018 2014-03-16T04:17:39Z Registration Private DomainsByProxy.com14747 N Northsight Blvd Suite 111, PMB 309 Scottsdale 85260 US DomainsByProxy.com14747 N Northsight Blvd Suite 111, PMB 309 Scottsdale 85260 US<br />
<br />
http://none yet<br />
<br />
http://www.alpheus.net 2003-03-27T23:14:33Z 27-mar-2018 2016-03-28T11:22:05Z Alpheus Firstcall 1301 Fannin St.20th Floor Houston 77002 US 1301 Fannin St.20th Floor Houston 77002 US</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Whois_Parser&diff=4789Whois Parser2016-07-11T19:06:20Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Internal]]<br />
[[Internal Classification::Legacy| ]]<br />
This wiki page is under Additional Links/WhoisParser<br />
<br />
The whoisParser was written by Kunal Shah on March 20, 2016 and is located <br />
repository: Web_Crawler<br />
branch: shoeb_patch/whoisparser<br />
directory: /WhoIsParser<br />
file: whoisParser.pl<br />
<br />
To use this parser, copy above perl program into a directory, make it current working directory (that is, use 'cd' command if needed) and run the following command. The directory should also have the input file(see below).<br />
<br />
perl WhoIsParser.pl -file=listofurls.txt -outfile=listofurls_processed.txt<br />
<br />
= NAME =<br />
<br />
WhoIs Parser - Retrieves and parses Whois information<br />
Specifically, takes a file with a column of domain names and populates the<br />
corresponding columns with information from the WhoIs API.<br />
<br />
= SYNOPSIS =<br />
<br />
perl whoisParser -file=<file> [-outfile=<file>] <br />
<br />
= OPTIONS =<br />
<br />
-file=<file>: Name of file of domain names. <br />
-outfile=<file>: The name of the outfile <br />
-h: Display help<br />
<br />
= USAGE & FEATURES =<br />
<br />
'''Arguments:''' <br />
<br />
A text file with a column of domain names<br />
<br />
'''Returns:''' <br />
<br />
A text file of the domain names with the next 12 columns populated with information pulled from the Whois API. A header specifying each column is inserted into the first row of the file. The columns of information outputed are:<br />
<br />
1. Domain Name<br />
<br />
2. Creation Date<br />
<br />
3. Expiration Date<br />
<br />
4. Update Date<br />
<br />
5. Registrant Name<br />
<br />
6. Registrant Street<br />
<br />
7. Registrant City<br />
<br />
8. Registrant Postal Code<br />
<br />
9. Registrant Country<br />
<br />
10. Admin Street<br />
<br />
11. Admin City<br />
<br />
12. Admin Postal Code<br />
<br />
13. Admin Country<br />
<br />
= BUGS & FEEDBACK =<br />
<br />
Worked as expected on all example files. Please report any discovered bugs to Kunal.<br />
<br />
Tested files:<br />
Input: example_file.txt<br />
<br />
Output: example_outfile.txt<br />
<br />
<br />
Input Text: <br />
<br />
<br />
[[Image:input.jpg|400px|thumb|right|WhoIs input file in Excel]]<br />
<br />
http://1986ventures.com<br />
<br />
http://2nd.md/<br />
<br />
http://www.2ndsquare.com<br />
<br />
http://www.32nddegree.com/<br />
<br />
http://www.80legs.com<br />
<br />
http://hotmailpasswordsupportnumber.info/<br />
<br />
http://www.MidtownDelivery.com<br />
<br />
http://accreu.com<br />
<br />
http://www.actionfigurelabs.com<br />
<br />
https://m.facebook.com/AddictivePerformance99<br />
<br />
http://www.additech.com/<br />
<br />
http://adknowledgents.wix.com/adknowledgents<br />
<br />
http://www.rmudata.com<br />
<br />
http://www.advancedcardiodr.com/<br />
<br />
http://alwii.org<br />
<br />
http://www.advancedseismic.com<br />
<br />
http://www.AdvoWire.com<br />
<br />
http://www.aggredyne.com<br />
<br />
http://www.akrostechlabs.com/<br />
<br />
http://www.aleedex.com<br />
<br />
http://www.alertlogic.com/<br />
<br />
http://www.aliceandlove.com<br />
<br />
https://www.alignedsigns.com/ppcregistration6.htm<br />
<br />
https://www.alliedwarranty.com/<br />
<br />
http://none yet<br />
<br />
http://www.alpheus.net<br />
<br />
Output Text:<br />
<br />
[[Image:output.jpg|400px|thumb|right|WhoIs output file in Excel]]<br />
<br />
Domain Name Creation Date Expiration Date Update Date Registrant Name Registrant Street Registrant City Registrant Postal Code Registrant Country Admin Street Admin City Admin Postal Code Admin Country<br />
<br />
http://1986ventures.com 2013-09-12T09:25:51Z 12-sep-2016 Domain Admin C/O ID#10760, PO Box 16 Note - Visit PrivacyProtect.org to contact the domain owner/operator Note - Visit PrivacyProtect.org to contact the domain owner/operator Nobby Beach QLD 4218 AU C/O ID#10760, PO Box 16 Note - Visit PrivacyProtect.org to contact the domain owner/operator Note - Visit PrivacyProtect.org to contact the domain owner/operator Nobby Beach QLD 4218 AU<br />
<br />
http://2nd.md/ 2010-11-17 2017-11-17 <br />
<br />
http://www.2ndsquare.com 2013-10-16T04:01:29Z 16-oct-2016 2015-10-16T20:38:12Z Sameer Khan 22215 Tower Terr San Antonio 78259 US 22215 Tower Terr San Antonio 78259 US<br />
<br />
http://www.32nddegree.com/ 2008-02-18T18:45:15Z 18-feb-2020 Cutshall, Wes 1321 Upland Dr. Houston 77043 US 1321 Upland Dr. Houston 77043 US<br />
<br />
http://www.80legs.com 2008-07-17T21:09:48Z 17-jul-2016 Shion Deysarkar 904 West Avenue Austin 78701 US 904 West Avenue Austin 78701 US<br />
<br />
http://hotmailpasswordsupportnumber.info/<br />
<br />
http://www.MidtownDelivery.com 2012-01-23T05:01:21Z 23-jan-2017 2015-01-05T05:24:56Z Jim Wiseheart 7655 S. Braeswood#21 Houston 77071 US 7655 S. Braeswood#21 Houston 77071 US<br />
<br />
http://accreu.com 2011-05-05T00:11:53.000Z 05-may-2016 Oneandone Private Registration 701 Lee Road Suite 300ATTN Chesterbrook 19087 US 701 Lee Road Suite 300ATTN Chesterbrook 19087 US<br />
<br />
http://www.actionfigurelabs.com 2011-02-18T17:40:24Z 18-feb-2017 Phillip Leech 2223 Willowby Dr Houston 77008 US 2223 Willowby Dr Houston 77008 US<br />
<br />
https://m.facebook.com/AddictivePerformance99<br />
<br />
http://www.additech.com/ 1997-01-24T05:00:00Z 25-jan-2018 Additech, Inc. 10925 Kinghurst Houston 77099 US 10925 Kinghurst Houston 77099 US<br />
<br />
http://adknowledgents.wix.com/adknowledgents<br />
<br />
http://www.rmudata.com 2000-04-13T17:09:54Z 13-apr-2017 PERFECT PRIVACY, LLC 12808 Gran Bay Parkway West Jacksonville 32258 US 12808 Gran Bay Parkway West Jacksonville 32258 US<br />
<br />
http://www.advancedcardiodr.com/ 2012-04-17T14:12:09Z 17-apr-2022 2015-01-08T22:09:14Z Sharafat Hussain Advanced Cardiovascular Care Center800 Peakwood Drive, Suite 8C Houston 77090 US Advanced Cardiovascular Care Center800 Peakwood Drive, Suite 8C Houston 77090 US<br />
<br />
http://alwii.org 2011-05-31T21:48:05Z Chi Mao 1917 Ashland St, 2nd FloorIn Select Specialty Hospital Houston 77008 US 1917 Ashland St, 2nd FloorIn Select Specialty Hospital Houston 77008 US<br />
<br />
http://www.advancedseismic.com 2009-10-30T19:00:47Z 30-oct-2016 2015-10-31T11:28:22Z na na na na 88888 US na na 88888 US<br />
<br />
http://www.AdvoWire.com 2013-07-13T08:43:39Z 13-jul-2018 2013-07-13T08:43:39Z Jason Pampell 6516 North Gessner Houston 77040 US 6516 North Gessner Houston 77040 US<br />
<br />
http://www.aggredyne.com 2011-04-01T21:03:52Z 01-apr-2018 Robert C. Hux 10530 Rockley Rd.,Suite 150 Houston 77099 US 10530 Rockley Rd.,Suite 150 Houston 77099 US<br />
<br />
http://www.akrostechlabs.com/ 2008-03-24T17:34:07Z 24-mar-2017 2015-03-24T01:54:15Z Registration Private DomainsByProxy.com14747 N Northsight Blvd Suite 111, PMB 309 Scottsdale 85260 US DomainsByProxy.com14747 N Northsight Blvd Suite 111, PMB 309 Scottsdale 85260 US<br />
<br />
http://www.aleedex.com 2012-12-27T20:15:55Z 10-jun-2019 2013-06-14T09:54:17Z Farid Premani 10500 Reserve at Fountain Lake Stafford 77477 US 10500 Reserve at Fountain Lake Stafford 77477 US<br />
<br />
http://www.alertlogic.com/ 2003-10-10T21:24:13Z 10-oct-2019 PERFECT PRIVACY, LLC 12808 Gran Bay Pkwy West Jacksonville 32258 US 12808 Gran Bay Pkwy West Jacksonville 32258 US<br />
<br />
http://www.aliceandlove.com 2014-08-07T01:42:29Z 07-aug-2016 c/o WHOIStrustee.com Limited Riverside View Thornes Lane WF1 5QW GB Riverside View Thornes Lane WF1 5QW GB<br />
<br />
https://www.alignedsigns.com/ppcregistration6.htm<br />
<br />
https://www.alliedwarranty.com/ 2004-03-31T20:07:28Z 31-mar-2018 2014-03-16T04:17:39Z Registration Private DomainsByProxy.com14747 N Northsight Blvd Suite 111, PMB 309 Scottsdale 85260 US DomainsByProxy.com14747 N Northsight Blvd Suite 111, PMB 309 Scottsdale 85260 US<br />
<br />
http://none yet<br />
<br />
http://www.alpheus.net 2003-03-27T23:14:33Z 27-mar-2018 2016-03-28T11:22:05Z Alpheus Firstcall 1301 Fannin St.20th Floor Houston 77002 US 1301 Fannin St.20th Floor Houston 77002 US</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Whois_Parser&diff=4788Whois Parser2016-07-11T19:00:55Z<p>ShoebMohammed: </p>
<hr />
<div>[[Category:Internal]]<br />
[[Internal Classification::Legacy| ]]<br />
This wiki page is under Additional Links/WhoisParser<br />
<br />
The whoisParser was written by Kunal Shah on March 20, 2016 and is located <br />
repository: Web_Crawler<br />
branch: kunal/whoisparser<br />
directory: /WhoIsParser<br />
file: whoisParser.pl<br />
<br />
To use this parser, copy above perl program into a directory, make it current working directory (that is, use 'cd' command if needed) and run the following command. The directory should also have the input file(see below).<br />
<br />
perl WhoIsParser.pl -file=listofurls.txt -outfile=listofurls_processed.txt<br />
<br />
= NAME =<br />
<br />
WhoIs Parser - Retrieves and parses Whois information<br />
Specifically, takes a file with a column of domain names and populates the<br />
corresponding columns with information from the WhoIs API.<br />
<br />
= SYNOPSIS =<br />
<br />
perl whoisParser -file=<file> [-outfile=<file>] <br />
<br />
= OPTIONS =<br />
<br />
-file=<file>: Name of file of domain names. <br />
-outfile=<file>: The name of the outfile <br />
-h: Display help<br />
<br />
= USAGE & FEATURES =<br />
<br />
'''Arguments:''' <br />
<br />
A text file with a column of domain names<br />
<br />
'''Returns:''' <br />
<br />
A text file of the domain names with the next 12 columns populated with information pulled from the Whois API. A header specifying each column is inserted into the first row of the file. The columns of information outputed are:<br />
<br />
1. Domain Name<br />
<br />
2. Creation Date<br />
<br />
3. Expiration Date<br />
<br />
4. Update Date<br />
<br />
5. Registrant Name<br />
<br />
6. Registrant Street<br />
<br />
7. Registrant City<br />
<br />
8. Registrant Postal Code<br />
<br />
9. Registrant Country<br />
<br />
10. Admin Street<br />
<br />
11. Admin City<br />
<br />
12. Admin Postal Code<br />
<br />
13. Admin Country<br />
<br />
= BUGS & FEEDBACK =<br />
<br />
Worked as expected on all example files. Please report any discovered bugs to Kunal.<br />
<br />
Tested files:<br />
Input: example_file.txt<br />
<br />
Output: example_outfile.txt<br />
<br />
<br />
Input Text: <br />
<br />
<br />
[[Image:input.jpg|400px|thumb|right|WhoIs input file in Excel]]<br />
<br />
http://1986ventures.com<br />
<br />
http://2nd.md/<br />
<br />
http://www.2ndsquare.com<br />
<br />
http://www.32nddegree.com/<br />
<br />
http://www.80legs.com<br />
<br />
http://hotmailpasswordsupportnumber.info/<br />
<br />
http://www.MidtownDelivery.com<br />
<br />
http://accreu.com<br />
<br />
http://www.actionfigurelabs.com<br />
<br />
https://m.facebook.com/AddictivePerformance99<br />
<br />
http://www.additech.com/<br />
<br />
http://adknowledgents.wix.com/adknowledgents<br />
<br />
http://www.rmudata.com<br />
<br />
http://www.advancedcardiodr.com/<br />
<br />
http://alwii.org<br />
<br />
http://www.advancedseismic.com<br />
<br />
http://www.AdvoWire.com<br />
<br />
http://www.aggredyne.com<br />
<br />
http://www.akrostechlabs.com/<br />
<br />
http://www.aleedex.com<br />
<br />
http://www.alertlogic.com/<br />
<br />
http://www.aliceandlove.com<br />
<br />
https://www.alignedsigns.com/ppcregistration6.htm<br />
<br />
https://www.alliedwarranty.com/<br />
<br />
http://none yet<br />
<br />
http://www.alpheus.net<br />
<br />
Output Text:<br />
<br />
[[Image:output.jpg|400px|thumb|right|WhoIs output file in Excel]]<br />
<br />
Domain Name Creation Date Expiration Date Update Date Registrant Name Registrant Street Registrant City Registrant Postal Code Registrant Country Admin Street Admin City Admin Postal Code Admin Country<br />
<br />
http://1986ventures.com 2013-09-12T09:25:51Z 12-sep-2016 Domain Admin C/O ID#10760, PO Box 16 Note - Visit PrivacyProtect.org to contact the domain owner/operator Note - Visit PrivacyProtect.org to contact the domain owner/operator Nobby Beach QLD 4218 AU C/O ID#10760, PO Box 16 Note - Visit PrivacyProtect.org to contact the domain owner/operator Note - Visit PrivacyProtect.org to contact the domain owner/operator Nobby Beach QLD 4218 AU<br />
<br />
http://2nd.md/ 2010-11-17 2017-11-17 <br />
<br />
http://www.2ndsquare.com 2013-10-16T04:01:29Z 16-oct-2016 2015-10-16T20:38:12Z Sameer Khan 22215 Tower Terr San Antonio 78259 US 22215 Tower Terr San Antonio 78259 US<br />
<br />
http://www.32nddegree.com/ 2008-02-18T18:45:15Z 18-feb-2020 Cutshall, Wes 1321 Upland Dr. Houston 77043 US 1321 Upland Dr. Houston 77043 US<br />
<br />
http://www.80legs.com 2008-07-17T21:09:48Z 17-jul-2016 Shion Deysarkar 904 West Avenue Austin 78701 US 904 West Avenue Austin 78701 US<br />
<br />
http://hotmailpasswordsupportnumber.info/<br />
<br />
http://www.MidtownDelivery.com 2012-01-23T05:01:21Z 23-jan-2017 2015-01-05T05:24:56Z Jim Wiseheart 7655 S. Braeswood#21 Houston 77071 US 7655 S. Braeswood#21 Houston 77071 US<br />
<br />
http://accreu.com 2011-05-05T00:11:53.000Z 05-may-2016 Oneandone Private Registration 701 Lee Road Suite 300ATTN Chesterbrook 19087 US 701 Lee Road Suite 300ATTN Chesterbrook 19087 US<br />
<br />
http://www.actionfigurelabs.com 2011-02-18T17:40:24Z 18-feb-2017 Phillip Leech 2223 Willowby Dr Houston 77008 US 2223 Willowby Dr Houston 77008 US<br />
<br />
https://m.facebook.com/AddictivePerformance99<br />
<br />
http://www.additech.com/ 1997-01-24T05:00:00Z 25-jan-2018 Additech, Inc. 10925 Kinghurst Houston 77099 US 10925 Kinghurst Houston 77099 US<br />
<br />
http://adknowledgents.wix.com/adknowledgents<br />
<br />
http://www.rmudata.com 2000-04-13T17:09:54Z 13-apr-2017 PERFECT PRIVACY, LLC 12808 Gran Bay Parkway West Jacksonville 32258 US 12808 Gran Bay Parkway West Jacksonville 32258 US<br />
<br />
http://www.advancedcardiodr.com/ 2012-04-17T14:12:09Z 17-apr-2022 2015-01-08T22:09:14Z Sharafat Hussain Advanced Cardiovascular Care Center800 Peakwood Drive, Suite 8C Houston 77090 US Advanced Cardiovascular Care Center800 Peakwood Drive, Suite 8C Houston 77090 US<br />
<br />
http://alwii.org 2011-05-31T21:48:05Z Chi Mao 1917 Ashland St, 2nd FloorIn Select Specialty Hospital Houston 77008 US 1917 Ashland St, 2nd FloorIn Select Specialty Hospital Houston 77008 US<br />
<br />
http://www.advancedseismic.com 2009-10-30T19:00:47Z 30-oct-2016 2015-10-31T11:28:22Z na na na na 88888 US na na 88888 US<br />
<br />
http://www.AdvoWire.com 2013-07-13T08:43:39Z 13-jul-2018 2013-07-13T08:43:39Z Jason Pampell 6516 North Gessner Houston 77040 US 6516 North Gessner Houston 77040 US<br />
<br />
http://www.aggredyne.com 2011-04-01T21:03:52Z 01-apr-2018 Robert C. Hux 10530 Rockley Rd.,Suite 150 Houston 77099 US 10530 Rockley Rd.,Suite 150 Houston 77099 US<br />
<br />
http://www.akrostechlabs.com/ 2008-03-24T17:34:07Z 24-mar-2017 2015-03-24T01:54:15Z Registration Private DomainsByProxy.com14747 N Northsight Blvd Suite 111, PMB 309 Scottsdale 85260 US DomainsByProxy.com14747 N Northsight Blvd Suite 111, PMB 309 Scottsdale 85260 US<br />
<br />
http://www.aleedex.com 2012-12-27T20:15:55Z 10-jun-2019 2013-06-14T09:54:17Z Farid Premani 10500 Reserve at Fountain Lake Stafford 77477 US 10500 Reserve at Fountain Lake Stafford 77477 US<br />
<br />
http://www.alertlogic.com/ 2003-10-10T21:24:13Z 10-oct-2019 PERFECT PRIVACY, LLC 12808 Gran Bay Pkwy West Jacksonville 32258 US 12808 Gran Bay Pkwy West Jacksonville 32258 US<br />
<br />
http://www.aliceandlove.com 2014-08-07T01:42:29Z 07-aug-2016 c/o WHOIStrustee.com Limited Riverside View Thornes Lane WF1 5QW GB Riverside View Thornes Lane WF1 5QW GB<br />
<br />
https://www.alignedsigns.com/ppcregistration6.htm<br />
<br />
https://www.alliedwarranty.com/ 2004-03-31T20:07:28Z 31-mar-2018 2014-03-16T04:17:39Z Registration Private DomainsByProxy.com14747 N Northsight Blvd Suite 111, PMB 309 Scottsdale 85260 US DomainsByProxy.com14747 N Northsight Blvd Suite 111, PMB 309 Scottsdale 85260 US<br />
<br />
http://none yet<br />
<br />
http://www.alpheus.net 2003-03-27T23:14:33Z 27-mar-2018 2016-03-28T11:22:05Z Alpheus Firstcall 1301 Fannin St.20th Floor Houston 77002 US 1301 Fannin St.20th Floor Houston 77002 US</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Bulk_Patent_Assignee_Processing&diff=4646Bulk Patent Assignee Processing2016-07-08T15:56:50Z<p>ShoebMohammed: /* Scripts for processing data */</p>
<hr />
<div>== USPTO Assignees Data ==<br />
<br />
We would like to download and absorb data from this location on the USPTO website into our tables. The objective is to determine whether this dataset is better than the current version of our patent data (a combination of the data in the patent_2015 and patentdata databases.<br />
<br />
== Steps Followed to Extract the Data ==<br />
<br />
===Extracting Data from XML Files ===<br />
<br />
All the historical USPTO data is available as XML files. Here is the tree structure for the XML files:<br />
<br />
<patent-assignment><br />
+<assignment-record><br />
+<patent-assignors><br />
+<patent-assignees><br />
+<patent-properties><br />
</patent-assignment><br />
<br />
Each of the above internal nodes is mandatory, and is a logical grouping of information fields. Each node has a corresponding table created with more or less the same fields as the XML elements.<br />
<br />
Corresponding tables are:<br />
*assignment-records : assignment<br />
*patent-assignors : assignors<br />
*patent-assignees : assignees<br />
*patent-properties : properties<br />
<br />
Additionally, for each file that is downloaded, there are some associated specs. All of these are stored in the PatentAssignment table. Here is the data model diagram.<br />
<br />
==== Assignment Records ====<br />
<br />
The fields in the assignment record are:<br />
* last_update_date<br />
* purge_indicator<br />
* recorded_date<br />
* correspondent_name<br />
* correspondent_address_1<br />
* correspondent_address_2<br />
* correspondent_address_3<br />
* correspondent_address_4<br />
* conveyance_text<br />
<br />
Here is the corresponding XML that we are mapping:<br />
<br />
-<assignment-record><br />
<reel-no>27132</reel-no><br />
<frame-no>841</frame-no><br />
-<last-update-date><br />
<date>20160122</date><br />
</last-update-date><br />
<purge-indicator>N</purge-indicator><br />
-<recorded-date><br />
<date>20111027</date><br />
</recorded-date><br />
<page-count>2</page-count><br />
-<correspondent><br />
<name>DOUGLAS B. MCKNIGHT</name><br />
<address-1>595 MINER ROAD</address-1><br />
<address-2>INTELLECTUAL PROPERTY & STANDARDS</address-2><br />
<address-3>CLEVELAND, OH 44143</address-3><br />
</correspondent><br />
<conveyance-text>ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS).</conveyance-text><br />
</assignment-record><br />
<br />
<br />
==== Assignors ====<br />
<br />
Here are the columns in the assignors table:<br />
* reel_no<br />
* frame_no<br />
* assignor_name<br />
* execution_date<br />
<br />
The corresponding XML node is :<br />
<br />
-<patent-assignors><br />
-<patent-assignor><br />
<name>WALKER, MATTHEW J.</name><br />
-<execution-date><br />
<date>20090512</date><br />
</execution-date><br />
</patent-assignor><br />
-<patent-assignor><br />
<name>OLSZEWSKI, MARK E.</name><br />
-<execution-date><br />
<date>20090512</date><br />
</execution-date><br />
</patent-assignor><br />
</patent-assignors><br />
<br />
==== Assignees ====<br />
<br />
Here are the columns in the assignees table:<br />
<br />
* reel_no<br />
* frame_no<br />
* assignee_name<br />
* assignee_address_1<br />
* assignee_address_2<br />
* assignee_city<br />
* assignee_state<br />
* assignee_country<br />
* assignee_postcode<br />
<br />
The corresponding XML nodes are:<br />
<br />
-<patent-assignees><br />
-<patent-assignee><br />
<name>KONINKLIJKE PHILIPS ELECTRONICS N V</name><br />
<address-1>GROENEWOUDSEWEG 1</address-1><br />
<city>EINDHOVEN</city><br />
<country-name>NETHERLANDS</country-name><br />
<postcode>5621 BA</postcode><br />
</patent-assignee><br />
</patent-assignees><br />
<br />
==== Patent Properties ====<br />
<br />
Here are the columns in the properties table:<br />
<br />
* reel_no<br />
* frame_no<br />
* documentid<br />
* country<br />
* kind<br />
* filingdate<br />
* invention_title<br />
<br />
The corresponding XML segment would be:<br />
<br />
-<patent-properties><br />
-<patent-property><br />
-<document-id><br />
<country>US</country><br />
<doc-number>14143589</doc-number><br />
<kind>X0</kind><br />
<date>20131230</date><br />
</document-id><br />
-<document-id><br />
<country>US</country><br />
<doc-number>20140260305</doc-number><br />
<kind>A1</kind><br />
<date>20140918</date><br />
</document-id><br />
<invention-title lang="en">LEAN AZIMUTHAL FLAME COMBUSTOR</invention-title><br />
</patent-property><br />
</patent-properties><br />
<br />
Patent properties have a many-to-one relationship : one patent can have more than one properties.<br />
Note: We are not sure what documents with kind 'X0' say<br />
<br />
<br />
==== Patent Assignment ====<br />
<br />
Every XML file download has some fields associated with it, in addition to a number of patent assignment nodes.<br />
<br />
Here are the columns in the table:<br />
<br />
* reel_no<br />
* frame_no<br />
* action_key_code<br />
* USPTO_Transaction_Date<br />
* USPTO_Date_Produced<br />
* version<br />
<br />
Here is what the XML in a downloaded file looks like:<br />
<br />
<?xml version="1.0" encoding="UTF-8"?><br />
<!DOCTYPE us-patent-assignments><br />
-<us-patent-assignments date-produced="20131101" dtd-version="1.0"><br />
<action-key-code>DA</action-key-code><br />
-<transaction-date><br />
<date>20160122</date><br />
</transaction-date><br />
-<patent-assignments><br />
+<patent-assignment><br />
+<patent-assignment><br />
+<patent-assignment><br />
+<patent-assignment><br />
+<patent-assignment><br />
+<patent-assignment><br />
.<br />
.<br />
.<br />
</patent-assignments><br />
</us-patent-assignments><br />
<br />
<br />
<br />
====DTD====<br />
Here is the DTD specified by the USPTO, which specifies optional fields and :<br />
<br />
<?xml version="1.0" encoding="utf-8"?> <br />
<!DOCTYPE us-patent-assignments [<!ELEMENT us-patent-assignments (action-key-code, transaction-date, patent-assignments)><br />
<!ATTLIST us-patent-assignments dtd-version CDATA #IMPLIED <br />
date-produced CDATA #IMPLIED> <br />
<!ELEMENT action-key-code (#PCDATA)> <br />
<!ELEMENT transaction-date (date)><br />
<!ELEMENT patent-assignments (data-available-code | patent-assignment+)><br />
<!ELEMENT date (#PCDATA)> <br />
<!ELEMENT data-available-code (#PCDATA)> <br />
<!ELEMENT patent-assignment (assignment-record, patent-assignors, patent-assignees, patent-properties)> <br />
<!ELEMENT assignment-record (reel-no, frame-no, last-update-date, purge-indicator, recorded-date, page-count?, correspondent, conveyance-text)> <br />
<!ELEMENT patent-assignors (patent-assignor+)> <br />
<!ELEMENT patent-assignees (patent-assignee+)> <br />
<!ELEMENT patent-properties (patent-property+)> <br />
<!ELEMENT reel-no (#PCDATA)> <br />
<!ELEMENT frame-no (#PCDATA)> <br />
<!ELEMENT last-update-date (date)> <br />
<!ELEMENT purge-indicator (#PCDATA)> <br />
<!ELEMENT recorded-date (date)> <br />
<!ELEMENT page-count (#PCDATA)> <br />
<!ELEMENT correspondent (name, address-1?, address-2?, address-3?, address-4?)> <br />
<!ELEMENT conveyance-text (#PCDATA)> <br />
<!ELEMENT patent-assignor (name, execution-date?, date-acknowledged?)> <br />
<!ELEMENT patent-assignee (name, address-1?, address-2?, city?, state?, country-name?, postcode?)> <br />
<!ELEMENT patent-property (document-id*, invention-title?)> <br />
<!ELEMENT name (#PCDATA)> <br />
<!ATTLIST name name-type (natural | legal) #IMPLIED> <br />
<!ELEMENT address-1 (#PCDATA)> <br />
<!ELEMENT address-2 (#PCDATA)> <br />
<!ELEMENT address-3 (#PCDATA)> <br />
<!ELEMENT address-4 (#PCDATA)> <br />
<!ELEMENT execution-date (date)><br />
<!ELEMENT date-acknowledged (date)> <br />
<!ELEMENT city (#PCDATA)><br />
<!ELEMENT state (#PCDATA)><br />
<!ELEMENT country-name (#PCDATA)><br />
<!ELEMENT postcode (#PCDATA)> <br />
<!ELEMENT document-id (country, doc-number, kind?, name?, date?)> <br />
<!ELEMENT invention-title (#PCDATA | b | i | u | sup | sub)*><br />
<!ATTLIST invention-title id ID #IMPLIED <br />
lang CDATA #REQUIRED> <br />
<!ELEMENT country (#PCDATA)><br />
<!ELEMENT doc-number (#PCDATA)><br />
<!ELEMENT kind (#PCDATA)><br />
<!--bold formatting for text--><br />
<!ELEMENT b (#PCDATA | i | u | smallcaps)*><br />
<!--italic formatting for text--><br />
<!ELEMENT i (#PCDATA | b | u | smallcaps)*><br />
<!--underscore: style - single is default--><br />
<!ELEMENT u (#PCDATA | b | i | smallcaps)*><br />
<!ATTLIST u style (single | double | dash | dots ) 'single' ><br />
<!--superscripted text--><br />
<!ELEMENT sup (#PCDATA | b | u | i)*><br />
<!--subscripted text--><br />
<!ELEMENT sub (#PCDATA | b | u | i)*><br />
<!--small capitals--><br />
<!ELEMENT smallcaps (#PCDATA | b | u | i)*><br />
]><br />
<br />
<br />
===Inserting Extracted Data into Tables ===<br />
<br />
===Clean Up ===<br />
<br />
<br />
<br />
== Scripts for processing data ==<br />
The programs/scripts (see details below) are located on our [[Software Repository|Bonobo Git Server]]. <br />
repository: Patent_Data_Parser <br />
branch: next<br />
directory: /uspto_assignees_xml_parser<br />
<br />
<br />
=== Downloading raw bulk data from USPTO ===<br />
repository: Patent_Data_Parser <br />
branch: next<br />
directory: /uspto_assignees_xml_parser<br />
file: USPTO_Assignee_Download.pl<br />
<br />
The down-loader script used to download XML files is essentially same, with minor changes, as the one used for downloading USPTO patent-data.<br />
That is, the current version of down-loader script downloads all files from the base URL: https://bulkdata.uspto.gov/data2/patent/assignment/<br />
<br />
=== Parsing the XML files ===<br />
repository: Patent_Data_Parser <br />
branch: next<br />
directory: /uspto_assignees_xml_parser<br />
file: uspto_assignees_XML_parser.plx<br />
<br />
==== NAME ====<br />
<br />
uspto_assignees_XML_parser.plx - Parses XML files and populates a database.<br />
<br />
Specifically, parses every file in a directory according to a schema (see above).<br />
Then populates a database on the RDP. <br />
<br />
==== SYNOPSIS ====<br />
<br />
perl uspto_assignees_XML_parser.plx /path/to/directory_containing_XML_files<br />
<br />
==== USAGE & FEATURES ====<br />
<br />
'''Arguments'''<br />
The full path to directory is provided as a command line argument. It should contain the XML files to parse and no other file.<br />
This path should be specified in Windows format (with '\') and NOT unix format.<br />
<br />
'''Features and Effects'''<br />
As each XML file is parsed, a database on local host (RDP) is populated. If at any point there is an error, for example a particular<br />
XML file is bad/invalid or the psql statement cannot be executed, the program aborts with a message.<br />
<br />
We choose to populate local database because remote connections are too slow. The database is eventually moved to DataBase server manually.<br />
<br />
==== TESTS ====<br />
The first version does the job as expected. It was used to populate the assignees database by parsing XML files from USPTO(see above).<br />
We parsed all XML files dated till 7/4/2016.<br />
<br />
==== TO DO ====<br />
*Add more command line options to improve usability.<br />
*Improve portability to allow Unix/Linux pathnames. This is straightforward to do with Perl modules File::Basename and File::Spec.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Bulk_Patent_Assignee_Processing&diff=4645Bulk Patent Assignee Processing2016-07-08T15:55:55Z<p>ShoebMohammed: /* Scripts for processing data */</p>
<hr />
<div>== USPTO Assignees Data ==<br />
<br />
We would like to download and absorb data from this location on the USPTO website into our tables. The objective is to determine whether this dataset is better than the current version of our patent data (a combination of the data in the patent_2015 and patentdata databases.<br />
<br />
== Steps Followed to Extract the Data ==<br />
<br />
===Extracting Data from XML Files ===<br />
<br />
All the historical USPTO data is available as XML files. Here is the tree structure for the XML files:<br />
<br />
<patent-assignment><br />
+<assignment-record><br />
+<patent-assignors><br />
+<patent-assignees><br />
+<patent-properties><br />
</patent-assignment><br />
<br />
Each of the above internal nodes is mandatory, and is a logical grouping of information fields. Each node has a corresponding table created with more or less the same fields as the XML elements.<br />
<br />
Corresponding tables are:<br />
*assignment-records : assignment<br />
*patent-assignors : assignors<br />
*patent-assignees : assignees<br />
*patent-properties : properties<br />
<br />
Additionally, for each file that is downloaded, there are some associated specs. All of these are stored in the PatentAssignment table. Here is the data model diagram.<br />
<br />
==== Assignment Records ====<br />
<br />
The fields in the assignment record are:<br />
* last_update_date<br />
* purge_indicator<br />
* recorded_date<br />
* correspondent_name<br />
* correspondent_address_1<br />
* correspondent_address_2<br />
* correspondent_address_3<br />
* correspondent_address_4<br />
* conveyance_text<br />
<br />
Here is the corresponding XML that we are mapping:<br />
<br />
-<assignment-record><br />
<reel-no>27132</reel-no><br />
<frame-no>841</frame-no><br />
-<last-update-date><br />
<date>20160122</date><br />
</last-update-date><br />
<purge-indicator>N</purge-indicator><br />
-<recorded-date><br />
<date>20111027</date><br />
</recorded-date><br />
<page-count>2</page-count><br />
-<correspondent><br />
<name>DOUGLAS B. MCKNIGHT</name><br />
<address-1>595 MINER ROAD</address-1><br />
<address-2>INTELLECTUAL PROPERTY & STANDARDS</address-2><br />
<address-3>CLEVELAND, OH 44143</address-3><br />
</correspondent><br />
<conveyance-text>ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS).</conveyance-text><br />
</assignment-record><br />
<br />
<br />
==== Assignors ====<br />
<br />
Here are the columns in the assignors table:<br />
* reel_no<br />
* frame_no<br />
* assignor_name<br />
* execution_date<br />
<br />
The corresponding XML node is :<br />
<br />
-<patent-assignors><br />
-<patent-assignor><br />
<name>WALKER, MATTHEW J.</name><br />
-<execution-date><br />
<date>20090512</date><br />
</execution-date><br />
</patent-assignor><br />
-<patent-assignor><br />
<name>OLSZEWSKI, MARK E.</name><br />
-<execution-date><br />
<date>20090512</date><br />
</execution-date><br />
</patent-assignor><br />
</patent-assignors><br />
<br />
==== Assignees ====<br />
<br />
Here are the columns in the assignees table:<br />
<br />
* reel_no<br />
* frame_no<br />
* assignee_name<br />
* assignee_address_1<br />
* assignee_address_2<br />
* assignee_city<br />
* assignee_state<br />
* assignee_country<br />
* assignee_postcode<br />
<br />
The corresponding XML nodes are:<br />
<br />
-<patent-assignees><br />
-<patent-assignee><br />
<name>KONINKLIJKE PHILIPS ELECTRONICS N V</name><br />
<address-1>GROENEWOUDSEWEG 1</address-1><br />
<city>EINDHOVEN</city><br />
<country-name>NETHERLANDS</country-name><br />
<postcode>5621 BA</postcode><br />
</patent-assignee><br />
</patent-assignees><br />
<br />
==== Patent Properties ====<br />
<br />
Here are the columns in the properties table:<br />
<br />
* reel_no<br />
* frame_no<br />
* documentid<br />
* country<br />
* kind<br />
* filingdate<br />
* invention_title<br />
<br />
The corresponding XML segment would be:<br />
<br />
-<patent-properties><br />
-<patent-property><br />
-<document-id><br />
<country>US</country><br />
<doc-number>14143589</doc-number><br />
<kind>X0</kind><br />
<date>20131230</date><br />
</document-id><br />
-<document-id><br />
<country>US</country><br />
<doc-number>20140260305</doc-number><br />
<kind>A1</kind><br />
<date>20140918</date><br />
</document-id><br />
<invention-title lang="en">LEAN AZIMUTHAL FLAME COMBUSTOR</invention-title><br />
</patent-property><br />
</patent-properties><br />
<br />
Patent properties have a many-to-one relationship : one patent can have more than one properties.<br />
Note: We are not sure what documents with kind 'X0' say<br />
<br />
<br />
==== Patent Assignment ====<br />
<br />
Every XML file download has some fields associated with it, in addition to a number of patent assignment nodes.<br />
<br />
Here are the columns in the table:<br />
<br />
* reel_no<br />
* frame_no<br />
* action_key_code<br />
* USPTO_Transaction_Date<br />
* USPTO_Date_Produced<br />
* version<br />
<br />
Here is what the XML in a downloaded file looks like:<br />
<br />
<?xml version="1.0" encoding="UTF-8"?><br />
<!DOCTYPE us-patent-assignments><br />
-<us-patent-assignments date-produced="20131101" dtd-version="1.0"><br />
<action-key-code>DA</action-key-code><br />
-<transaction-date><br />
<date>20160122</date><br />
</transaction-date><br />
-<patent-assignments><br />
+<patent-assignment><br />
+<patent-assignment><br />
+<patent-assignment><br />
+<patent-assignment><br />
+<patent-assignment><br />
+<patent-assignment><br />
.<br />
.<br />
.<br />
</patent-assignments><br />
</us-patent-assignments><br />
<br />
<br />
<br />
====DTD====<br />
Here is the DTD specified by the USPTO, which specifies optional fields and :<br />
<br />
<?xml version="1.0" encoding="utf-8"?> <br />
<!DOCTYPE us-patent-assignments [<!ELEMENT us-patent-assignments (action-key-code, transaction-date, patent-assignments)><br />
<!ATTLIST us-patent-assignments dtd-version CDATA #IMPLIED <br />
date-produced CDATA #IMPLIED> <br />
<!ELEMENT action-key-code (#PCDATA)> <br />
<!ELEMENT transaction-date (date)><br />
<!ELEMENT patent-assignments (data-available-code | patent-assignment+)><br />
<!ELEMENT date (#PCDATA)> <br />
<!ELEMENT data-available-code (#PCDATA)> <br />
<!ELEMENT patent-assignment (assignment-record, patent-assignors, patent-assignees, patent-properties)> <br />
<!ELEMENT assignment-record (reel-no, frame-no, last-update-date, purge-indicator, recorded-date, page-count?, correspondent, conveyance-text)> <br />
<!ELEMENT patent-assignors (patent-assignor+)> <br />
<!ELEMENT patent-assignees (patent-assignee+)> <br />
<!ELEMENT patent-properties (patent-property+)> <br />
<!ELEMENT reel-no (#PCDATA)> <br />
<!ELEMENT frame-no (#PCDATA)> <br />
<!ELEMENT last-update-date (date)> <br />
<!ELEMENT purge-indicator (#PCDATA)> <br />
<!ELEMENT recorded-date (date)> <br />
<!ELEMENT page-count (#PCDATA)> <br />
<!ELEMENT correspondent (name, address-1?, address-2?, address-3?, address-4?)> <br />
<!ELEMENT conveyance-text (#PCDATA)> <br />
<!ELEMENT patent-assignor (name, execution-date?, date-acknowledged?)> <br />
<!ELEMENT patent-assignee (name, address-1?, address-2?, city?, state?, country-name?, postcode?)> <br />
<!ELEMENT patent-property (document-id*, invention-title?)> <br />
<!ELEMENT name (#PCDATA)> <br />
<!ATTLIST name name-type (natural | legal) #IMPLIED> <br />
<!ELEMENT address-1 (#PCDATA)> <br />
<!ELEMENT address-2 (#PCDATA)> <br />
<!ELEMENT address-3 (#PCDATA)> <br />
<!ELEMENT address-4 (#PCDATA)> <br />
<!ELEMENT execution-date (date)><br />
<!ELEMENT date-acknowledged (date)> <br />
<!ELEMENT city (#PCDATA)><br />
<!ELEMENT state (#PCDATA)><br />
<!ELEMENT country-name (#PCDATA)><br />
<!ELEMENT postcode (#PCDATA)> <br />
<!ELEMENT document-id (country, doc-number, kind?, name?, date?)> <br />
<!ELEMENT invention-title (#PCDATA | b | i | u | sup | sub)*><br />
<!ATTLIST invention-title id ID #IMPLIED <br />
lang CDATA #REQUIRED> <br />
<!ELEMENT country (#PCDATA)><br />
<!ELEMENT doc-number (#PCDATA)><br />
<!ELEMENT kind (#PCDATA)><br />
<!--bold formatting for text--><br />
<!ELEMENT b (#PCDATA | i | u | smallcaps)*><br />
<!--italic formatting for text--><br />
<!ELEMENT i (#PCDATA | b | u | smallcaps)*><br />
<!--underscore: style - single is default--><br />
<!ELEMENT u (#PCDATA | b | i | smallcaps)*><br />
<!ATTLIST u style (single | double | dash | dots ) 'single' ><br />
<!--superscripted text--><br />
<!ELEMENT sup (#PCDATA | b | u | i)*><br />
<!--subscripted text--><br />
<!ELEMENT sub (#PCDATA | b | u | i)*><br />
<!--small capitals--><br />
<!ELEMENT smallcaps (#PCDATA | b | u | i)*><br />
]><br />
<br />
<br />
===Inserting Extracted Data into Tables ===<br />
<br />
===Clean Up ===<br />
<br />
<br />
<br />
== Scripts for processing data ==<br />
The programs/scripts are located on our [[Software Repository|Bonobo Git Server]].<br />
repository: Patent_Data_Parser <br />
branch: next<br />
directory: /uspto_assignees_xml_parser<br />
<br />
<br />
=== Downloading raw bulk data from USPTO ===<br />
repository: Patent_Data_Parser <br />
branch: next<br />
directory: /uspto_assignees_xml_parser<br />
file: USPTO_Assignee_Download.pl<br />
<br />
The down-loader script used to download XML files is essentially same, with minor changes, as the one used for downloading USPTO patent-data.<br />
That is, the current version of down-loader script downloads all files from the base URL: https://bulkdata.uspto.gov/data2/patent/assignment/<br />
<br />
=== Parsing the XML files ===<br />
repository: Patent_Data_Parser <br />
branch: next<br />
directory: /uspto_assignees_xml_parser<br />
file: uspto_assignees_XML_parser.plx<br />
<br />
==== NAME ====<br />
<br />
uspto_assignees_XML_parser.plx - Parses XML files and populates a database.<br />
<br />
Specifically, parses every file in a directory according to a schema (see above).<br />
Then populates a database on the RDP. <br />
<br />
==== SYNOPSIS ====<br />
<br />
perl uspto_assignees_XML_parser.plx /path/to/directory_containing_XML_files<br />
<br />
==== USAGE & FEATURES ====<br />
<br />
'''Arguments'''<br />
The full path to directory is provided as a command line argument. It should contain the XML files to parse and no other file.<br />
This path should be specified in Windows format (with '\') and NOT unix format.<br />
<br />
'''Features and Effects'''<br />
As each XML file is parsed, a database on local host (RDP) is populated. If at any point there is an error, for example a particular<br />
XML file is bad/invalid or the psql statement cannot be executed, the program aborts with a message.<br />
<br />
We choose to populate local database because remote connections are too slow. The database is eventually moved to DataBase server manually.<br />
<br />
==== TESTS ====<br />
The first version does the job as expected. It was used to populate the assignees database by parsing XML files from USPTO(see above).<br />
We parsed all XML files dated till 7/4/2016.<br />
<br />
==== TO DO ====<br />
*Add more command line options to improve usability.<br />
*Improve portability to allow Unix/Linux pathnames. This is straightforward to do with Perl modules File::Basename and File::Spec.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Software_Repository&diff=4641Software Repository2016-07-08T15:50:22Z<p>ShoebMohammed: /* Our Git workflow */</p>
<hr />
<div>[[Category: McNair Admin]]<br />
<br />
==Background==<br />
Given the amount of software that has been written by past computer science interns and more being written, we felt the need to have some kind of source code management system put into place so that developers can work without ever being in fear of breaking production and facing Ed's wrath (you do not want that dude angry! Wherever you go, he will find you! No escape.).<br />
<br />
To enforce efficient source control we(Ed) chose to host our own git server on the RDP machine using [https://bonobogitserver.com/ Bonobo Git Server] that makes use of the windows IIS platform and is open source.<br />
<br />
Installing Bonobo git server is pretty simple:<br />
* dowload the zip file from the Bonobo website.<br />
* extract its contents. It should be a single folder containing directories like App_Data, bin etc.<br />
* rename that folder to anything you want. I used the name "codebase"<br />
* copy the codebase folder to C:\inetpub\wwwroot\<br />
* Allow IIS User to modify C:\inetpub\wwwroot\codebase\App_Data folder. To do so:<br />
**select Properties of App_Data folder,<br />
**go to Security tab,<br />
**click edit,<br />
**select IIS user (in my case IIS_IUSRS) and add Modify and Write permission,<br />
**confirm these settings with Apply button.<br />
*Convert ''codebase'' to Application in IIS<br />
**Run IIS Manager and navigate to Sites -> Default Web Site. You should see Bonobo.Git.Server.<br />
**Right click on 'codebase' and convert to application.<br />
**Check if the selected application pool runs on .NET 4.0 and convert the site.<br />
*Enable Anonymous Authentication in IIS and disable the others. To do so, select the application in the left pane, double-click on the authentication icon in the right pane and set the value to of Anonymous Authentication to Enabled<br />
*Launch your browser and go to http://localhost/codebase. Now you can see the initial page of the Bonobo Git Server and everything should work.<br />
**default credentials are ''username'': '''admin''', ''password'': '''admin'''<br />
**[6-22-2016]: Can also use https://localhost/codebase which is preferable, otherwise username/passwords are transmitted plain text. The browser will show a security error because we have a self signed certificate. This is ok if we are restricted to intranet. If we want to allow public access, we probably need to get a certificate from a Certificate Authority like Verisign etc.<br />
<br />
==Our Git Server==<br />
We have already done the set up of the git server on the RDP machine. Here are the admin credentials:<br />
*Username: '''boss'''<br />
*Name: '''Ed'''<br />
*Surname: '''Egan'''<br />
*Email: '''Edward.Egan@rice.edu'''<br />
*Password: '''you_seriously_thought_Id_write_that_in_here??'''<br />
<br />
To access this from your computer and not the RDP you can go to http://128.42.44.182/codebase where it will prompt you for your username and password.<br />
**[6-22-2016]: Can also use https://128.42.44.182/codebase which is preferable. The browser will show a security error because we have a self signed certificate. This is ok if we are restricted to intranet. If we want to allow public access, we probably need to get a certificate from a Certificate Authority like Verisign etc.<br />
<br />
<br />
==Our Git workflow==<br />
We chose a simple git workflow.<br />
<br />
Our aim is not to break things in the master branch. All commits on the master should work.<br />
<br />
1.<br />
When adding a new feature or fixing a bug, ALWAYS check out a new feature branch from the master.<br />
NEVER checkout a feature branch from next (see below). The feature branch should be named user/feature_name. <br />
<br />
2.<br />
After feature development is complete merge your feature-branch into next.<br />
<br />
3.<br />
The next branch is intended for testing and confirming things do not break. So, after feature branches are merged into next and conflicts resolved, we merge into master.<br />
After this, you can end the feature branches if you want.<br />
<br />
==Quick and dirty github tutorial==<br />
For a cool interactive tutorial see http://learngitbranching.js.org/.<br />
<br />
***<br />
You can also use SourceTree which is a GUI interface for git-client. This is installed on the RDP.<br />
Like using git from CLI (see below), SourceTree constructs appropriate commands. But the good thing<br />
is it automatically generates all error check/logging options with each command that are difficult<br />
to recall from memory. SourceTree is freely available from Altassian at https://www.sourcetreeapp.com/<br />
***<br />
<br />
To use SourceTree you should have basic understanding of git (like branches,commits etc). The interactive tutorial above is very good for this purpose.<br />
<br />
<br />
<br />
*''Installing - '' Depending on your operating system you can install git in three different ways:<br />
** If you are a windows or a mac user user, you can simply download & install the latest release from [https://git-scm.com/ git scm website]<br />
** If you use ubuntu then all you need to do is type <code><big>sudo apt-get install git</big></code><br />
* Check your installation by typing <code><big>git</big></code> in terminal or windows powershell<br />
*Basic git operations:<br />
<div style="text-align: left; direction: ltr; margin-left: 2em;"><br />
* to checkout code from remote repository, use the <code><big>git clone</big></code> command. This will create a local repository on your disk as well as download the source code of the project you wish to work on. Here's an example:<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git clone http://128.42.44.182/codebase/Matcher.git</big></code></div><br />
<br />
* to update your repository to include others' work in your project use the <code><big>git update</big></code> command. Its always a good practice to update your code before you commit to ensure that others' code doesn't break yours. Also, you cannot push to remote unless your local repository is up to date. If you commit on a stale local repository that is fine, just that this would mean you are likely to have more trouble merging your code with others later on thanks to all the conflicts that you'll face when you actually try to update your repository later. See example:<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git update <optional folder path></big></code></div><br />
<br />
* to commit your changes to your local repository use the <code><big>git commit</big></code> command. Committing your changes is an essential step whether you are adding/removing items from the repository or changing existing items. See example :<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git commit -m "mandatory commit message"</big></code></div><br />
<br />
* to push your changes to remote repository use the <code><big>git push <optional file/folder name></big></code> command. Whatever you need to be pushed to the server must be committed to your local repository first. By default this command will push everything from current folder if no item is specified.<br />
<br />
* to add new files to your repository use the <code><big>git add</big></code> command. After that you must commit to ensure that your repository actually has the new file. See example:<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git add <filename/folder name></big></code></div><br />
<br />
* to remove items from your repository use the <code><big>git remove</big></code> command. After that you delete the file that you wanted removed from the repository and commit to ensure that your repository actually has the change persisted. Finally, you push to server to make sure the server has those items removed as well and that nobody in your team works under the assumption that those items are stills there. See example:<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git remove <filename></big></code></div><br />
<div style="text-align: left; direction: ltr; margin-left: 2em;"><br />
''Note'': if removing a non empty folder use the -r flag to recursively remove all contents of that folder as well :<br />
</div><br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git remove -r <foldername></big></code></div><br />
<br />
</div><br />
<br />
[[admin_classification::IT Build| ]]</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Software_Repository&diff=4639Software Repository2016-07-08T15:47:56Z<p>ShoebMohammed: /* Background */</p>
<hr />
<div>[[Category: McNair Admin]]<br />
<br />
==Background==<br />
Given the amount of software that has been written by past computer science interns and more being written, we felt the need to have some kind of source code management system put into place so that developers can work without ever being in fear of breaking production and facing Ed's wrath (you do not want that dude angry! Wherever you go, he will find you! No escape.).<br />
<br />
To enforce efficient source control we(Ed) chose to host our own git server on the RDP machine using [https://bonobogitserver.com/ Bonobo Git Server] that makes use of the windows IIS platform and is open source.<br />
<br />
Installing Bonobo git server is pretty simple:<br />
* dowload the zip file from the Bonobo website.<br />
* extract its contents. It should be a single folder containing directories like App_Data, bin etc.<br />
* rename that folder to anything you want. I used the name "codebase"<br />
* copy the codebase folder to C:\inetpub\wwwroot\<br />
* Allow IIS User to modify C:\inetpub\wwwroot\codebase\App_Data folder. To do so:<br />
**select Properties of App_Data folder,<br />
**go to Security tab,<br />
**click edit,<br />
**select IIS user (in my case IIS_IUSRS) and add Modify and Write permission,<br />
**confirm these settings with Apply button.<br />
*Convert ''codebase'' to Application in IIS<br />
**Run IIS Manager and navigate to Sites -> Default Web Site. You should see Bonobo.Git.Server.<br />
**Right click on 'codebase' and convert to application.<br />
**Check if the selected application pool runs on .NET 4.0 and convert the site.<br />
*Enable Anonymous Authentication in IIS and disable the others. To do so, select the application in the left pane, double-click on the authentication icon in the right pane and set the value to of Anonymous Authentication to Enabled<br />
*Launch your browser and go to http://localhost/codebase. Now you can see the initial page of the Bonobo Git Server and everything should work.<br />
**default credentials are ''username'': '''admin''', ''password'': '''admin'''<br />
**[6-22-2016]: Can also use https://localhost/codebase which is preferable, otherwise username/passwords are transmitted plain text. The browser will show a security error because we have a self signed certificate. This is ok if we are restricted to intranet. If we want to allow public access, we probably need to get a certificate from a Certificate Authority like Verisign etc.<br />
<br />
==Our Git Server==<br />
We have already done the set up of the git server on the RDP machine. Here are the admin credentials:<br />
*Username: '''boss'''<br />
*Name: '''Ed'''<br />
*Surname: '''Egan'''<br />
*Email: '''Edward.Egan@rice.edu'''<br />
*Password: '''you_seriously_thought_Id_write_that_in_here??'''<br />
<br />
To access this from your computer and not the RDP you can go to http://128.42.44.182/codebase where it will prompt you for your username and password.<br />
**[6-22-2016]: Can also use https://128.42.44.182/codebase which is preferable. The browser will show a security error because we have a self signed certificate. This is ok if we are restricted to intranet. If we want to allow public access, we probably need to get a certificate from a Certificate Authority like Verisign etc.<br />
<br />
<br />
==Our Git workflow==<br />
We chose a simple git workflow.<br />
<br />
Our aim is not to break things in the master branch. All commits on the master should work.<br />
<br />
1.<br />
When adding a new feature or fixing a bug (well, why fix it, was that not a feature?), ALWAYS check out a new feature branch from the master.<br />
NEVER checkout a feature branch from next (see below). The feature branch should be named user/feature_name. <br />
<br />
2.<br />
After feature development is complete merge your feature-branch into next.<br />
<br />
3.<br />
The next branch is intended for testing and confirm things do not break. So, after feature branches are merged into next and conflicts resolved, if things work we push it to master.<br />
You can end the feature branches if you want. <br />
<br />
<br />
==Quick and dirty github tutorial==<br />
For a cool interactive tutorial see http://learngitbranching.js.org/.<br />
<br />
***<br />
You can also use SourceTree which is a GUI interface for git-client. This is installed on the RDP.<br />
Like using git from CLI (see below), SourceTree constructs appropriate commands. But the good thing<br />
is it automatically generates all error check/logging options with each command that are difficult<br />
to recall from memory. SourceTree is freely available from Altassian at https://www.sourcetreeapp.com/<br />
***<br />
<br />
To use SourceTree you should have basic understanding of git (like branches,commits etc). The interactive tutorial above is very good for this purpose.<br />
<br />
<br />
<br />
*''Installing - '' Depending on your operating system you can install git in three different ways:<br />
** If you are a windows or a mac user user, you can simply download & install the latest release from [https://git-scm.com/ git scm website]<br />
** If you use ubuntu then all you need to do is type <code><big>sudo apt-get install git</big></code><br />
* Check your installation by typing <code><big>git</big></code> in terminal or windows powershell<br />
*Basic git operations:<br />
<div style="text-align: left; direction: ltr; margin-left: 2em;"><br />
* to checkout code from remote repository, use the <code><big>git clone</big></code> command. This will create a local repository on your disk as well as download the source code of the project you wish to work on. Here's an example:<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git clone http://128.42.44.182/codebase/Matcher.git</big></code></div><br />
<br />
* to update your repository to include others' work in your project use the <code><big>git update</big></code> command. Its always a good practice to update your code before you commit to ensure that others' code doesn't break yours. Also, you cannot push to remote unless your local repository is up to date. If you commit on a stale local repository that is fine, just that this would mean you are likely to have more trouble merging your code with others later on thanks to all the conflicts that you'll face when you actually try to update your repository later. See example:<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git update <optional folder path></big></code></div><br />
<br />
* to commit your changes to your local repository use the <code><big>git commit</big></code> command. Committing your changes is an essential step whether you are adding/removing items from the repository or changing existing items. See example :<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git commit -m "mandatory commit message"</big></code></div><br />
<br />
* to push your changes to remote repository use the <code><big>git push <optional file/folder name></big></code> command. Whatever you need to be pushed to the server must be committed to your local repository first. By default this command will push everything from current folder if no item is specified.<br />
<br />
* to add new files to your repository use the <code><big>git add</big></code> command. After that you must commit to ensure that your repository actually has the new file. See example:<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git add <filename/folder name></big></code></div><br />
<br />
* to remove items from your repository use the <code><big>git remove</big></code> command. After that you delete the file that you wanted removed from the repository and commit to ensure that your repository actually has the change persisted. Finally, you push to server to make sure the server has those items removed as well and that nobody in your team works under the assumption that those items are stills there. See example:<br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git remove <filename></big></code></div><br />
<div style="text-align: left; direction: ltr; margin-left: 2em;"><br />
''Note'': if removing a non empty folder use the -r flag to recursively remove all contents of that folder as well :<br />
</div><br />
<div style="text-align: left; direction: ltr; margin-left: 5em;"><code><big>git remove -r <foldername></big></code></div><br />
<br />
</div><br />
<br />
[[admin_classification::IT Build| ]]</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Bulk_Patent_Assignee_Processing&diff=4636Bulk Patent Assignee Processing2016-07-08T15:38:04Z<p>ShoebMohammed: /* Downloading raw bulk data from USPTO */</p>
<hr />
<div>== USPTO Assignees Data ==<br />
<br />
We would like to download and absorb data from this location on the USPTO website into our tables. The objective is to determine whether this dataset is better than the current version of our patent data (a combination of the data in the patent_2015 and patentdata databases.<br />
<br />
== Steps Followed to Extract the Data ==<br />
<br />
===Extracting Data from XML Files ===<br />
<br />
All the historical USPTO data is available as XML files. Here is the tree structure for the XML files:<br />
<br />
<patent-assignment><br />
+<assignment-record><br />
+<patent-assignors><br />
+<patent-assignees><br />
+<patent-properties><br />
</patent-assignment><br />
<br />
Each of the above internal nodes is mandatory, and is a logical grouping of information fields. Each node has a corresponding table created with more or less the same fields as the XML elements.<br />
<br />
Corresponding tables are:<br />
*assignment-records : assignment<br />
*patent-assignors : assignors<br />
*patent-assignees : assignees<br />
*patent-properties : properties<br />
<br />
Additionally, for each file that is downloaded, there are some associated specs. All of these are stored in the PatentAssignment table. Here is the data model diagram.<br />
<br />
==== Assignment Records ====<br />
<br />
The fields in the assignment record are:<br />
* last_update_date<br />
* purge_indicator<br />
* recorded_date<br />
* correspondent_name<br />
* correspondent_address_1<br />
* correspondent_address_2<br />
* correspondent_address_3<br />
* correspondent_address_4<br />
* conveyance_text<br />
<br />
Here is the corresponding XML that we are mapping:<br />
<br />
-<assignment-record><br />
<reel-no>27132</reel-no><br />
<frame-no>841</frame-no><br />
-<last-update-date><br />
<date>20160122</date><br />
</last-update-date><br />
<purge-indicator>N</purge-indicator><br />
-<recorded-date><br />
<date>20111027</date><br />
</recorded-date><br />
<page-count>2</page-count><br />
-<correspondent><br />
<name>DOUGLAS B. MCKNIGHT</name><br />
<address-1>595 MINER ROAD</address-1><br />
<address-2>INTELLECTUAL PROPERTY & STANDARDS</address-2><br />
<address-3>CLEVELAND, OH 44143</address-3><br />
</correspondent><br />
<conveyance-text>ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS).</conveyance-text><br />
</assignment-record><br />
<br />
<br />
==== Assignors ====<br />
<br />
Here are the columns in the assignors table:<br />
* reel_no<br />
* frame_no<br />
* assignor_name<br />
* execution_date<br />
<br />
The corresponding XML node is :<br />
<br />
-<patent-assignors><br />
-<patent-assignor><br />
<name>WALKER, MATTHEW J.</name><br />
-<execution-date><br />
<date>20090512</date><br />
</execution-date><br />
</patent-assignor><br />
-<patent-assignor><br />
<name>OLSZEWSKI, MARK E.</name><br />
-<execution-date><br />
<date>20090512</date><br />
</execution-date><br />
</patent-assignor><br />
</patent-assignors><br />
<br />
==== Assignees ====<br />
<br />
Here are the columns in the assignees table:<br />
<br />
* reel_no<br />
* frame_no<br />
* assignee_name<br />
* assignee_address_1<br />
* assignee_address_2<br />
* assignee_city<br />
* assignee_state<br />
* assignee_country<br />
* assignee_postcode<br />
<br />
The corresponding XML nodes are:<br />
<br />
-<patent-assignees><br />
-<patent-assignee><br />
<name>KONINKLIJKE PHILIPS ELECTRONICS N V</name><br />
<address-1>GROENEWOUDSEWEG 1</address-1><br />
<city>EINDHOVEN</city><br />
<country-name>NETHERLANDS</country-name><br />
<postcode>5621 BA</postcode><br />
</patent-assignee><br />
</patent-assignees><br />
<br />
==== Patent Properties ====<br />
<br />
Here are the columns in the properties table:<br />
<br />
* reel_no<br />
* frame_no<br />
* documentid<br />
* country<br />
* kind<br />
* filingdate<br />
* invention_title<br />
<br />
The corresponding XML segment would be:<br />
<br />
-<patent-properties><br />
-<patent-property><br />
-<document-id><br />
<country>US</country><br />
<doc-number>14143589</doc-number><br />
<kind>X0</kind><br />
<date>20131230</date><br />
</document-id><br />
-<document-id><br />
<country>US</country><br />
<doc-number>20140260305</doc-number><br />
<kind>A1</kind><br />
<date>20140918</date><br />
</document-id><br />
<invention-title lang="en">LEAN AZIMUTHAL FLAME COMBUSTOR</invention-title><br />
</patent-property><br />
</patent-properties><br />
<br />
Patent properties have a many-to-one relationship : one patent can have more than one properties.<br />
Note: We are not sure what documents with kind 'X0' say<br />
<br />
<br />
==== Patent Assignment ====<br />
<br />
Every XML file download has some fields associated with it, in addition to a number of patent assignment nodes.<br />
<br />
Here are the columns in the table:<br />
<br />
* reel_no<br />
* frame_no<br />
* action_key_code<br />
* USPTO_Transaction_Date<br />
* USPTO_Date_Produced<br />
* version<br />
<br />
Here is what the XML in a downloaded file looks like:<br />
<br />
<?xml version="1.0" encoding="UTF-8"?><br />
<!DOCTYPE us-patent-assignments><br />
-<us-patent-assignments date-produced="20131101" dtd-version="1.0"><br />
<action-key-code>DA</action-key-code><br />
-<transaction-date><br />
<date>20160122</date><br />
</transaction-date><br />
-<patent-assignments><br />
+<patent-assignment><br />
+<patent-assignment><br />
+<patent-assignment><br />
+<patent-assignment><br />
+<patent-assignment><br />
+<patent-assignment><br />
.<br />
.<br />
.<br />
</patent-assignments><br />
</us-patent-assignments><br />
<br />
<br />
<br />
====DTD====<br />
Here is the DTD specified by the USPTO, which specifies optional fields and :<br />
<br />
<?xml version="1.0" encoding="utf-8"?> <br />
<!DOCTYPE us-patent-assignments [<!ELEMENT us-patent-assignments (action-key-code, transaction-date, patent-assignments)><br />
<!ATTLIST us-patent-assignments dtd-version CDATA #IMPLIED <br />
date-produced CDATA #IMPLIED> <br />
<!ELEMENT action-key-code (#PCDATA)> <br />
<!ELEMENT transaction-date (date)><br />
<!ELEMENT patent-assignments (data-available-code | patent-assignment+)><br />
<!ELEMENT date (#PCDATA)> <br />
<!ELEMENT data-available-code (#PCDATA)> <br />
<!ELEMENT patent-assignment (assignment-record, patent-assignors, patent-assignees, patent-properties)> <br />
<!ELEMENT assignment-record (reel-no, frame-no, last-update-date, purge-indicator, recorded-date, page-count?, correspondent, conveyance-text)> <br />
<!ELEMENT patent-assignors (patent-assignor+)> <br />
<!ELEMENT patent-assignees (patent-assignee+)> <br />
<!ELEMENT patent-properties (patent-property+)> <br />
<!ELEMENT reel-no (#PCDATA)> <br />
<!ELEMENT frame-no (#PCDATA)> <br />
<!ELEMENT last-update-date (date)> <br />
<!ELEMENT purge-indicator (#PCDATA)> <br />
<!ELEMENT recorded-date (date)> <br />
<!ELEMENT page-count (#PCDATA)> <br />
<!ELEMENT correspondent (name, address-1?, address-2?, address-3?, address-4?)> <br />
<!ELEMENT conveyance-text (#PCDATA)> <br />
<!ELEMENT patent-assignor (name, execution-date?, date-acknowledged?)> <br />
<!ELEMENT patent-assignee (name, address-1?, address-2?, city?, state?, country-name?, postcode?)> <br />
<!ELEMENT patent-property (document-id*, invention-title?)> <br />
<!ELEMENT name (#PCDATA)> <br />
<!ATTLIST name name-type (natural | legal) #IMPLIED> <br />
<!ELEMENT address-1 (#PCDATA)> <br />
<!ELEMENT address-2 (#PCDATA)> <br />
<!ELEMENT address-3 (#PCDATA)> <br />
<!ELEMENT address-4 (#PCDATA)> <br />
<!ELEMENT execution-date (date)><br />
<!ELEMENT date-acknowledged (date)> <br />
<!ELEMENT city (#PCDATA)><br />
<!ELEMENT state (#PCDATA)><br />
<!ELEMENT country-name (#PCDATA)><br />
<!ELEMENT postcode (#PCDATA)> <br />
<!ELEMENT document-id (country, doc-number, kind?, name?, date?)> <br />
<!ELEMENT invention-title (#PCDATA | b | i | u | sup | sub)*><br />
<!ATTLIST invention-title id ID #IMPLIED <br />
lang CDATA #REQUIRED> <br />
<!ELEMENT country (#PCDATA)><br />
<!ELEMENT doc-number (#PCDATA)><br />
<!ELEMENT kind (#PCDATA)><br />
<!--bold formatting for text--><br />
<!ELEMENT b (#PCDATA | i | u | smallcaps)*><br />
<!--italic formatting for text--><br />
<!ELEMENT i (#PCDATA | b | u | smallcaps)*><br />
<!--underscore: style - single is default--><br />
<!ELEMENT u (#PCDATA | b | i | smallcaps)*><br />
<!ATTLIST u style (single | double | dash | dots ) 'single' ><br />
<!--superscripted text--><br />
<!ELEMENT sup (#PCDATA | b | u | i)*><br />
<!--subscripted text--><br />
<!ELEMENT sub (#PCDATA | b | u | i)*><br />
<!--small capitals--><br />
<!ELEMENT smallcaps (#PCDATA | b | u | i)*><br />
]><br />
<br />
<br />
===Inserting Extracted Data into Tables ===<br />
<br />
===Clean Up ===<br />
<br />
<br />
<br />
== Scripts for processing data ==<br />
The programs/scripts are located at our [[Software Repository]].<br />
repository: Patent_Data_Parser <br />
branch: next<br />
directory: /uspto_assignees_xml_parser<br />
<br />
<br />
=== Downloading raw bulk data from USPTO ===<br />
repository: Patent_Data_Parser <br />
branch: next<br />
directory: /uspto_assignees_xml_parser<br />
file: USPTO_Assignee_Download.pl<br />
<br />
The down-loader script used to download XML files is essentially same, with minor changes, as the one used for downloading USPTO patent-data.<br />
That is, the current version of down-loader script downloads all files from the base URL: https://bulkdata.uspto.gov/data2/patent/assignment/<br />
<br />
=== Parsing the XML files ===<br />
repository: Patent_Data_Parser <br />
branch: next<br />
directory: /uspto_assignees_xml_parser<br />
file: uspto_assignees_XML_parser.plx<br />
<br />
==== NAME ====<br />
<br />
uspto_assignees_XML_parser.plx - Parses XML files and populates a database.<br />
<br />
Specifically, parses every file in a directory according to a schema (see above).<br />
Then populates a database on the RDP. <br />
<br />
==== SYNOPSIS ====<br />
<br />
perl uspto_assignees_XML_parser.plx /path/to/directory_containing_XML_files<br />
<br />
==== USAGE & FEATURES ====<br />
<br />
'''Arguments'''<br />
The full path to directory is provided as a command line argument. It should contain the XML files to parse and no other file.<br />
This path should be specified in Windows format (with '\') and NOT unix format.<br />
<br />
'''Features and Effects'''<br />
As each XML file is parsed, a database on local host (RDP) is populated. If at any point there is an error, for example a particular<br />
XML file is bad/invalid or the psql statement cannot be executed, the program aborts with a message.<br />
<br />
We choose to populate local database because remote connections are too slow. The database is eventually moved to DataBase server manually.<br />
<br />
==== TESTS ====<br />
The first version does the job as expected. It was used to populate the assignees database by parsing XML files from USPTO(see above).<br />
We parsed all XML files dated till 7/4/2016.<br />
<br />
==== TO DO ====<br />
*Add more command line options to improve usability.<br />
*Improve portability to allow Unix/Linux pathnames. This is straightforward to do with Perl modules File::Basename and File::Spec.</div>ShoebMohammedhttp://www.edegan.com/mediawiki/index.php?title=Bulk_Patent_Assignee_Processing&diff=4420Bulk Patent Assignee Processing2016-07-05T20:37:21Z<p>ShoebMohammed: /* Scripts for processing data */</p>
<hr />
<div>== USPTO Assignees Data ==<br />
<br />
We would like to download and absorb data from this location on the USPTO website into our tables. The objective is to determine whether this dataset is better than the current version of our patent data (a combination of the data in the patent_2015 and patentdata databases.<br />
<br />
== Steps Followed to Extract the Data ==<br />
<br />
===Extracting Data from XML Files ===<br />
<br />
All the historical USPTO data is available as XML files. Here is the tree structure for the XML files:<br />
<br />
<patent-assignment><br />
+<assignment-record><br />
+<patent-assignors><br />
+<patent-assignees><br />
+<patent-properties><br />
</patent-assignment><br />
<br />
Each of the above internal nodes is mandatory, and is a logical grouping of information fields. Each node has a corresponding table created with more or less the same fields as the XML elements.<br />
<br />
Corresponding tables are:<br />
*assignment-records : assignment<br />
*patent-assignors : assignors<br />
*patent-assignees : assignees<br />
*patent-properties : properties<br />
<br />
Additionally, for each file that is downloaded, there are some associated specs. All of these are stored in the PatentAssignment table. Here is the data model diagram.<br />
<br />
==== Assignment Records ====<br />
<br />
The fields in the assignment record are:<br />
* last_update_date<br />
* purge_indicator<br />
* recorded_date<br />
* correspondent_name<br />
* correspondent_address_1<br />
* correspondent_address_2<br />
* correspondent_address_3<br />
* correspondent_address_4<br />
* conveyance_text<br />
<br />
Here is the corresponding XML that we are mapping:<br />
<br />
-<assignment-record><br />
<reel-no>27132</reel-no><br />
<frame-no>841</frame-no><br />
-<last-update-date><br />
<date>20160122</date><br />
</last-update-date><br />
<purge-indicator>N</purge-indicator><br />
-<recorded-date><br />
<date>20111027</date><br />
</recorded-date><br />
<page-count>2</page-count><br />
-<correspondent><br />
<name>DOUGLAS B. MCKNIGHT</name><br />
<address-1>595 MINER ROAD</address-1><br />
<address-2>INTELLECTUAL PROPERTY & STANDARDS</address-2><br />
<address-3>CLEVELAND, OH 44143</address-3><br />
</correspondent><br />
<conveyance-text>ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS).</conveyance-text><br />
</assignment-record><br />
<br />
<br />
==== Assignors ====<br />
<br />
Here are the columns in the assignors table:<br />
* reel_no<br />
* frame_no<br />
* assignor_name<br />
* execution_date<br />
<br />
The corresponding XML node is :<br />
<br />
-<patent-assignors><br />
-<patent-assignor><br />
<name>WALKER, MATTHEW J.</name><br />
-<execution-date><br />
<date>20090512</date><br />
</execution-date><br />
</patent-assignor><br />
-<patent-assignor><br />
<name>OLSZEWSKI, MARK E.</name><br />
-<execution-date><br />
<date>20090512</date><br />
</execution-date><br />
</patent-assignor><br />
</patent-assignors><br />
<br />
==== Assignees ====<br />
<br />
Here are the columns in the assignees table:<br />
<br />
* reel_no<br />
* frame_no<br />
* assignee_name<br />
* assignee_address_1<br />
* assignee_address_2<br />
* assignee_city<br />
* assignee_state<br />
* assignee_country<br />
* assignee_postcode<br />
<br />
The corresponding XML nodes are:<br />
<br />
-<patent-assignees><br />
-<patent-assignee><br />
<name>KONINKLIJKE PHILIPS ELECTRONICS N V</name><br />
<address-1>GROENEWOUDSEWEG 1</address-1><br />
<city>EINDHOVEN</city><br />
<country-name>NETHERLANDS</country-name><br />
<postcode>5621 BA</postcode><br />
</patent-assignee><br />
</patent-assignees><br />
<br />
==== Patent Properties ====<br />
<br />
Here are the columns in the properties table:<br />
<br />
* reel_no<br />
* frame_no<br />
* documentid<br />
* country<br />
* kind<br />
* filingdate<br />
* invention_title<br />
<br />
The corresponding XML segment would be:<br />
<br />
-<patent-properties><br />
-<patent-property><br />
-<document-id><br />
<country>US</country><br />
<doc-number>14143589</doc-number><br />
<kind>X0</kind><br />
<date>20131230</date><br />
</document-id><br />
-<document-id><br />
<country>US</country><br />
<doc-number>20140260305</doc-number><br />
<kind>A1</kind><br />
<date>20140918</date><br />
</document-id><br />
<invention-title lang="en">LEAN AZIMUTHAL FLAME COMBUSTOR</invention-title><br />
</patent-property><br />
</patent-properties><br />
<br />
Patent properties have a many-to-one relationship : one patent can have more than one properties.<br />
Note: We are not sure what documents with kind 'X0' say<br />
<br />
<br />
==== Patent Assignment ====<br />
<br />
Every XML file download has some fields associated with it, in addition to a number of patent assignment nodes.<br />
<br />
Here are the columns in the table:<br />
<br />
* reel_no<br />
* frame_no<br />
* action_key_code<br />
* USPTO_Transaction_Date<br />
* USPTO_Date_Produced<br />
* version<br />
<br />
Here is what the XML in a downloaded file looks like:<br />
<br />
<?xml version="1.0" encoding="UTF-8"?><br />
<!DOCTYPE us-patent-assignments><br />
-<us-patent-assignments date-produced="20131101" dtd-version="1.0"><br />
<action-key-code>DA</action-key-code><br />
-<transaction-date><br />
<date>20160122</date><br />
</transaction-date><br />
-<patent-assignments><br />
+<patent-assignment><br />
+<patent-assignment><br />
+<patent-assignment><br />
+<patent-assignment><br />
+<patent-assignment><br />
+<patent-assignment><br />
.<br />
.<br />
.<br />
</patent-assignments><br />
</us-patent-assignments><br />
<br />
<br />
<br />
====DTD====<br />
Here is the DTD specified by the USPTO, which specifies optional fields and :<br />
<br />
<?xml version="1.0" encoding="utf-8"?> <br />
<!DOCTYPE us-patent-assignments [<!ELEMENT us-patent-assignments (action-key-code, transaction-date, patent-assignments)><br />
<!ATTLIST us-patent-assignments dtd-version CDATA #IMPLIED <br />
date-produced CDATA #IMPLIED> <br />
<!ELEMENT action-key-code (#PCDATA)> <br />
<!ELEMENT transaction-date (date)><br />
<!ELEMENT patent-assignments (data-available-code | patent-assignment+)><br />
<!ELEMENT date (#PCDATA)> <br />
<!ELEMENT data-available-code (#PCDATA)> <br />
<!ELEMENT patent-assignment (assignment-record, patent-assignors, patent-assignees, patent-properties)> <br />
<!ELEMENT assignment-record (reel-no, frame-no, last-update-date, purge-indicator, recorded-date, page-count?, correspondent, conveyance-text)> <br />
<!ELEMENT patent-assignors (patent-assignor+)> <br />
<!ELEMENT patent-assignees (patent-assignee+)> <br />
<!ELEMENT patent-properties (patent-property+)> <br />
<!ELEMENT reel-no (#PCDATA)> <br />
<!ELEMENT frame-no (#PCDATA)> <br />
<!ELEMENT last-update-date (date)> <br />
<!ELEMENT purge-indicator (#PCDATA)> <br />
<!ELEMENT recorded-date (date)> <br />
<!ELEMENT page-count (#PCDATA)> <br />
<!ELEMENT correspondent (name, address-1?, address-2?, address-3?, address-4?)> <br />
<!ELEMENT conveyance-text (#PCDATA)> <br />
<!ELEMENT patent-assignor (name, execution-date?, date-acknowledged?)> <br />
<!ELEMENT patent-assignee (name, address-1?, address-2?, city?, state?, country-name?, postcode?)> <br />
<!ELEMENT patent-property (document-id*, invention-title?)> <br />
<!ELEMENT name (#PCDATA)> <br />
<!ATTLIST name name-type (natural | legal) #IMPLIED> <br />
<!ELEMENT address-1 (#PCDATA)> <br />
<!ELEMENT address-2 (#PCDATA)> <br />
<!ELEMENT address-3 (#PCDATA)> <br />
<!ELEMENT address-4 (#PCDATA)> <br />
<!ELEMENT execution-date (date)><br />
<!ELEMENT date-acknowledged (date)> <br />
<!ELEMENT city (#PCDATA)><br />
<!ELEMENT state (#PCDATA)><br />
<!ELEMENT country-name (#PCDATA)><br />
<!ELEMENT postcode (#PCDATA)> <br />
<!ELEMENT document-id (country, doc-number, kind?, name?, date?)> <br />
<!ELEMENT invention-title (#PCDATA | b | i | u | sup | sub)*><br />
<!ATTLIST invention-title id ID #IMPLIED <br />
lang CDATA #REQUIRED> <br />
<!ELEMENT country (#PCDATA)><br />
<!ELEMENT doc-number (#PCDATA)><br />
<!ELEMENT kind (#PCDATA)><br />
<!--bold formatting for text--><br />
<!ELEMENT b (#PCDATA | i | u | smallcaps)*><br />
<!--italic formatting for text--><br />
<!ELEMENT i (#PCDATA | b | u | smallcaps)*><br />
<!--underscore: style - single is default--><br />
<!ELEMENT u (#PCDATA | b | i | smallcaps)*><br />
<!ATTLIST u style (single | double | dash | dots ) 'single' ><br />
<!--superscripted text--><br />
<!ELEMENT sup (#PCDATA | b | u | i)*><br />
<!--subscripted text--><br />
<!ELEMENT sub (#PCDATA | b | u | i)*><br />
<!--small capitals--><br />
<!ELEMENT smallcaps (#PCDATA | b | u | i)*><br />
]><br />
<br />
<br />
===Inserting Extracted Data into Tables ===<br />
<br />
===Clean Up ===<br />
<br />
<br />
<br />
== Scripts for processing data ==<br />
The programs/scripts are located at our [[Software Repository]].<br />
repository: Patent_Data_Parser <br />
branch: next<br />
directory: /uspto_assignees_xml_parser<br />
<br />
<br />
=== Downloading raw bulk data from USPTO ===<br />
repository: Patent_Data_Parser <br />
branch: next<br />
directory: /uspto_assignees_xml_parser<br />
file: USPTO_Assignee_Download.pl<br />
<br />
The XML files are available at https://bulkdata.uspto.gov/data2/patent/assignment/<br />
<br />
The down-loader script used to download XML files is essentially same, with minor changes, as the one used for downloading USPTO patent-data.<br />
That is, the current version of down-loader script downloads all files from the base URL (see above).<br />
<br />
<br />
=== Parsing the XML files ===<br />
repository: Patent_Data_Parser <br />
branch: next<br />
directory: /uspto_assignees_xml_parser<br />
file: uspto_assignees_XML_parser.plx<br />
<br />
==== NAME ====<br />
<br />
uspto_assignees_XML_parser.plx - Parses XML files and populates a database.<br />
<br />
Specifically, parses every file in a directory according to a schema (see above).<br />
Then populates a database on the RDP. <br />
<br />
==== SYNOPSIS ====<br />
<br />
perl uspto_assignees_XML_parser.plx /path/to/directory_containing_XML_files<br />
<br />
==== USAGE & FEATURES ====<br />
<br />
'''Arguments'''<br />
The full path to directory is provided as a command line argument. It should contain the XML files to parse and no other file.<br />
This path should be specified in Windows format (with '\') and NOT unix format.<br />
<br />
'''Features and Effects'''<br />
As each XML file is parsed, a database on local host (RDP) is populated. If at any point there is an error, for example a particular<br />
XML file is bad/invalid or the psql statement cannot be executed, the program aborts with a message.<br />
<br />
We choose to populate local database because remote connections are too slow. The database is eventually moved to DataBase server manually.<br />
<br />
==== TESTS ====<br />
The first version does the job as expected. It was used to populate the assignees database by parsing XML files from USPTO(see above).<br />
We parsed all XML files dated till 7/4/2016.<br />
<br />
==== TO DO ====<br />
*Add more command line options to improve usability.<br />
*Improve portability to allow Unix/Linux pathnames. This is straightforward to do with Perl modules File::Basename and File::Spec.</div>ShoebMohammed