PTLR Webcrawler - Revision history

ChristyW at 19:00, 12 December 2017

2017-12-12T19:00:05Z

ChristyW at 16:46, 8 December 2017

2017-12-08T16:46:57Z

ChristyW at 22:24, 28 November 2017

2017-11-28T22:24:05Z

LaurenB: /* Download PDFs */

2017-11-28T22:18:20Z

Download PDFs

ChristyW at 20:46, 2 November 2017

2017-11-02T20:46:23Z

ChristyW at 21:12, 10 October 2017

2017-10-10T21:12:12Z

ChristyW at 21:11, 10 October 2017

2017-10-10T21:11:25Z

ChristyW at 22:06, 3 October 2017

2017-10-03T22:06:38Z

LaurenB: /* Lauren's LOG */

2017-09-29T17:25:42Z

Lauren's LOG

ChristyW at 20:24, 28 September 2017

2017-09-28T20:24:39Z

@@ Line 10: / Line 10: @@
 =Code=
+/12/17: [[Scholar Crawler Main Program]]
 =Steps=

← Older revision		Revision as of 16:46, 8 December 2017
Line 8:		Line 8:

	Thursday: 2:15-3:45		Thursday: 2:15-3:45
		+
		+	=Code=
		+

	=Steps=		=Steps=

← Older revision		Revision as of 22:24, 28 November 2017
Line 70:		Line 70:

	Things are good! Today made the program so that we can get however many pages of search results we want and get the PDF links for all the ones we can see. Towards the end of the day, google scholar picked up that we were a robot and started blocking me. Hopefully the block goes away when I am back on Monday. Now working on parsing apart the txt file to go to the websites we saved and download the PDFs. Should not be particularly difficult.		Things are good! Today made the program so that we can get however many pages of search results we want and get the PDF links for all the ones we can see. Towards the end of the day, google scholar picked up that we were a robot and started blocking me. Hopefully the block goes away when I am back on Monday. Now working on parsing apart the txt file to go to the websites we saved and download the PDFs. Should not be particularly difficult.
		+
		+	'''11/28'''
		+
		+	Basically everything is ready to go, so long as Google Scholar leaves me alone. We currently have a program which will take in a search term and number of pages you want to search. The crawler will pull as many PDFs from this many pages as possible (it'll go slowly to avoid getting caught). Next, it will download all the PDFs discovered by the crawler (also possibly save the links for journals whose PDFs were not linked on scholar). It will then convert all the PDFs to text. Finally, it will search through the paper for a list of terms and for any definitions of patent thickets. I will be making documentation for these pieces of code today.

	=Lauren's LOG=		=Lauren's LOG=

← Older revision		Revision as of 22:18, 28 November 2017
Line 22:		Line 22:

	Incomplete, struggling to find links.		Incomplete, struggling to find links.
		+
		+	==Keywords List==
		+
		+	Find a copy of the Keywords List in the Dropbox: https://www.dropbox.com/s/mw5ep33fv7vz1rp/Keywords%20%3A%20Categories.xlsx?dl=0

	=Christy's LOG=		=Christy's LOG=

← Older revision		Revision as of 20:46, 2 November 2017
Line 62:		Line 62:

	Code located at E:/McNair/Software/Google_Scholar_Crawler/downloadPDFs.py		Code located at E:/McNair/Software/Google_Scholar_Crawler/downloadPDFs.py
		+
		+	'''11/02'''
		+
		+	Things are good! Today made the program so that we can get however many pages of search results we want and get the PDF links for all the ones we can see. Towards the end of the day, google scholar picked up that we were a robot and started blocking me. Hopefully the block goes away when I am back on Monday. Now working on parsing apart the txt file to go to the websites we saved and download the PDFs. Should not be particularly difficult.

	=Lauren's LOG=		=Lauren's LOG=

← Older revision		Revision as of 21:12, 10 October 2017
Line 60:		Line 60:

	Found a way to get past google scholar blocking my crawling so spent time writing selenium code. I can certainly download 10 search result BibTeXs when you search for a certain term automatically now which is awesome. I am part of the way through having the crawler save the pdf link once it has saved the BibTex for the search results. Yay selenium :')))		Found a way to get past google scholar blocking my crawling so spent time writing selenium code. I can certainly download 10 search result BibTeXs when you search for a certain term automatically now which is awesome. I am part of the way through having the crawler save the pdf link once it has saved the BibTex for the search results. Yay selenium :')))
		+
		+	Code located at E:/McNair/Software/Google_Scholar_Crawler/downloadPDFs.py

	=Lauren's LOG=		=Lauren's LOG=

← Older revision		Revision as of 21:11, 10 October 2017
Line 57:		Line 57:


		+	'''10/10'''
		+
		+	Found a way to get past google scholar blocking my crawling so spent time writing selenium code. I can certainly download 10 search result BibTeXs when you search for a certain term automatically now which is awesome. I am part of the way through having the crawler save the pdf link once it has saved the BibTex for the search results. Yay selenium :')))

	=Lauren's LOG=		=Lauren's LOG=

← Older revision		Revision as of 22:06, 3 October 2017
Line 40:		Line 40:

	2) Classifying papers based on the matrix of term appearances that the current program builds.		2) Classifying papers based on the matrix of term appearances that the current program builds.
		+
		+
		+	'''10/02'''
		+
		+	Program finally outputting something useful YAY. In FindKeyTerms.py (under McNair/Software/Google_Scholar_Crawler) I can input the path of a folder of txt files and it will scan all of them and seek they key words. It will put reports for every file in a new folder called KeyTerms that will appear in the input folder once the program terminates. An example file will be emailed to Lauren for corrections and adjustment. The file currently takes all the categories in the codification page and says 1) How many terms in that category appeared 2) How many times each of those terms appeared. At the bottom, it suggests potential definitions for patent thicket in the paper, but this part is pretty poor for now and needs adjustment. On the bright side, the program executes absurdly quickly and we can get through hundreds of files in less than a minute. In addition, while the program is running I am outputting a bag of words vector into a folder called WordBags in the input folder for future neural net usage to classify the papers. Need a training dataset that is relatively large.
		+
		+	Stuff to work on:
		+
		+	1) Neural net classification (computer suggesting which kind of paper it is)
		+
		+	2) Improving patent thicket definition finding
		+
		+	3) Finding the authors and having this as a contributing factor of the vectors
		+
		+	4) Potentially going back to the google scholar problem to try to find the PDFs automatically.
		+
		+

	=Lauren's LOG=		=Lauren's LOG=

← Older revision		Revision as of 17:25, 29 September 2017
Line 74:		Line 74:

	Cremers Schliessler (2014) - Patent Litigation Settlement in Germany - Why Parties Settle During Trial.pdf		Cremers Schliessler (2014) - Patent Litigation Settlement in Germany - Why Parties Settle During Trial.pdf
		+
		+
		+	09/28
		+
		+	I added a section to the PTLR Codification page titled "Individual Terms." Ed would like to have all downloaded papers searched for these terms and record the frequency of which they appear.

@@ Line 25: / Line 25: @@
 =Christy's LOG=
-/27
+'''09/27'''
 Created file FindKeyTerms.py in Software/Google_Scholar_Crawler which takes in a text file and returns counts of the key terms from the codification page.
@@ Line 31: / Line 31: @@
-/28
+'''09/28'''
 Thought that the pdf to text converter wasn't working, but realized that it does just sloooowly (70 papers converted overnight). Should be fine since we are still developing the rest of the code and we only need to convert them to txt once.