Changes

Jump to navigation Jump to search
===scholarcrawl.py===
====Overview====
This code is the work-in-progress replacement for downloadPDFs.py. The issue with downloadPDFs was that its impossible to discover the sweet spot of not being discovered by Google, since you cannot find any info online about how many clicks/ how fast gets you marked as a robot. scholarcrawl.py tries to work around the issue by catching every time Google stops us, and waiting 24 hours before trying again, leaving off on the same page you were stopped on previously. It is being tested as of Friday Dec 8, 2017. It is continuing to run as expected as of Dec 12, 2017 and has searched through 34 pages.
====How to Use====
272

edits

Navigation menu