479
edits
Changes
Jump to navigation
Jump to search
LinkedIn Crawler (Python) (view source)
Revision as of 16:05, 12 September 2017
, 16:05, 12 September 2017no edit summary
==Accelerator Founders Search==
'''These results are for the paper: The Jockey, The Horse, or the RaceTrack'''
Our LinkedIn Recruiter Pro account has expired. Unfortunately, it turns out that profiles cannot be viewed through LinkedIn if the target profile is 3rd degree away or further. However, a Google search on such a LinkedIn profile will still let you view the profile, provided that an account has been logged into prior to the search.
===Piggybacking Google===
In order to get our data, we will piggyback on Google's web crawler to work around the LinkedIn protective wall. The crawler begins by logging into our test LinkedIn Account (credentials displayed at the top), and then launching a Google search for each query. By adding "LinkedIn" before the query, and "Founder" after the query, we can turn up relevant search results. The top 5 results on Google search are explored, scraped, and saved.
We ended up not opting to use the Google method for various reasons.
===Crunchbase API===
Instead, we opted to use data from Crunchbase we have access to through a license. A wiki page on the crunchbase data and how to use the API can be found [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here]. The data can be accessed either through the web API (discussed on the Crunchbase Data wiki page), or through the bulk download we have in our SQL server.
The web API has the nice added feature of having a '''Founders''' section. The API returns a JSON when a GET request is submitted using the correct company identifier. The Founders section of this JSON contains information on the Founders of the accelerator if Crunchbase has said data. Details about the data can be found on the [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data Crunchbase Data Page].
The script that queried the API is called '''crunchbase_founders.py''' and can be found:
E:\McNair\Projects\Accelerators\crunchbase_founders.py
The resulting text file, called '''founders_linkedin.txt''', containing names and linkedin URLs of founders after messing around with the database can be found:
E:\McNair\Projects\Accelerators\founders_linkedin.txt
=Previous Posts about the LinkedIn Crawler=
