Crunchbase Accelerator Founders - Revision history

Ed at 17:41, 21 September 2020

2020-09-21T17:41:42Z

Ed at 17:34, 21 September 2020

2020-09-21T17:34:17Z

Ed at 17:19, 20 September 2020

2020-09-20T17:19:56Z

Ed at 17:12, 20 September 2020

2020-09-20T17:12:09Z

GraceTan at 21:28, 30 July 2018

2018-07-30T21:28:28Z

GraceTan at 21:27, 30 July 2018

2018-07-30T21:27:19Z

GraceTan at 21:27, 30 July 2018

2018-07-30T21:27:03Z

GraceTan at 21:23, 30 July 2018

2018-07-30T21:23:57Z

GraceTan at 21:08, 30 July 2018

2018-07-30T21:08:26Z

GraceTan at 21:07, 30 July 2018

2018-07-30T21:07:25Z

@@ Line 1: / Line 1: @@
 {{Project
-|Has project output=
+|Has project output=Data
 |Has sponsor=McNair Center
 |Has title=Crunchbase Accelerator Founders

@@ Line 1: / Line 1: @@
 {{Project
 |Has sponsor=McNair Center
 |Has title=Crunchbase Accelerator Founders

@@ Line 1: / Line 1: @@
 {{Project
 |Has title=Crunchbase Accelerator Founders
 |Has owner=Grace Tan,

@@ Line 1: / Line 1: @@
-{{McNair Projects
+{{Project
 |Has title=Crunchbase Accelerator Founders
 |Has owner=Grace Tan,

@@ Line 19: / Line 19: @@
 To get the founder UUIDs from each accelerator, input the accelerator UUID (or name all lowercase if its one work) into this link:
    https://api.crunchbase.com/v3.1/organizations/ + UUID of organization + ?relationships=founders&user_key=662e263576fe3e4ea5991edfbcfb9883
 ===scrapefounders.py===

@@ Line 15: / Line 15: @@
 ==Project Introduction==
 This project uses the Crunchbase Data and API to find founders of the accelerators we are interested in. We then take the founders and run their names through the LinkedIn Crawler to find information about them.
 ==Part 1: Getting Data==

@@ Line 44: / Line 44: @@
 Main function for the crawler which opens up a window and enters the known linkedin urls of each founder and puts the information into txt files. It then uses the search box to search for founder name + company and selects the href from the html and opens the profile on a new page. If a founder cannot be found(there are no search results), the founder name and company is put into unavailable_profiles.txt (might be called something else but should have "unavailable" in the name).
 ===linked_in_crawler.py===
@@ Line 50: / Line 49: @@
 Relies heavily on locating the xpath of an element of that we want on the webpage. I ran into a lot of difficulty with this because the xpath in the code no longer exists in the new linkedin html which includes dynamic ids. To get around this, I located different aspects that we could use to find an xpath to the element we were looking for and also located elements by their css.
 ===New Test Account===
@@ Line 56: / Line 54: @@
    Password: McNair2018
 Use the selenium computer on Rice Visitor wifi.
 After logging in a couple of times, LinkedIn will get suspicious and ask you to confirm that you are not a robot using reCaptcha. I got around this by delaying the program by 3 minutes so that I had time to complete the reCaptcha test. However, sometimes reCaptcha loses connection and it forces you to continue the tests which can be frustrating. When this happens, I disconnect and reconnect from the wifi as well as switch between the test accounts.
 I never figured out how to stop reCaptcha from losing connection so I spent a lot of time completing reCaptcha tests.

← Older revision		Revision as of 21:23, 30 July 2018
Line 42:		Line 42:

	Output: 3 txt files - founders_education.txt, founders_experience.txt, founders_main.txt		Output: 3 txt files - founders_education.txt, founders_experience.txt, founders_main.txt
		+
		+	Main function for the crawler which opens up a window and enters the known linkedin urls of each founder and puts the information into txt files. It then uses the search box to search for founder name + company and selects the href from the html and opens the profile on a new page. If a founder cannot be found(there are no search results), the founder name and company is put into unavailable_profiles.txt (might be called something else but should have "unavailable" in the name).


		+	===linked_in_crawler.py===
		+	This contains the LinkedInCrawler class that includes functions(login, logout, search, etc) that the driver calls to crawl the LinkedIn website.

		+	Relies heavily on locating the xpath of an element of that we want on the webpage. I ran into a lot of difficulty with this because the xpath in the code no longer exists in the new linkedin html which includes dynamic ids. To get around this, I located different aspects that we could use to find an xpath to the element we were looking for and also located elements by their css.


Line 55:		Line 60:

	I never figured out how to stop reCaptcha from losing connection so I spent a lot of time completing reCaptcha tests.		I never figured out how to stop reCaptcha from losing connection so I spent a lot of time completing reCaptcha tests.
−
−	~~==Code==~~
−	~~===linkedin_crawler_main.py===~~
−	Main function for the crawler which opens up a window and enters the known linkedin urls of each founder and puts the information into txt files. It then uses the search box to search for founder name + company and selects the href from the html and opens the profile on a new page. If a founder cannot be found(there are no search results), the founder name and company is put into unavailable_profiles.txt (might be called something else but should have "unavailable" in the name).
−
−	~~===linked_in_crawler.py===~~
−	Contains the LinkedInCrawler class that contains functions for login, logout, search, etc. Relies heavily on locating the xpath of an element of that we want on the webpage. I ran into a lot of difficulty with this because the xpath in the code no longer exists in the new linkedin html which includes dynamic ids. To get around this, I located different aspects that we could use to find an xpath to the element we were looking for and also located elements by their css.

@@ Line 39: / Line 39: @@
 This contains the main function that will run the LinkedIn Crawler. It includes two test accounts at the top which I went back and forth on to prevent LinkedIn from finding me.
-Inputs (set outside of function) : username(of test account), password(of test account), query_filepath(txt file that includes name of accelerator, first_name, last_name, linkedin_url)
+Inputs (set outside of function): username(of test account), password(of test account), query_filepath(txt file that includes name of accelerator, first_name, last_name, linkedin_url)
 Output: 3 txt files - founders_education.txt, founders_experience.txt, founders_main.txt

@@ Line 38: / Line 38: @@
 ===linkedin_crawler_main.py===
 This contains the main function that will run the LinkedIn Crawler. It includes two test accounts at the top which I went back and forth on to prevent LinkedIn from finding me.
-Inputs (set outside of function) : username(of test account), password(of test account). query_filepath(txt file that includes name of accelerator, first_name, last_name, linkedin_url).
 Output: 3 txt files - founders_education.txt, founders_experience.txt, founders_main.txt