U.S. Seed Accelerators

McNair Project
U.S. Seed Accelerators
Project Information
Project Title	U.S. Seed Accelerators
Owner	Connor Rothschild
Start Date	06/18/2018
Deadline
Keywords	accelerators, data
Primary Billing
Notes	[[Has notes::Continuation of Accelerator Data]]
Has project status	Active
Is dependent on	Industry Classifier, Demo Day Page Parser, Accelerator Demo Day, Demo Day Page Google Classifier, Crunchbase Accelerator Equity, Crunchbase Accelerator Founders, Crunchbase Data
Subsumes:	Accelerator Data, Accelerator Seed List (Data)
	Copyright © 2016 edegan.com. All Rights Reserved.

Project Location

The master file can be found at

/bulk/McNair/Projects/Accelerators/Summer 2018/The File to Rule Them All.xlsx

Relevant Former Projects

This page serves as an updated and tidied version of the data and work presented on the Accelerator Seed List (Data) Project, which subsumed Accelerator Data. Both of these projects (and as a corollary, this project) are dependent on the Demo Day Page Parser, Industry Classifier, and the Whois Parser.

Update for Hira

After our Skype call, I did the following:

Recoding Founders

I've began recoding the Founders variable. I can categorize 232 of the 824 listed job titles (there are 466 distinct job titles)into 9 categories (the most frequent results as given in a PivotTable found in the sheet).

We will need to talk about how to categorize the most extraneous job titles.

Multiple campuses and cohorts

The File to Rule Them All contains an updated address variable. 139 of our 166 accelerators have addresses that are available online. The ones we could not find information for are left blank.

The collaborative sheet that Hira, Maxine, and I worked on to update this list can be accessed here:

https://docs.google.com/spreadsheets/d/1nktgJZfm3L8IsSCHgYbPasSdvKb7QHKxZp8K5Si2iNo/edit?usp=sharing

We have also added a new cohort list. Under The File To Rule Them All, the sheet Multiple Campuses contains the different locations for accelerators with multiple campuses. Column B can be used to filter out multiple location accelerators.

Fixed Google Sheet

We created a new sheet with only data we want to keep, and cleaned it up. That sheet is called "Good Data Only", at the same link:

https://docs.google.com/spreadsheets/d/16Suyp364lMkmUuUmK2dy_9MeSoS1X4DfFl3dYYDGPT4/edit?usp=sharing

I first used our "recap" and "announced" classification to standardize and fix the dates.

Columns N-R contain our new data. Please note that all of these columns are based on formulas and will be made erroneous if edited.
Column N is the # of weeks for an accelerator program, gathered via VLookup from The File to Rule Them All.
Column O is the Actual Date we want to record, and was gathered by subtracting the # of weeks from a date if the listed page was a recap.
Columns P and Q are the Month and Years stripped from Column O.
Finally, Column R is the season variable, as Ed said it should be coded.

We have also gone through and removed all bad data, all duplicates, and all rows without timing info. These is the most complete list possible.

Recode employee count

I have added a new column in Cohorts Final (of the File to Rule Them All) yet left the old column in case you would prefer to edit/classify differently. Column AB (emp_count_scale) contains a variable coded on a scale of 1 to 9, with each number corresponding to one of the employee_count classifications (1 the lowest, 9 the highest). The exact output can be modified (1 could instead be tiny, 2 be small... 9 be huge).

The employee count column is standardized and can easily be edited given some modification of the Excel formula.

Normalized investment amount

I've been trying to fix the investment amount. But I think its smart we discuss this before I move forward. I've done some tentative standardization (finding the average in a range if given), but so many accelerators "take up to __%" equity and "invest up to $___" that I think its smartest we think hard about how to standardize first. Also notable is that some accelerators say they provide "$____ up front and another $___ in follow up funding for each stage." How do we deal with these? Message me if you'd like to talk more about this.

I refrained from creating max and min investment columns lest spending time on the data before we discuss it.

Remaning to do

Founders Experience: code job title
Founders Education: remove unknowns, code degree and code major

Recent Work

Here's a project update on the work that has been done since coming to McNair. The most recent file is

/bulk/McNair/Projects/Accelerators/Summer 2018/The File to Rule Them All.xlsx

Merging Cohort Companies with Crunchbase Info

More information on this part of the project can be found on the page Merging Existing Data with Crunchbase.

The newest updated sheet of cohort company info is under the Cohorts Final sheet of The File to Rule Them All.xlsx.

Working with Maxine Tao, we have matched companies to their respective pages and information found in Crunchbase (via UUID). We ensured single matches by doing a 1-1-1-1 match with our data and with Crunchbase (using the Matcher). We then received additional information on 8092 companies. The following new information (on top of what we already had) is included in the sheet:

short_description
long_description
category_list (details the company's category)
category_group_list (a less refined, more all-encompassing category classification)
founded_on date
employee_count
linkedin_url
address

And the following information was also pulled from Crunchbase and merged with our existing data:

URL (was merged with courl cells)
city (was merged with colocation)
state_code (was merged with colocation)
country_code (was merged with colocation)
status (was merged with costatus)

The Equity Variables

Maxine Tao and I have added six new variables to the Accelerators Final sheet. Those variables are:

Terms of joining - terms of joining accelerator and important details about program
Equity? (1/0) - cells contain a 1 if the accelerator take equity, a 0 if an accelerator definitively does not, and is blank if we could not find that information
Equity Amount - the % of equity the accelerator will take (can sometimes be a range (eg. 5-7%))
Equity Amount Normalized - this copies the previous column but only keeps %>0, and if a range was given (e.g. 5-7%) it returns the average (e.g. 6%)
Investment Amount - the $ the accelerator invests in a company to begin, if relevant (also could be a range or a "up to $######")
Investment Notes - anything to comment on previous 4 columns

These six variables tell us more about the characteristics of accelerators; specifically, which ones take equity and which ones do not, and how much equity accelerators take.

Relevant information:

82 accelerators take equity, 42 do not, and we lack information for 37.
The average % of equity among accelerators who take equity (rough estimate--do not use for anything official) is 6.49% (got this number by only looking at accelerators who take equity, averaging equity amount for accelerators who report a range (e.g. 4%-10% equity would be coded as 7% equity) and took mean.

Matching Accelerators to UUIDs via Crunchbase

We've also added UUIDs for 163 of our 166 accelerators. The UUIDs can be found in Column AE of the Accelerators Final sheet.

The file with accelerators matched to Crunchbase UUIDs can be found at:

/bulk/McNair/Projects/Accelerators/Summer 2018/Accelerators and UUIDs.xlsx

This is the master file and should never be modified unless we find a UUID changed. ALL OTHER SHEETS with UUIDs are linked to this sheet so its changes will be reflected elsewhere.

More information can be found on the Crunchbase Data page.

Linking Accelerators to Founders/LinkedIn Crawling

Grace Tan got the LinkedIn Crawler (Python) to work, which means we currently have the following information about accelerator founders:

Current Job Title
Location
Employer
Job(s) Title
Dates Employed
Time Employed
Location of jobs
Extra Description
School Name
Degree Name
Major
Attended
Graduated
Societies

This information can be found in the various Founders sheets in The File to Rule Them All.

Finding Company URLs

Excel master datasets are in:

E:\McNair\Projects\Accelerators\Summer 2018

Code and files specific to this URL finder are in:

E:\McNair\Projects\Accelerators\Summer 2018\url finder

Results

I used STEP1_crawl.py and STEP2_findcorrecturl.py to add approximately 1000 more URLs into 'The File to Rule Them All.xlsx'.

Testing

In this file (sheet: 'Most Recent Merged Data' note that this is just a copy of 'Cohorts Final' in 'The File to Rule Them All'):

E:\McNair\Projects\Accelerators\Summer 2018\Merged W Crunchbase Data as of July 17.xlx

We filter for companies (~4000) that did not receive VC, are not in crunchbase, and do not have URLs. Using a Google crawler(STEP1_crawl.py) and URL matching script(STEP2_findcorrecturl.py), we will try to find as many URLs as possible.

To test, I ran about 40 companies from "smallcompanylist.txt", using only the company name as a search term and taking the first 4 valid results (see don't collect list in code). The google crawler and URL matcher was able to correctly identify around 20 URLs. It also misidentifies some URLs that look really similar to the company name, but it is accurate for the most part if the name is not too generic. I then tried to run the 20 unfound company names through the crawler again, but this time I used company name + startup as the search term. This did not identify any more correct URLs.

It seems reasonable to assume that if the company URL cannot be found within the first 4 valid search results, then that company probably does not have URL at all. This is the case for many of the unfound 20 URLs from my test run above.

Actual Run Info

The companies we needed to find URLs for are in a file called 'ACTUALNEEDEDCOMPANIES.txt'.

The first four results for every company, as found by STEP1_crawl.py, are in 'ACTUAL_crawled_company_urls.txt'.

The results after the matching done by STEP2_findcorrecturl.py, are in 'ACTUAL_finalurls.txt'.

Note that in the end, I decided to only take URLs that were given a match score of greater than 0.9 by setting this restriction in STEP2_findcorrecturl.py. Then I manually removed any duplicates/inaccurate results. If you want, you can set the threshold lower in STEP2 and use STEP3_clean.py to find the URL with the highest score for each company.

Using Python files

To use STEP1_crawl.py:

INPUT: a list of company names (or anything) you would like to find websites for by searching on google
OUTPUT: a list of company names and the top X number of results from google

1. Change LIST_FILEPATH in line 26 to be the name of the file that contains the list of things you would like to search.

2. Change NUMRESULT to be however many results you would like from Google.

3. Adjust DONT_COLLECT to include any websites that you don't want.

4. If you would like to add another search keyword, add this in line 87 which is queries.append(name + "whatever you want here")

5. Change line 127 to be the name of your output file.

To use STEP2_findcorrecturl.py:

INPUT: output file from STEP1
OUTPUT: a file formatted the same as the output of STEP1, but URLs that do not match over the threshold value you set will be replaced with "no match"

1. Change file f to be the output file name from STEP1. Change g to be the desired name of the output file for this part.

2. In the if statement on line 44, set your desired threshold. Note that anything greater than 0.6 is generally considered a decent match. It might be safer to use 0.75 and my use of 0.9 ensures almost exact matches. However, you should consider that if your list of companies (or whatever you are searching) includes really common names, then a 90%+ match might not be exactly what you're looking for.

To use STEP3_clean.py:

Note this is an optional step depending on the accuracy level you need and what kind of data you crawled earlier. I chose not to use this and instead set a more restrictive threshold in STEP2.

1. Change file f to be the output file from STEP2. Change g to be the desired name of the output file for this part.

Your output should be a text file containing the company name and the URL that had the highest assigned score in STEP2. In case of more than 1 URL with the highest score, the script should take the first one.

An Overview

This project will be used to determine which accelerators are the most effective at churning out successful startups, as well as what characteristics are exhibited by these accelerators. First, we need to gather as much data as we can about as many accelerators as we can in order to look at factors that differentiate successful vs. unsuccessful ventures. Next, we need to create a web crawling program which will gather information about accelerators across the world by accessing their websites and extracting information. I believe that our overall goal with this research project is to gain insight into the methods of successful accelerators, as well as to find out what exactly differentiates very successful accelerators from dead accelerators.

Helpful Links: http://seedrankings.com/

Remaining To Dos

The last update on Accelerator Seed List (Data) said the following needed to be done:

Cross-reference sheet with data from Peter's old accelerator consolidation file ("accelerator_data_noflag" and "accelerator_data" in "All Relevant Files") and fill in missing data
Variables that are 100% NOT in these 2 files:
- Cohort Breakout?
- Subtype
- Designed for Students?
- Campuses
- Stage
- Software Tech
- What stage do they look for?

(Outdated) Necessary Steps

Acquiring the necessary data to complete the Accelerator Master Variable List and the Cohort List will require the following (not necessarily in this order):

Step Zero: Connect to Crunchbase and Link Data

Complete - more info: Crunchbase Data

Step One: LinkedIn Founders Data

This project will begin by working with Grace Tan and Maxine Tao to connect accelerators to their founders and cohort companies using Crunchbase and LinkedIn crawlers. Grace and Maxine will go through Crunchbase and find the UUID for companies and their founders (reference Crunchbase Data, Crunchbase Accelerator Founders, Crunchbase Accelerator Equity). Connect them using SQL and feed the names of founders into our LinkedIn crawler (headed by Grace Tan).

The list of founders for accelerators can be found at

McNair/Projects/Accelerators/Fall 2017/founders_linkedin.txt

The Unfound Founders file codes a 0 for all companies not listed within the LinkedIn Founders file, and a 1 for those that do have founders listed.

Given the founders' names, we will then be able to use the LinkedIn Crawler (Python) to find the relevant details of an accelerator founder (education, work experience, etc.) This data on founders will help us solve the horse, jockey, racetrack question to detect what variables affect a startup's success (the accelerator, the founders, the environment/city).

Step Two: Linking Accelerators to Cohorts Using Investments on Crunchbase

In this step we focus on accelerators who take equity from the companies that engage in their program. We do this to prevent looking at accelerators who may also run funds/invest in various companies but do not take equity. This would provide us misleading results and lead us to believe some companies are in cohorts at accelerators that they are really not.

Maxine will acquire the list of accelerators who take equity from companies from the following sheet:

E://McNair/Projects/Accelerators/All Relevant Files/accelerator_data_noflag.txt

Looking at the file, however, shows that very few are actually categorized well and the equity variable is messy. Moving forward, we need to check/refine/fix this classification.

This file has 266 rows. The most recent, actual version of our accelerator database (found at E://McNair/Projects/Accelerators/Summer 2018/Connor Accelerator Work/Accelerator Master Variable List - Revised by Ed V2.xlsx under the sheet Master Variable List) only has 167 rows, meaning the accelerator_data_noflag.txt file has too many rows.

We will need to do a left join of Accelerator Master List with accelerator_data_noflag.txt to get rid of the accelerator names that are in accelerator_data_noflag but NOT in Accelerator Master List.

Once this is finished, we should have an “Equity” classification variable for every accelerator in Accelerator Master List. The accelerators that have a Y (or maybe it’s a 1) are companies that do take equity. These are the companies we’ll be able to do your Crunchbase work on to see when accelerators take equity.

We then look at the accelerators investments (or companies and the entities which invested in them), cross-reference the list of companies/accelerators, and once we find a match, we know that a company went through an accelerator and during which year they went through a cohort.

From this, we get the following data:

Accelerator a given company went through
Year said company went through a cohort/Specific cohort company went through

Step Three: Demo Day Crawler

This part of the project relies on the contributions of the wonderful Minh Le. Better documentation for the project can be found on the Demo Day Page Parser, Demo Day Page Google Classifier, and Accelerator Demo Day project pages.

Essentially, this part of the accelerator data project will use the Demo Day Page Parser to look through accelerator websites for pages which list a cohort's 'Demo Day', or the day in which accelerators present their companies to a group of special investors (here's an example FAQ page from Y Combinator). The Demo Day Page Google Classifier will then determine if the page is, in fact, a demo day page.

Given a cohort's demo day, we can gather a few pieces of key information (check with Ed to make sure this is the correct information to gather from Demo Days):

The date a cohort began/the season the cohort went through the accelerator
- This is acquired by looking at the cohort's demo day date, and subtracting the number of weeks/months of a cohort for that given accelerator. The length of a cohort can be found in the file:

E://McNair/Projects/Accelerators/Summer 2018/Connor Accelerator Work/Accelerator Master Variable List - Revised by Ed V2.xlsx

I assume we can also acquire the companies in a specific cohort, as in we will have the list of all companies in the (for example) Spring 2018 Cohort of Techstars.

Step Four: Non-profit Finder

More at Non-profit Finder

Another important step in this project is finding which accelerators are non-profits.

A comprehensive list of nonprofits taken from the IRS can be found here:

E://McNair/Projects/Accelerators/Summer 2018/Connor Accelerator Work/Nonprofits in US.xlsx

Warning: this file has 1 million rows

This file should be cross-referenced with the list of accelerators to find which ones are listed--those accelerators are non-profits. We'll need someone good in SQL to do this with a match.

Potential problem:

The names of listed non-profits will likely be somewhat different than the names of accelerators, because companies often file for tax exemption with different names than they show the public.

Workflow Image

(the color coding for this image is a very rough and preliminary estimate subject to change) I imagine the project will look something like this, in that it will require the following information to fully complete the project: