http://www.edegan.com/mediawiki/api.php?action=feedcontributions&feedformat=atom&user=Peterjalbert 2026-05-17T10:53:54Z User contributions MediaWiki 1.34.2 http://www.edegan.com/mediawiki/index.php?title=Selenium_Documentation&diff=22387 2017-12-22T17:37:24Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has Image=selenium.jpg<br /> |Has title=Selenium Documentation<br /> |Has owner=Peter Jalbert,<br /> |Has project status=Active<br /> }}<br /> [http://www.seleniumhq.org/projects/webdriver/ Selenium Web Driver] is a framework often used for automated web application testing. It uses an API to launch a web browser and browse sites from the client's perspective. Popular Selenium bindings exist for [http://selenium-python.readthedocs.io/ Python], [http://seleniumhq.github.io/selenium/docs/api/java/index.html Java], [https://www.npmjs.com/package/selenium-webdriver Javascript], and other languages. This documentation covers Selenium Web Driver using Python3.<br /> <br /> ==Installation==<br /> A full list of installation documentation can be found [http://selenium-python.readthedocs.io/installation.html# here].<br /> <br /> This documentation assumes you have Python 3.6 or later installed. If you do not, visit the [https://www.python.org/downloads/ Python Download page].<br /> <br /> From the command line, enter<br /> pip install selenium<br /> <br /> <br /> ==The Basics==<br /> <br /> A folder with tutorial code can be found on the RDP in:<br /> E:\McNair\Software\Selenium Tutorial<br /> <br /> ===Launching a Driver===<br /> The first step is to launch a driver. This is an object that has information on the current page including its url and web elements, and is the object you interact with to do any sort of navigation. First, import the webdriver:<br /> from selenium import webdriver<br /> <br /> Then, create an instance of the web driver. The RDP has bindings for Google Chrome and Mozilla Firefox. The following will launch a web browser on Google Chrome.<br /> driver = webdriver.Chrome()<br /> <br /> The GET method is used to visit a website. The get() command in Selenium takes a string url.<br /> driver.get(&quot;http://www.google.com&quot;)<br /> <br /> From here, different methods can be used to interact with the page. Most interactions involve some type of exchange with a web element. Selenium comes with many different ways to locate specific elements. To see the attributes of the element you want to work with, it is often a good idea to visit that page on your own browser, right click on the element you want your program to interact with, and select INSPECT. This will bring up the developer console and display the HTML representation of that element. From there, you can use one of the following selectors that best matches what you need.<br /> <br /> ===Selectors===<br /> These functions deal with web elements on the current page the driver is on. Any function that contains find_element_by returns a single web element, and any function that contains find_elements_by returns a list of web elements.<br /> <br /> driver.find_element_by_class_name(class_name)<br /> This function takes a string class name of the element you're looking for, and finds the first element on the page that has that class name. If there is a possibility that more than one web element on the page has the same class, you are probably better of using find_elements_by_class_name.<br /> <br /> driver.find_element_by_name(name)<br /> This function takes a string name of the element you're looking for, and finds the first element on the page that has a name attribute matching the string. Similar to the find_element_by_class_name function, this is not your best bet if there are multiple objects with the same name attribute.<br /> <br /> driver.find_element_by_id(id)<br /> This function takes a string id of the element you're looking for, and finds the element on the page that has an id attribute matching the string. Since ids are guaranteed to be unique, this will always find the element you're looking for. This function is not helpful if the element you want to select does not have an id attribute.<br /> <br /> driver.find_element_by_xpath(xpath)<br /> This function takes an XPATH, and returns the first web element that matches the path. This should not be your first choice if many elements can share the same XPATH. Contrary to all the above functions, XPATH can be used to find any web element, regardless of its attributes. However, XPATH takes some time to learn, and is more complex than all of the above. After investing some time, XPATH is the most secure way to find the elements you're looking for.<br /> <br /> driver.find_elements_by_class_name(class_name)<br /> This is similar to find_element_by_class_name, except it returns a list of all matches with the class name. This allows you to iterate over the results or index them accordingly. This is often useful for search results, or any sort of list based queries.<br /> <br /> driver.find_elements_by_name(name)<br /> Same as find_element_by_name, except it returns a list of all matches.<br /> <br /> driver.find_elements_by_xpath(xpath)<br /> Same as find_element_by_xpath, but returns a list of all matches.<br /> <br /> A tutorial on XPATH can be found [https://www.w3schools.com/xml/xpath_intro.asp here].<br /> <br /> ===Javascript===<br /> If you are familiar with Javascript, you can inject Javascript into the driver to envoke certain behaviors. Simply use:<br /> driver.execute_script(someJavascriptCode)<br /> <br /> For example, the following could be used to scroll to the bottom of the page:<br /> driver.execute_script(&quot;window.scrollTo(0, document.body.scrollHeight);&quot;)<br /> <br /> ===Clicking===<br /> If an element is clickable (such as a link or a button), you can click on that element by doing:<br /> element.click()<br /> <br /> This will route you to the linked page, or execute the action of the button. The driver will now be on the new page, and commands will deal with elements on the new page.<br /> <br /> ===Web Elements===<br /> This area contains methods that can be called on a web element.<br /> <br /> element.get_attribute(attr)<br /> <br /> This function gets an attribute of the web element by using the attribute identifier. For example, element.get_attribute(&quot;href&quot;) would get the URL of a button or link.<br /> <br /> ===Open a New Window===<br /> OPTION 1: Cntrl + Click<br /> -----------------------<br /> <br /> This method will utilize Action Chains and Keys. Action Chains simply queue commands, and the string of commands are executed once the perform() method is called. First, import these dependencies:<br /> from selenium.webdriver.common.action_chains import ActionChains<br /> from selenium.webdriver.common.keys import Keys<br /> <br /> Next, select the element using one of the selectors listed above. We will assume this web element is stored in the variable called element. Then, add a key down command to Action Chains using the Shift Key to simulate the shift button being pressed down:<br /> ActionChains(driver).key_down(Keys.SHIFT).perform()<br /> <br /> The Shift button is now &quot;pressed&quot; down. A click will now be a Shift + Click.<br /> element.click()<br /> <br /> This will simulate a Shift + Click on a link. A new window should open with the new link.<br /> Execute a key up command with Action Chains on the Shift key so your future commands are not combined with shift:<br /> ActionChains(driver).key_up(Keys.SHIFT).perform()<br /> <br /> Now the browser is displaying a new window, but the driver thinks it is still on the previous window. To change this, access the window handlers:<br /> handles = driver.window_handles<br /> <br /> And switch the driver's &quot;focus&quot; to the new window (the last window in this list of windows is the newest one):<br /> driver.switch_to_window(handles[-1])<br /> <br /> Now you can execute commands on the new window! To exit this window, use:<br /> driver.close()<br /> <br /> When closing a window, you will also need to return the driver to the previous handler by repeating the code:<br /> handles = driver.window_handles<br /> driver.switch_to_window(handles[-1])<br /> <br /> <br /> OPTION 2: Open Blank Window + get()<br /> -----------------------------------<br /> <br /> This method finds URLs that you wish to traverse, and opens up a new window for each URL you would like to visit. It utlizies Javascript injection and the GET method.<br /> <br /> PROS: This method allows you to check a URL string for correctness before visiting the page. This can save you trouble from TimeOut Errors if someone has linked a rotten link on their webpage.<br /> <br /> This assumes you have a URL retrieved by using element.get_attribute(&quot;href&quot;) or some other method. We assume this is stored in the variable url.<br /> <br /> First, open a new window by injecting javascript:<br /> driver.execute_script(&quot;window.open('');&quot;)<br /> <br /> Now the browser is displaying a new window, but the driver thinks it is still on the previous window. To change this, access the window handlers:<br /> handles = driver.window_handles<br /> <br /> And switch the driver's &quot;focus&quot; to the new window (the last window in this list of windows is the newest one):<br /> driver.switch_to_window(handles[-1])<br /> <br /> Now, you can execute commands on the new window. To get to the webpage you want to visit:<br /> driver.get(url)<br /> <br /> Now you can interact with anything on the new page using the current driver. To exit this window, use:<br /> driver.close()<br /> <br /> When closing a window, you will also need to return the driver to the previous handler by repeating the code:<br /> handles = driver.window_handles<br /> driver.switch_to_window(handles[-1])<br /> <br /> <br /> ===Downloading Files===<br /> In general, I recommend using Selenium to do browsing, and use another method to download the file.<br /> driver.current_url<br /> <br /> This will get the current url. From there, standard libraries such as wget or urlretrieve can be used to download the file if the url ends in .pdf, or the html of the page if it is a regular webpage.<br /> <br /> If you are trying to retrieve a body of text, find the element using selectors. Then:<br /> element.text<br /> <br /> will retrieve the text in that element as a string. This can then be written to a file in any way you see fit.<br /> <br /> [http://stackabuse.com/download-files-with-python/ Helpful link for downloading files in python]<br /> <br /> <br /> <br /> ==Helpful Links==<br /> [https://stackoverflow.com/questions/22676/how-do-i-download-a-file-over-http-using-python How to Download a file in Python with a URL]<br /> <br /> [https://stackoverflow.com/questions/20986631/how-can-i-scroll-a-web-page-using-selenium-webdriver-in-python Selenium: Scroll a Webpage]<br /> <br /> [https://stackoverflow.com/questions/24795198/selenium-python-get-all-children-elements Selenium: Get Children Elements]<br /> <br /> ==Advanced==<br /> The folder for the Web Driver Executables can be found:<br /> C:\SeleniumDriver<br /> <br /> chromedriver.exe is an executable to launch Google Chrome, and geckodriver.exe is an executable to launch Mozilla Firefox. Any new drivers for different web browsers should be placed in this folder.</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Selenium_Documentation&diff=22386 2017-12-21T21:03:56Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has Image=selenium.jpg<br /> |Has title=Selenium Documentation<br /> |Has owner=Peter Jalbert,<br /> |Has project status=Active<br /> }}<br /> [http://www.seleniumhq.org/projects/webdriver/ Selenium Web Driver] is a framework often used for automated web application testing. It uses an API to launch a web browser and browse sites from the client's perspective. Popular Selenium bindings exist for [http://selenium-python.readthedocs.io/ Python], [http://seleniumhq.github.io/selenium/docs/api/java/index.html Java], [https://www.npmjs.com/package/selenium-webdriver Javascript], and other languages. This documentation covers Selenium Web Driver using Python3.<br /> <br /> ==Installation==<br /> A full list of installation documentation can be found [http://selenium-python.readthedocs.io/installation.html# here].<br /> <br /> This documentation assumes you have Python 3.6 or later installed. If you do not, visit the [https://www.python.org/downloads/ Python Download page].<br /> <br /> From the command line, enter<br /> pip install selenium<br /> <br /> <br /> ==The Basics==<br /> <br /> A folder with tutorial code can be found on the RDP in:<br /> E:\McNair\Software\Selenium Tutorial<br /> <br /> ===Launching a Driver===<br /> The first step is to launch a driver. This is an object that has information on the current page including its url and web elements, and is the object you interact with to do any sort of navigation. First, import the webdriver:<br /> from selenium import webdriver<br /> <br /> Then, create an instance of the web driver. The RDP has bindings for Google Chrome and Mozilla Firefox. The following will launch a web browser on Google Chrome.<br /> driver = webdriver.Chrome()<br /> <br /> The GET method is used to visit a website. The get() command in Selenium takes a string url.<br /> driver.get(&quot;http://www.google.com&quot;)<br /> <br /> From here, different methods can be used to interact with the page. Most interactions involve some type of exchange with a web element. Selenium comes with many different ways to locate specific elements. To see the attributes of the element you want to work with, it is often a good idea to visit that page on your own browser, right click on the element you want your program to interact with, and select INSPECT. This will bring up the developer console and display the HTML representation of that element. From there, you can use one of the following selectors that best matches what you need.<br /> <br /> ===Selectors===<br /> These functions deal with web elements on the current page the driver is on. Any function that contains find_element_by returns a single web element, and any function that contains find_elements_by returns a list of web elements.<br /> <br /> driver.find_element_by_class_name(class_name)<br /> This function takes a string class name of the element you're looking for, and finds the first element on the page that has that class name. If there is a possibility that more than one web element on the page has the same class, you are probably better of using find_elements_by_class_name.<br /> <br /> driver.find_element_by_name(name)<br /> This function takes a string name of the element you're looking for, and finds the first element on the page that has a name attribute matching the string. Similar to the find_element_by_class_name function, this is not your best bet if there are multiple objects with the same name attribute.<br /> <br /> driver.find_element_by_id(id)<br /> This function takes a string id of the element you're looking for, and finds the element on the page that has an id attribute matching the string. Since ids are guaranteed to be unique, this will always find the element you're looking for. This function is not helpful if the element you want to select does not have an id attribute.<br /> <br /> driver.find_element_by_xpath(xpath)<br /> This function takes an XPATH, and returns the first web element that matches the path. This should not be your first choice if many elements can share the same XPATH. Contrary to all the above functions, XPATH can be used to find any web element, regardless of its attributes. However, XPATH takes some time to learn, and is more complex than all of the above. After investing some time, XPATH is the most secure way to find the elements you're looking for.<br /> <br /> driver.find_elements_by_class_name(class_name)<br /> This is similar to find_element_by_class_name, except it returns a list of all matches with the class name. This allows you to iterate over the results or index them accordingly. This is often useful for search results, or any sort of list based queries.<br /> <br /> driver.find_elements_by_name(name)<br /> Same as find_element_by_name, except it returns a list of all matches.<br /> <br /> driver.find_elements_by_xpath(xpath)<br /> Same as find_element_by_xpath, but returns a list of all matches.<br /> <br /> A tutorial on XPATH can be found [https://www.w3schools.com/xml/xpath_intro.asp here].<br /> <br /> ===Javascript===<br /> If you are familiar with Javascript, you can inject Javascript into the driver to envoke certain behaviors. Simply use:<br /> driver.execute_script(someJavascriptCode)<br /> <br /> For example, the following could be used to scroll to the bottom of the page:<br /> driver.execute_script(&quot;window.scrollTo(0, document.body.scrollHeight);&quot;)<br /> <br /> ===Clicking===<br /> If an element is clickable (such as a link or a button), you can click on that element by doing:<br /> element.click()<br /> <br /> This will route you to the linked page, or execute the action of the button. The driver will now be on the new page, and commands will deal with elements on the new page.<br /> <br /> ===Open a New Window===<br /> OPTION 1: Cntrl + Click<br /> -----------------------<br /> <br /> This method will utilize Action Chains and Keys. Action Chains simply queue commands, and the string of commands are executed once the perform() method is called. First, import these dependencies:<br /> from selenium.webdriver.common.action_chains import ActionChains<br /> from selenium.webdriver.common.keys import Keys<br /> <br /> OPTION 2: Open Blank Window + get()<br /> -----------------------------------<br /> <br /> ===Downloading Files===<br /> In general, I recommend using Selenium to do browsing, and use another method to download the file.<br /> driver.current_url<br /> <br /> This will get the current url. From there, standard libraries such as wget or urlretrieve can be used to download the file if the url ends in .pdf, or the html of the page if it is a regular webpage.<br /> <br /> If you are trying to retrieve a body of text, find the element using selectors. Then:<br /> element.text<br /> <br /> will retrieve the text in that element as a string. This can then be written to a file in any way you see fit.<br /> <br /> [http://stackabuse.com/download-files-with-python/ Helpful link for downloading files in python]<br /> <br /> <br /> <br /> ==Helpful Links==<br /> [https://stackoverflow.com/questions/22676/how-do-i-download-a-file-over-http-using-python How to Download a file in Python with a URL]<br /> <br /> [https://stackoverflow.com/questions/20986631/how-can-i-scroll-a-web-page-using-selenium-webdriver-in-python Selenium: Scroll a Webpage]<br /> <br /> [https://stackoverflow.com/questions/24795198/selenium-python-get-all-children-elements Selenium: Get Children Elements]<br /> <br /> ==Advanced==<br /> The folder for the Web Driver Executables can be found:<br /> C:\SeleniumDriver<br /> <br /> chromedriver.exe is an executable to launch Google Chrome, and geckodriver.exe is an executable to launch Mozilla Firefox. Any new drivers for different web browsers should be placed in this folder.</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Selenium_Documentation&diff=22385 2017-12-21T20:48:51Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has Image=selenium.jpg<br /> |Has title=Selenium Documentation<br /> |Has owner=Peter Jalbert,<br /> |Has project status=Active<br /> }}<br /> [http://www.seleniumhq.org/projects/webdriver/ Selenium Web Driver] is a framework often used for automated web application testing. It uses an API to launch a web browser and browse sites from the client's perspective. Popular Selenium bindings exist for [http://selenium-python.readthedocs.io/ Python], [http://seleniumhq.github.io/selenium/docs/api/java/index.html Java], [https://www.npmjs.com/package/selenium-webdriver Javascript], and other languages. This documentation covers Selenium Web Driver using Python3.<br /> <br /> ==Installation==<br /> A full list of installation documentation can be found [http://selenium-python.readthedocs.io/installation.html# here].<br /> <br /> This documentation assumes you have Python 3.6 or later installed. If you do not, visit the [https://www.python.org/downloads/ Python Download page].<br /> <br /> From the command line, enter<br /> pip install selenium<br /> <br /> <br /> ==The Basics==<br /> <br /> A folder with tutorial code can be found on the RDP in:<br /> E:\McNair\Software\Selenium Tutorial<br /> <br /> ===Launching a Driver===<br /> The first step is to launch a driver. This is an object that has information on the current page including its url and web elements, and is the object you interact with to do any sort of navigation. First, import the webdriver:<br /> from selenium import webdriver<br /> <br /> Then, create an instance of the web driver. The RDP has bindings for Google Chrome and Mozilla Firefox. The following will launch a web browser on Google Chrome.<br /> driver = webdriver.Chrome()<br /> <br /> The GET method is used to visit a website. The get() command in Selenium takes a string url.<br /> driver.get(&quot;http://www.google.com&quot;)<br /> <br /> From here, different methods can be used to interact with the page. Most interactions involve some type of exchange with a web element. Selenium comes with many different ways to locate specific elements. To see the attributes of the element you want to work with, it is often a good idea to visit that page on your own browser, right click on the element you want your program to interact with, and select INSPECT. This will bring up the developer console and display the HTML representation of that element. From there, you can use one of the following selectors that best matches what you need.<br /> <br /> ===Selectors===<br /> These functions deal with web elements on the current page the driver is on. Any function that contains find_element_by returns a single web element, and any function that contains find_elements_by returns a list of web elements.<br /> <br /> driver.find_element_by_class_name(class_name)<br /> This function takes a string class name of the element you're looking for, and finds the first element on the page that has that class name. If there is a possibility that more than one web element on the page has the same class, you are probably better of using find_elements_by_class_name.<br /> <br /> driver.find_element_by_name(name)<br /> This function takes a string name of the element you're looking for, and finds the first element on the page that has a name attribute matching the string. Similar to the find_element_by_class_name function, this is not your best bet if there are multiple objects with the same name attribute.<br /> <br /> driver.find_element_by_id(id)<br /> This function takes a string id of the element you're looking for, and finds the element on the page that has an id attribute matching the string. Since ids are guaranteed to be unique, this will always find the element you're looking for. This function is not helpful if the element you want to select does not have an id attribute.<br /> <br /> driver.find_element_by_xpath(xpath)<br /> This function takes an XPATH, and returns the first web element that matches the path. This should not be your first choice if many elements can share the same XPATH. Contrary to all the above functions, XPATH can be used to find any web element, regardless of its attributes. However, XPATH takes some time to learn, and is more complex than all of the above. After investing some time, XPATH is the most secure way to find the elements you're looking for.<br /> <br /> driver.find_elements_by_class_name(class_name)<br /> This is similar to find_element_by_class_name, except it returns a list of all matches with the class name. This allows you to iterate over the results or index them accordingly. This is often useful for search results, or any sort of list based queries.<br /> <br /> driver.find_elements_by_name(name)<br /> Same as find_element_by_name, except it returns a list of all matches.<br /> <br /> driver.find_elements_by_xpath(xpath)<br /> Same as find_element_by_xpath, but returns a list of all matches.<br /> <br /> A tutorial on XPATH can be found [https://www.w3schools.com/xml/xpath_intro.asp here].<br /> <br /> ===Javascript===<br /> If you are familiar with Javascript, you can inject Javascript into the driver to envoke certain behaviors. Simply use:<br /> driver.execute_script(someJavascriptCode)<br /> <br /> For example, the following could be used to scroll to the bottom of the page:<br /> driver.execute_script(&quot;window.scrollTo(0, document.body.scrollHeight);&quot;)<br /> <br /> <br /> ==Helpful Links==<br /> [https://stackoverflow.com/questions/22676/how-do-i-download-a-file-over-http-using-python How to Download a file in Python with a URL]<br /> <br /> [https://stackoverflow.com/questions/20986631/how-can-i-scroll-a-web-page-using-selenium-webdriver-in-python Selenium: Scroll a Webpage]<br /> <br /> [https://stackoverflow.com/questions/24795198/selenium-python-get-all-children-elements Selenium: Get Children Elements]<br /> <br /> ==Advanced==<br /> The folder for the Web Driver Executables can be found:<br /> C:\SeleniumDriver<br /> <br /> chromedriver.exe is an executable to launch Google Chrome, and geckodriver.exe is an executable to launch Mozilla Firefox. Any new drivers for different web browsers should be placed in this folder.</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Selenium_Documentation&diff=22384 2017-12-21T20:47:37Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has image=selenium.jpg<br /> |Has title=Selenium Documentation<br /> |Has owner=Peter Jalbert,<br /> |Has project status=Active<br /> }}<br /> [http://www.seleniumhq.org/projects/webdriver/ Selenium Web Driver] is a framework often used for automated web application testing. It uses an API to launch a web browser and browse sites from the client's perspective. Popular Selenium bindings exist for [http://selenium-python.readthedocs.io/ Python], [http://seleniumhq.github.io/selenium/docs/api/java/index.html Java], [https://www.npmjs.com/package/selenium-webdriver Javascript], and other languages. This documentation covers Selenium Web Driver using Python3.<br /> <br /> ==Installation==<br /> A full list of installation documentation can be found [http://selenium-python.readthedocs.io/installation.html# here].<br /> <br /> This documentation assumes you have Python 3.6 or later installed. If you do not, visit the [https://www.python.org/downloads/ Python Download page].<br /> <br /> From the command line, enter<br /> pip install selenium<br /> <br /> <br /> ==The Basics==<br /> <br /> A folder with tutorial code can be found on the RDP in:<br /> E:\McNair\Software\Selenium Tutorial<br /> <br /> ===Launching a Driver===<br /> The first step is to launch a driver. This is an object that has information on the current page including its url and web elements, and is the object you interact with to do any sort of navigation. First, import the webdriver:<br /> from selenium import webdriver<br /> <br /> Then, create an instance of the web driver. The RDP has bindings for Google Chrome and Mozilla Firefox. The following will launch a web browser on Google Chrome.<br /> driver = webdriver.Chrome()<br /> <br /> The GET method is used to visit a website. The get() command in Selenium takes a string url.<br /> driver.get(&quot;http://www.google.com&quot;)<br /> <br /> From here, different methods can be used to interact with the page. Most interactions involve some type of exchange with a web element. Selenium comes with many different ways to locate specific elements. To see the attributes of the element you want to work with, it is often a good idea to visit that page on your own browser, right click on the element you want your program to interact with, and select INSPECT. This will bring up the developer console and display the HTML representation of that element. From there, you can use one of the following selectors that best matches what you need.<br /> <br /> ===Selectors===<br /> These functions deal with web elements on the current page the driver is on. Any function that contains find_element_by returns a single web element, and any function that contains find_elements_by returns a list of web elements.<br /> <br /> driver.find_element_by_class_name(class_name)<br /> This function takes a string class name of the element you're looking for, and finds the first element on the page that has that class name. If there is a possibility that more than one web element on the page has the same class, you are probably better of using find_elements_by_class_name.<br /> <br /> driver.find_element_by_name(name)<br /> This function takes a string name of the element you're looking for, and finds the first element on the page that has a name attribute matching the string. Similar to the find_element_by_class_name function, this is not your best bet if there are multiple objects with the same name attribute.<br /> <br /> driver.find_element_by_id(id)<br /> This function takes a string id of the element you're looking for, and finds the element on the page that has an id attribute matching the string. Since ids are guaranteed to be unique, this will always find the element you're looking for. This function is not helpful if the element you want to select does not have an id attribute.<br /> <br /> driver.find_element_by_xpath(xpath)<br /> This function takes an XPATH, and returns the first web element that matches the path. This should not be your first choice if many elements can share the same XPATH. Contrary to all the above functions, XPATH can be used to find any web element, regardless of its attributes. However, XPATH takes some time to learn, and is more complex than all of the above. After investing some time, XPATH is the most secure way to find the elements you're looking for.<br /> <br /> driver.find_elements_by_class_name(class_name)<br /> This is similar to find_element_by_class_name, except it returns a list of all matches with the class name. This allows you to iterate over the results or index them accordingly. This is often useful for search results, or any sort of list based queries.<br /> <br /> driver.find_elements_by_name(name)<br /> Same as find_element_by_name, except it returns a list of all matches.<br /> <br /> driver.find_elements_by_xpath(xpath)<br /> Same as find_element_by_xpath, but returns a list of all matches.<br /> <br /> A tutorial on XPATH can be found [https://www.w3schools.com/xml/xpath_intro.asp here].<br /> <br /> ===Javascript===<br /> If you are familiar with Javascript, you can inject Javascript into the driver to envoke certain behaviors. Simply use:<br /> driver.execute_script(someJavascriptCode)<br /> <br /> For example, the following could be used to scroll to the bottom of the page:<br /> driver.execute_script(&quot;window.scrollTo(0, document.body.scrollHeight);&quot;)<br /> <br /> <br /> ==Helpful Links==<br /> [https://stackoverflow.com/questions/22676/how-do-i-download-a-file-over-http-using-python How to Download a file in Python with a URL]<br /> <br /> [https://stackoverflow.com/questions/20986631/how-can-i-scroll-a-web-page-using-selenium-webdriver-in-python Selenium: Scroll a Webpage]<br /> <br /> [https://stackoverflow.com/questions/24795198/selenium-python-get-all-children-elements Selenium: Get Children Elements]<br /> <br /> ==Advanced==<br /> The folder for the Web Driver Executables can be found:<br /> C:\SeleniumDriver<br /> <br /> chromedriver.exe is an executable to launch Google Chrome, and geckodriver.exe is an executable to launch Mozilla Firefox. Any new drivers for different web browsers should be placed in this folder.</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=File:Selenium.jpg&diff=22383 2017-12-21T20:46:36Z

<p>Peterjalbert: </p> <hr /> <div></div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Peter_Jalbert_(Work_Log)&diff=22382 2017-12-21T15:58:46Z

<p>Peterjalbert: </p> <hr /> <div>===Fall 2017===<br /> &lt;onlyinclude&gt; <br /> <br /> [[Peter Jalbert]] [[Work Logs]] [[Peter Jalbert (Work Log)|(log page)]] <br /> <br /> 2017-12-21: Last minute adjustments to the Moroccan Data. Continued working on [http://mcnair.bakerinstitute.org/wiki/Selenium_Documentation Selenium Documentation].<br /> <br /> 2017-12-20: Working on Selenium Documentation. Wrote 2 demo files. Wiki Page is avaiable [http://mcnair.bakerinstitute.org/wiki/Selenium_Documentation here]. Created 3 spreadsheets for the Moroccan data.<br /> <br /> 2017-12-19: Finished fixing the Demo Day Crawler. Changed files and installed as appropriate to make linked in crawler compatible with the RDP. Removed some of the bells and whistles.<br /> <br /> 2017-12-18: Continued finding errors with the Demo Day Crawler analysis. Rewrote the parser to remove any search terms that were in the top 10000 most common English words according to Google. Finished uploading and submitting Moroccan data.<br /> <br /> 2017-12-15: Found errors with the Demo Day Crawler. Fixed scripts to download Moroccan Law Data.<br /> <br /> 2017-12-14: Uploading Morocco Parliament Written Questions. Creating script for next Morocco Parliament download. Begin writing Selenium documentation. Continuing to download TIGER data.<br /> <br /> 2017-12-06: Running Morocco Parliament Written Questions script. Analyzing Demo Day Crawler results. Continued downloading for TIGER geocoder.<br /> <br /> 2017-11-28: Debugging Morocco Parliament Crawler. Running Demo Day Crawler for all accelerators and 10 pages per accelerator. TIGER geocoder is back to Forbidden Error.<br /> <br /> 2017-11-27: Rerunning Morocco Parliament Crawler. Fixed KeyTerms.py and running it again. Continued downloading for TIGER geocoder.<br /> <br /> 2017-11-20: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-16: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-15: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote a script to extract counts that were greater than 2 from Keyword Matcher. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder]. Finished re-formatting work logs.<br /> <br /> 2017-11-14: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote an HTML to Text parser. See Parser Demo Day Page for file location. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder].<br /> <br /> 2017-11-13: Built [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser].<br /> <br /> 2017-11-09: Running demo version of Demo Day crawler (Accelerator Google Crawler). Fixing work log format. <br /> <br /> 2017-11-07: Created file with 0s and 1s detailing whether crunchbase has the founder information for an accelerator. Details posted as a TODO on [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List] page. Still waiting for feedback on the PostGIS installation from [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder]. Continued working on Accelerator Google Crawler.<br /> <br /> 2017-11-06: Contacted Geography Center for the US Census Bureau, [https://www.census.gov/geo/about/contact.html here], and began email exchange on PostGIS installation problems. Began working on the [http://mcnair.bakerinstitute.org/wiki/Selenium_Documentation Selenium Documentation]. Also began working on an Accelerator Google Crawler that will be used with Yang and ML to find Demo Days for cohort companies.<br /> <br /> 2017-11-01: Attempted to continue downloading, however ran into HTTP Forbidden errors. Listed the errors on the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder Page].<br /> <br /> 2017-10-31: Began downloading blocks of data for individual states for the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder] project. Wrote out the new wiki page for installation, and beginning to write documentation on usage.<br /> <br /> 2017-10-30: With Ed's help, was able to get the national data from Tiger installed onto a database server. The process required much jumping around and changing users, and all the things we learned are outlined in [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation#Editing_Users the database server documentation] under &quot;Editing Users&quot;.<br /> <br /> 2017-10-25: Continued working on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation TigerCoder Installation].<br /> <br /> 2017-10-24: Throw some addresses into a database, use address normalizer and geocoder. May need to install things. Details on the installation process can be found on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation page].<br /> <br /> 2017-10-23: Finished Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-19: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-18: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_DistrictHouston Innovation District Project].<br /> <br /> 2017-10-17: Constructed ArcGIS maps for the agglomeration project. Finished maps of points for every year in the state of California. Finished maps of Route 128. Began working on selenium Yelp crawler to get cafe locations within the 610-loop.<br /> <br /> 2017-10-16: Assisted Harrison on the USITC project. Looked for natural language processing tools to extract complaintants and defendants along with their location from case files. Experimented with pulling based on parts of speech tags, as well as using geotext or geograpy to pull locations from a case segment.<br /> <br /> 2017-10-13: Updated various project wiki pages.<br /> <br /> 2017-10-12: Continued work on Patent Thicket project, awaiting further project specs.<br /> <br /> 2017-10-05: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-04: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-02: Worked on ArcGIS data. See Harrison's Work Log for the details.<br /> <br /> 2017-09-28: Added collaborative editing feature to PyCharm. <br /> <br /> 2017-09-27: Worked on big database file.<br /> <br /> 2017-09-25: New task -- Create text file with company, description, and company type.<br /> #[http://mcnair.bakerinstitute.org/wiki/VC_Database_Rebuild VC Database Rebuild]<br /> #psql vcdb2<br /> #table name, sdccompanybasecore2<br /> #Combine with Crunchbasebulk<br /> <br /> #TODO: Write wiki on linkedin crawler, write wiki on creating accounts.<br /> <br /> 2017-09-21: Wrote wiki on Linkedin crawler, met with Laura about patents project. <br /> <br /> 2017-09-20: Finished running linkedin crawler. Transferred data to RDP. Will write wikis next.<br /> <br /> 2017-09-19: Began running linkedin crawler. Helped Yang create RDP account, get permissions, and get wiki setup.<br /> <br /> 2017-09-18: Finished implementation of Experience Crawler, continued working on Education Crawler for LinkedIn.<br /> <br /> 2017-09-14: Continued implementing LinkedIn Crawler for profiles.<br /> <br /> 2017-09-13: Implemented LinkedIn Crawler for main portion of profiles. Began working on crawling Experience section of profiles.<br /> <br /> 2017-09-12: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. Added to the wiki on this topic.<br /> <br /> 2017-09-11: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. <br /> <br /> 2017-09-06: Combined founders data retrieved with the Crunchbase API with the crunchbasebulk data to get linkedin urls for different accelerator founders. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-09-05: Post Harvey. Finished retrieving names from the Crunchbase API on founders. Next step is to query crunchbase bulk database to get linkedin urls. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-08-24: Began using the Crunchbase API to retrieve founder information for accelerators. Halfway through compiling a dictionary that translates accelerator names into proper Crunchbase API URLs.<br /> <br /> 2017-08-23: Decided with Ed to abandon LinkedIn crawling to retrieve accelerator founder data, and instead use crunchbase. Spent the day navigating the crunchbasebulk database, and seeing what useful information was contained in it.<br /> <br /> 2017-08-22: Discovered that LinkedIn Profiles cannot be viewed through LinkedIn if the target is 3rd degree or further. However, if entering LinkedIn through a Google search, the profile can still be viewed if the user has previously logged into LinkedIn. Devising a workaround crawler that utilizes Google search. Continued blog post [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) here] under Section 4.<br /> <br /> 2017-08-21: Began work on extracting founders for accelerators through LinkedIn Crawler. Discovered that Python3 is not installed on RDP, so the virtual environment for the project cannot be fired up. Continued working on Ubuntu machine.<br /> &lt;/onlyinclude&gt;<br /> <br /> ===Spring 2017===<br /> <br /> 2017-05-01: Continued work on HTML Parser. Uploaded all semester projects to git server.<br /> <br /> 2017-04-20: Finished the HTML Parser for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Ran HTML parser on accelerator founders. Data is stored in projects/accelerators/LinkedIn Founder Data.<br /> <br /> 2017-04-19: Made updates to the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler] Wikipage. Ran LinkedIn Crawler on accelerator data. Working on an html parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-18: Ran LinkedIn Crawler on matches between Crunchbase Snapshot and the accelerator data.<br /> <br /> 2017-04-17: Worked on ways to get correct search results from the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Worked on an HTML Parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-13: Worked on debugging the logout procedure for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Began formulation of process to search for founders of startups using a combination of the LinkedIn Crawler with the data resources from the [http://mcnair.bakerinstitute.org/wiki/Crunchbase_2013_Snapshot CrunchBase Snapshot].<br /> <br /> 2017-04-12: Work on bugs with the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler].<br /> <br /> 2017-04-11: Completed functional [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) crawler of LinkedIn Recruiter Pro]. Basic search functions work and download profile information for a given person. <br /> <br /> 2017-04-10: Began writing functioning crawler of LinkedIn. <br /> <br /> 2017-04-06: Continued working on debugging and documenting the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Wrote a test program that logs in, searches for a query, navigates through search pages, and logs out. Recruiter program can now login and search.<br /> <br /> 2017-04-05: Began work on the LinkedIn Crawler. Researched on launching Python Virtual Environment.<br /> <br /> 2017-04-03: Finished debugging points for the Enclosing Circle Algorithm. Added Command Line functionality to the Industry Classifier.<br /> <br /> 2017-03-29: Worked on debugging points for the Enclosing Circle Algorithm.<br /> <br /> 2017-03-28: Finished running the Enclosing Circle Algorithm. Worked on removing incorrect points from the data set(see above). <br /> <br /> 2017-03-27: Worked on debugging the Enclosing Circle Algorithm. Implemented a way to remove interior circles, and determined that translation to latitude and longitude coordinates resulted in slightly off center circles.<br /> <br /> 2017-03-23: Finished debugging the brute force algorithm for [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm]. Implemented a method to plot the points and circles on a graph. Analyzed runtime of the brute force algorithm.<br /> <br /> 2017-03-21: Coded a brute force algorithm for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-20: Worked on debugging the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-09: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Finished script to draw Enclosing Circles on a Google Map.<br /> <br /> 2017-03-08: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Created script to draw outcome of the Enclosing Circle Algorithm on Google Maps.<br /> <br /> 2017-03-07: Redetermined the top 50 cities which Enclosing Circle should be run on. Data on the [http://mcnair.bakerinstitute.org/wiki/Top_Cities_for_VC_Backed_Companies Top 50 Cities for VC Backed Companies can be found here.] Ran [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities.<br /> <br /> 2017-03-06: Ran script to determine the top 50 cities which Enclosing Circle should be run on. Fixed the VC Circles script to take in a new data format.<br /> <br /> 2017-03-02: Cleaned up data for the VC Circles Project. Created histogram of data in Excel. See [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Began work on the [http://mcnair.bakerinstitute.org/wiki/LinkedInCrawlerPython LinkedIn Crawler].<br /> <br /> 2017-03-01: Created statistics for the VC Circles Project.<br /> <br /> 2017-02-28: Finished downloading geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Found bug in Enclosing Circle Algorithm.<br /> <br /> 2017-02-27: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-23: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Installed C++ Compiler for Python. Ran tests on difference between Python and C wrapped Python.<br /> <br /> 2017-02-22: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Helped out with [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-21: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Researched into C++ Compilers for Python so that the Enclosing Circle Algorithm could be wrapped in C. Found a recommended one [https://www.microsoft.com/en-us/download/details.aspx?id=44266 here].<br /> <br /> 2017-02-20: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-16: Reworked [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] to create a file of geocoded data. Began work on wrapping the algorithm in C to improve speed.<br /> <br /> 2017-02-15: Finished [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] applied to the VC study. Enclosing Circle algorithm still needs adjustment, but the program runs with the temporary fixes.<br /> <br /> 2017-02-14: Worked on the application of the Enclosing Circle algorithm to the VC study. Working on bug fixes in the Enclosing Circle algorithm. Created wiki page for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-02-13: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-08: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-07: Fixed bugs in parse_cohort_data.py, the script for parsing the cohort data from the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Added descriptive statistics to cohort data excel file.<br /> <br /> 2017-02-02: Out sick, independent research and work from RDP. Brief research into the [http://jorgeg.scripts.mit.edu/homepage/wp-content/uploads/2016/03/Guzman-Stern-State-of-American-Entrepreneurship-FINAL.pdf Stern-Guzman algorithm]. Research into [http://mcnair.bakerinstitute.org/wiki/interactive_maps Interactive Maps]. No helpful additions to map embedding problem.<br /> <br /> 2017-02-01: Notes from Session with Ed: Project on US university patenting and entrepreneurship programs (writing code to identify universities in assignees), search Wikipedia (XML then bulk download), student pop, faculty pop, etc.<br /> Circle project for VC data will end up being a joint project to join accelerator data. <br /> Pull descriptions for VC. Founders of accelerators in linkedin. LinkedIn cannot be caught(pretend to not be a bot). Can eventually get academic backgrounds through linkedin. <br /> Pull business registration data, Stern/Guzman Algorithm. <br /> GIS ontop of geocoded data.<br /> Maps that works on wiki or blog (CartoDB), Maps API and R.<br /> NLP Projects, Description Classifier.<br /> <br /> 2017-01-31: Built WayBack Machine Crawler. Updated documentation for coordinates script. Updated profile page to include locations of code.<br /> <br /> <br /> 2017-01-30: Optimized enclosing circle algorithm through memoization. Developed script to read addresses from accelerator data and return latitude and longitude coordinates.<br /> <br /> 2017-01-26: Continued working on Google sitesearch project. Discovered crunchbase, changed project priority. Priority 1, split accelerator data up by flag, priority 2, use crunchbase to get web urls for cohorts, priority 3, make internet archive wayback machine driver. Located [http://mcnair.bakerinstitute.org/wiki/Whois_Parser Whois Parser].<br /> <br /> <br /> 2017-01-25: Finished parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Some data files still need proofreading as they are not in an acceptable format. Began working on Google sitesearch project.<br /> <br /> 2017-01-24: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Cohort data file created, debugging is almost complete. Will begin work on the google accelerator search soon.<br /> <br /> 2017-01-23: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Preliminary code is written, working on debugging.<br /> <br /> 2017-01-19: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project]. Created parser for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project], completed creation of final data set(yay!). Began working on cohort parser.<br /> <br /> 2017-01-18: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-13: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-12: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-11: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-10: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> <br /> ===Fall 2016===<br /> <br /> 2016-12-08: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-07: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-06: Learned how to use git. Committed software projects from the semester to the McNair git repository. Projects can be found at; [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Executive Order Crawler], [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Foreign Government Web Crawlers], [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) F6S Crawler and Parser].<br /> <br /> 2016-12-02: Built and ran web crawler for Center for Middle East Studies on Kuwait. Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-01: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Built tool for the [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report E&amp;I Governance Report Project] with Christy. Adds a column of data that shows whether or not the bill has been passed.<br /> <br /> 2016-11-29: Began pulling data from the accelerators listed [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here]. Made text files for about 18 accelerators.<br /> <br /> 2016-11-22: Transferred downloaded Morocco Written Bills to provided SeaGate Drive. Made a &quot;gentle&quot; F6S crawler to retrieve HTMLs of possible accelerator pages documented [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here].<br /> <br /> 2016-11-18: Converted Executive Order PDFs to text files using adobe acrobat DC. See [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Wikipage] for details.<br /> <br /> 2016-11-17: Wrote a crawler to retrieve information about executive orders, and their corresponding pdfs. They can be found [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report here.] Next step is to run code to convert the pdfs to text files, then use the parser fixed by Christy. <br /> <br /> 2016-11-15: Finished download of Moroccan Written Question pdfs. Wrote a parser with Christy to be used for parsing bills from Congress and eventually executive orders. Found bug in the system Python that was worked out and rebooted.<br /> <br /> 2016-11-11: Continued to download Moroccan data in the background. Attempted to find bug fixes for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] crawler.<br /> <br /> 2016-11-10: Continued to download Moroccan data and Kuwait data in the background. Began work on [http://mcnair.bakerinstitute.org/wiki/Google_Scholar_Crawler Google Scholar Crawler]. Wrote a crawler for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] to get the HTML files of hundreds of accelerators. The crawler ended up failing; it appears to have been due to HTTPS. <br /> <br /> 2016-11-08: Continued to download Moroccan data in the background. Finished writing code for the embedded files on the Kuwait Site. Spent time debugging the frame errors due to the dynamically generated content. Never found an answer to the bug, and instead found a workaround that sacrificed run time for the ability to work. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-04: Continued to download Moroccan data in the background. Finished writing initial Kuwait Web Crawler/Driver for the Middle East Studies Department. Middle East Studies Department asked for additional embedded files in the Kuwait website. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-03: Continued to download Moroccan data in the background. Dr. Egan fixed systems requirements to run the GovTrack Web Crawler. Made significant progress on the Kuwait Web Crawler/Driver for the Middle East Studies Department. <br /> <br /> 2016-11-01: Continued to download Moroccan data in the background. Went over code for GovTracker Web Crawler, continued learning Perl. [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] Began Kuwait Web Crawler/Driver.<br /> <br /> 2016-10-21: Continued to download data for the Moroccan Parliament Written and Oral Questions. Looked over [http://mcnair.bakerinstitute.org/wiki/Christy_Warden_(Twitter_Crawler_Application_1) Christy's Twitter Crawler] to see how I can be helpful. Dr. Egan asked me to think about how to potentially make multiple tools to get cohorts and other sorts of data from accelerator sites. See [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator List] He also asked me to look at the [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] for potential ideas on how to bring this project to fruition.<br /> <br /> 2016-10-20: Continued to download data for the Moroccan Parliament Written and Oral Questions. Updated Wiki page. Started working on Twitter project with Christy. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-18: Finished code for Oral Questions web driver and Written Questions web driver using selenium. Now, the data for the dates of questions can be found using the crawler, and the pdfs of the questions will be downloaded using selenium. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-14: Finished Oral Questions crawler. Finished Written Questions crawler. Waiting for further details on whether that data needs to be tweaked in any way. Updated the Moroccan Web Driver/Web Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-13: Completed download of Moroccan Bills. Working on either a web driver screenshot approach or a webcrawler approach to download the Moroccan oral and written questions data. Began building Web Crawler for Oral and Written Questions site. Edited Moroccan Web Driver/Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-11: Fixed arabic bug, files can now be saved with arabic titles. Monarchy bills downloaded and ready for shipment. House of Representatives Bill mostly downloaded, ratified bills prepared for download. Started learning scrapy library in python for web scraping. Discussed idea of screenshot-ing questions instead of scraping. <br /> <br /> 2016-10-07: Learned unicode and utf8 encoding and decoding in arabic. Still working on transforming an ascii url into printable unicode.<br /> <br /> 2016-10-06: Discussed with Dr. Elbadawy about the desired file names for Moroccan data download. The consensus was that the bill programs are ready to launch once the files can be named properly, and the questions data must be retrieved using a web crawler which I need to learn how to implement. The naming of files is currently drawing errors in going from arabic, to url, to download, to filename. Debugging in process. Also built a demo selenium program for Dr. Egan that drives the McNair blog site on an infinite loop.<br /> <br /> 2016-10-03: Moroccan Web Driver projects completed for driving of the Monarchy proposed bills, the House of Representatives proposed bills, and the Ratified bills sites. Begun process of devising a naming system for the files that does not require scraping. Tinkered with naming through regular expression parsing of the URL. Structure for the oral questions and written questions drivers is set up, but need fixes due to the differences in the sites. Fixed bug on McNair wiki for women's biz team where email was plain text instead of an email link. Took a glimpse at Kuwait Parliament website, and it appears to be very different from the Moroccan setup.<br /> <br /> 2016-09-30: Selenium program selects view pdf option from the website, and goes to the pdf webpage. Program then switches handle to the new page. CTRL S is sent to the page to launch save dialog window. Text cannot be sent to this window. Brainstorm ways around this issue. Explored Chrome Options for saving automatically without a dialog window. Looking into other libraries besides selenium that may help.<br /> <br /> 2016-09-29: Re-enroll in Microsoft Remote Desktop with proper authentication, set up Selenium environment and Komodo IDE on Remote Desktop, wrote program using Selenium that goes to a link and opens up the print dialog box. Developed computational recipe for a different approach to the problem.<br /> <br /> 2016-09-26: Set up Staff wiki page, work log page; registered for Slack, Microsoft Remote Desktop; downloaded Selenium on personal computer, read Selenium docs. Created wiki page for Moroccan Web Driver Project.<br /> <br /> '''Notes'''<br /> <br /> *Ed moved the Morocco Data to E:\McNair\Projects from C:\Users\PeterJ\Documents<br /> * C Drive files moved to E:\McNair\Users\PeterJ<br /> <br /> [[Category:Work Log]]</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Selenium_Documentation&diff=22381 2017-12-20T17:21:21Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Selenium Documentation<br /> |Has owner=Peter Jalbert,<br /> |Has project status=Active<br /> }}<br /> [http://www.seleniumhq.org/projects/webdriver/ Selenium Web Driver] is a framework often used for automated web application testing. It uses an API to launch a web browser and browse sites from the client's perspective. Popular Selenium bindings exist for [http://selenium-python.readthedocs.io/ Python], [http://seleniumhq.github.io/selenium/docs/api/java/index.html Java], [https://www.npmjs.com/package/selenium-webdriver Javascript], and other languages. This documentation covers Selenium Web Driver using Python3.<br /> <br /> ==Installation==<br /> A full list of installation documentation can be found [http://selenium-python.readthedocs.io/installation.html# here].<br /> <br /> This documentation assumes you have Python 3.6 or later installed. If you do not, visit the [https://www.python.org/downloads/ Python Download page].<br /> <br /> From the command line, enter<br /> pip install selenium<br /> <br /> <br /> ==The Basics==<br /> <br /> A folder with tutorial code can be found on the RDP in:<br /> E:\McNair\Software\Selenium Tutorial<br /> <br /> ===Launching a Driver===<br /> The first step is to launch a driver. This is an object that has information on the current page including its url and web elements, and is the object you interact with to do any sort of navigation. First, import the webdriver:<br /> from selenium import webdriver<br /> <br /> Then, create an instance of the web driver. The RDP has bindings for Google Chrome and Mozilla Firefox. The following will launch a web browser on Google Chrome.<br /> driver = webdriver.Chrome()<br /> <br /> The GET method is used to visit a website. The get() command in Selenium takes a string url.<br /> driver.get(&quot;http://www.google.com&quot;)<br /> <br /> From here, different methods can be used to interact with the page. Most interactions involve some type of exchange with a web element. Selenium comes with many different ways to locate specific elements. To see the attributes of the element you want to work with, it is often a good idea to visit that page on your own browser, right click on the element you want your program to interact with, and select INSPECT. This will bring up the developer console and display the HTML representation of that element. From there, you can use one of the following selectors that best matches what you need.<br /> <br /> ===Selectors===<br /> These functions deal with web elements on the current page the driver is on. Any function that contains find_element_by returns a single web element, and any function that contains find_elements_by returns a list of web elements.<br /> <br /> driver.find_element_by_class_name(class_name)<br /> This function takes a string class name of the element you're looking for, and finds the first element on the page that has that class name. If there is a possibility that more than one web element on the page has the same class, you are probably better of using find_elements_by_class_name.<br /> <br /> driver.find_element_by_name(name)<br /> This function takes a string name of the element you're looking for, and finds the first element on the page that has a name attribute matching the string. Similar to the find_element_by_class_name function, this is not your best bet if there are multiple objects with the same name attribute.<br /> <br /> driver.find_element_by_id(id)<br /> This function takes a string id of the element you're looking for, and finds the element on the page that has an id attribute matching the string. Since ids are guaranteed to be unique, this will always find the element you're looking for. This function is not helpful if the element you want to select does not have an id attribute.<br /> <br /> driver.find_element_by_xpath(xpath)<br /> This function takes an XPATH, and returns the first web element that matches the path. This should not be your first choice if many elements can share the same XPATH. Contrary to all the above functions, XPATH can be used to find any web element, regardless of its attributes. However, XPATH takes some time to learn, and is more complex than all of the above. After investing some time, XPATH is the most secure way to find the elements you're looking for.<br /> <br /> driver.find_elements_by_class_name(class_name)<br /> This is similar to find_element_by_class_name, except it returns a list of all matches with the class name. This allows you to iterate over the results or index them accordingly. This is often useful for search results, or any sort of list based queries.<br /> <br /> driver.find_elements_by_name(name)<br /> Same as find_element_by_name, except it returns a list of all matches.<br /> <br /> driver.find_elements_by_xpath(xpath)<br /> Same as find_element_by_xpath, but returns a list of all matches.<br /> <br /> A tutorial on XPATH can be found [https://www.w3schools.com/xml/xpath_intro.asp here].<br /> <br /> ===Javascript===<br /> If you are familiar with Javascript, you can inject Javascript into the driver to envoke certain behaviors. Simply use:<br /> driver.execute_script(someJavascriptCode)<br /> <br /> For example, the following could be used to scroll to the bottom of the page:<br /> driver.execute_script(&quot;window.scrollTo(0, document.body.scrollHeight);&quot;)<br /> <br /> <br /> ==Helpful Links==<br /> [https://stackoverflow.com/questions/22676/how-do-i-download-a-file-over-http-using-python How to Download a file in Python with a URL]<br /> <br /> [https://stackoverflow.com/questions/20986631/how-can-i-scroll-a-web-page-using-selenium-webdriver-in-python Selenium: Scroll a Webpage]<br /> <br /> [https://stackoverflow.com/questions/24795198/selenium-python-get-all-children-elements Selenium: Get Children Elements]<br /> <br /> ==Advanced==<br /> The folder for the Web Driver Executables can be found:<br /> C:\SeleniumDriver<br /> <br /> chromedriver.exe is an executable to launch Google Chrome, and geckodriver.exe is an executable to launch Mozilla Firefox. Any new drivers for different web browsers should be placed in this folder.</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Selenium_Documentation&diff=22380 2017-12-20T17:19:15Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Selenium Documentation<br /> |Has owner=Peter Jalbert,<br /> |Has project status=Active<br /> }}<br /> [http://www.seleniumhq.org/projects/webdriver/ Selenium Web Driver] is a framework often used for automated web application testing. It uses an API to launch a web browser and browse sites from the client's perspective. Popular Selenium bindings exist for [http://selenium-python.readthedocs.io/ Python], [http://seleniumhq.github.io/selenium/docs/api/java/index.html Java], [https://www.npmjs.com/package/selenium-webdriver Javascript], and other languages. This documentation covers Selenium Web Driver using Python3.<br /> <br /> ==Installation==<br /> A full list of installation documentation can be found [http://selenium-python.readthedocs.io/installation.html# here].<br /> <br /> This documentation assumes you have Python 3.6 or later installed. If you do not, visit the [https://www.python.org/downloads/ Python Download page].<br /> <br /> From the command line, enter<br /> pip install selenium<br /> <br /> <br /> ==The Basics==<br /> <br /> A folder with tutorial code can be found on the RDP in:<br /> E:\McNair\Software\Selenium Tutorial<br /> <br /> ===Launching a Driver===<br /> The first step is to launch a driver. This is an object that has information on the current page including its url and web elements, and is the object you interact with to do any sort of navigation. First, import the webdriver:<br /> from selenium import webdriver<br /> <br /> Then, create an instance of the web driver. The RDP has bindings for Google Chrome and Mozilla Firefox. The following will launch a web browser on Google Chrome.<br /> driver = webdriver.Chrome()<br /> <br /> The GET method is used to visit a website. The get() command in Selenium takes a string url.<br /> driver.get(&quot;http://www.google.com&quot;)<br /> <br /> From here, different methods can be used to interact with the page. Most interactions involve some type of exchange with a web element. Selenium comes with many different ways to locate specific elements. To see the attributes of the element you want to work with, it is often a good idea to visit that page on your own browser, right click on the element you want your program to interact with, and select INSPECT. This will bring up the developer console and display the HTML representation of that element. From there, you can use one of the following selectors that best matches what you need.<br /> <br /> ===Selectors===<br /> These functions deal with web elements on the current page the driver is on. Any function that contains find_element_by returns a single web element, and any function that contains find_elements_by returns a list of web elements.<br /> <br /> driver.find_element_by_class_name(class_name)<br /> This function takes a string class name of the element you're looking for, and finds the first element on the page that has that class name. If there is a possibility that more than one web element on the page has the same class, you are probably better of using find_elements_by_class_name.<br /> <br /> driver.find_element_by_name(name)<br /> This function takes a string name of the element you're looking for, and finds the first element on the page that has a name attribute matching the string. Similar to the find_element_by_class_name function, this is not your best bet if there are multiple objects with the same name attribute.<br /> <br /> driver.find_element_by_id(id)<br /> This function takes a string id of the element you're looking for, and finds the element on the page that has an id attribute matching the string. Since ids are guaranteed to be unique, this will always find the element you're looking for. This function is not helpful if the element you want to select does not have an id attribute.<br /> <br /> driver.find_element_by_xpath(xpath)<br /> This function takes an XPATH, and returns the first web element that matches the path. This should not be your first choice if many elements can share the same XPATH. Contrary to all the above functions, XPATH can be used to find any web element, regardless of its attributes. However, XPATH takes some time to learn, and is more complex than all of the above. After investing some time, XPATH is the most secure way to find the elements you're looking for.<br /> <br /> driver.find_elements_by_class_name(class_name)<br /> This is similar to find_element_by_class_name, except it returns a list of all matches with the class name. This allows you to iterate over the results or index them accordingly. This is often useful for search results, or any sort of list based queries.<br /> <br /> driver.find_elements_by_name(name)<br /> Same as find_element_by_name, except it returns a list of all matches.<br /> <br /> driver.find_elements_by_xpath(xpath)<br /> Same as find_element_by_xpath, but returns a list of all matches.<br /> <br /> A tutorial on XPATH can be found [https://www.w3schools.com/xml/xpath_intro.asp here].<br /> <br /> <br /> ==Helpful Links==<br /> [https://stackoverflow.com/questions/22676/how-do-i-download-a-file-over-http-using-python How to Download a file in Python with a URL]<br /> <br /> [https://stackoverflow.com/questions/20986631/how-can-i-scroll-a-web-page-using-selenium-webdriver-in-python Selenium: Scroll a Webpage]<br /> <br /> [https://stackoverflow.com/questions/24795198/selenium-python-get-all-children-elements Selenium: Get Children Elements]<br /> <br /> ==Advanced==<br /> The folder for the Web Driver Executables can be found:<br /> C:\SeleniumDriver<br /> <br /> chromedriver.exe is an executable to launch Google Chrome, and geckodriver.exe is an executable to launch Mozilla Firefox. Any new drivers for different web browsers should be placed in this folder.</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Selenium_Documentation&diff=22379 2017-12-20T16:58:32Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Selenium Documentation<br /> |Has owner=Peter Jalbert,<br /> |Has project status=Active<br /> }}<br /> [http://www.seleniumhq.org/projects/webdriver/ Selenium Web Driver] is a framework often used for automated web application testing. It uses an API to launch a web browser and browse sites from the client's perspective. Popular Selenium bindings exist for [http://selenium-python.readthedocs.io/ Python], [http://seleniumhq.github.io/selenium/docs/api/java/index.html Java], [https://www.npmjs.com/package/selenium-webdriver Javascript], and other languages. This documentation covers Selenium Web Driver using Python3.<br /> <br /> ==Installation==<br /> A full list of installation documentation can be found [http://selenium-python.readthedocs.io/installation.html# here].<br /> <br /> This documentation assumes you have Python 3.6 or later installed. If you do not, visit the [https://www.python.org/downloads/ Python Download page].<br /> <br /> From the command line, enter<br /> pip install selenium<br /> <br /> <br /> ==The Basics==<br /> <br /> A folder with tutorial code can be found on the RDP in:<br /> E:\McNair\Software\Selenium Tutorial<br /> <br /> ===Launching a Driver===<br /> The first step is to launch a driver. This is an object that has information on the current page including its url and web elements, and is the object you interact with to do any sort of navigation. First, import the webdriver:<br /> from selenium import webdriver<br /> <br /> Then, create an instance of the web driver. The RDP has bindings for Google Chrome and Mozilla Firefox. The following will launch a web browser on Google Chrome.<br /> driver = webdriver.Chrome()<br /> <br /> The GET method is used to visit a website. The get() command in Selenium takes a string url.<br /> driver.get(&quot;http://www.google.com&quot;)<br /> <br /> From here, different methods can be used to interact with the page. Most interactions involve some type of exchange with a web element. Selenium comes with many different ways to locate specific elements. To see the attributes of the element you want to work with, it is often a good idea to visit that page on your own browser, right click on the element you want your program to interact with, and select INSPECT. This will bring up the developer console and display the HTML representation of that element. From there, you can use one of the following selectors that best matches what you need.<br /> <br /> ===Selectors===<br /> <br /> <br /> ==Helpful Links==<br /> [https://stackoverflow.com/questions/22676/how-do-i-download-a-file-over-http-using-python How to Download a file in Python with a URL]<br /> <br /> [https://stackoverflow.com/questions/20986631/how-can-i-scroll-a-web-page-using-selenium-webdriver-in-python Selenium: Scroll a Webpage]<br /> <br /> [https://stackoverflow.com/questions/24795198/selenium-python-get-all-children-elements Selenium: Get Children Elements]<br /> <br /> ==Advanced==<br /> The folder for the Web Driver Executables can be found:<br /> C:\SeleniumDriver<br /> <br /> chromedriver.exe is an executable to launch Google Chrome, and geckodriver.exe is an executable to launch Mozilla Firefox. Any new drivers for different web browsers should be placed in this folder.</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Selenium_Documentation&diff=22378 2017-12-20T16:58:02Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Selenium Documentation<br /> |Has owner=Peter Jalbert,<br /> }}<br /> [http://www.seleniumhq.org/projects/webdriver/ Selenium Web Driver] is a framework often used for automated web application testing. It uses an API to launch a web browser and browse sites from the client's perspective. Popular Selenium bindings exist for [http://selenium-python.readthedocs.io/ Python], [http://seleniumhq.github.io/selenium/docs/api/java/index.html Java], [https://www.npmjs.com/package/selenium-webdriver Javascript], and other languages. This documentation covers Selenium Web Driver using Python3.<br /> <br /> ==Installation==<br /> A full list of installation documentation can be found [http://selenium-python.readthedocs.io/installation.html# here].<br /> <br /> This documentation assumes you have Python 3.6 or later installed. If you do not, visit the [https://www.python.org/downloads/ Python Download page].<br /> <br /> From the command line, enter<br /> pip install selenium<br /> <br /> <br /> ==The Basics==<br /> <br /> A folder with tutorial code can be found on the RDP in:<br /> E:\McNair\Software\Selenium Tutorial<br /> <br /> ===Launching a Driver===<br /> The first step is to launch a driver. This is an object that has information on the current page including its url and web elements, and is the object you interact with to do any sort of navigation. First, import the webdriver:<br /> from selenium import webdriver<br /> <br /> Then, create an instance of the web driver. The RDP has bindings for Google Chrome and Mozilla Firefox. The following will launch a web browser on Google Chrome.<br /> driver = webdriver.Chrome()<br /> <br /> The GET method is used to visit a website. The get() command in Selenium takes a string url.<br /> driver.get(&quot;http://www.google.com&quot;)<br /> <br /> From here, different methods can be used to interact with the page. Most interactions involve some type of exchange with a web element. Selenium comes with many different ways to locate specific elements. To see the attributes of the element you want to work with, it is often a good idea to visit that page on your own browser, right click on the element you want your program to interact with, and select INSPECT. This will bring up the developer console and display the HTML representation of that element. From there, you can use one of the following selectors that best matches what you need.<br /> <br /> ===Selectors===<br /> <br /> <br /> ==Helpful Links==<br /> [https://stackoverflow.com/questions/22676/how-do-i-download-a-file-over-http-using-python How to Download a file in Python with a URL]<br /> <br /> [https://stackoverflow.com/questions/20986631/how-can-i-scroll-a-web-page-using-selenium-webdriver-in-python Selenium: Scroll a Webpage]<br /> <br /> [https://stackoverflow.com/questions/24795198/selenium-python-get-all-children-elements Selenium: Get Children Elements]<br /> <br /> ==Advanced==<br /> The folder for the Web Driver Executables can be found:<br /> C:\SeleniumDriver<br /> <br /> chromedriver.exe is an executable to launch Google Chrome, and geckodriver.exe is an executable to launch Mozilla Firefox. Any new drivers for different web browsers should be placed in this folder.</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Selenium_Documentation&diff=22377 2017-12-20T16:23:54Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Selenium Documentation<br /> |Has owner=Peter Jalbert,<br /> }}<br /> [http://www.seleniumhq.org/projects/webdriver/ Selenium Web Driver] is a framework often used for automated web application testing. It uses an API to launch a web browser and browse sites from the client's perspective. Popular Selenium bindings exist for [http://selenium-python.readthedocs.io/ Python], [http://seleniumhq.github.io/selenium/docs/api/java/index.html Java], [https://www.npmjs.com/package/selenium-webdriver Javascript], and other languages. This documentation covers Selenium Web Driver using Python3.<br /> <br /> ==Installation==<br /> A full list of installation documentation can be found [http://selenium-python.readthedocs.io/installation.html# here].<br /> <br /> This documentation assumes you have Python 3.6 or later installed. If you do not, visit the [https://www.python.org/downloads/ Python Download page].<br /> <br /> From the command line, enter<br /> pip install selenium<br /> <br /> <br /> ==The Basics==<br /> <br /> A folder with tutorial code can be found on the RDP in:<br /> E:\McNair\Software\Selenium Tutorial<br /> <br /> ===Launching a Driver===<br /> The first step is to launch a driver. This is an object that has information on the current page including its url and web elements, and is the object you interact with to do any sort of navigation. First, import the webdriver:<br /> from selenium import webdriver<br /> <br /> ==Helpful Links==<br /> [https://stackoverflow.com/questions/22676/how-do-i-download-a-file-over-http-using-python How to Download a file in Python with a URL]<br /> <br /> [https://stackoverflow.com/questions/20986631/how-can-i-scroll-a-web-page-using-selenium-webdriver-in-python Selenium: Scroll a Webpage]<br /> <br /> [https://stackoverflow.com/questions/24795198/selenium-python-get-all-children-elements Selenium: Get Children Elements]</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Peter_Jalbert_(Work_Log)&diff=22376 2017-12-20T16:21:13Z

<p>Peterjalbert: </p> <hr /> <div>===Fall 2017===<br /> &lt;onlyinclude&gt; <br /> <br /> [[Peter Jalbert]] [[Work Logs]] [[Peter Jalbert (Work Log)|(log page)]] <br /> <br /> 2017-12-20: Working on Selenium Documentation. Wrote 2 demo files. Wiki Page is avaiable [http://mcnair.bakerinstitute.org/wiki/Selenium_Documentation here].<br /> <br /> 2017-12-19: Finished fixing the Demo Day Crawler. Changed files and installed as appropriate to make linked in crawler compatible with the RDP. Removed some of the bells and whistles.<br /> <br /> 2017-12-18: Continued finding errors with the Demo Day Crawler analysis. Rewrote the parser to remove any search terms that were in the top 10000 most common English words according to Google. Finished uploading and submitting Moroccan data.<br /> <br /> 2017-12-15: Found errors with the Demo Day Crawler. Fixed scripts to download Moroccan Law Data.<br /> <br /> 2017-12-14: Uploading Morocco Parliament Written Questions. Creating script for next Morocco Parliament download. Begin writing Selenium documentation. Continuing to download TIGER data.<br /> <br /> 2017-12-06: Running Morocco Parliament Written Questions script. Analyzing Demo Day Crawler results. Continued downloading for TIGER geocoder.<br /> <br /> 2017-11-28: Debugging Morocco Parliament Crawler. Running Demo Day Crawler for all accelerators and 10 pages per accelerator. TIGER geocoder is back to Forbidden Error.<br /> <br /> 2017-11-27: Rerunning Morocco Parliament Crawler. Fixed KeyTerms.py and running it again. Continued downloading for TIGER geocoder.<br /> <br /> 2017-11-20: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-16: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-15: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote a script to extract counts that were greater than 2 from Keyword Matcher. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder]. Finished re-formatting work logs.<br /> <br /> 2017-11-14: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote an HTML to Text parser. See Parser Demo Day Page for file location. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder].<br /> <br /> 2017-11-13: Built [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser].<br /> <br /> 2017-11-09: Running demo version of Demo Day crawler (Accelerator Google Crawler). Fixing work log format. <br /> <br /> 2017-11-07: Created file with 0s and 1s detailing whether crunchbase has the founder information for an accelerator. Details posted as a TODO on [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List] page. Still waiting for feedback on the PostGIS installation from [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder]. Continued working on Accelerator Google Crawler.<br /> <br /> 2017-11-06: Contacted Geography Center for the US Census Bureau, [https://www.census.gov/geo/about/contact.html here], and began email exchange on PostGIS installation problems. Began working on the [http://mcnair.bakerinstitute.org/wiki/Selenium_Documentation Selenium Documentation]. Also began working on an Accelerator Google Crawler that will be used with Yang and ML to find Demo Days for cohort companies.<br /> <br /> 2017-11-01: Attempted to continue downloading, however ran into HTTP Forbidden errors. Listed the errors on the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder Page].<br /> <br /> 2017-10-31: Began downloading blocks of data for individual states for the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder] project. Wrote out the new wiki page for installation, and beginning to write documentation on usage.<br /> <br /> 2017-10-30: With Ed's help, was able to get the national data from Tiger installed onto a database server. The process required much jumping around and changing users, and all the things we learned are outlined in [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation#Editing_Users the database server documentation] under &quot;Editing Users&quot;.<br /> <br /> 2017-10-25: Continued working on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation TigerCoder Installation].<br /> <br /> 2017-10-24: Throw some addresses into a database, use address normalizer and geocoder. May need to install things. Details on the installation process can be found on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation page].<br /> <br /> 2017-10-23: Finished Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-19: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-18: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_DistrictHouston Innovation District Project].<br /> <br /> 2017-10-17: Constructed ArcGIS maps for the agglomeration project. Finished maps of points for every year in the state of California. Finished maps of Route 128. Began working on selenium Yelp crawler to get cafe locations within the 610-loop.<br /> <br /> 2017-10-16: Assisted Harrison on the USITC project. Looked for natural language processing tools to extract complaintants and defendants along with their location from case files. Experimented with pulling based on parts of speech tags, as well as using geotext or geograpy to pull locations from a case segment.<br /> <br /> 2017-10-13: Updated various project wiki pages.<br /> <br /> 2017-10-12: Continued work on Patent Thicket project, awaiting further project specs.<br /> <br /> 2017-10-05: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-04: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-02: Worked on ArcGIS data. See Harrison's Work Log for the details.<br /> <br /> 2017-09-28: Added collaborative editing feature to PyCharm. <br /> <br /> 2017-09-27: Worked on big database file.<br /> <br /> 2017-09-25: New task -- Create text file with company, description, and company type.<br /> #[http://mcnair.bakerinstitute.org/wiki/VC_Database_Rebuild VC Database Rebuild]<br /> #psql vcdb2<br /> #table name, sdccompanybasecore2<br /> #Combine with Crunchbasebulk<br /> <br /> #TODO: Write wiki on linkedin crawler, write wiki on creating accounts.<br /> <br /> 2017-09-21: Wrote wiki on Linkedin crawler, met with Laura about patents project. <br /> <br /> 2017-09-20: Finished running linkedin crawler. Transferred data to RDP. Will write wikis next.<br /> <br /> 2017-09-19: Began running linkedin crawler. Helped Yang create RDP account, get permissions, and get wiki setup.<br /> <br /> 2017-09-18: Finished implementation of Experience Crawler, continued working on Education Crawler for LinkedIn.<br /> <br /> 2017-09-14: Continued implementing LinkedIn Crawler for profiles.<br /> <br /> 2017-09-13: Implemented LinkedIn Crawler for main portion of profiles. Began working on crawling Experience section of profiles.<br /> <br /> 2017-09-12: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. Added to the wiki on this topic.<br /> <br /> 2017-09-11: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. <br /> <br /> 2017-09-06: Combined founders data retrieved with the Crunchbase API with the crunchbasebulk data to get linkedin urls for different accelerator founders. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-09-05: Post Harvey. Finished retrieving names from the Crunchbase API on founders. Next step is to query crunchbase bulk database to get linkedin urls. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-08-24: Began using the Crunchbase API to retrieve founder information for accelerators. Halfway through compiling a dictionary that translates accelerator names into proper Crunchbase API URLs.<br /> <br /> 2017-08-23: Decided with Ed to abandon LinkedIn crawling to retrieve accelerator founder data, and instead use crunchbase. Spent the day navigating the crunchbasebulk database, and seeing what useful information was contained in it.<br /> <br /> 2017-08-22: Discovered that LinkedIn Profiles cannot be viewed through LinkedIn if the target is 3rd degree or further. However, if entering LinkedIn through a Google search, the profile can still be viewed if the user has previously logged into LinkedIn. Devising a workaround crawler that utilizes Google search. Continued blog post [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) here] under Section 4.<br /> <br /> 2017-08-21: Began work on extracting founders for accelerators through LinkedIn Crawler. Discovered that Python3 is not installed on RDP, so the virtual environment for the project cannot be fired up. Continued working on Ubuntu machine.<br /> &lt;/onlyinclude&gt;<br /> <br /> ===Spring 2017===<br /> <br /> 2017-05-01: Continued work on HTML Parser. Uploaded all semester projects to git server.<br /> <br /> 2017-04-20: Finished the HTML Parser for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Ran HTML parser on accelerator founders. Data is stored in projects/accelerators/LinkedIn Founder Data.<br /> <br /> 2017-04-19: Made updates to the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler] Wikipage. Ran LinkedIn Crawler on accelerator data. Working on an html parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-18: Ran LinkedIn Crawler on matches between Crunchbase Snapshot and the accelerator data.<br /> <br /> 2017-04-17: Worked on ways to get correct search results from the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Worked on an HTML Parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-13: Worked on debugging the logout procedure for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Began formulation of process to search for founders of startups using a combination of the LinkedIn Crawler with the data resources from the [http://mcnair.bakerinstitute.org/wiki/Crunchbase_2013_Snapshot CrunchBase Snapshot].<br /> <br /> 2017-04-12: Work on bugs with the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler].<br /> <br /> 2017-04-11: Completed functional [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) crawler of LinkedIn Recruiter Pro]. Basic search functions work and download profile information for a given person. <br /> <br /> 2017-04-10: Began writing functioning crawler of LinkedIn. <br /> <br /> 2017-04-06: Continued working on debugging and documenting the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Wrote a test program that logs in, searches for a query, navigates through search pages, and logs out. Recruiter program can now login and search.<br /> <br /> 2017-04-05: Began work on the LinkedIn Crawler. Researched on launching Python Virtual Environment.<br /> <br /> 2017-04-03: Finished debugging points for the Enclosing Circle Algorithm. Added Command Line functionality to the Industry Classifier.<br /> <br /> 2017-03-29: Worked on debugging points for the Enclosing Circle Algorithm.<br /> <br /> 2017-03-28: Finished running the Enclosing Circle Algorithm. Worked on removing incorrect points from the data set(see above). <br /> <br /> 2017-03-27: Worked on debugging the Enclosing Circle Algorithm. Implemented a way to remove interior circles, and determined that translation to latitude and longitude coordinates resulted in slightly off center circles.<br /> <br /> 2017-03-23: Finished debugging the brute force algorithm for [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm]. Implemented a method to plot the points and circles on a graph. Analyzed runtime of the brute force algorithm.<br /> <br /> 2017-03-21: Coded a brute force algorithm for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-20: Worked on debugging the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-09: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Finished script to draw Enclosing Circles on a Google Map.<br /> <br /> 2017-03-08: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Created script to draw outcome of the Enclosing Circle Algorithm on Google Maps.<br /> <br /> 2017-03-07: Redetermined the top 50 cities which Enclosing Circle should be run on. Data on the [http://mcnair.bakerinstitute.org/wiki/Top_Cities_for_VC_Backed_Companies Top 50 Cities for VC Backed Companies can be found here.] Ran [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities.<br /> <br /> 2017-03-06: Ran script to determine the top 50 cities which Enclosing Circle should be run on. Fixed the VC Circles script to take in a new data format.<br /> <br /> 2017-03-02: Cleaned up data for the VC Circles Project. Created histogram of data in Excel. See [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Began work on the [http://mcnair.bakerinstitute.org/wiki/LinkedInCrawlerPython LinkedIn Crawler].<br /> <br /> 2017-03-01: Created statistics for the VC Circles Project.<br /> <br /> 2017-02-28: Finished downloading geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Found bug in Enclosing Circle Algorithm.<br /> <br /> 2017-02-27: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-23: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Installed C++ Compiler for Python. Ran tests on difference between Python and C wrapped Python.<br /> <br /> 2017-02-22: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Helped out with [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-21: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Researched into C++ Compilers for Python so that the Enclosing Circle Algorithm could be wrapped in C. Found a recommended one [https://www.microsoft.com/en-us/download/details.aspx?id=44266 here].<br /> <br /> 2017-02-20: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-16: Reworked [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] to create a file of geocoded data. Began work on wrapping the algorithm in C to improve speed.<br /> <br /> 2017-02-15: Finished [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] applied to the VC study. Enclosing Circle algorithm still needs adjustment, but the program runs with the temporary fixes.<br /> <br /> 2017-02-14: Worked on the application of the Enclosing Circle algorithm to the VC study. Working on bug fixes in the Enclosing Circle algorithm. Created wiki page for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-02-13: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-08: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-07: Fixed bugs in parse_cohort_data.py, the script for parsing the cohort data from the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Added descriptive statistics to cohort data excel file.<br /> <br /> 2017-02-02: Out sick, independent research and work from RDP. Brief research into the [http://jorgeg.scripts.mit.edu/homepage/wp-content/uploads/2016/03/Guzman-Stern-State-of-American-Entrepreneurship-FINAL.pdf Stern-Guzman algorithm]. Research into [http://mcnair.bakerinstitute.org/wiki/interactive_maps Interactive Maps]. No helpful additions to map embedding problem.<br /> <br /> 2017-02-01: Notes from Session with Ed: Project on US university patenting and entrepreneurship programs (writing code to identify universities in assignees), search Wikipedia (XML then bulk download), student pop, faculty pop, etc.<br /> Circle project for VC data will end up being a joint project to join accelerator data. <br /> Pull descriptions for VC. Founders of accelerators in linkedin. LinkedIn cannot be caught(pretend to not be a bot). Can eventually get academic backgrounds through linkedin. <br /> Pull business registration data, Stern/Guzman Algorithm. <br /> GIS ontop of geocoded data.<br /> Maps that works on wiki or blog (CartoDB), Maps API and R.<br /> NLP Projects, Description Classifier.<br /> <br /> 2017-01-31: Built WayBack Machine Crawler. Updated documentation for coordinates script. Updated profile page to include locations of code.<br /> <br /> <br /> 2017-01-30: Optimized enclosing circle algorithm through memoization. Developed script to read addresses from accelerator data and return latitude and longitude coordinates.<br /> <br /> 2017-01-26: Continued working on Google sitesearch project. Discovered crunchbase, changed project priority. Priority 1, split accelerator data up by flag, priority 2, use crunchbase to get web urls for cohorts, priority 3, make internet archive wayback machine driver. Located [http://mcnair.bakerinstitute.org/wiki/Whois_Parser Whois Parser].<br /> <br /> <br /> 2017-01-25: Finished parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Some data files still need proofreading as they are not in an acceptable format. Began working on Google sitesearch project.<br /> <br /> 2017-01-24: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Cohort data file created, debugging is almost complete. Will begin work on the google accelerator search soon.<br /> <br /> 2017-01-23: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Preliminary code is written, working on debugging.<br /> <br /> 2017-01-19: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project]. Created parser for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project], completed creation of final data set(yay!). Began working on cohort parser.<br /> <br /> 2017-01-18: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-13: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-12: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-11: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-10: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> <br /> ===Fall 2016===<br /> <br /> 2016-12-08: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-07: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-06: Learned how to use git. Committed software projects from the semester to the McNair git repository. Projects can be found at; [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Executive Order Crawler], [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Foreign Government Web Crawlers], [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) F6S Crawler and Parser].<br /> <br /> 2016-12-02: Built and ran web crawler for Center for Middle East Studies on Kuwait. Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-01: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Built tool for the [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report E&amp;I Governance Report Project] with Christy. Adds a column of data that shows whether or not the bill has been passed.<br /> <br /> 2016-11-29: Began pulling data from the accelerators listed [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here]. Made text files for about 18 accelerators.<br /> <br /> 2016-11-22: Transferred downloaded Morocco Written Bills to provided SeaGate Drive. Made a &quot;gentle&quot; F6S crawler to retrieve HTMLs of possible accelerator pages documented [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here].<br /> <br /> 2016-11-18: Converted Executive Order PDFs to text files using adobe acrobat DC. See [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Wikipage] for details.<br /> <br /> 2016-11-17: Wrote a crawler to retrieve information about executive orders, and their corresponding pdfs. They can be found [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report here.] Next step is to run code to convert the pdfs to text files, then use the parser fixed by Christy. <br /> <br /> 2016-11-15: Finished download of Moroccan Written Question pdfs. Wrote a parser with Christy to be used for parsing bills from Congress and eventually executive orders. Found bug in the system Python that was worked out and rebooted.<br /> <br /> 2016-11-11: Continued to download Moroccan data in the background. Attempted to find bug fixes for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] crawler.<br /> <br /> 2016-11-10: Continued to download Moroccan data and Kuwait data in the background. Began work on [http://mcnair.bakerinstitute.org/wiki/Google_Scholar_Crawler Google Scholar Crawler]. Wrote a crawler for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] to get the HTML files of hundreds of accelerators. The crawler ended up failing; it appears to have been due to HTTPS. <br /> <br /> 2016-11-08: Continued to download Moroccan data in the background. Finished writing code for the embedded files on the Kuwait Site. Spent time debugging the frame errors due to the dynamically generated content. Never found an answer to the bug, and instead found a workaround that sacrificed run time for the ability to work. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-04: Continued to download Moroccan data in the background. Finished writing initial Kuwait Web Crawler/Driver for the Middle East Studies Department. Middle East Studies Department asked for additional embedded files in the Kuwait website. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-03: Continued to download Moroccan data in the background. Dr. Egan fixed systems requirements to run the GovTrack Web Crawler. Made significant progress on the Kuwait Web Crawler/Driver for the Middle East Studies Department. <br /> <br /> 2016-11-01: Continued to download Moroccan data in the background. Went over code for GovTracker Web Crawler, continued learning Perl. [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] Began Kuwait Web Crawler/Driver.<br /> <br /> 2016-10-21: Continued to download data for the Moroccan Parliament Written and Oral Questions. Looked over [http://mcnair.bakerinstitute.org/wiki/Christy_Warden_(Twitter_Crawler_Application_1) Christy's Twitter Crawler] to see how I can be helpful. Dr. Egan asked me to think about how to potentially make multiple tools to get cohorts and other sorts of data from accelerator sites. See [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator List] He also asked me to look at the [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] for potential ideas on how to bring this project to fruition.<br /> <br /> 2016-10-20: Continued to download data for the Moroccan Parliament Written and Oral Questions. Updated Wiki page. Started working on Twitter project with Christy. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-18: Finished code for Oral Questions web driver and Written Questions web driver using selenium. Now, the data for the dates of questions can be found using the crawler, and the pdfs of the questions will be downloaded using selenium. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-14: Finished Oral Questions crawler. Finished Written Questions crawler. Waiting for further details on whether that data needs to be tweaked in any way. Updated the Moroccan Web Driver/Web Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-13: Completed download of Moroccan Bills. Working on either a web driver screenshot approach or a webcrawler approach to download the Moroccan oral and written questions data. Began building Web Crawler for Oral and Written Questions site. Edited Moroccan Web Driver/Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-11: Fixed arabic bug, files can now be saved with arabic titles. Monarchy bills downloaded and ready for shipment. House of Representatives Bill mostly downloaded, ratified bills prepared for download. Started learning scrapy library in python for web scraping. Discussed idea of screenshot-ing questions instead of scraping. <br /> <br /> 2016-10-07: Learned unicode and utf8 encoding and decoding in arabic. Still working on transforming an ascii url into printable unicode.<br /> <br /> 2016-10-06: Discussed with Dr. Elbadawy about the desired file names for Moroccan data download. The consensus was that the bill programs are ready to launch once the files can be named properly, and the questions data must be retrieved using a web crawler which I need to learn how to implement. The naming of files is currently drawing errors in going from arabic, to url, to download, to filename. Debugging in process. Also built a demo selenium program for Dr. Egan that drives the McNair blog site on an infinite loop.<br /> <br /> 2016-10-03: Moroccan Web Driver projects completed for driving of the Monarchy proposed bills, the House of Representatives proposed bills, and the Ratified bills sites. Begun process of devising a naming system for the files that does not require scraping. Tinkered with naming through regular expression parsing of the URL. Structure for the oral questions and written questions drivers is set up, but need fixes due to the differences in the sites. Fixed bug on McNair wiki for women's biz team where email was plain text instead of an email link. Took a glimpse at Kuwait Parliament website, and it appears to be very different from the Moroccan setup.<br /> <br /> 2016-09-30: Selenium program selects view pdf option from the website, and goes to the pdf webpage. Program then switches handle to the new page. CTRL S is sent to the page to launch save dialog window. Text cannot be sent to this window. Brainstorm ways around this issue. Explored Chrome Options for saving automatically without a dialog window. Looking into other libraries besides selenium that may help.<br /> <br /> 2016-09-29: Re-enroll in Microsoft Remote Desktop with proper authentication, set up Selenium environment and Komodo IDE on Remote Desktop, wrote program using Selenium that goes to a link and opens up the print dialog box. Developed computational recipe for a different approach to the problem.<br /> <br /> 2016-09-26: Set up Staff wiki page, work log page; registered for Slack, Microsoft Remote Desktop; downloaded Selenium on personal computer, read Selenium docs. Created wiki page for Moroccan Web Driver Project.<br /> <br /> '''Notes'''<br /> <br /> *Ed moved the Morocco Data to E:\McNair\Projects from C:\Users\PeterJ\Documents<br /> * C Drive files moved to E:\McNair\Users\PeterJ<br /> <br /> [[Category:Work Log]]</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Selenium_Documentation&diff=22375 2017-12-20T16:20:02Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Selenium Documentation<br /> |Has owner=Peter Jalbert,<br /> }}<br /> [http://www.seleniumhq.org/projects/webdriver/ Selenium Web Driver] is a framework often used for automated web application testing. It uses an API to launch a web browser and browse sites from the client's perspective. Popular Selenium bindings exist for [http://selenium-python.readthedocs.io/ Python], [http://seleniumhq.github.io/selenium/docs/api/java/index.html Java], [https://www.npmjs.com/package/selenium-webdriver Javascript], and other languages. This documentation covers Selenium Web Driver using Python3.<br /> <br /> ==Installation==<br /> A full list of installation documentation can be found [http://selenium-python.readthedocs.io/installation.html# here].<br /> <br /> This documentation assumes you have Python 3.6 or later installed. If you do not, visit the [https://www.python.org/downloads/ Python Download page].<br /> <br /> From the command line, enter<br /> pip install selenium<br /> <br /> <br /> ==The Basics==<br /> <br /> A folder with tutorial code can be found on the RDP in:<br /> E:\McNair\Software\Selenium Tutorial<br /> <br /> ==Helpful Links==<br /> [https://stackoverflow.com/questions/22676/how-do-i-download-a-file-over-http-using-python How to Download a file in Python with a URL]<br /> <br /> [https://stackoverflow.com/questions/20986631/how-can-i-scroll-a-web-page-using-selenium-webdriver-in-python Selenium: Scroll a Webpage]<br /> <br /> [https://stackoverflow.com/questions/24795198/selenium-python-get-all-children-elements Selenium: Get Children Elements]</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Selenium_Documentation&diff=22374 2017-12-20T16:19:48Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Selenium Documentation<br /> |Has owner=Peter Jalbert,<br /> }}<br /> [http://www.seleniumhq.org/projects/webdriver/ Selenium Web Driver] is a framework often used for automated web application testing. It uses an API to launch a web browser and browse sites from the client's perspective. Popular Selenium bindings exist for [http://selenium-python.readthedocs.io/ Python], [http://seleniumhq.github.io/selenium/docs/api/java/index.html Java], [https://www.npmjs.com/package/selenium-webdriver Javascript], and other languages. This documentation covers Selenium Web Driver using Python3.<br /> <br /> ==Installation==<br /> A full list of installation documentation can be found [http://selenium-python.readthedocs.io/installation.html# here].<br /> <br /> This documentation assumes you have Python 3.6 or later installed. If you do not, visit the [https://www.python.org/downloads/ Python Download page].<br /> <br /> From the command line, enter<br /> pip install selenium<br /> <br /> <br /> ==The Basics==<br /> <br /> A folder with tutorial code can be found on the RDP in:<br /> E:\McNair\Software\Selenium Tutorial<br /> <br /> ==Helpful Links==<br /> [https://stackoverflow.com/questions/22676/how-do-i-download-a-file-over-http-using-python How to Download a file in Python with a URL]<br /> [https://stackoverflow.com/questions/20986631/how-can-i-scroll-a-web-page-using-selenium-webdriver-in-python Selenium: Scroll a Webpage]<br /> [https://stackoverflow.com/questions/24795198/selenium-python-get-all-children-elements Selenium: Get Children Elements]</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Demo_Day_Page_Parser&diff=22373 2017-12-19T19:35:13Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Demo Day Page Parser<br /> |Has owner=Peter Jalbert,<br /> |Has project status=Active<br /> }}<br /> <br /> ==Project Specs==<br /> The goal of this project is to leverage data mining with Selenium and Machine Learning to get good candidate web pages for Demo Days for accelerators. Relevant information on the project can be found on the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Data Accelerator Data] page.<br /> <br /> ==Code Location==<br /> The code directory for this project can be found:<br /> E:\McNair\Software\Accelerators<br /> <br /> The Selenium-based crawler can be found in the file below. This script runs a google search on accelerator names and keywords, and saves the urls and html pages for future use:<br /> DemoDayCrawler.py<br /> <br /> <br /> A script to rip from HTML to TXT can be found below. This script reads HTML files from the DemoDayHTML directory, and writes them to the DemoDayTxt directory:<br /> htmlToText.py<br /> <br /> A script to match Keywords (Accelerator and Cohort names) against the resulting text pages can be found in KeyTerms.py. The script takes the Keywords located in CohortAndAcceleratorsFullList.txt, and the text files in DemoDayTxt, and creates a file with the number of matches of each keyword against each text file.<br /> <br /> The script can be found:<br /> KeyTerms.py<br /> <br /> The Keyword matches text file can be found:<br /> DemoDayTxt\KeyTermFile\KeyTerms.txt<br /> <br /> A script to determine the text files of webpages that have at least one hit of these key words can be found:<br /> DemoDayHits.py<br /> <br /> ==Downloading HTML Files with Selenium==<br /> The code for utilizing Selenium to download HTML files can be found in the DemoDayCrawler.py file.<br /> <br /> The initial observation set over the data scraped 100 links for each of 20 sample accelerators from the list of overall accelerators. These sample pages were turned to text, and scored to remove web pages with no mention of relevant accelerators or companies.<br /> <br /> Once the process was tweaked in response to the initial sample testing, the process ran again over all accelerators. The test determined that we needed take no more than 10 links for each accelerator, and that 'Demo Day' was a suitable search term.<br /> <br /> COMPLETE FILES<br /> -----------------<br /> These files hold data for all the accelerators: not just the test set.<br /> <br /> The full list of accelerators:<br /> ListOfAccs.txt<br /> <br /> The full list of search terms to match with the text versions of news articles:<br /> CohortAndAcceleratorsFullList.txt<br /> <br /> A list of accelerators, queries, and urls:<br /> demoday_crawl_full.txt<br /> <br /> A directory with HTML files for all accelerator demo day results:<br /> DemoDayHTMLFull<br /> <br /> A directory with TXT files for all accelerator demo day results:<br /> DemoDayTxtFull<br /> <br /> A file with the name of the results that passed keyword matching:<br /> DemoDayHitsFull.txt<br /> <br /> A file with an analysis of the most frequent matched words in each text file:<br /> topWordsFull.txt<br /> <br /> ==Faulty Results==<br /> The first pass through the data revealed articles that had thousands of hits for keyword matches. This seemed highly suspicious, so we dug in deeper to investigate the cause of this issue. <br /> <br /> The following script in the same directory analyzes the keyword matches to determine the words with the highest number of hits.<br /> DemoDayAnalysis.py<br /> <br /> After investigation, it was found that many company names were taken after common english words. Here are some of the companies causing issues along with their associated accelerator:<br /> <br /> the, L-Spark<br /> <br /> Matter, This., [https://matter.vc/portfolio/this/ website]<br /> <br /> Fledge, HERE, [http://fledge.co/fledgling/here/ website]<br /> <br /> StartupBootCamp, We...<br /> <br /> LightBank Start, Zero<br /> <br /> Entrepreneurs Roundtable Accelerator, SELECT<br /> <br /> Y Combinator, Her<br /> <br /> Y Combinator, Final<br /> <br /> AngelCube, class<br /> <br /> Matter, common<br /> <br /> L-Spark, Company<br /> <br /> Techstars, Hot<br /> <br /> Rather than removing these companies from the list of search terms, we opted to not include as search terms any words that were considered among the top 10000 most common English words. For reference, we used the top 10000 most common English words according to a Google research study. The github documentation of the study can be found [https://github.com/first20hours/google-10000-english here].<br /> <br /> The file containing the 10000 most common English words can be found:<br /> E:\McNair\Software\Accelerators\10000_common_words.txt<br /> <br /> The results seemed much more plausible after removing these words. Some company words still appeared many times, but in the correct context.</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Demo_Day_Page_Parser&diff=22372 2017-12-19T19:34:03Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Demo Day Page Parser<br /> |Has owner=Peter Jalbert,<br /> |Has project status=Active<br /> }}<br /> <br /> ==Project Specs==<br /> The goal of this project is to leverage data mining with Selenium and Machine Learning to get good candidate web pages for Demo Days for accelerators. Relevant information on the project can be found on the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Data Accelerator Data] page.<br /> <br /> ==Code Location==<br /> The code directory for this project can be found:<br /> E:\McNair\Software\Accelerators<br /> <br /> The Selenium-based crawler can be found in the file below. This script runs a google search on accelerator names and keywords, and saves the urls and html pages for future use:<br /> DemoDayCrawler.py<br /> <br /> <br /> A script to rip from HTML to TXT can be found below. This script reads HTML files from the DemoDayHTML directory, and writes them to the DemoDayTxt directory:<br /> htmlToText.py<br /> <br /> A script to match Keywords (Accelerator and Cohort names) against the resulting text pages can be found in KeyTerms.py. The script takes the Keywords located in CohortAndAcceleratorsFullList.txt, and the text files in DemoDayTxt, and creates a file with the number of matches of each keyword against each text file.<br /> <br /> The script can be found:<br /> KeyTerms.py<br /> <br /> The Keyword matches text file can be found:<br /> DemoDayTxt\KeyTermFile\KeyTerms.txt<br /> <br /> A script to determine the text files of webpages that have at least one hit of these key words can be found:<br /> DemoDayHits.py<br /> <br /> ==Downloading HTML Files with Selenium==<br /> The code for utilizing Selenium to download HTML files can be found in the DemoDayCrawler.py file.<br /> <br /> The initial observation set over the data scraped 100 links for each of 20 sample accelerators from the list of overall accelerators. These sample pages were turned to text, and scored to remove web pages with no mention of relevant accelerators or companies.<br /> <br /> Once the process was tweaked in response to the initial sample testing, the process ran again over all accelerators. The test determined that we needed take no more than 10 links for each accelerator, and that 'Demo Day' was a suitable search term.<br /> <br /> COMPLETE FILES<br /> -----------------<br /> These files hold data for all the accelerators: not just the test set.<br /> <br /> The full list of accelerators:<br /> ListOfAccs.txt<br /> <br /> The full list of search terms to match with the text versions of news articles:<br /> CohortAndAcceleratorsFullList.txt<br /> <br /> A list of accelerators, queries, and urls:<br /> demoday_crawl_full.txt<br /> <br /> A directory with HTML files for all accelerator demo day results:<br /> DemoDayHTMLFull<br /> <br /> A directory with TXT files for all accelerator demo day results:<br /> DemoDayTxtFull<br /> <br /> A file with the name of the results that passed keyword matching:<br /> DemoDayHitsFull.txt<br /> <br /> A file with an analysis of the most frequent matched words in each text file:<br /> topWordsNew.txt<br /> <br /> ==Faulty Results==<br /> The first pass through the data revealed articles that had thousands of hits for keyword matches. This seemed highly suspicious, so we dug in deeper to investigate the cause of this issue. <br /> <br /> The following script in the same directory analyzes the keyword matches to determine the words with the highest number of hits.<br /> DemoDayAnalysis.py<br /> <br /> After investigation, it was found that many company names were taken after common english words. Here are some of the companies causing issues along with their associated accelerator:<br /> <br /> the, L-Spark<br /> <br /> Matter, This., [https://matter.vc/portfolio/this/ website]<br /> <br /> Fledge, HERE, [http://fledge.co/fledgling/here/ website]<br /> <br /> StartupBootCamp, We...<br /> <br /> LightBank Start, Zero<br /> <br /> Entrepreneurs Roundtable Accelerator, SELECT<br /> <br /> Y Combinator, Her<br /> <br /> Y Combinator, Final<br /> <br /> AngelCube, class<br /> <br /> Matter, common<br /> <br /> L-Spark, Company<br /> <br /> Techstars, Hot<br /> <br /> Rather than removing these companies from the list of search terms, we opted to not include as search terms any words that were considered among the top 10000 most common English words. For reference, we used the top 10000 most common English words according to a Google research study. The github documentation of the study can be found [https://github.com/first20hours/google-10000-english here].<br /> <br /> The file containing the 10000 most common English words can be found:<br /> E:\McNair\Software\Accelerators\10000_common_words.txt<br /> <br /> The results seemed much more plausible after removing these words. Some company words still appeared many times, but in the correct context.</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Demo_Day_Page_Parser&diff=22371 2017-12-19T19:31:32Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Demo Day Page Parser<br /> |Has owner=Peter Jalbert,<br /> |Has project status=Active<br /> }}<br /> <br /> ==Project Specs==<br /> The goal of this project is to leverage data mining with Selenium and Machine Learning to get good candidate web pages for Demo Days for accelerators. Relevant information on the project can be found on the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Data Accelerator Data] page.<br /> <br /> ==Code Location==<br /> The code directory for this project can be found:<br /> E:\McNair\Software\Accelerators<br /> <br /> The Selenium-based crawler can be found in the file below. This script runs a google search on accelerator names and keywords, and saves the urls and html pages for future use:<br /> DemoDayCrawler.py<br /> <br /> <br /> A script to rip from HTML to TXT can be found below. This script reads HTML files from the DemoDayHTML directory, and writes them to the DemoDayTxt directory:<br /> htmlToText.py<br /> <br /> A script to match Keywords (Accelerator and Cohort names) against the resulting text pages can be found in KeyTerms.py. The script takes the Keywords located in CohortAndAcceleratorsFullList.txt, and the text files in DemoDayTxt, and creates a file with the number of matches of each keyword against each text file.<br /> <br /> The script can be found:<br /> KeyTerms.py<br /> <br /> The Keyword matches text file can be found:<br /> DemoDayTxt\KeyTermFile\KeyTerms.txt<br /> <br /> A script to determine the text files of webpages that have at least one hit of these key words can be found:<br /> DemoDayHits.py<br /> <br /> ==Downloading HTML Files with Selenium==<br /> The code for utilizing Selenium to download HTML files can be found in the DemoDayCrawler.py file.<br /> <br /> The initial observation set over the data scraped 100 links for each of 20 sample accelerators from the list of overall accelerators. These sample pages were turned to text, and scored to remove web pages with no mention of relevant accelerators or companies.<br /> <br /> Once the process was tweaked in response to the initial sample testing, the process ran again over all accelerators. The test determined that we needed take no more than 10 links for each accelerator, and that 'Demo Day' was a suitable search term.<br /> <br /> COMPLETE FILES<br /> -----------------<br /> These files hold data for all the accelerators: not just the test set.<br /> <br /> The full list of accelerators:<br /> ListOfAccs.txt<br /> <br /> The full list of potential keywords (used for throwing out irrelevant results):<br /> Keywords.txt<br /> <br /> A list of accelerators, queries, and urls:<br /> demoday_crawl_full.txt<br /> <br /> A directory with HTML files for all accelerator demo day results:<br /> DemoDayHTMLFull<br /> <br /> A directory with TXT files for all accelerator demo day results:<br /> DemoDayTxtFull<br /> <br /> A file with the name of the results that passed keyword matching:<br /> DemoDayHitsFull.txt<br /> <br /> ==Faulty Results==<br /> The first pass through the data revealed articles that had thousands of hits for keyword matches. This seemed highly suspicious, so we dug in deeper to investigate the cause of this issue. <br /> <br /> The following script in the same directory analyzes the keyword matches to determine the words with the highest number of hits.<br /> DemoDayAnalysis.py<br /> <br /> After investigation, it was found that many company names were taken after common english words. Here are some of the companies causing issues along with their associated accelerator:<br /> <br /> the, L-Spark<br /> <br /> Matter, This., [https://matter.vc/portfolio/this/ website]<br /> <br /> Fledge, HERE, [http://fledge.co/fledgling/here/ website]<br /> <br /> StartupBootCamp, We...<br /> <br /> LightBank Start, Zero<br /> <br /> Entrepreneurs Roundtable Accelerator, SELECT<br /> <br /> Y Combinator, Her<br /> <br /> Y Combinator, Final<br /> <br /> AngelCube, class<br /> <br /> Matter, common<br /> <br /> L-Spark, Company<br /> <br /> Techstars, Hot<br /> <br /> Rather than removing these companies from the list of search terms, we opted to not include as search terms any words that were considered among the top 10000 most common English words. For reference, we used the top 10000 most common English words according to a Google research study. The github documentation of the study can be found [https://github.com/first20hours/google-10000-english here].<br /> <br /> The file containing the 10000 most common English words can be found:<br /> E:\McNair\Software\Accelerators\10000_common_words.txt<br /> <br /> The results seemed much more plausible after removing these words. Some company words still appeared many times, but in the correct context.</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=LinkedIn_Crawler_(Python)&diff=22370 2017-12-19T19:27:50Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has Image=Web-crawler.jpg<br /> |Has title=LinkedIn Crawler (Python)<br /> |Has start date=April 3, 2017<br /> |Has keywords=Selenium, LinkedIn, Crawler,Tool<br /> }}<br /> =Overview=<br /> <br /> Files for this project can be found on our Git Server under the directory LinkedIn_Crawler.<br /> <br /> This page is dedicated to a new LinkedIn Crawler built using Selenium and Python. The goal of this project is to be able to crawl LinkedIn without being caught by LinkedIn's aggressive [https://www.linkedin.com/help/linkedin/answer/56347/prohibition-of-scraping-software?lang=en anti-scraping rules.] To do this, we will use Selenium to behave like a human, and use time delays to hide bot-like tendencies.<br /> <br /> The documentation for Selenium Web Driver can be found [here http://selenium-python.readthedocs.io/index.html].<br /> <br /> Relevant scripts can be found in the following directory:<br /> E:\McNair\Projects\LinkedIn Crawler<br /> <br /> The resulting data for accelerator founders can be found:<br /> E:\McNair\Projects\LinkedIn Crawler\LinkedIn_Crawler\linkedin\accelerator_founders_data<br /> <br /> The code from the original Summer 2016 Project can be found in:<br /> web_crawler\linkedin<br /> <br /> The next section will provide details on the construction and functionality of the scripts located in the linkedin directory.<br /> <br /> The old documentation said that the programs/scripts (see details below) are located on our [[Software Repository|Bonobo Git Server]]. <br /> repository: Web_Crawler<br /> branch: researcher/linkedin<br /> directory: /linkedin<br /> <br /> <br /> <br /> ==Accounts==<br /> Test Account:<br /> <br /> email: testapplicat6@gmail.com<br /> <br /> pass: McNair2017<br /> <br /> Real Account:<br /> <br /> email: ed.edgan@rice.edu<br /> <br /> pass: This area has intentionally been left blank.<br /> <br /> =LinkedIn Scripts=<br /> ==Overview==<br /> This section provides a file by file breakdown of the contents of the folder located at:<br /> E:\McNair\Projects\LinkedIn Crawler\web_crawler\linkedin<br /> The main script to run is:<br /> run_linkedin_recruiter.py<br /> <br /> ==run_linkedin_recruiter.py==<br /> This script executes the linkedin recruiter crawler. At the top of the file, just below the imports, are three fields: username, password, and query_filepath. The username and password fields are for the desired recruiter pro account you would like to log into, and query_filepath is a pathname to a text file that contains a list of properly formatted queries that can be read by the LinkedIn Crawler's simple_search method. The following are the functions listed in the script.<br /> <br /> ===main()===<br /> This function runs the LinkedIn Crawler and will automatically begin when called from the command line. If you only want to go through some of the queries, you can change the range of the splice in line 32, and if you wish to only look at a certain number of search results, you can change the range of the splice in line 40.<br /> <br /> ===open_new_window(driver, element)===<br /> This function does a shift click on a web element to open the link in a new window. It then changes the window handler to the new window. This method makes it simple to view search results and close them in a quick manner.<br /> <br /> ===close_window_and_return(driver)===<br /> This function closes the current window, and returns to the main window. It is used in conjunction with open_new_window() to view search results and close them in an iterative manner.<br /> <br /> ===close_tab(driver)===<br /> When necessary, this function is used to close the current tab and return to the main tab. It is similar to close_window_and_return(). This function is used to log out of the account.<br /> <br /> ==crawlererror.py==<br /> This script is a simple class construction for error messages. It is used in other scripts to raise errors to the user when errors with the crawler occur. Please continue.<br /> <br /> ==linked_in_crawler.py==<br /> This script constructs a class that provides navigation functionality around the traditional LinkedIn site. The beginning section lists some global xpaths that will be used by Selenium throughout the process. These xpaths are used to locate elements within the HTML. The following are some important functions to keep in mind when designing original programs using this code.<br /> <br /> === login(self, username, password)===<br /> This function takes a username and password, and logs in to LinkedIn. During the process, the function uses the MouseMove move_random() function to move the mouse randomly across the screen like a crazy person.<br /> <br /> ===logout(self)===<br /> This function logs out of LinkedIn. It works by clicking on the profile picture, and then selecting logout.<br /> <br /> ===go_back(self)===<br /> This function goes back a page if you ever need to do such a thing.This function also doesn't seem to work.<br /> <br /> ===simple_search(self, query)===<br /> This function takes a string as a query, and searches it using the search box. At the end of the functions run, a page with search results relevant to your search query will be on the screen.<br /> <br /> ===advance_search(self, query)===<br /> This function uses the advanced search feature of LinkedIn. Instead of a string, this function takes in a dictionary mapping predetermined keywords to their necessary values. This function has not been debugged yet.<br /> <br /> ===get_search_results_on_page(self)===<br /> This function is supposed to return all the search results on the current page. This function has not been debugged yet.<br /> <br /> ===get_next_search_page(self)===<br /> This function is supposed to click and load the next search page if one exists. This function has not been debugged yet.<br /> <br /> ==linked_in_crawler_recruiter.py==<br /> This script constructs a class called LinkedInCrawlerRecruiter that implements functionality specifically for the Recruiter Pro feature of LinkedIn. Similar to the regular linked_in_crawler, the program begins with a list of relevant xpaths. It is followed by multiple functions. Their functionalities are listed below.<br /> <br /> ===login(self, username, password)===<br /> This function logs into a normal LinkedIn account, and then launches the Recruiter Pro session from the LinkedIn home page. At the end of the function run, there will be a window with the Recruiter Pro feature open, and the Selenium web frame will be on that window.<br /> <br /> ===simple_search(self, query)===<br /> Similar to the original LinkedIn Crawler, this function implements a basic string query search for the Recruiter Pro feature. At the end of the function run, a page will be up with the relevant search results of the search query.<br /> <br /> ===help_search_handler_stuff(self)===<br /> This function does some things on the current page in an attempt to appear more human. As of now, the function has a notes feature that will randomly jot down notes on the current page.<br /> <br /> <br /> ==utils.py==<br /> This file contains a few useful functions for waiting and moving the mouse. This is the human file for this project.<br /> <br /> ===sleep_secs(secs)===<br /> This is a simple function that has the browser wait for a specified number of seconds.<br /> <br /> ===sleep_rand(limit=__SLEEP_LIMIT__)===<br /> This function has the browser wait for a random amount of time less than the user provided limit. If the user does not provide a limit, the browser waits for a random time less than 5 seconds.<br /> <br /> ===move_strategy1(self)===<br /> This is a function within the MouseMove class. This function moves the mouse randomly across the window. It uses autopy to move the mouse across the window visibly to the user.<br /> <br /> ===move_to(self, x=None, y=None)===<br /> This function is a function within the MouseMove class. Given an x and y coordinates on the screen, this function will move the mouse to that given point. <br /> <br /> ===move_random(self)===<br /> This function chooses a random MouseMove method and executes it.<br /> <br /> ==web_driver.py==<br /> This file contains the relevant functions from the Selenium library that is used for web driving.<br /> <br /> =Constructing Your Query=<br /> <br /> Using Recruiter to search generic terms such as &quot;CompanyName Founder&quot; does not turn up valuable search results. For optimal performance, it is recommended that you determine through another source the exact person you are looking for. Methods to get such information will be listed below.<br /> <br /> ==format_founders.py==<br /> Script location:<br /> TBD<br /> <br /> This python script takes a textfile of company names, and uses the Crunchbase Snapshot to determine the founder names of each company. If Crunchbase does not have the records of the founder, it is unlikely that a generic search on LinkedIn will provide any useful results. The script returns a new textfile with each company name replaced with &quot;CompanyName Founder FounderName&quot; for each founder of the company listed in the Crunchbase Snapshot. This new textfile can then be used directly with the LinkedIn Crawler to generate accurate search results, and retrieve accurate html pages.<br /> <br /> The following lists the functionality of functions in the format_founders.py script.<br /> <br /> ===create_pickle()===<br /> This function creates a pickled python dictionary of the Crunchbase Snapshot, people.csv. If a different dataset should be used in the future, one should pickle a dictionary in a similar fashion to this function, and then use that pickled result in the next function to reformat your queries.<br /> <br /> ===reformat(pathname, output_filename)===<br /> This function takes a textfile pathname and an output filename, and converts the textfile to a searchable term by using the data from the pickled Crunchbase Snapshot. The new textfile with the corrected queries are saved to the output filename.<br /> <br /> ===Results with Accelerator Data===<br /> Of the 265 recorded accelerators we have data on, 94 of them have founders listed through the Crunchbase Snapshot. Some of these companies will have multiple founders with profiles, and some of these founders will not have LinkedIn profiles.<br /> <br /> The final data is a text file with accelerator name, founder name, profile summary, experience, and education. It can be found at:<br /> E:\McNair\Projects\Accelerators\LinkedIn Founders Data<br /> <br /> =Fall 2017=<br /> <br /> ==Accelerator Founders Search==<br /> <br /> '''These results are for the paper: The Jockey, The Horse, or the RaceTrack'''<br /> <br /> <br /> Our LinkedIn Recruiter Pro account has expired. Unfortunately, it turns out that profiles cannot be viewed through LinkedIn if the target profile is 3rd degree away or further. However, a Google search on such a LinkedIn profile will still let you view the profile, provided that an account has been logged into prior to the search. <br /> <br /> ===Piggybacking Google===<br /> <br /> In order to get our data, we will piggyback on Google's web crawler to work around the LinkedIn protective wall. The crawler begins by logging into our test LinkedIn Account (credentials displayed at the top), and then launching a Google search for each query. By adding &quot;LinkedIn&quot; before the query, and &quot;Founder&quot; after the query, we can turn up relevant search results. The top 5 results on Google search are explored, scraped, and saved.<br /> <br /> We ended up not opting to use the Google method for various reasons.<br /> <br /> ===Crunchbase API===<br /> <br /> Instead, we opted to use data from Crunchbase we have access to through a license. A wiki page on the crunchbase data and how to use the API can be found [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here]. The data can be accessed either through the web API (discussed on the Crunchbase Data wiki page), or through the bulk download we have in our SQL server.<br /> <br /> The web API has the nice added feature of having a '''Founders''' section. The API returns a JSON when a GET request is submitted using the correct company identifier. The Founders section of this JSON contains information on the Founders of the accelerator if Crunchbase has said data. Details about the data can be found on the [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data Crunchbase Data Page]. <br /> <br /> The script that queried the API is called '''crunchbase_founders.py''' and can be found:<br /> E:\McNair\Projects\Accelerators\crunchbase_founders.py<br /> <br /> The resulting text file, called '''founders_linkedin.txt''', containing names and linkedin URLs of founders after messing around with the database can be found:<br /> E:\McNair\Projects\Accelerators\founders_linkedin.txt<br /> <br /> ===Crawling LinkedIn===<br /> <br /> The next step of the process uses this data to get information about these founders from their LinkedIn profiles. For the founders we have linkedin URLs for, we will use those. For those we do not have linkedin URLs for, we will do a simple LinkedIn search with their name and accelerator name. The code for this crawler, '''linkedin_founders.py''' can be found:<br /> E:\McNair\Projects\LinkedIn Crawler\LinkedIn_Crawler\linkedin\linkedin_founders.py<br /> <br /> NOTE: Right now, this code needs to run in a virtual environment that contains Python3. This is due to the origins of the project, and this needs to be addressed when we have a lull in the development process. The only virtual environment we have managed to get working is on the Ubuntu machine sitting in the corner of the room. <br /> <br /> ===Using the Ubuntu Virtual Environment===<br /> <br /> Step 1: Login using the researcher credentials. If you don't know what these are, ask someone.<br /> <br /> Step 2: Open the command prompt. Type:<br /> source dev/python3_venv_linkedin/bin/activate<br /> <br /> Your screen should now have (python3_venv_linkedin) next to any command you write. The virtual enivornment has been activated.<br /> <br /> Step 3: Change directories to: <br /> ~/dev/web_crawler/linkedin<br /> <br /> Step 4: All the files for any sort of LinkedIn Crawler are here. The file for this project is:<br /> linkedin_founders.py<br /> <br /> This file executes the crawler on all of the information stored in the file founders_linkedin.txt. Any file with the format company-tab-first name-tab-last name-tab-linkedin url-newline- will work.<br /> The output of the data will be stored in founders_linkedin_main.txt, founders_linkedin_experience.txt, and founders_linkedin_education.txt.<br /> <br /> Step 5: To run the file, enter:<br /> python linkedin_founders.py<br /> <br /> The crawler will begin running automatically.<br /> <br /> Step 6: If you want to leave the virtual environment and return to the normal environment, simply enter the following in the command prompt:<br /> deactivate<br /> <br /> ==LinkedIn Crawler on the RDP==<br /> As of 12/18/2017, the linkedin crawler has been updated to be compatible with the RDP. Some of the bells and whistles have been removed from the ubuntu version due to download failures related to a missing vcvarsall.bat. <br /> <br /> Relevant files are located: <br /> E:\McNair\Projects\LinkedIn Crawler\LinkedIn_Crawler\linkedin<br /> <br /> ===Crawling Google for unknown LinkedIn accounts===<br /> For accelerator founders without a recorded LinkedIn profile, a quick google search will most likely get the correct page if the person has a LinkedIn profile. The script to run this process is in the same folder, and is called:<br /> goog_linkedin_founders.py<br /> This file uses the same formatted text file for its queries.<br /> <br /> <br /> <br /> =Previous Posts about the LinkedIn Crawler=<br /> == To what extent are we able to reproduce the network structure in LinkedIn (From Previous) == <br /> <br /> Example 1: 1st degree contact- You are connected to his profile<br /> Albert Nabiullin (485 connections)<br /> <br /> Example 2: 2nd degree contact- You are connected to someone who is connected to him<br /> Amir Kazempour Esmati (63 connections)<br /> <br /> Example 3: 3rd degree contact- You are connected to someone who is connected to someone else who is connected to her. <br /> Linda Szabados(500+ connections) <br /> <br /> Any profile with a distance greater than three is defined as out your network. <br /> <br /> Summary: Individual specific network information are not accessible even for the first degree connections. Therefore, any such plans to construct a network structure based on the connection of every individuals is not feasible. <br /> <br /> It seems that the only possible direction would be using the advanced search feature.</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Peter_Jalbert_(Work_Log)&diff=22369 2017-12-19T19:24:55Z

<p>Peterjalbert: /* Fall 2017 */</p> <hr /> <div>===Fall 2017===<br /> &lt;onlyinclude&gt; <br /> <br /> [[Peter Jalbert]] [[Work Logs]] [[Peter Jalbert (Work Log)|(log page)]] <br /> <br /> 2017-12-19: Finished fixing the Demo Day Crawler. Changed files and installed as appropriate to make linked in crawler compatible with the RDP. Removed some of the bells and whistles.<br /> <br /> 2017-12-18: Continued finding errors with the Demo Day Crawler analysis. Rewrote the parser to remove any search terms that were in the top 10000 most common English words according to Google. Finished uploading and submitting Moroccan data.<br /> <br /> 2017-12-15: Found errors with the Demo Day Crawler. Fixed scripts to download Moroccan Law Data.<br /> <br /> 2017-12-14: Uploading Morocco Parliament Written Questions. Creating script for next Morocco Parliament download. Begin writing Selenium documentation. Continuing to download TIGER data.<br /> <br /> 2017-12-06: Running Morocco Parliament Written Questions script. Analyzing Demo Day Crawler results. Continued downloading for TIGER geocoder.<br /> <br /> 2017-11-28: Debugging Morocco Parliament Crawler. Running Demo Day Crawler for all accelerators and 10 pages per accelerator. TIGER geocoder is back to Forbidden Error.<br /> <br /> 2017-11-27: Rerunning Morocco Parliament Crawler. Fixed KeyTerms.py and running it again. Continued downloading for TIGER geocoder.<br /> <br /> 2017-11-20: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-16: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-15: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote a script to extract counts that were greater than 2 from Keyword Matcher. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder]. Finished re-formatting work logs.<br /> <br /> 2017-11-14: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote an HTML to Text parser. See Parser Demo Day Page for file location. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder].<br /> <br /> 2017-11-13: Built [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser].<br /> <br /> 2017-11-09: Running demo version of Demo Day crawler (Accelerator Google Crawler). Fixing work log format. <br /> <br /> 2017-11-07: Created file with 0s and 1s detailing whether crunchbase has the founder information for an accelerator. Details posted as a TODO on [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List] page. Still waiting for feedback on the PostGIS installation from [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder]. Continued working on Accelerator Google Crawler.<br /> <br /> 2017-11-06: Contacted Geography Center for the US Census Bureau, [https://www.census.gov/geo/about/contact.html here], and began email exchange on PostGIS installation problems. Began working on the [http://mcnair.bakerinstitute.org/wiki/Selenium_Documentation Selenium Documentation]. Also began working on an Accelerator Google Crawler that will be used with Yang and ML to find Demo Days for cohort companies.<br /> <br /> 2017-11-01: Attempted to continue downloading, however ran into HTTP Forbidden errors. Listed the errors on the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder Page].<br /> <br /> 2017-10-31: Began downloading blocks of data for individual states for the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder] project. Wrote out the new wiki page for installation, and beginning to write documentation on usage.<br /> <br /> 2017-10-30: With Ed's help, was able to get the national data from Tiger installed onto a database server. The process required much jumping around and changing users, and all the things we learned are outlined in [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation#Editing_Users the database server documentation] under &quot;Editing Users&quot;.<br /> <br /> 2017-10-25: Continued working on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation TigerCoder Installation].<br /> <br /> 2017-10-24: Throw some addresses into a database, use address normalizer and geocoder. May need to install things. Details on the installation process can be found on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation page].<br /> <br /> 2017-10-23: Finished Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-19: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-18: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_DistrictHouston Innovation District Project].<br /> <br /> 2017-10-17: Constructed ArcGIS maps for the agglomeration project. Finished maps of points for every year in the state of California. Finished maps of Route 128. Began working on selenium Yelp crawler to get cafe locations within the 610-loop.<br /> <br /> 2017-10-16: Assisted Harrison on the USITC project. Looked for natural language processing tools to extract complaintants and defendants along with their location from case files. Experimented with pulling based on parts of speech tags, as well as using geotext or geograpy to pull locations from a case segment.<br /> <br /> 2017-10-13: Updated various project wiki pages.<br /> <br /> 2017-10-12: Continued work on Patent Thicket project, awaiting further project specs.<br /> <br /> 2017-10-05: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-04: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-02: Worked on ArcGIS data. See Harrison's Work Log for the details.<br /> <br /> 2017-09-28: Added collaborative editing feature to PyCharm. <br /> <br /> 2017-09-27: Worked on big database file.<br /> <br /> 2017-09-25: New task -- Create text file with company, description, and company type.<br /> #[http://mcnair.bakerinstitute.org/wiki/VC_Database_Rebuild VC Database Rebuild]<br /> #psql vcdb2<br /> #table name, sdccompanybasecore2<br /> #Combine with Crunchbasebulk<br /> <br /> #TODO: Write wiki on linkedin crawler, write wiki on creating accounts.<br /> <br /> 2017-09-21: Wrote wiki on Linkedin crawler, met with Laura about patents project. <br /> <br /> 2017-09-20: Finished running linkedin crawler. Transferred data to RDP. Will write wikis next.<br /> <br /> 2017-09-19: Began running linkedin crawler. Helped Yang create RDP account, get permissions, and get wiki setup.<br /> <br /> 2017-09-18: Finished implementation of Experience Crawler, continued working on Education Crawler for LinkedIn.<br /> <br /> 2017-09-14: Continued implementing LinkedIn Crawler for profiles.<br /> <br /> 2017-09-13: Implemented LinkedIn Crawler for main portion of profiles. Began working on crawling Experience section of profiles.<br /> <br /> 2017-09-12: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. Added to the wiki on this topic.<br /> <br /> 2017-09-11: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. <br /> <br /> 2017-09-06: Combined founders data retrieved with the Crunchbase API with the crunchbasebulk data to get linkedin urls for different accelerator founders. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-09-05: Post Harvey. Finished retrieving names from the Crunchbase API on founders. Next step is to query crunchbase bulk database to get linkedin urls. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-08-24: Began using the Crunchbase API to retrieve founder information for accelerators. Halfway through compiling a dictionary that translates accelerator names into proper Crunchbase API URLs.<br /> <br /> 2017-08-23: Decided with Ed to abandon LinkedIn crawling to retrieve accelerator founder data, and instead use crunchbase. Spent the day navigating the crunchbasebulk database, and seeing what useful information was contained in it.<br /> <br /> 2017-08-22: Discovered that LinkedIn Profiles cannot be viewed through LinkedIn if the target is 3rd degree or further. However, if entering LinkedIn through a Google search, the profile can still be viewed if the user has previously logged into LinkedIn. Devising a workaround crawler that utilizes Google search. Continued blog post [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) here] under Section 4.<br /> <br /> 2017-08-21: Began work on extracting founders for accelerators through LinkedIn Crawler. Discovered that Python3 is not installed on RDP, so the virtual environment for the project cannot be fired up. Continued working on Ubuntu machine.<br /> &lt;/onlyinclude&gt;<br /> <br /> ===Spring 2017===<br /> <br /> 2017-05-01: Continued work on HTML Parser. Uploaded all semester projects to git server.<br /> <br /> 2017-04-20: Finished the HTML Parser for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Ran HTML parser on accelerator founders. Data is stored in projects/accelerators/LinkedIn Founder Data.<br /> <br /> 2017-04-19: Made updates to the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler] Wikipage. Ran LinkedIn Crawler on accelerator data. Working on an html parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-18: Ran LinkedIn Crawler on matches between Crunchbase Snapshot and the accelerator data.<br /> <br /> 2017-04-17: Worked on ways to get correct search results from the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Worked on an HTML Parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-13: Worked on debugging the logout procedure for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Began formulation of process to search for founders of startups using a combination of the LinkedIn Crawler with the data resources from the [http://mcnair.bakerinstitute.org/wiki/Crunchbase_2013_Snapshot CrunchBase Snapshot].<br /> <br /> 2017-04-12: Work on bugs with the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler].<br /> <br /> 2017-04-11: Completed functional [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) crawler of LinkedIn Recruiter Pro]. Basic search functions work and download profile information for a given person. <br /> <br /> 2017-04-10: Began writing functioning crawler of LinkedIn. <br /> <br /> 2017-04-06: Continued working on debugging and documenting the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Wrote a test program that logs in, searches for a query, navigates through search pages, and logs out. Recruiter program can now login and search.<br /> <br /> 2017-04-05: Began work on the LinkedIn Crawler. Researched on launching Python Virtual Environment.<br /> <br /> 2017-04-03: Finished debugging points for the Enclosing Circle Algorithm. Added Command Line functionality to the Industry Classifier.<br /> <br /> 2017-03-29: Worked on debugging points for the Enclosing Circle Algorithm.<br /> <br /> 2017-03-28: Finished running the Enclosing Circle Algorithm. Worked on removing incorrect points from the data set(see above). <br /> <br /> 2017-03-27: Worked on debugging the Enclosing Circle Algorithm. Implemented a way to remove interior circles, and determined that translation to latitude and longitude coordinates resulted in slightly off center circles.<br /> <br /> 2017-03-23: Finished debugging the brute force algorithm for [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm]. Implemented a method to plot the points and circles on a graph. Analyzed runtime of the brute force algorithm.<br /> <br /> 2017-03-21: Coded a brute force algorithm for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-20: Worked on debugging the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-09: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Finished script to draw Enclosing Circles on a Google Map.<br /> <br /> 2017-03-08: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Created script to draw outcome of the Enclosing Circle Algorithm on Google Maps.<br /> <br /> 2017-03-07: Redetermined the top 50 cities which Enclosing Circle should be run on. Data on the [http://mcnair.bakerinstitute.org/wiki/Top_Cities_for_VC_Backed_Companies Top 50 Cities for VC Backed Companies can be found here.] Ran [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities.<br /> <br /> 2017-03-06: Ran script to determine the top 50 cities which Enclosing Circle should be run on. Fixed the VC Circles script to take in a new data format.<br /> <br /> 2017-03-02: Cleaned up data for the VC Circles Project. Created histogram of data in Excel. See [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Began work on the [http://mcnair.bakerinstitute.org/wiki/LinkedInCrawlerPython LinkedIn Crawler].<br /> <br /> 2017-03-01: Created statistics for the VC Circles Project.<br /> <br /> 2017-02-28: Finished downloading geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Found bug in Enclosing Circle Algorithm.<br /> <br /> 2017-02-27: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-23: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Installed C++ Compiler for Python. Ran tests on difference between Python and C wrapped Python.<br /> <br /> 2017-02-22: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Helped out with [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-21: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Researched into C++ Compilers for Python so that the Enclosing Circle Algorithm could be wrapped in C. Found a recommended one [https://www.microsoft.com/en-us/download/details.aspx?id=44266 here].<br /> <br /> 2017-02-20: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-16: Reworked [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] to create a file of geocoded data. Began work on wrapping the algorithm in C to improve speed.<br /> <br /> 2017-02-15: Finished [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] applied to the VC study. Enclosing Circle algorithm still needs adjustment, but the program runs with the temporary fixes.<br /> <br /> 2017-02-14: Worked on the application of the Enclosing Circle algorithm to the VC study. Working on bug fixes in the Enclosing Circle algorithm. Created wiki page for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-02-13: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-08: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-07: Fixed bugs in parse_cohort_data.py, the script for parsing the cohort data from the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Added descriptive statistics to cohort data excel file.<br /> <br /> 2017-02-02: Out sick, independent research and work from RDP. Brief research into the [http://jorgeg.scripts.mit.edu/homepage/wp-content/uploads/2016/03/Guzman-Stern-State-of-American-Entrepreneurship-FINAL.pdf Stern-Guzman algorithm]. Research into [http://mcnair.bakerinstitute.org/wiki/interactive_maps Interactive Maps]. No helpful additions to map embedding problem.<br /> <br /> 2017-02-01: Notes from Session with Ed: Project on US university patenting and entrepreneurship programs (writing code to identify universities in assignees), search Wikipedia (XML then bulk download), student pop, faculty pop, etc.<br /> Circle project for VC data will end up being a joint project to join accelerator data. <br /> Pull descriptions for VC. Founders of accelerators in linkedin. LinkedIn cannot be caught(pretend to not be a bot). Can eventually get academic backgrounds through linkedin. <br /> Pull business registration data, Stern/Guzman Algorithm. <br /> GIS ontop of geocoded data.<br /> Maps that works on wiki or blog (CartoDB), Maps API and R.<br /> NLP Projects, Description Classifier.<br /> <br /> 2017-01-31: Built WayBack Machine Crawler. Updated documentation for coordinates script. Updated profile page to include locations of code.<br /> <br /> <br /> 2017-01-30: Optimized enclosing circle algorithm through memoization. Developed script to read addresses from accelerator data and return latitude and longitude coordinates.<br /> <br /> 2017-01-26: Continued working on Google sitesearch project. Discovered crunchbase, changed project priority. Priority 1, split accelerator data up by flag, priority 2, use crunchbase to get web urls for cohorts, priority 3, make internet archive wayback machine driver. Located [http://mcnair.bakerinstitute.org/wiki/Whois_Parser Whois Parser].<br /> <br /> <br /> 2017-01-25: Finished parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Some data files still need proofreading as they are not in an acceptable format. Began working on Google sitesearch project.<br /> <br /> 2017-01-24: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Cohort data file created, debugging is almost complete. Will begin work on the google accelerator search soon.<br /> <br /> 2017-01-23: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Preliminary code is written, working on debugging.<br /> <br /> 2017-01-19: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project]. Created parser for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project], completed creation of final data set(yay!). Began working on cohort parser.<br /> <br /> 2017-01-18: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-13: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-12: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-11: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-10: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> <br /> ===Fall 2016===<br /> <br /> 2016-12-08: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-07: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-06: Learned how to use git. Committed software projects from the semester to the McNair git repository. Projects can be found at; [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Executive Order Crawler], [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Foreign Government Web Crawlers], [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) F6S Crawler and Parser].<br /> <br /> 2016-12-02: Built and ran web crawler for Center for Middle East Studies on Kuwait. Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-01: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Built tool for the [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report E&amp;I Governance Report Project] with Christy. Adds a column of data that shows whether or not the bill has been passed.<br /> <br /> 2016-11-29: Began pulling data from the accelerators listed [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here]. Made text files for about 18 accelerators.<br /> <br /> 2016-11-22: Transferred downloaded Morocco Written Bills to provided SeaGate Drive. Made a &quot;gentle&quot; F6S crawler to retrieve HTMLs of possible accelerator pages documented [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here].<br /> <br /> 2016-11-18: Converted Executive Order PDFs to text files using adobe acrobat DC. See [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Wikipage] for details.<br /> <br /> 2016-11-17: Wrote a crawler to retrieve information about executive orders, and their corresponding pdfs. They can be found [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report here.] Next step is to run code to convert the pdfs to text files, then use the parser fixed by Christy. <br /> <br /> 2016-11-15: Finished download of Moroccan Written Question pdfs. Wrote a parser with Christy to be used for parsing bills from Congress and eventually executive orders. Found bug in the system Python that was worked out and rebooted.<br /> <br /> 2016-11-11: Continued to download Moroccan data in the background. Attempted to find bug fixes for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] crawler.<br /> <br /> 2016-11-10: Continued to download Moroccan data and Kuwait data in the background. Began work on [http://mcnair.bakerinstitute.org/wiki/Google_Scholar_Crawler Google Scholar Crawler]. Wrote a crawler for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] to get the HTML files of hundreds of accelerators. The crawler ended up failing; it appears to have been due to HTTPS. <br /> <br /> 2016-11-08: Continued to download Moroccan data in the background. Finished writing code for the embedded files on the Kuwait Site. Spent time debugging the frame errors due to the dynamically generated content. Never found an answer to the bug, and instead found a workaround that sacrificed run time for the ability to work. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-04: Continued to download Moroccan data in the background. Finished writing initial Kuwait Web Crawler/Driver for the Middle East Studies Department. Middle East Studies Department asked for additional embedded files in the Kuwait website. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-03: Continued to download Moroccan data in the background. Dr. Egan fixed systems requirements to run the GovTrack Web Crawler. Made significant progress on the Kuwait Web Crawler/Driver for the Middle East Studies Department. <br /> <br /> 2016-11-01: Continued to download Moroccan data in the background. Went over code for GovTracker Web Crawler, continued learning Perl. [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] Began Kuwait Web Crawler/Driver.<br /> <br /> 2016-10-21: Continued to download data for the Moroccan Parliament Written and Oral Questions. Looked over [http://mcnair.bakerinstitute.org/wiki/Christy_Warden_(Twitter_Crawler_Application_1) Christy's Twitter Crawler] to see how I can be helpful. Dr. Egan asked me to think about how to potentially make multiple tools to get cohorts and other sorts of data from accelerator sites. See [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator List] He also asked me to look at the [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] for potential ideas on how to bring this project to fruition.<br /> <br /> 2016-10-20: Continued to download data for the Moroccan Parliament Written and Oral Questions. Updated Wiki page. Started working on Twitter project with Christy. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-18: Finished code for Oral Questions web driver and Written Questions web driver using selenium. Now, the data for the dates of questions can be found using the crawler, and the pdfs of the questions will be downloaded using selenium. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-14: Finished Oral Questions crawler. Finished Written Questions crawler. Waiting for further details on whether that data needs to be tweaked in any way. Updated the Moroccan Web Driver/Web Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-13: Completed download of Moroccan Bills. Working on either a web driver screenshot approach or a webcrawler approach to download the Moroccan oral and written questions data. Began building Web Crawler for Oral and Written Questions site. Edited Moroccan Web Driver/Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-11: Fixed arabic bug, files can now be saved with arabic titles. Monarchy bills downloaded and ready for shipment. House of Representatives Bill mostly downloaded, ratified bills prepared for download. Started learning scrapy library in python for web scraping. Discussed idea of screenshot-ing questions instead of scraping. <br /> <br /> 2016-10-07: Learned unicode and utf8 encoding and decoding in arabic. Still working on transforming an ascii url into printable unicode.<br /> <br /> 2016-10-06: Discussed with Dr. Elbadawy about the desired file names for Moroccan data download. The consensus was that the bill programs are ready to launch once the files can be named properly, and the questions data must be retrieved using a web crawler which I need to learn how to implement. The naming of files is currently drawing errors in going from arabic, to url, to download, to filename. Debugging in process. Also built a demo selenium program for Dr. Egan that drives the McNair blog site on an infinite loop.<br /> <br /> 2016-10-03: Moroccan Web Driver projects completed for driving of the Monarchy proposed bills, the House of Representatives proposed bills, and the Ratified bills sites. Begun process of devising a naming system for the files that does not require scraping. Tinkered with naming through regular expression parsing of the URL. Structure for the oral questions and written questions drivers is set up, but need fixes due to the differences in the sites. Fixed bug on McNair wiki for women's biz team where email was plain text instead of an email link. Took a glimpse at Kuwait Parliament website, and it appears to be very different from the Moroccan setup.<br /> <br /> 2016-09-30: Selenium program selects view pdf option from the website, and goes to the pdf webpage. Program then switches handle to the new page. CTRL S is sent to the page to launch save dialog window. Text cannot be sent to this window. Brainstorm ways around this issue. Explored Chrome Options for saving automatically without a dialog window. Looking into other libraries besides selenium that may help.<br /> <br /> 2016-09-29: Re-enroll in Microsoft Remote Desktop with proper authentication, set up Selenium environment and Komodo IDE on Remote Desktop, wrote program using Selenium that goes to a link and opens up the print dialog box. Developed computational recipe for a different approach to the problem.<br /> <br /> 2016-09-26: Set up Staff wiki page, work log page; registered for Slack, Microsoft Remote Desktop; downloaded Selenium on personal computer, read Selenium docs. Created wiki page for Moroccan Web Driver Project.<br /> <br /> '''Notes'''<br /> <br /> *Ed moved the Morocco Data to E:\McNair\Projects from C:\Users\PeterJ\Documents<br /> * C Drive files moved to E:\McNair\Users\PeterJ<br /> <br /> [[Category:Work Log]]</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Demo_Day_Page_Parser&diff=22365 2017-12-15T20:17:40Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Demo Day Page Parser<br /> |Has owner=Peter Jalbert,<br /> |Has project status=Active<br /> }}<br /> <br /> ==Project Specs==<br /> The goal of this project is to leverage data mining with Selenium and Machine Learning to get good candidate web pages for Demo Days for accelerators. Relevant information on the project can be found on the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Data Accelerator Data] page.<br /> <br /> ==Code Location==<br /> The code directory for this project can be found:<br /> E:\McNair\Software\Accelerators<br /> <br /> The Selenium-based crawler can be found in the file below. This script runs a google search on accelerator names and keywords, and saves the urls and html pages for future use:<br /> DemoDayCrawler.py<br /> <br /> <br /> A script to rip from HTML to TXT can be found below. This script reads HTML files from the DemoDayHTML directory, and writes them to the DemoDayTxt directory:<br /> htmlToText.py<br /> <br /> A script to match Keywords (Accelerator and Cohort names) against the resulting text pages can be found in KeyTerms.py. The script takes the Keywords located in CohortAndAcceleratorsFullList.txt, and the text files in DemoDayTxt, and creates a file with the number of matches of each keyword against each text file.<br /> <br /> The script can be found:<br /> KeyTerms.py<br /> <br /> The Keyword matches text file can be found:<br /> DemoDayTxt\KeyTermFile\KeyTerms.txt<br /> <br /> A script to determine the text files of webpages that have at least one hit of these key words can be found:<br /> DemoDayHits.py<br /> <br /> ==Downloading HTML Files with Selenium==<br /> The code for utilizing Selenium to download HTML files can be found in the DemoDayCrawler.py file.<br /> <br /> The initial observation set over the data scraped 100 links for each of 20 sample accelerators from the list of overall accelerators. These sample pages were turned to text, and scored to remove web pages with no mention of relevant accelerators or companies.<br /> <br /> Once the process was tweaked in response to the initial sample testing, the process ran again over all accelerators. The test determined that we needed take no more than 10 links for each accelerator, and that 'Demo Day' was a suitable search term.<br /> <br /> COMPLETE FILES<br /> -----------------<br /> These files hold data for all the accelerators: not just the test set.<br /> <br /> The full list of accelerators:<br /> ListOfAccs.txt<br /> <br /> The full list of potential keywords (used for throwing out irrelevant results):<br /> Keywords.txt<br /> <br /> A list of accelerators, queries, and urls:<br /> demoday_crawl_full.txt<br /> <br /> A directory with HTML files for all accelerator demo day results:<br /> DemoDayHTMLFull<br /> <br /> A directory with TXT files for all accelerator demo day results:<br /> DemoDayTxtFull<br /> <br /> A file with the name of the results that passed keyword matching:<br /> DemoDayHitsFull.txt<br /> <br /> ==Faulty Results==<br /> The first pass through the data revealed articles that had thousands of hits for keyword matches. This seemed highly suspicious, so we dug in deeper to investigate the cause of this issue. <br /> <br /> The following script in the same directory analyzes the keyword matches to determine the words with the highest number of hits.<br /> DemoDayAnalysis.py<br /> <br /> After investigation, it was found that many company names were taken after common english words. Here are some of the companies causing issues along with their associated accelerator:<br /> <br /> the, L-Spark<br /> <br /> Matter, This., [https://matter.vc/portfolio/this/ website]<br /> <br /> Fledge, HERE, [http://fledge.co/fledgling/here/ website]<br /> <br /> StartupBootCamp, We...<br /> <br /> LightBank Start, Zero<br /> <br /> Entrepreneurs Roundtable Accelerator, SELECT<br /> <br /> Y Combinator, Her<br /> <br /> Y Combinator, Final<br /> <br /> AngelCube, class<br /> <br /> Matter, common<br /> <br /> L-Spark, Company<br /> <br /> Techstars, Hot<br /> <br /> After removing these companies from consideration as keywords,</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Demo_Day_Page_Parser&diff=22364 2017-12-15T20:12:48Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Demo Day Page Parser<br /> |Has owner=Peter Jalbert,<br /> |Has project status=Active<br /> }}<br /> <br /> ==Project Specs==<br /> The goal of this project is to leverage data mining with Selenium and Machine Learning to get good candidate web pages for Demo Days for accelerators. Relevant information on the project can be found on the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Data Accelerator Data] page.<br /> <br /> ==Code Location==<br /> The code directory for this project can be found:<br /> E:\McNair\Software\Accelerators<br /> <br /> The Selenium-based crawler can be found in the file below. This script runs a google search on accelerator names and keywords, and saves the urls and html pages for future use:<br /> DemoDayCrawler.py<br /> <br /> <br /> A script to rip from HTML to TXT can be found below. This script reads HTML files from the DemoDayHTML directory, and writes them to the DemoDayTxt directory:<br /> htmlToText.py<br /> <br /> A script to match Keywords (Accelerator and Cohort names) against the resulting text pages can be found in KeyTerms.py. The script takes the Keywords located in CohortAndAcceleratorsFullList.txt, and the text files in DemoDayTxt, and creates a file with the number of matches of each keyword against each text file.<br /> <br /> The script can be found:<br /> KeyTerms.py<br /> <br /> The Keyword matches text file can be found:<br /> DemoDayTxt\KeyTermFile\KeyTerms.txt<br /> <br /> A script to determine the text files of webpages that have at least one hit of these key words can be found:<br /> DemoDayHits.py<br /> <br /> ==Downloading HTML Files with Selenium==<br /> The code for utilizing Selenium to download HTML files can be found in the DemoDayCrawler.py file.<br /> <br /> The initial observation set over the data scraped 100 links for each of 20 sample accelerators from the list of overall accelerators. These sample pages were turned to text, and scored to remove web pages with no mention of relevant accelerators or companies.<br /> <br /> Once the process was tweaked in response to the initial sample testing, the process ran again over all accelerators. The test determined that we needed take no more than 10 links for each accelerator, and that 'Demo Day' was a suitable search term.<br /> <br /> COMPLETE FILES<br /> -----------------<br /> These files hold data for all the accelerators: not just the test set.<br /> <br /> The full list of accelerators:<br /> ListOfAccs.txt<br /> <br /> The full list of potential keywords (used for throwing out irrelevant results):<br /> Keywords.txt<br /> <br /> A list of accelerators, queries, and urls:<br /> demoday_crawl_full.txt<br /> <br /> A directory with HTML files for all accelerator demo day results:<br /> DemoDayHTMLFull<br /> <br /> A directory with TXT files for all accelerator demo day results:<br /> DemoDayTxtFull<br /> <br /> A file with the name of the results that passed keyword matching:<br /> DemoDayHitsFull.txt<br /> <br /> ==Faulty Results==<br /> The first pass through the data revealed articles that had thousands of hits for keyword matches. This seemed highly suspicious, so we dug in deeper to investigate the cause of this issue. <br /> <br /> The following script in the same directory analyzes the keyword matches to determine the words with the highest number of hits.<br /> DemoDayAnalysis.py<br /> <br /> After investigation, it was found that many company names were taken after common english words. Here are some of the companies causing issues along with their associated accelerator:<br /> <br /> the, L-Spark<br /> <br /> Matter, This., [https://matter.vc/portfolio/this/ website]<br /> <br /> Fledge, HERE, [http://fledge.co/fledgling/here/ website]<br /> <br /> StartupBootCamp, We...<br /> <br /> LightBank Start, Zero<br /> <br /> Entrepreneurs Roundtable Accelerator, SELECT<br /> <br /> Y Combinator, Her<br /> <br /> Y Combinator, Final<br /> <br /> AngelCube, class<br /> <br /> Matter, common<br /> <br /> L-Spark, Company<br /> <br /> After removing these companies from consideration as keywords,</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Demo_Day_Page_Parser&diff=22363 2017-12-15T20:11:23Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Demo Day Page Parser<br /> |Has owner=Peter Jalbert,<br /> |Has project status=Active<br /> }}<br /> <br /> ==Project Specs==<br /> The goal of this project is to leverage data mining with Selenium and Machine Learning to get good candidate web pages for Demo Days for accelerators. Relevant information on the project can be found on the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Data Accelerator Data] page.<br /> <br /> ==Code Location==<br /> The code directory for this project can be found:<br /> E:\McNair\Software\Accelerators<br /> <br /> The Selenium-based crawler can be found in the file below. This script runs a google search on accelerator names and keywords, and saves the urls and html pages for future use:<br /> DemoDayCrawler.py<br /> <br /> <br /> A script to rip from HTML to TXT can be found below. This script reads HTML files from the DemoDayHTML directory, and writes them to the DemoDayTxt directory:<br /> htmlToText.py<br /> <br /> A script to match Keywords (Accelerator and Cohort names) against the resulting text pages can be found in KeyTerms.py. The script takes the Keywords located in CohortAndAcceleratorsFullList.txt, and the text files in DemoDayTxt, and creates a file with the number of matches of each keyword against each text file.<br /> <br /> The script can be found:<br /> KeyTerms.py<br /> <br /> The Keyword matches text file can be found:<br /> DemoDayTxt\KeyTermFile\KeyTerms.txt<br /> <br /> A script to determine the text files of webpages that have at least one hit of these key words can be found:<br /> DemoDayHits.py<br /> <br /> ==Downloading HTML Files with Selenium==<br /> The code for utilizing Selenium to download HTML files can be found in the DemoDayCrawler.py file.<br /> <br /> The initial observation set over the data scraped 100 links for each of 20 sample accelerators from the list of overall accelerators. These sample pages were turned to text, and scored to remove web pages with no mention of relevant accelerators or companies.<br /> <br /> Once the process was tweaked in response to the initial sample testing, the process ran again over all accelerators. The test determined that we needed take no more than 10 links for each accelerator, and that 'Demo Day' was a suitable search term.<br /> <br /> COMPLETE FILES<br /> -----------------<br /> These files hold data for all the accelerators: not just the test set.<br /> <br /> The full list of accelerators:<br /> ListOfAccs.txt<br /> <br /> The full list of potential keywords (used for throwing out irrelevant results):<br /> Keywords.txt<br /> <br /> A list of accelerators, queries, and urls:<br /> demoday_crawl_full.txt<br /> <br /> A directory with HTML files for all accelerator demo day results:<br /> DemoDayHTMLFull<br /> <br /> A directory with TXT files for all accelerator demo day results:<br /> DemoDayTxtFull<br /> <br /> A file with the name of the results that passed keyword matching:<br /> DemoDayHitsFull.txt<br /> <br /> ==Faulty Results==<br /> The first pass through the data revealed articles that had thousands of hits for keyword matches. This seemed highly suspicious, so we dug in deeper to investigate the cause of this issue. <br /> <br /> The following script in the same directory analyzes the keyword matches to determine the words with the highest number of hits.<br /> DemoDayAnalysis.py<br /> <br /> After investigation, it was found that many company names were taken after common english words. Here are some of the companies causing issues along with their associated accelerator:<br /> <br /> the, L-Spark<br /> <br /> Matter, This.<br /> <br /> Fledge, HERE<br /> <br /> StartupBootCamp, We...<br /> <br /> LightBank Start, Zero<br /> <br /> Entrepreneurs Roundtable Accelerator, SELECT<br /> <br /> Y Combinator, Her<br /> <br /> Y Combinator, Final<br /> <br /> AngelCube, class<br /> <br /> Matter, common<br /> <br /> L-Spark, Company<br /> <br /> After removing these companies from consideration as keywords,</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Peter_Jalbert_(Work_Log)&diff=22362 2017-12-15T20:03:47Z

<p>Peterjalbert: </p> <hr /> <div>===Fall 2017===<br /> &lt;onlyinclude&gt; <br /> <br /> [[Peter Jalbert]] [[Work Logs]] [[Peter Jalbert (Work Log)|(log page)]] <br /> <br /> 2017-12-15: Found the error! Ex: L-Spark the, Matter This., Fledge HERE, StartupBootCamp We..., LightBank Start Zero, Entrepreneurs Roundtable Accelerator SELECT, Y Combinator Her, Y Combinator Final, AngelCube class, Matter common, L-Spark Company<br /> <br /> 2017-12-14: Uploading Morocco Parliament Written Questions. Creating script for next Morocco Parliament download. Begin writing Selenium documentation. Continuing to download TIGER data.<br /> <br /> 2017-12-06: Running Morocco Parliament Written Questions script. Analyzing Demo Day Crawler results. Continued downloading for TIGER geocoder.<br /> <br /> 2017-11-28: Debugging Morocco Parliament Crawler. Running Demo Day Crawler for all accelerators and 10 pages per accelerator. TIGER geocoder is back to Forbidden Error.<br /> <br /> 2017-11-27: Rerunning Morocco Parliament Crawler. Fixed KeyTerms.py and running it again. Continued downloading for TIGER geocoder.<br /> <br /> 2017-11-20: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-16: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-15: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote a script to extract counts that were greater than 2 from Keyword Matcher. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder]. Finished re-formatting work logs.<br /> <br /> 2017-11-14: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote an HTML to Text parser. See Parser Demo Day Page for file location. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder].<br /> <br /> 2017-11-13: Built [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser].<br /> <br /> 2017-11-09: Running demo version of Demo Day crawler (Accelerator Google Crawler). Fixing work log format. <br /> <br /> 2017-11-07: Created file with 0s and 1s detailing whether crunchbase has the founder information for an accelerator. Details posted as a TODO on [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List] page. Still waiting for feedback on the PostGIS installation from [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder]. Continued working on Accelerator Google Crawler.<br /> <br /> 2017-11-06: Contacted Geography Center for the US Census Bureau, [https://www.census.gov/geo/about/contact.html here], and began email exchange on PostGIS installation problems. Began working on the [http://mcnair.bakerinstitute.org/wiki/Selenium_Documentation Selenium Documentation]. Also began working on an Accelerator Google Crawler that will be used with Yang and ML to find Demo Days for cohort companies.<br /> <br /> 2017-11-01: Attempted to continue downloading, however ran into HTTP Forbidden errors. Listed the errors on the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder Page].<br /> <br /> 2017-10-31: Began downloading blocks of data for individual states for the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder] project. Wrote out the new wiki page for installation, and beginning to write documentation on usage.<br /> <br /> 2017-10-30: With Ed's help, was able to get the national data from Tiger installed onto a database server. The process required much jumping around and changing users, and all the things we learned are outlined in [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation#Editing_Users the database server documentation] under &quot;Editing Users&quot;.<br /> <br /> 2017-10-25: Continued working on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation TigerCoder Installation].<br /> <br /> 2017-10-24: Throw some addresses into a database, use address normalizer and geocoder. May need to install things. Details on the installation process can be found on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation page].<br /> <br /> 2017-10-23: Finished Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-19: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-18: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_DistrictHouston Innovation District Project].<br /> <br /> 2017-10-17: Constructed ArcGIS maps for the agglomeration project. Finished maps of points for every year in the state of California. Finished maps of Route 128. Began working on selenium Yelp crawler to get cafe locations within the 610-loop.<br /> <br /> 2017-10-16: Assisted Harrison on the USITC project. Looked for natural language processing tools to extract complaintants and defendants along with their location from case files. Experimented with pulling based on parts of speech tags, as well as using geotext or geograpy to pull locations from a case segment.<br /> <br /> 2017-10-13: Updated various project wiki pages.<br /> <br /> 2017-10-12: Continued work on Patent Thicket project, awaiting further project specs.<br /> <br /> 2017-10-05: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-04: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-02: Worked on ArcGIS data. See Harrison's Work Log for the details.<br /> <br /> 2017-09-28: Added collaborative editing feature to PyCharm. <br /> <br /> 2017-09-27: Worked on big database file.<br /> <br /> 2017-09-25: New task -- Create text file with company, description, and company type.<br /> #[http://mcnair.bakerinstitute.org/wiki/VC_Database_Rebuild VC Database Rebuild]<br /> #psql vcdb2<br /> #table name, sdccompanybasecore2<br /> #Combine with Crunchbasebulk<br /> <br /> #TODO: Write wiki on linkedin crawler, write wiki on creating accounts.<br /> <br /> 2017-09-21: Wrote wiki on Linkedin crawler, met with Laura about patents project. <br /> <br /> 2017-09-20: Finished running linkedin crawler. Transferred data to RDP. Will write wikis next.<br /> <br /> 2017-09-19: Began running linkedin crawler. Helped Yang create RDP account, get permissions, and get wiki setup.<br /> <br /> 2017-09-18: Finished implementation of Experience Crawler, continued working on Education Crawler for LinkedIn.<br /> <br /> 2017-09-14: Continued implementing LinkedIn Crawler for profiles.<br /> <br /> 2017-09-13: Implemented LinkedIn Crawler for main portion of profiles. Began working on crawling Experience section of profiles.<br /> <br /> 2017-09-12: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. Added to the wiki on this topic.<br /> <br /> 2017-09-11: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. <br /> <br /> 2017-09-06: Combined founders data retrieved with the Crunchbase API with the crunchbasebulk data to get linkedin urls for different accelerator founders. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-09-05: Post Harvey. Finished retrieving names from the Crunchbase API on founders. Next step is to query crunchbase bulk database to get linkedin urls. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-08-24: Began using the Crunchbase API to retrieve founder information for accelerators. Halfway through compiling a dictionary that translates accelerator names into proper Crunchbase API URLs.<br /> <br /> 2017-08-23: Decided with Ed to abandon LinkedIn crawling to retrieve accelerator founder data, and instead use crunchbase. Spent the day navigating the crunchbasebulk database, and seeing what useful information was contained in it.<br /> <br /> 2017-08-22: Discovered that LinkedIn Profiles cannot be viewed through LinkedIn if the target is 3rd degree or further. However, if entering LinkedIn through a Google search, the profile can still be viewed if the user has previously logged into LinkedIn. Devising a workaround crawler that utilizes Google search. Continued blog post [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) here] under Section 4.<br /> <br /> 2017-08-21: Began work on extracting founders for accelerators through LinkedIn Crawler. Discovered that Python3 is not installed on RDP, so the virtual environment for the project cannot be fired up. Continued working on Ubuntu machine.<br /> &lt;/onlyinclude&gt; <br /> <br /> ===Spring 2017===<br /> <br /> 2017-05-01: Continued work on HTML Parser. Uploaded all semester projects to git server.<br /> <br /> 2017-04-20: Finished the HTML Parser for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Ran HTML parser on accelerator founders. Data is stored in projects/accelerators/LinkedIn Founder Data.<br /> <br /> 2017-04-19: Made updates to the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler] Wikipage. Ran LinkedIn Crawler on accelerator data. Working on an html parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-18: Ran LinkedIn Crawler on matches between Crunchbase Snapshot and the accelerator data.<br /> <br /> 2017-04-17: Worked on ways to get correct search results from the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Worked on an HTML Parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-13: Worked on debugging the logout procedure for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Began formulation of process to search for founders of startups using a combination of the LinkedIn Crawler with the data resources from the [http://mcnair.bakerinstitute.org/wiki/Crunchbase_2013_Snapshot CrunchBase Snapshot].<br /> <br /> 2017-04-12: Work on bugs with the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler].<br /> <br /> 2017-04-11: Completed functional [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) crawler of LinkedIn Recruiter Pro]. Basic search functions work and download profile information for a given person. <br /> <br /> 2017-04-10: Began writing functioning crawler of LinkedIn. <br /> <br /> 2017-04-06: Continued working on debugging and documenting the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Wrote a test program that logs in, searches for a query, navigates through search pages, and logs out. Recruiter program can now login and search.<br /> <br /> 2017-04-05: Began work on the LinkedIn Crawler. Researched on launching Python Virtual Environment.<br /> <br /> 2017-04-03: Finished debugging points for the Enclosing Circle Algorithm. Added Command Line functionality to the Industry Classifier.<br /> <br /> 2017-03-29: Worked on debugging points for the Enclosing Circle Algorithm.<br /> <br /> 2017-03-28: Finished running the Enclosing Circle Algorithm. Worked on removing incorrect points from the data set(see above). <br /> <br /> 2017-03-27: Worked on debugging the Enclosing Circle Algorithm. Implemented a way to remove interior circles, and determined that translation to latitude and longitude coordinates resulted in slightly off center circles.<br /> <br /> 2017-03-23: Finished debugging the brute force algorithm for [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm]. Implemented a method to plot the points and circles on a graph. Analyzed runtime of the brute force algorithm.<br /> <br /> 2017-03-21: Coded a brute force algorithm for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-20: Worked on debugging the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-09: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Finished script to draw Enclosing Circles on a Google Map.<br /> <br /> 2017-03-08: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Created script to draw outcome of the Enclosing Circle Algorithm on Google Maps.<br /> <br /> 2017-03-07: Redetermined the top 50 cities which Enclosing Circle should be run on. Data on the [http://mcnair.bakerinstitute.org/wiki/Top_Cities_for_VC_Backed_Companies Top 50 Cities for VC Backed Companies can be found here.] Ran [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities.<br /> <br /> 2017-03-06: Ran script to determine the top 50 cities which Enclosing Circle should be run on. Fixed the VC Circles script to take in a new data format.<br /> <br /> 2017-03-02: Cleaned up data for the VC Circles Project. Created histogram of data in Excel. See [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Began work on the [http://mcnair.bakerinstitute.org/wiki/LinkedInCrawlerPython LinkedIn Crawler].<br /> <br /> 2017-03-01: Created statistics for the VC Circles Project.<br /> <br /> 2017-02-28: Finished downloading geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Found bug in Enclosing Circle Algorithm.<br /> <br /> 2017-02-27: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-23: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Installed C++ Compiler for Python. Ran tests on difference between Python and C wrapped Python.<br /> <br /> 2017-02-22: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Helped out with [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-21: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Researched into C++ Compilers for Python so that the Enclosing Circle Algorithm could be wrapped in C. Found a recommended one [https://www.microsoft.com/en-us/download/details.aspx?id=44266 here].<br /> <br /> 2017-02-20: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-16: Reworked [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] to create a file of geocoded data. Began work on wrapping the algorithm in C to improve speed.<br /> <br /> 2017-02-15: Finished [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] applied to the VC study. Enclosing Circle algorithm still needs adjustment, but the program runs with the temporary fixes.<br /> <br /> 2017-02-14: Worked on the application of the Enclosing Circle algorithm to the VC study. Working on bug fixes in the Enclosing Circle algorithm. Created wiki page for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-02-13: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-08: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-07: Fixed bugs in parse_cohort_data.py, the script for parsing the cohort data from the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Added descriptive statistics to cohort data excel file.<br /> <br /> 2017-02-02: Out sick, independent research and work from RDP. Brief research into the [http://jorgeg.scripts.mit.edu/homepage/wp-content/uploads/2016/03/Guzman-Stern-State-of-American-Entrepreneurship-FINAL.pdf Stern-Guzman algorithm]. Research into [http://mcnair.bakerinstitute.org/wiki/interactive_maps Interactive Maps]. No helpful additions to map embedding problem.<br /> <br /> 2017-02-01: Notes from Session with Ed: Project on US university patenting and entrepreneurship programs (writing code to identify universities in assignees), search Wikipedia (XML then bulk download), student pop, faculty pop, etc.<br /> Circle project for VC data will end up being a joint project to join accelerator data. <br /> Pull descriptions for VC. Founders of accelerators in linkedin. LinkedIn cannot be caught(pretend to not be a bot). Can eventually get academic backgrounds through linkedin. <br /> Pull business registration data, Stern/Guzman Algorithm. <br /> GIS ontop of geocoded data.<br /> Maps that works on wiki or blog (CartoDB), Maps API and R.<br /> NLP Projects, Description Classifier.<br /> <br /> 2017-01-31: Built WayBack Machine Crawler. Updated documentation for coordinates script. Updated profile page to include locations of code.<br /> <br /> <br /> 2017-01-30: Optimized enclosing circle algorithm through memoization. Developed script to read addresses from accelerator data and return latitude and longitude coordinates.<br /> <br /> 2017-01-26: Continued working on Google sitesearch project. Discovered crunchbase, changed project priority. Priority 1, split accelerator data up by flag, priority 2, use crunchbase to get web urls for cohorts, priority 3, make internet archive wayback machine driver. Located [http://mcnair.bakerinstitute.org/wiki/Whois_Parser Whois Parser].<br /> <br /> <br /> 2017-01-25: Finished parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Some data files still need proofreading as they are not in an acceptable format. Began working on Google sitesearch project.<br /> <br /> 2017-01-24: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Cohort data file created, debugging is almost complete. Will begin work on the google accelerator search soon.<br /> <br /> 2017-01-23: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Preliminary code is written, working on debugging.<br /> <br /> 2017-01-19: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project]. Created parser for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project], completed creation of final data set(yay!). Began working on cohort parser.<br /> <br /> 2017-01-18: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-13: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-12: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-11: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-10: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> <br /> ===Fall 2016===<br /> <br /> 2016-12-08: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-07: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-06: Learned how to use git. Committed software projects from the semester to the McNair git repository. Projects can be found at; [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Executive Order Crawler], [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Foreign Government Web Crawlers], [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) F6S Crawler and Parser].<br /> <br /> 2016-12-02: Built and ran web crawler for Center for Middle East Studies on Kuwait. Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-01: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Built tool for the [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report E&amp;I Governance Report Project] with Christy. Adds a column of data that shows whether or not the bill has been passed.<br /> <br /> 2016-11-29: Began pulling data from the accelerators listed [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here]. Made text files for about 18 accelerators.<br /> <br /> 2016-11-22: Transferred downloaded Morocco Written Bills to provided SeaGate Drive. Made a &quot;gentle&quot; F6S crawler to retrieve HTMLs of possible accelerator pages documented [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here].<br /> <br /> 2016-11-18: Converted Executive Order PDFs to text files using adobe acrobat DC. See [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Wikipage] for details.<br /> <br /> 2016-11-17: Wrote a crawler to retrieve information about executive orders, and their corresponding pdfs. They can be found [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report here.] Next step is to run code to convert the pdfs to text files, then use the parser fixed by Christy. <br /> <br /> 2016-11-15: Finished download of Moroccan Written Question pdfs. Wrote a parser with Christy to be used for parsing bills from Congress and eventually executive orders. Found bug in the system Python that was worked out and rebooted.<br /> <br /> 2016-11-11: Continued to download Moroccan data in the background. Attempted to find bug fixes for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] crawler.<br /> <br /> 2016-11-10: Continued to download Moroccan data and Kuwait data in the background. Began work on [http://mcnair.bakerinstitute.org/wiki/Google_Scholar_Crawler Google Scholar Crawler]. Wrote a crawler for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] to get the HTML files of hundreds of accelerators. The crawler ended up failing; it appears to have been due to HTTPS. <br /> <br /> 2016-11-08: Continued to download Moroccan data in the background. Finished writing code for the embedded files on the Kuwait Site. Spent time debugging the frame errors due to the dynamically generated content. Never found an answer to the bug, and instead found a workaround that sacrificed run time for the ability to work. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-04: Continued to download Moroccan data in the background. Finished writing initial Kuwait Web Crawler/Driver for the Middle East Studies Department. Middle East Studies Department asked for additional embedded files in the Kuwait website. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-03: Continued to download Moroccan data in the background. Dr. Egan fixed systems requirements to run the GovTrack Web Crawler. Made significant progress on the Kuwait Web Crawler/Driver for the Middle East Studies Department. <br /> <br /> 2016-11-01: Continued to download Moroccan data in the background. Went over code for GovTracker Web Crawler, continued learning Perl. [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] Began Kuwait Web Crawler/Driver.<br /> <br /> 2016-10-21: Continued to download data for the Moroccan Parliament Written and Oral Questions. Looked over [http://mcnair.bakerinstitute.org/wiki/Christy_Warden_(Twitter_Crawler_Application_1) Christy's Twitter Crawler] to see how I can be helpful. Dr. Egan asked me to think about how to potentially make multiple tools to get cohorts and other sorts of data from accelerator sites. See [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator List] He also asked me to look at the [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] for potential ideas on how to bring this project to fruition.<br /> <br /> 2016-10-20: Continued to download data for the Moroccan Parliament Written and Oral Questions. Updated Wiki page. Started working on Twitter project with Christy. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-18: Finished code for Oral Questions web driver and Written Questions web driver using selenium. Now, the data for the dates of questions can be found using the crawler, and the pdfs of the questions will be downloaded using selenium. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-14: Finished Oral Questions crawler. Finished Written Questions crawler. Waiting for further details on whether that data needs to be tweaked in any way. Updated the Moroccan Web Driver/Web Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-13: Completed download of Moroccan Bills. Working on either a web driver screenshot approach or a webcrawler approach to download the Moroccan oral and written questions data. Began building Web Crawler for Oral and Written Questions site. Edited Moroccan Web Driver/Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-11: Fixed arabic bug, files can now be saved with arabic titles. Monarchy bills downloaded and ready for shipment. House of Representatives Bill mostly downloaded, ratified bills prepared for download. Started learning scrapy library in python for web scraping. Discussed idea of screenshot-ing questions instead of scraping. <br /> <br /> 2016-10-07: Learned unicode and utf8 encoding and decoding in arabic. Still working on transforming an ascii url into printable unicode.<br /> <br /> 2016-10-06: Discussed with Dr. Elbadawy about the desired file names for Moroccan data download. The consensus was that the bill programs are ready to launch once the files can be named properly, and the questions data must be retrieved using a web crawler which I need to learn how to implement. The naming of files is currently drawing errors in going from arabic, to url, to download, to filename. Debugging in process. Also built a demo selenium program for Dr. Egan that drives the McNair blog site on an infinite loop.<br /> <br /> 2016-10-03: Moroccan Web Driver projects completed for driving of the Monarchy proposed bills, the House of Representatives proposed bills, and the Ratified bills sites. Begun process of devising a naming system for the files that does not require scraping. Tinkered with naming through regular expression parsing of the URL. Structure for the oral questions and written questions drivers is set up, but need fixes due to the differences in the sites. Fixed bug on McNair wiki for women's biz team where email was plain text instead of an email link. Took a glimpse at Kuwait Parliament website, and it appears to be very different from the Moroccan setup.<br /> <br /> 2016-09-30: Selenium program selects view pdf option from the website, and goes to the pdf webpage. Program then switches handle to the new page. CTRL S is sent to the page to launch save dialog window. Text cannot be sent to this window. Brainstorm ways around this issue. Explored Chrome Options for saving automatically without a dialog window. Looking into other libraries besides selenium that may help.<br /> <br /> 2016-09-29: Re-enroll in Microsoft Remote Desktop with proper authentication, set up Selenium environment and Komodo IDE on Remote Desktop, wrote program using Selenium that goes to a link and opens up the print dialog box. Developed computational recipe for a different approach to the problem.<br /> <br /> 2016-09-26: Set up Staff wiki page, work log page; registered for Slack, Microsoft Remote Desktop; downloaded Selenium on personal computer, read Selenium docs. Created wiki page for Moroccan Web Driver Project.<br /> <br /> '''Notes'''<br /> <br /> *Ed moved the Morocco Data to E:\McNair\Projects from C:\Users\PeterJ\Documents<br /> * C Drive files moved to E:\McNair\Users\PeterJ<br /> <br /> [[Category:Work Log]]</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Peter_Jalbert_(Work_Log)&diff=22361 2017-12-15T20:01:42Z

<p>Peterjalbert: </p> <hr /> <div>===Fall 2017===<br /> &lt;onlyinclude&gt; <br /> <br /> [[Peter Jalbert]] [[Work Logs]] [[Peter Jalbert (Work Log)|(log page)]] <br /> <br /> 2017-12-15: Found the error! Ex: L-Spark the, Matter This., Fledge HERE, StartupBootCamp We..., LightBank Start Zero, Entrepreneurs Roundtable Accelerator SELECT, Y Combinator Her, Y Combinator Final, AngelCube class, Matter common<br /> <br /> 2017-12-14: Uploading Morocco Parliament Written Questions. Creating script for next Morocco Parliament download. Begin writing Selenium documentation. Continuing to download TIGER data.<br /> <br /> 2017-12-06: Running Morocco Parliament Written Questions script. Analyzing Demo Day Crawler results. Continued downloading for TIGER geocoder.<br /> <br /> 2017-11-28: Debugging Morocco Parliament Crawler. Running Demo Day Crawler for all accelerators and 10 pages per accelerator. TIGER geocoder is back to Forbidden Error.<br /> <br /> 2017-11-27: Rerunning Morocco Parliament Crawler. Fixed KeyTerms.py and running it again. Continued downloading for TIGER geocoder.<br /> <br /> 2017-11-20: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-16: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-15: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote a script to extract counts that were greater than 2 from Keyword Matcher. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder]. Finished re-formatting work logs.<br /> <br /> 2017-11-14: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote an HTML to Text parser. See Parser Demo Day Page for file location. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder].<br /> <br /> 2017-11-13: Built [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser].<br /> <br /> 2017-11-09: Running demo version of Demo Day crawler (Accelerator Google Crawler). Fixing work log format. <br /> <br /> 2017-11-07: Created file with 0s and 1s detailing whether crunchbase has the founder information for an accelerator. Details posted as a TODO on [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List] page. Still waiting for feedback on the PostGIS installation from [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder]. Continued working on Accelerator Google Crawler.<br /> <br /> 2017-11-06: Contacted Geography Center for the US Census Bureau, [https://www.census.gov/geo/about/contact.html here], and began email exchange on PostGIS installation problems. Began working on the [http://mcnair.bakerinstitute.org/wiki/Selenium_Documentation Selenium Documentation]. Also began working on an Accelerator Google Crawler that will be used with Yang and ML to find Demo Days for cohort companies.<br /> <br /> 2017-11-01: Attempted to continue downloading, however ran into HTTP Forbidden errors. Listed the errors on the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder Page].<br /> <br /> 2017-10-31: Began downloading blocks of data for individual states for the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder] project. Wrote out the new wiki page for installation, and beginning to write documentation on usage.<br /> <br /> 2017-10-30: With Ed's help, was able to get the national data from Tiger installed onto a database server. The process required much jumping around and changing users, and all the things we learned are outlined in [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation#Editing_Users the database server documentation] under &quot;Editing Users&quot;.<br /> <br /> 2017-10-25: Continued working on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation TigerCoder Installation].<br /> <br /> 2017-10-24: Throw some addresses into a database, use address normalizer and geocoder. May need to install things. Details on the installation process can be found on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation page].<br /> <br /> 2017-10-23: Finished Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-19: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-18: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_DistrictHouston Innovation District Project].<br /> <br /> 2017-10-17: Constructed ArcGIS maps for the agglomeration project. Finished maps of points for every year in the state of California. Finished maps of Route 128. Began working on selenium Yelp crawler to get cafe locations within the 610-loop.<br /> <br /> 2017-10-16: Assisted Harrison on the USITC project. Looked for natural language processing tools to extract complaintants and defendants along with their location from case files. Experimented with pulling based on parts of speech tags, as well as using geotext or geograpy to pull locations from a case segment.<br /> <br /> 2017-10-13: Updated various project wiki pages.<br /> <br /> 2017-10-12: Continued work on Patent Thicket project, awaiting further project specs.<br /> <br /> 2017-10-05: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-04: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-02: Worked on ArcGIS data. See Harrison's Work Log for the details.<br /> <br /> 2017-09-28: Added collaborative editing feature to PyCharm. <br /> <br /> 2017-09-27: Worked on big database file.<br /> <br /> 2017-09-25: New task -- Create text file with company, description, and company type.<br /> #[http://mcnair.bakerinstitute.org/wiki/VC_Database_Rebuild VC Database Rebuild]<br /> #psql vcdb2<br /> #table name, sdccompanybasecore2<br /> #Combine with Crunchbasebulk<br /> <br /> #TODO: Write wiki on linkedin crawler, write wiki on creating accounts.<br /> <br /> 2017-09-21: Wrote wiki on Linkedin crawler, met with Laura about patents project. <br /> <br /> 2017-09-20: Finished running linkedin crawler. Transferred data to RDP. Will write wikis next.<br /> <br /> 2017-09-19: Began running linkedin crawler. Helped Yang create RDP account, get permissions, and get wiki setup.<br /> <br /> 2017-09-18: Finished implementation of Experience Crawler, continued working on Education Crawler for LinkedIn.<br /> <br /> 2017-09-14: Continued implementing LinkedIn Crawler for profiles.<br /> <br /> 2017-09-13: Implemented LinkedIn Crawler for main portion of profiles. Began working on crawling Experience section of profiles.<br /> <br /> 2017-09-12: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. Added to the wiki on this topic.<br /> <br /> 2017-09-11: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. <br /> <br /> 2017-09-06: Combined founders data retrieved with the Crunchbase API with the crunchbasebulk data to get linkedin urls for different accelerator founders. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-09-05: Post Harvey. Finished retrieving names from the Crunchbase API on founders. Next step is to query crunchbase bulk database to get linkedin urls. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-08-24: Began using the Crunchbase API to retrieve founder information for accelerators. Halfway through compiling a dictionary that translates accelerator names into proper Crunchbase API URLs.<br /> <br /> 2017-08-23: Decided with Ed to abandon LinkedIn crawling to retrieve accelerator founder data, and instead use crunchbase. Spent the day navigating the crunchbasebulk database, and seeing what useful information was contained in it.<br /> <br /> 2017-08-22: Discovered that LinkedIn Profiles cannot be viewed through LinkedIn if the target is 3rd degree or further. However, if entering LinkedIn through a Google search, the profile can still be viewed if the user has previously logged into LinkedIn. Devising a workaround crawler that utilizes Google search. Continued blog post [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) here] under Section 4.<br /> <br /> 2017-08-21: Began work on extracting founders for accelerators through LinkedIn Crawler. Discovered that Python3 is not installed on RDP, so the virtual environment for the project cannot be fired up. Continued working on Ubuntu machine.<br /> &lt;/onlyinclude&gt; <br /> <br /> ===Spring 2017===<br /> <br /> 2017-05-01: Continued work on HTML Parser. Uploaded all semester projects to git server.<br /> <br /> 2017-04-20: Finished the HTML Parser for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Ran HTML parser on accelerator founders. Data is stored in projects/accelerators/LinkedIn Founder Data.<br /> <br /> 2017-04-19: Made updates to the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler] Wikipage. Ran LinkedIn Crawler on accelerator data. Working on an html parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-18: Ran LinkedIn Crawler on matches between Crunchbase Snapshot and the accelerator data.<br /> <br /> 2017-04-17: Worked on ways to get correct search results from the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Worked on an HTML Parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-13: Worked on debugging the logout procedure for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Began formulation of process to search for founders of startups using a combination of the LinkedIn Crawler with the data resources from the [http://mcnair.bakerinstitute.org/wiki/Crunchbase_2013_Snapshot CrunchBase Snapshot].<br /> <br /> 2017-04-12: Work on bugs with the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler].<br /> <br /> 2017-04-11: Completed functional [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) crawler of LinkedIn Recruiter Pro]. Basic search functions work and download profile information for a given person. <br /> <br /> 2017-04-10: Began writing functioning crawler of LinkedIn. <br /> <br /> 2017-04-06: Continued working on debugging and documenting the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Wrote a test program that logs in, searches for a query, navigates through search pages, and logs out. Recruiter program can now login and search.<br /> <br /> 2017-04-05: Began work on the LinkedIn Crawler. Researched on launching Python Virtual Environment.<br /> <br /> 2017-04-03: Finished debugging points for the Enclosing Circle Algorithm. Added Command Line functionality to the Industry Classifier.<br /> <br /> 2017-03-29: Worked on debugging points for the Enclosing Circle Algorithm.<br /> <br /> 2017-03-28: Finished running the Enclosing Circle Algorithm. Worked on removing incorrect points from the data set(see above). <br /> <br /> 2017-03-27: Worked on debugging the Enclosing Circle Algorithm. Implemented a way to remove interior circles, and determined that translation to latitude and longitude coordinates resulted in slightly off center circles.<br /> <br /> 2017-03-23: Finished debugging the brute force algorithm for [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm]. Implemented a method to plot the points and circles on a graph. Analyzed runtime of the brute force algorithm.<br /> <br /> 2017-03-21: Coded a brute force algorithm for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-20: Worked on debugging the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-09: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Finished script to draw Enclosing Circles on a Google Map.<br /> <br /> 2017-03-08: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Created script to draw outcome of the Enclosing Circle Algorithm on Google Maps.<br /> <br /> 2017-03-07: Redetermined the top 50 cities which Enclosing Circle should be run on. Data on the [http://mcnair.bakerinstitute.org/wiki/Top_Cities_for_VC_Backed_Companies Top 50 Cities for VC Backed Companies can be found here.] Ran [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities.<br /> <br /> 2017-03-06: Ran script to determine the top 50 cities which Enclosing Circle should be run on. Fixed the VC Circles script to take in a new data format.<br /> <br /> 2017-03-02: Cleaned up data for the VC Circles Project. Created histogram of data in Excel. See [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Began work on the [http://mcnair.bakerinstitute.org/wiki/LinkedInCrawlerPython LinkedIn Crawler].<br /> <br /> 2017-03-01: Created statistics for the VC Circles Project.<br /> <br /> 2017-02-28: Finished downloading geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Found bug in Enclosing Circle Algorithm.<br /> <br /> 2017-02-27: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-23: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Installed C++ Compiler for Python. Ran tests on difference between Python and C wrapped Python.<br /> <br /> 2017-02-22: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Helped out with [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-21: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Researched into C++ Compilers for Python so that the Enclosing Circle Algorithm could be wrapped in C. Found a recommended one [https://www.microsoft.com/en-us/download/details.aspx?id=44266 here].<br /> <br /> 2017-02-20: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-16: Reworked [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] to create a file of geocoded data. Began work on wrapping the algorithm in C to improve speed.<br /> <br /> 2017-02-15: Finished [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] applied to the VC study. Enclosing Circle algorithm still needs adjustment, but the program runs with the temporary fixes.<br /> <br /> 2017-02-14: Worked on the application of the Enclosing Circle algorithm to the VC study. Working on bug fixes in the Enclosing Circle algorithm. Created wiki page for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-02-13: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-08: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-07: Fixed bugs in parse_cohort_data.py, the script for parsing the cohort data from the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Added descriptive statistics to cohort data excel file.<br /> <br /> 2017-02-02: Out sick, independent research and work from RDP. Brief research into the [http://jorgeg.scripts.mit.edu/homepage/wp-content/uploads/2016/03/Guzman-Stern-State-of-American-Entrepreneurship-FINAL.pdf Stern-Guzman algorithm]. Research into [http://mcnair.bakerinstitute.org/wiki/interactive_maps Interactive Maps]. No helpful additions to map embedding problem.<br /> <br /> 2017-02-01: Notes from Session with Ed: Project on US university patenting and entrepreneurship programs (writing code to identify universities in assignees), search Wikipedia (XML then bulk download), student pop, faculty pop, etc.<br /> Circle project for VC data will end up being a joint project to join accelerator data. <br /> Pull descriptions for VC. Founders of accelerators in linkedin. LinkedIn cannot be caught(pretend to not be a bot). Can eventually get academic backgrounds through linkedin. <br /> Pull business registration data, Stern/Guzman Algorithm. <br /> GIS ontop of geocoded data.<br /> Maps that works on wiki or blog (CartoDB), Maps API and R.<br /> NLP Projects, Description Classifier.<br /> <br /> 2017-01-31: Built WayBack Machine Crawler. Updated documentation for coordinates script. Updated profile page to include locations of code.<br /> <br /> <br /> 2017-01-30: Optimized enclosing circle algorithm through memoization. Developed script to read addresses from accelerator data and return latitude and longitude coordinates.<br /> <br /> 2017-01-26: Continued working on Google sitesearch project. Discovered crunchbase, changed project priority. Priority 1, split accelerator data up by flag, priority 2, use crunchbase to get web urls for cohorts, priority 3, make internet archive wayback machine driver. Located [http://mcnair.bakerinstitute.org/wiki/Whois_Parser Whois Parser].<br /> <br /> <br /> 2017-01-25: Finished parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Some data files still need proofreading as they are not in an acceptable format. Began working on Google sitesearch project.<br /> <br /> 2017-01-24: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Cohort data file created, debugging is almost complete. Will begin work on the google accelerator search soon.<br /> <br /> 2017-01-23: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Preliminary code is written, working on debugging.<br /> <br /> 2017-01-19: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project]. Created parser for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project], completed creation of final data set(yay!). Began working on cohort parser.<br /> <br /> 2017-01-18: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-13: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-12: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-11: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-10: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> <br /> ===Fall 2016===<br /> <br /> 2016-12-08: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-07: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-06: Learned how to use git. Committed software projects from the semester to the McNair git repository. Projects can be found at; [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Executive Order Crawler], [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Foreign Government Web Crawlers], [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) F6S Crawler and Parser].<br /> <br /> 2016-12-02: Built and ran web crawler for Center for Middle East Studies on Kuwait. Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-01: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Built tool for the [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report E&amp;I Governance Report Project] with Christy. Adds a column of data that shows whether or not the bill has been passed.<br /> <br /> 2016-11-29: Began pulling data from the accelerators listed [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here]. Made text files for about 18 accelerators.<br /> <br /> 2016-11-22: Transferred downloaded Morocco Written Bills to provided SeaGate Drive. Made a &quot;gentle&quot; F6S crawler to retrieve HTMLs of possible accelerator pages documented [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here].<br /> <br /> 2016-11-18: Converted Executive Order PDFs to text files using adobe acrobat DC. See [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Wikipage] for details.<br /> <br /> 2016-11-17: Wrote a crawler to retrieve information about executive orders, and their corresponding pdfs. They can be found [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report here.] Next step is to run code to convert the pdfs to text files, then use the parser fixed by Christy. <br /> <br /> 2016-11-15: Finished download of Moroccan Written Question pdfs. Wrote a parser with Christy to be used for parsing bills from Congress and eventually executive orders. Found bug in the system Python that was worked out and rebooted.<br /> <br /> 2016-11-11: Continued to download Moroccan data in the background. Attempted to find bug fixes for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] crawler.<br /> <br /> 2016-11-10: Continued to download Moroccan data and Kuwait data in the background. Began work on [http://mcnair.bakerinstitute.org/wiki/Google_Scholar_Crawler Google Scholar Crawler]. Wrote a crawler for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] to get the HTML files of hundreds of accelerators. The crawler ended up failing; it appears to have been due to HTTPS. <br /> <br /> 2016-11-08: Continued to download Moroccan data in the background. Finished writing code for the embedded files on the Kuwait Site. Spent time debugging the frame errors due to the dynamically generated content. Never found an answer to the bug, and instead found a workaround that sacrificed run time for the ability to work. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-04: Continued to download Moroccan data in the background. Finished writing initial Kuwait Web Crawler/Driver for the Middle East Studies Department. Middle East Studies Department asked for additional embedded files in the Kuwait website. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-03: Continued to download Moroccan data in the background. Dr. Egan fixed systems requirements to run the GovTrack Web Crawler. Made significant progress on the Kuwait Web Crawler/Driver for the Middle East Studies Department. <br /> <br /> 2016-11-01: Continued to download Moroccan data in the background. Went over code for GovTracker Web Crawler, continued learning Perl. [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] Began Kuwait Web Crawler/Driver.<br /> <br /> 2016-10-21: Continued to download data for the Moroccan Parliament Written and Oral Questions. Looked over [http://mcnair.bakerinstitute.org/wiki/Christy_Warden_(Twitter_Crawler_Application_1) Christy's Twitter Crawler] to see how I can be helpful. Dr. Egan asked me to think about how to potentially make multiple tools to get cohorts and other sorts of data from accelerator sites. See [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator List] He also asked me to look at the [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] for potential ideas on how to bring this project to fruition.<br /> <br /> 2016-10-20: Continued to download data for the Moroccan Parliament Written and Oral Questions. Updated Wiki page. Started working on Twitter project with Christy. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-18: Finished code for Oral Questions web driver and Written Questions web driver using selenium. Now, the data for the dates of questions can be found using the crawler, and the pdfs of the questions will be downloaded using selenium. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-14: Finished Oral Questions crawler. Finished Written Questions crawler. Waiting for further details on whether that data needs to be tweaked in any way. Updated the Moroccan Web Driver/Web Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-13: Completed download of Moroccan Bills. Working on either a web driver screenshot approach or a webcrawler approach to download the Moroccan oral and written questions data. Began building Web Crawler for Oral and Written Questions site. Edited Moroccan Web Driver/Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-11: Fixed arabic bug, files can now be saved with arabic titles. Monarchy bills downloaded and ready for shipment. House of Representatives Bill mostly downloaded, ratified bills prepared for download. Started learning scrapy library in python for web scraping. Discussed idea of screenshot-ing questions instead of scraping. <br /> <br /> 2016-10-07: Learned unicode and utf8 encoding and decoding in arabic. Still working on transforming an ascii url into printable unicode.<br /> <br /> 2016-10-06: Discussed with Dr. Elbadawy about the desired file names for Moroccan data download. The consensus was that the bill programs are ready to launch once the files can be named properly, and the questions data must be retrieved using a web crawler which I need to learn how to implement. The naming of files is currently drawing errors in going from arabic, to url, to download, to filename. Debugging in process. Also built a demo selenium program for Dr. Egan that drives the McNair blog site on an infinite loop.<br /> <br /> 2016-10-03: Moroccan Web Driver projects completed for driving of the Monarchy proposed bills, the House of Representatives proposed bills, and the Ratified bills sites. Begun process of devising a naming system for the files that does not require scraping. Tinkered with naming through regular expression parsing of the URL. Structure for the oral questions and written questions drivers is set up, but need fixes due to the differences in the sites. Fixed bug on McNair wiki for women's biz team where email was plain text instead of an email link. Took a glimpse at Kuwait Parliament website, and it appears to be very different from the Moroccan setup.<br /> <br /> 2016-09-30: Selenium program selects view pdf option from the website, and goes to the pdf webpage. Program then switches handle to the new page. CTRL S is sent to the page to launch save dialog window. Text cannot be sent to this window. Brainstorm ways around this issue. Explored Chrome Options for saving automatically without a dialog window. Looking into other libraries besides selenium that may help.<br /> <br /> 2016-09-29: Re-enroll in Microsoft Remote Desktop with proper authentication, set up Selenium environment and Komodo IDE on Remote Desktop, wrote program using Selenium that goes to a link and opens up the print dialog box. Developed computational recipe for a different approach to the problem.<br /> <br /> 2016-09-26: Set up Staff wiki page, work log page; registered for Slack, Microsoft Remote Desktop; downloaded Selenium on personal computer, read Selenium docs. Created wiki page for Moroccan Web Driver Project.<br /> <br /> '''Notes'''<br /> <br /> *Ed moved the Morocco Data to E:\McNair\Projects from C:\Users\PeterJ\Documents<br /> * C Drive files moved to E:\McNair\Users\PeterJ<br /> <br /> [[Category:Work Log]]</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Peter_Jalbert_(Work_Log)&diff=22360 2017-12-15T19:57:45Z

<p>Peterjalbert: </p> <hr /> <div>===Fall 2017===<br /> &lt;onlyinclude&gt; <br /> <br /> [[Peter Jalbert]] [[Work Logs]] [[Peter Jalbert (Work Log)|(log page)]] <br /> <br /> 2017-12-15: Found the error! Ex: L-Spark the, Matter This., Fledge HERE, StartupBootCamp We..., LightBank Start Zero, Entrepreneurs Roundtable Accelerator SELECT, Y Combinator Her, Y Combinator Final, AngelCube class<br /> <br /> 2017-12-14: Uploading Morocco Parliament Written Questions. Creating script for next Morocco Parliament download. Begin writing Selenium documentation. Continuing to download TIGER data.<br /> <br /> 2017-12-06: Running Morocco Parliament Written Questions script. Analyzing Demo Day Crawler results. Continued downloading for TIGER geocoder.<br /> <br /> 2017-11-28: Debugging Morocco Parliament Crawler. Running Demo Day Crawler for all accelerators and 10 pages per accelerator. TIGER geocoder is back to Forbidden Error.<br /> <br /> 2017-11-27: Rerunning Morocco Parliament Crawler. Fixed KeyTerms.py and running it again. Continued downloading for TIGER geocoder.<br /> <br /> 2017-11-20: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-16: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-15: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote a script to extract counts that were greater than 2 from Keyword Matcher. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder]. Finished re-formatting work logs.<br /> <br /> 2017-11-14: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote an HTML to Text parser. See Parser Demo Day Page for file location. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder].<br /> <br /> 2017-11-13: Built [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser].<br /> <br /> 2017-11-09: Running demo version of Demo Day crawler (Accelerator Google Crawler). Fixing work log format. <br /> <br /> 2017-11-07: Created file with 0s and 1s detailing whether crunchbase has the founder information for an accelerator. Details posted as a TODO on [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List] page. Still waiting for feedback on the PostGIS installation from [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder]. Continued working on Accelerator Google Crawler.<br /> <br /> 2017-11-06: Contacted Geography Center for the US Census Bureau, [https://www.census.gov/geo/about/contact.html here], and began email exchange on PostGIS installation problems. Began working on the [http://mcnair.bakerinstitute.org/wiki/Selenium_Documentation Selenium Documentation]. Also began working on an Accelerator Google Crawler that will be used with Yang and ML to find Demo Days for cohort companies.<br /> <br /> 2017-11-01: Attempted to continue downloading, however ran into HTTP Forbidden errors. Listed the errors on the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder Page].<br /> <br /> 2017-10-31: Began downloading blocks of data for individual states for the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder] project. Wrote out the new wiki page for installation, and beginning to write documentation on usage.<br /> <br /> 2017-10-30: With Ed's help, was able to get the national data from Tiger installed onto a database server. The process required much jumping around and changing users, and all the things we learned are outlined in [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation#Editing_Users the database server documentation] under &quot;Editing Users&quot;.<br /> <br /> 2017-10-25: Continued working on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation TigerCoder Installation].<br /> <br /> 2017-10-24: Throw some addresses into a database, use address normalizer and geocoder. May need to install things. Details on the installation process can be found on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation page].<br /> <br /> 2017-10-23: Finished Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-19: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-18: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_DistrictHouston Innovation District Project].<br /> <br /> 2017-10-17: Constructed ArcGIS maps for the agglomeration project. Finished maps of points for every year in the state of California. Finished maps of Route 128. Began working on selenium Yelp crawler to get cafe locations within the 610-loop.<br /> <br /> 2017-10-16: Assisted Harrison on the USITC project. Looked for natural language processing tools to extract complaintants and defendants along with their location from case files. Experimented with pulling based on parts of speech tags, as well as using geotext or geograpy to pull locations from a case segment.<br /> <br /> 2017-10-13: Updated various project wiki pages.<br /> <br /> 2017-10-12: Continued work on Patent Thicket project, awaiting further project specs.<br /> <br /> 2017-10-05: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-04: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-02: Worked on ArcGIS data. See Harrison's Work Log for the details.<br /> <br /> 2017-09-28: Added collaborative editing feature to PyCharm. <br /> <br /> 2017-09-27: Worked on big database file.<br /> <br /> 2017-09-25: New task -- Create text file with company, description, and company type.<br /> #[http://mcnair.bakerinstitute.org/wiki/VC_Database_Rebuild VC Database Rebuild]<br /> #psql vcdb2<br /> #table name, sdccompanybasecore2<br /> #Combine with Crunchbasebulk<br /> <br /> #TODO: Write wiki on linkedin crawler, write wiki on creating accounts.<br /> <br /> 2017-09-21: Wrote wiki on Linkedin crawler, met with Laura about patents project. <br /> <br /> 2017-09-20: Finished running linkedin crawler. Transferred data to RDP. Will write wikis next.<br /> <br /> 2017-09-19: Began running linkedin crawler. Helped Yang create RDP account, get permissions, and get wiki setup.<br /> <br /> 2017-09-18: Finished implementation of Experience Crawler, continued working on Education Crawler for LinkedIn.<br /> <br /> 2017-09-14: Continued implementing LinkedIn Crawler for profiles.<br /> <br /> 2017-09-13: Implemented LinkedIn Crawler for main portion of profiles. Began working on crawling Experience section of profiles.<br /> <br /> 2017-09-12: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. Added to the wiki on this topic.<br /> <br /> 2017-09-11: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. <br /> <br /> 2017-09-06: Combined founders data retrieved with the Crunchbase API with the crunchbasebulk data to get linkedin urls for different accelerator founders. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-09-05: Post Harvey. Finished retrieving names from the Crunchbase API on founders. Next step is to query crunchbase bulk database to get linkedin urls. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-08-24: Began using the Crunchbase API to retrieve founder information for accelerators. Halfway through compiling a dictionary that translates accelerator names into proper Crunchbase API URLs.<br /> <br /> 2017-08-23: Decided with Ed to abandon LinkedIn crawling to retrieve accelerator founder data, and instead use crunchbase. Spent the day navigating the crunchbasebulk database, and seeing what useful information was contained in it.<br /> <br /> 2017-08-22: Discovered that LinkedIn Profiles cannot be viewed through LinkedIn if the target is 3rd degree or further. However, if entering LinkedIn through a Google search, the profile can still be viewed if the user has previously logged into LinkedIn. Devising a workaround crawler that utilizes Google search. Continued blog post [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) here] under Section 4.<br /> <br /> 2017-08-21: Began work on extracting founders for accelerators through LinkedIn Crawler. Discovered that Python3 is not installed on RDP, so the virtual environment for the project cannot be fired up. Continued working on Ubuntu machine.<br /> &lt;/onlyinclude&gt; <br /> <br /> ===Spring 2017===<br /> <br /> 2017-05-01: Continued work on HTML Parser. Uploaded all semester projects to git server.<br /> <br /> 2017-04-20: Finished the HTML Parser for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Ran HTML parser on accelerator founders. Data is stored in projects/accelerators/LinkedIn Founder Data.<br /> <br /> 2017-04-19: Made updates to the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler] Wikipage. Ran LinkedIn Crawler on accelerator data. Working on an html parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-18: Ran LinkedIn Crawler on matches between Crunchbase Snapshot and the accelerator data.<br /> <br /> 2017-04-17: Worked on ways to get correct search results from the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Worked on an HTML Parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-13: Worked on debugging the logout procedure for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Began formulation of process to search for founders of startups using a combination of the LinkedIn Crawler with the data resources from the [http://mcnair.bakerinstitute.org/wiki/Crunchbase_2013_Snapshot CrunchBase Snapshot].<br /> <br /> 2017-04-12: Work on bugs with the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler].<br /> <br /> 2017-04-11: Completed functional [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) crawler of LinkedIn Recruiter Pro]. Basic search functions work and download profile information for a given person. <br /> <br /> 2017-04-10: Began writing functioning crawler of LinkedIn. <br /> <br /> 2017-04-06: Continued working on debugging and documenting the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Wrote a test program that logs in, searches for a query, navigates through search pages, and logs out. Recruiter program can now login and search.<br /> <br /> 2017-04-05: Began work on the LinkedIn Crawler. Researched on launching Python Virtual Environment.<br /> <br /> 2017-04-03: Finished debugging points for the Enclosing Circle Algorithm. Added Command Line functionality to the Industry Classifier.<br /> <br /> 2017-03-29: Worked on debugging points for the Enclosing Circle Algorithm.<br /> <br /> 2017-03-28: Finished running the Enclosing Circle Algorithm. Worked on removing incorrect points from the data set(see above). <br /> <br /> 2017-03-27: Worked on debugging the Enclosing Circle Algorithm. Implemented a way to remove interior circles, and determined that translation to latitude and longitude coordinates resulted in slightly off center circles.<br /> <br /> 2017-03-23: Finished debugging the brute force algorithm for [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm]. Implemented a method to plot the points and circles on a graph. Analyzed runtime of the brute force algorithm.<br /> <br /> 2017-03-21: Coded a brute force algorithm for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-20: Worked on debugging the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-09: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Finished script to draw Enclosing Circles on a Google Map.<br /> <br /> 2017-03-08: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Created script to draw outcome of the Enclosing Circle Algorithm on Google Maps.<br /> <br /> 2017-03-07: Redetermined the top 50 cities which Enclosing Circle should be run on. Data on the [http://mcnair.bakerinstitute.org/wiki/Top_Cities_for_VC_Backed_Companies Top 50 Cities for VC Backed Companies can be found here.] Ran [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities.<br /> <br /> 2017-03-06: Ran script to determine the top 50 cities which Enclosing Circle should be run on. Fixed the VC Circles script to take in a new data format.<br /> <br /> 2017-03-02: Cleaned up data for the VC Circles Project. Created histogram of data in Excel. See [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Began work on the [http://mcnair.bakerinstitute.org/wiki/LinkedInCrawlerPython LinkedIn Crawler].<br /> <br /> 2017-03-01: Created statistics for the VC Circles Project.<br /> <br /> 2017-02-28: Finished downloading geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Found bug in Enclosing Circle Algorithm.<br /> <br /> 2017-02-27: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-23: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Installed C++ Compiler for Python. Ran tests on difference between Python and C wrapped Python.<br /> <br /> 2017-02-22: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Helped out with [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-21: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Researched into C++ Compilers for Python so that the Enclosing Circle Algorithm could be wrapped in C. Found a recommended one [https://www.microsoft.com/en-us/download/details.aspx?id=44266 here].<br /> <br /> 2017-02-20: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-16: Reworked [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] to create a file of geocoded data. Began work on wrapping the algorithm in C to improve speed.<br /> <br /> 2017-02-15: Finished [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] applied to the VC study. Enclosing Circle algorithm still needs adjustment, but the program runs with the temporary fixes.<br /> <br /> 2017-02-14: Worked on the application of the Enclosing Circle algorithm to the VC study. Working on bug fixes in the Enclosing Circle algorithm. Created wiki page for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-02-13: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-08: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-07: Fixed bugs in parse_cohort_data.py, the script for parsing the cohort data from the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Added descriptive statistics to cohort data excel file.<br /> <br /> 2017-02-02: Out sick, independent research and work from RDP. Brief research into the [http://jorgeg.scripts.mit.edu/homepage/wp-content/uploads/2016/03/Guzman-Stern-State-of-American-Entrepreneurship-FINAL.pdf Stern-Guzman algorithm]. Research into [http://mcnair.bakerinstitute.org/wiki/interactive_maps Interactive Maps]. No helpful additions to map embedding problem.<br /> <br /> 2017-02-01: Notes from Session with Ed: Project on US university patenting and entrepreneurship programs (writing code to identify universities in assignees), search Wikipedia (XML then bulk download), student pop, faculty pop, etc.<br /> Circle project for VC data will end up being a joint project to join accelerator data. <br /> Pull descriptions for VC. Founders of accelerators in linkedin. LinkedIn cannot be caught(pretend to not be a bot). Can eventually get academic backgrounds through linkedin. <br /> Pull business registration data, Stern/Guzman Algorithm. <br /> GIS ontop of geocoded data.<br /> Maps that works on wiki or blog (CartoDB), Maps API and R.<br /> NLP Projects, Description Classifier.<br /> <br /> 2017-01-31: Built WayBack Machine Crawler. Updated documentation for coordinates script. Updated profile page to include locations of code.<br /> <br /> <br /> 2017-01-30: Optimized enclosing circle algorithm through memoization. Developed script to read addresses from accelerator data and return latitude and longitude coordinates.<br /> <br /> 2017-01-26: Continued working on Google sitesearch project. Discovered crunchbase, changed project priority. Priority 1, split accelerator data up by flag, priority 2, use crunchbase to get web urls for cohorts, priority 3, make internet archive wayback machine driver. Located [http://mcnair.bakerinstitute.org/wiki/Whois_Parser Whois Parser].<br /> <br /> <br /> 2017-01-25: Finished parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Some data files still need proofreading as they are not in an acceptable format. Began working on Google sitesearch project.<br /> <br /> 2017-01-24: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Cohort data file created, debugging is almost complete. Will begin work on the google accelerator search soon.<br /> <br /> 2017-01-23: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Preliminary code is written, working on debugging.<br /> <br /> 2017-01-19: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project]. Created parser for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project], completed creation of final data set(yay!). Began working on cohort parser.<br /> <br /> 2017-01-18: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-13: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-12: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-11: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-10: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> <br /> ===Fall 2016===<br /> <br /> 2016-12-08: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-07: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-06: Learned how to use git. Committed software projects from the semester to the McNair git repository. Projects can be found at; [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Executive Order Crawler], [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Foreign Government Web Crawlers], [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) F6S Crawler and Parser].<br /> <br /> 2016-12-02: Built and ran web crawler for Center for Middle East Studies on Kuwait. Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-01: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Built tool for the [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report E&amp;I Governance Report Project] with Christy. Adds a column of data that shows whether or not the bill has been passed.<br /> <br /> 2016-11-29: Began pulling data from the accelerators listed [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here]. Made text files for about 18 accelerators.<br /> <br /> 2016-11-22: Transferred downloaded Morocco Written Bills to provided SeaGate Drive. Made a &quot;gentle&quot; F6S crawler to retrieve HTMLs of possible accelerator pages documented [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here].<br /> <br /> 2016-11-18: Converted Executive Order PDFs to text files using adobe acrobat DC. See [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Wikipage] for details.<br /> <br /> 2016-11-17: Wrote a crawler to retrieve information about executive orders, and their corresponding pdfs. They can be found [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report here.] Next step is to run code to convert the pdfs to text files, then use the parser fixed by Christy. <br /> <br /> 2016-11-15: Finished download of Moroccan Written Question pdfs. Wrote a parser with Christy to be used for parsing bills from Congress and eventually executive orders. Found bug in the system Python that was worked out and rebooted.<br /> <br /> 2016-11-11: Continued to download Moroccan data in the background. Attempted to find bug fixes for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] crawler.<br /> <br /> 2016-11-10: Continued to download Moroccan data and Kuwait data in the background. Began work on [http://mcnair.bakerinstitute.org/wiki/Google_Scholar_Crawler Google Scholar Crawler]. Wrote a crawler for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] to get the HTML files of hundreds of accelerators. The crawler ended up failing; it appears to have been due to HTTPS. <br /> <br /> 2016-11-08: Continued to download Moroccan data in the background. Finished writing code for the embedded files on the Kuwait Site. Spent time debugging the frame errors due to the dynamically generated content. Never found an answer to the bug, and instead found a workaround that sacrificed run time for the ability to work. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-04: Continued to download Moroccan data in the background. Finished writing initial Kuwait Web Crawler/Driver for the Middle East Studies Department. Middle East Studies Department asked for additional embedded files in the Kuwait website. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-03: Continued to download Moroccan data in the background. Dr. Egan fixed systems requirements to run the GovTrack Web Crawler. Made significant progress on the Kuwait Web Crawler/Driver for the Middle East Studies Department. <br /> <br /> 2016-11-01: Continued to download Moroccan data in the background. Went over code for GovTracker Web Crawler, continued learning Perl. [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] Began Kuwait Web Crawler/Driver.<br /> <br /> 2016-10-21: Continued to download data for the Moroccan Parliament Written and Oral Questions. Looked over [http://mcnair.bakerinstitute.org/wiki/Christy_Warden_(Twitter_Crawler_Application_1) Christy's Twitter Crawler] to see how I can be helpful. Dr. Egan asked me to think about how to potentially make multiple tools to get cohorts and other sorts of data from accelerator sites. See [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator List] He also asked me to look at the [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] for potential ideas on how to bring this project to fruition.<br /> <br /> 2016-10-20: Continued to download data for the Moroccan Parliament Written and Oral Questions. Updated Wiki page. Started working on Twitter project with Christy. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-18: Finished code for Oral Questions web driver and Written Questions web driver using selenium. Now, the data for the dates of questions can be found using the crawler, and the pdfs of the questions will be downloaded using selenium. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-14: Finished Oral Questions crawler. Finished Written Questions crawler. Waiting for further details on whether that data needs to be tweaked in any way. Updated the Moroccan Web Driver/Web Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-13: Completed download of Moroccan Bills. Working on either a web driver screenshot approach or a webcrawler approach to download the Moroccan oral and written questions data. Began building Web Crawler for Oral and Written Questions site. Edited Moroccan Web Driver/Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-11: Fixed arabic bug, files can now be saved with arabic titles. Monarchy bills downloaded and ready for shipment. House of Representatives Bill mostly downloaded, ratified bills prepared for download. Started learning scrapy library in python for web scraping. Discussed idea of screenshot-ing questions instead of scraping. <br /> <br /> 2016-10-07: Learned unicode and utf8 encoding and decoding in arabic. Still working on transforming an ascii url into printable unicode.<br /> <br /> 2016-10-06: Discussed with Dr. Elbadawy about the desired file names for Moroccan data download. The consensus was that the bill programs are ready to launch once the files can be named properly, and the questions data must be retrieved using a web crawler which I need to learn how to implement. The naming of files is currently drawing errors in going from arabic, to url, to download, to filename. Debugging in process. Also built a demo selenium program for Dr. Egan that drives the McNair blog site on an infinite loop.<br /> <br /> 2016-10-03: Moroccan Web Driver projects completed for driving of the Monarchy proposed bills, the House of Representatives proposed bills, and the Ratified bills sites. Begun process of devising a naming system for the files that does not require scraping. Tinkered with naming through regular expression parsing of the URL. Structure for the oral questions and written questions drivers is set up, but need fixes due to the differences in the sites. Fixed bug on McNair wiki for women's biz team where email was plain text instead of an email link. Took a glimpse at Kuwait Parliament website, and it appears to be very different from the Moroccan setup.<br /> <br /> 2016-09-30: Selenium program selects view pdf option from the website, and goes to the pdf webpage. Program then switches handle to the new page. CTRL S is sent to the page to launch save dialog window. Text cannot be sent to this window. Brainstorm ways around this issue. Explored Chrome Options for saving automatically without a dialog window. Looking into other libraries besides selenium that may help.<br /> <br /> 2016-09-29: Re-enroll in Microsoft Remote Desktop with proper authentication, set up Selenium environment and Komodo IDE on Remote Desktop, wrote program using Selenium that goes to a link and opens up the print dialog box. Developed computational recipe for a different approach to the problem.<br /> <br /> 2016-09-26: Set up Staff wiki page, work log page; registered for Slack, Microsoft Remote Desktop; downloaded Selenium on personal computer, read Selenium docs. Created wiki page for Moroccan Web Driver Project.<br /> <br /> '''Notes'''<br /> <br /> *Ed moved the Morocco Data to E:\McNair\Projects from C:\Users\PeterJ\Documents<br /> * C Drive files moved to E:\McNair\Users\PeterJ<br /> <br /> [[Category:Work Log]]</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Peter_Jalbert_(Work_Log)&diff=22359 2017-12-15T18:00:00Z

<p>Peterjalbert: </p> <hr /> <div>===Fall 2017===<br /> &lt;onlyinclude&gt; <br /> <br /> [[Peter Jalbert]] [[Work Logs]] [[Peter Jalbert (Work Log)|(log page)]] <br /> <br /> 2017-12-15: Found the error! Ex: L-Spark the, Matter This.<br /> <br /> 2017-12-14: Uploading Morocco Parliament Written Questions. Creating script for next Morocco Parliament download. Begin writing Selenium documentation. Continuing to download TIGER data.<br /> <br /> 2017-12-06: Running Morocco Parliament Written Questions script. Analyzing Demo Day Crawler results. Continued downloading for TIGER geocoder.<br /> <br /> 2017-11-28: Debugging Morocco Parliament Crawler. Running Demo Day Crawler for all accelerators and 10 pages per accelerator. TIGER geocoder is back to Forbidden Error.<br /> <br /> 2017-11-27: Rerunning Morocco Parliament Crawler. Fixed KeyTerms.py and running it again. Continued downloading for TIGER geocoder.<br /> <br /> 2017-11-20: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-16: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-15: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote a script to extract counts that were greater than 2 from Keyword Matcher. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder]. Finished re-formatting work logs.<br /> <br /> 2017-11-14: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote an HTML to Text parser. See Parser Demo Day Page for file location. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder].<br /> <br /> 2017-11-13: Built [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser].<br /> <br /> 2017-11-09: Running demo version of Demo Day crawler (Accelerator Google Crawler). Fixing work log format. <br /> <br /> 2017-11-07: Created file with 0s and 1s detailing whether crunchbase has the founder information for an accelerator. Details posted as a TODO on [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List] page. Still waiting for feedback on the PostGIS installation from [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder]. Continued working on Accelerator Google Crawler.<br /> <br /> 2017-11-06: Contacted Geography Center for the US Census Bureau, [https://www.census.gov/geo/about/contact.html here], and began email exchange on PostGIS installation problems. Began working on the [http://mcnair.bakerinstitute.org/wiki/Selenium_Documentation Selenium Documentation]. Also began working on an Accelerator Google Crawler that will be used with Yang and ML to find Demo Days for cohort companies.<br /> <br /> 2017-11-01: Attempted to continue downloading, however ran into HTTP Forbidden errors. Listed the errors on the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder Page].<br /> <br /> 2017-10-31: Began downloading blocks of data for individual states for the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder] project. Wrote out the new wiki page for installation, and beginning to write documentation on usage.<br /> <br /> 2017-10-30: With Ed's help, was able to get the national data from Tiger installed onto a database server. The process required much jumping around and changing users, and all the things we learned are outlined in [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation#Editing_Users the database server documentation] under &quot;Editing Users&quot;.<br /> <br /> 2017-10-25: Continued working on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation TigerCoder Installation].<br /> <br /> 2017-10-24: Throw some addresses into a database, use address normalizer and geocoder. May need to install things. Details on the installation process can be found on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation page].<br /> <br /> 2017-10-23: Finished Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-19: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-18: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_DistrictHouston Innovation District Project].<br /> <br /> 2017-10-17: Constructed ArcGIS maps for the agglomeration project. Finished maps of points for every year in the state of California. Finished maps of Route 128. Began working on selenium Yelp crawler to get cafe locations within the 610-loop.<br /> <br /> 2017-10-16: Assisted Harrison on the USITC project. Looked for natural language processing tools to extract complaintants and defendants along with their location from case files. Experimented with pulling based on parts of speech tags, as well as using geotext or geograpy to pull locations from a case segment.<br /> <br /> 2017-10-13: Updated various project wiki pages.<br /> <br /> 2017-10-12: Continued work on Patent Thicket project, awaiting further project specs.<br /> <br /> 2017-10-05: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-04: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-02: Worked on ArcGIS data. See Harrison's Work Log for the details.<br /> <br /> 2017-09-28: Added collaborative editing feature to PyCharm. <br /> <br /> 2017-09-27: Worked on big database file.<br /> <br /> 2017-09-25: New task -- Create text file with company, description, and company type.<br /> #[http://mcnair.bakerinstitute.org/wiki/VC_Database_Rebuild VC Database Rebuild]<br /> #psql vcdb2<br /> #table name, sdccompanybasecore2<br /> #Combine with Crunchbasebulk<br /> <br /> #TODO: Write wiki on linkedin crawler, write wiki on creating accounts.<br /> <br /> 2017-09-21: Wrote wiki on Linkedin crawler, met with Laura about patents project. <br /> <br /> 2017-09-20: Finished running linkedin crawler. Transferred data to RDP. Will write wikis next.<br /> <br /> 2017-09-19: Began running linkedin crawler. Helped Yang create RDP account, get permissions, and get wiki setup.<br /> <br /> 2017-09-18: Finished implementation of Experience Crawler, continued working on Education Crawler for LinkedIn.<br /> <br /> 2017-09-14: Continued implementing LinkedIn Crawler for profiles.<br /> <br /> 2017-09-13: Implemented LinkedIn Crawler for main portion of profiles. Began working on crawling Experience section of profiles.<br /> <br /> 2017-09-12: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. Added to the wiki on this topic.<br /> <br /> 2017-09-11: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. <br /> <br /> 2017-09-06: Combined founders data retrieved with the Crunchbase API with the crunchbasebulk data to get linkedin urls for different accelerator founders. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-09-05: Post Harvey. Finished retrieving names from the Crunchbase API on founders. Next step is to query crunchbase bulk database to get linkedin urls. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-08-24: Began using the Crunchbase API to retrieve founder information for accelerators. Halfway through compiling a dictionary that translates accelerator names into proper Crunchbase API URLs.<br /> <br /> 2017-08-23: Decided with Ed to abandon LinkedIn crawling to retrieve accelerator founder data, and instead use crunchbase. Spent the day navigating the crunchbasebulk database, and seeing what useful information was contained in it.<br /> <br /> 2017-08-22: Discovered that LinkedIn Profiles cannot be viewed through LinkedIn if the target is 3rd degree or further. However, if entering LinkedIn through a Google search, the profile can still be viewed if the user has previously logged into LinkedIn. Devising a workaround crawler that utilizes Google search. Continued blog post [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) here] under Section 4.<br /> <br /> 2017-08-21: Began work on extracting founders for accelerators through LinkedIn Crawler. Discovered that Python3 is not installed on RDP, so the virtual environment for the project cannot be fired up. Continued working on Ubuntu machine.<br /> &lt;/onlyinclude&gt; <br /> <br /> ===Spring 2017===<br /> <br /> 2017-05-01: Continued work on HTML Parser. Uploaded all semester projects to git server.<br /> <br /> 2017-04-20: Finished the HTML Parser for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Ran HTML parser on accelerator founders. Data is stored in projects/accelerators/LinkedIn Founder Data.<br /> <br /> 2017-04-19: Made updates to the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler] Wikipage. Ran LinkedIn Crawler on accelerator data. Working on an html parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-18: Ran LinkedIn Crawler on matches between Crunchbase Snapshot and the accelerator data.<br /> <br /> 2017-04-17: Worked on ways to get correct search results from the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Worked on an HTML Parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-13: Worked on debugging the logout procedure for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Began formulation of process to search for founders of startups using a combination of the LinkedIn Crawler with the data resources from the [http://mcnair.bakerinstitute.org/wiki/Crunchbase_2013_Snapshot CrunchBase Snapshot].<br /> <br /> 2017-04-12: Work on bugs with the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler].<br /> <br /> 2017-04-11: Completed functional [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) crawler of LinkedIn Recruiter Pro]. Basic search functions work and download profile information for a given person. <br /> <br /> 2017-04-10: Began writing functioning crawler of LinkedIn. <br /> <br /> 2017-04-06: Continued working on debugging and documenting the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Wrote a test program that logs in, searches for a query, navigates through search pages, and logs out. Recruiter program can now login and search.<br /> <br /> 2017-04-05: Began work on the LinkedIn Crawler. Researched on launching Python Virtual Environment.<br /> <br /> 2017-04-03: Finished debugging points for the Enclosing Circle Algorithm. Added Command Line functionality to the Industry Classifier.<br /> <br /> 2017-03-29: Worked on debugging points for the Enclosing Circle Algorithm.<br /> <br /> 2017-03-28: Finished running the Enclosing Circle Algorithm. Worked on removing incorrect points from the data set(see above). <br /> <br /> 2017-03-27: Worked on debugging the Enclosing Circle Algorithm. Implemented a way to remove interior circles, and determined that translation to latitude and longitude coordinates resulted in slightly off center circles.<br /> <br /> 2017-03-23: Finished debugging the brute force algorithm for [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm]. Implemented a method to plot the points and circles on a graph. Analyzed runtime of the brute force algorithm.<br /> <br /> 2017-03-21: Coded a brute force algorithm for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-20: Worked on debugging the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-09: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Finished script to draw Enclosing Circles on a Google Map.<br /> <br /> 2017-03-08: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Created script to draw outcome of the Enclosing Circle Algorithm on Google Maps.<br /> <br /> 2017-03-07: Redetermined the top 50 cities which Enclosing Circle should be run on. Data on the [http://mcnair.bakerinstitute.org/wiki/Top_Cities_for_VC_Backed_Companies Top 50 Cities for VC Backed Companies can be found here.] Ran [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities.<br /> <br /> 2017-03-06: Ran script to determine the top 50 cities which Enclosing Circle should be run on. Fixed the VC Circles script to take in a new data format.<br /> <br /> 2017-03-02: Cleaned up data for the VC Circles Project. Created histogram of data in Excel. See [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Began work on the [http://mcnair.bakerinstitute.org/wiki/LinkedInCrawlerPython LinkedIn Crawler].<br /> <br /> 2017-03-01: Created statistics for the VC Circles Project.<br /> <br /> 2017-02-28: Finished downloading geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Found bug in Enclosing Circle Algorithm.<br /> <br /> 2017-02-27: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-23: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Installed C++ Compiler for Python. Ran tests on difference between Python and C wrapped Python.<br /> <br /> 2017-02-22: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Helped out with [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-21: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Researched into C++ Compilers for Python so that the Enclosing Circle Algorithm could be wrapped in C. Found a recommended one [https://www.microsoft.com/en-us/download/details.aspx?id=44266 here].<br /> <br /> 2017-02-20: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-16: Reworked [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] to create a file of geocoded data. Began work on wrapping the algorithm in C to improve speed.<br /> <br /> 2017-02-15: Finished [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] applied to the VC study. Enclosing Circle algorithm still needs adjustment, but the program runs with the temporary fixes.<br /> <br /> 2017-02-14: Worked on the application of the Enclosing Circle algorithm to the VC study. Working on bug fixes in the Enclosing Circle algorithm. Created wiki page for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-02-13: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-08: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-07: Fixed bugs in parse_cohort_data.py, the script for parsing the cohort data from the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Added descriptive statistics to cohort data excel file.<br /> <br /> 2017-02-02: Out sick, independent research and work from RDP. Brief research into the [http://jorgeg.scripts.mit.edu/homepage/wp-content/uploads/2016/03/Guzman-Stern-State-of-American-Entrepreneurship-FINAL.pdf Stern-Guzman algorithm]. Research into [http://mcnair.bakerinstitute.org/wiki/interactive_maps Interactive Maps]. No helpful additions to map embedding problem.<br /> <br /> 2017-02-01: Notes from Session with Ed: Project on US university patenting and entrepreneurship programs (writing code to identify universities in assignees), search Wikipedia (XML then bulk download), student pop, faculty pop, etc.<br /> Circle project for VC data will end up being a joint project to join accelerator data. <br /> Pull descriptions for VC. Founders of accelerators in linkedin. LinkedIn cannot be caught(pretend to not be a bot). Can eventually get academic backgrounds through linkedin. <br /> Pull business registration data, Stern/Guzman Algorithm. <br /> GIS ontop of geocoded data.<br /> Maps that works on wiki or blog (CartoDB), Maps API and R.<br /> NLP Projects, Description Classifier.<br /> <br /> 2017-01-31: Built WayBack Machine Crawler. Updated documentation for coordinates script. Updated profile page to include locations of code.<br /> <br /> <br /> 2017-01-30: Optimized enclosing circle algorithm through memoization. Developed script to read addresses from accelerator data and return latitude and longitude coordinates.<br /> <br /> 2017-01-26: Continued working on Google sitesearch project. Discovered crunchbase, changed project priority. Priority 1, split accelerator data up by flag, priority 2, use crunchbase to get web urls for cohorts, priority 3, make internet archive wayback machine driver. Located [http://mcnair.bakerinstitute.org/wiki/Whois_Parser Whois Parser].<br /> <br /> <br /> 2017-01-25: Finished parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Some data files still need proofreading as they are not in an acceptable format. Began working on Google sitesearch project.<br /> <br /> 2017-01-24: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Cohort data file created, debugging is almost complete. Will begin work on the google accelerator search soon.<br /> <br /> 2017-01-23: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Preliminary code is written, working on debugging.<br /> <br /> 2017-01-19: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project]. Created parser for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project], completed creation of final data set(yay!). Began working on cohort parser.<br /> <br /> 2017-01-18: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-13: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-12: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-11: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-10: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> <br /> ===Fall 2016===<br /> <br /> 2016-12-08: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-07: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-06: Learned how to use git. Committed software projects from the semester to the McNair git repository. Projects can be found at; [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Executive Order Crawler], [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Foreign Government Web Crawlers], [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) F6S Crawler and Parser].<br /> <br /> 2016-12-02: Built and ran web crawler for Center for Middle East Studies on Kuwait. Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-01: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Built tool for the [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report E&amp;I Governance Report Project] with Christy. Adds a column of data that shows whether or not the bill has been passed.<br /> <br /> 2016-11-29: Began pulling data from the accelerators listed [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here]. Made text files for about 18 accelerators.<br /> <br /> 2016-11-22: Transferred downloaded Morocco Written Bills to provided SeaGate Drive. Made a &quot;gentle&quot; F6S crawler to retrieve HTMLs of possible accelerator pages documented [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here].<br /> <br /> 2016-11-18: Converted Executive Order PDFs to text files using adobe acrobat DC. See [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Wikipage] for details.<br /> <br /> 2016-11-17: Wrote a crawler to retrieve information about executive orders, and their corresponding pdfs. They can be found [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report here.] Next step is to run code to convert the pdfs to text files, then use the parser fixed by Christy. <br /> <br /> 2016-11-15: Finished download of Moroccan Written Question pdfs. Wrote a parser with Christy to be used for parsing bills from Congress and eventually executive orders. Found bug in the system Python that was worked out and rebooted.<br /> <br /> 2016-11-11: Continued to download Moroccan data in the background. Attempted to find bug fixes for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] crawler.<br /> <br /> 2016-11-10: Continued to download Moroccan data and Kuwait data in the background. Began work on [http://mcnair.bakerinstitute.org/wiki/Google_Scholar_Crawler Google Scholar Crawler]. Wrote a crawler for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] to get the HTML files of hundreds of accelerators. The crawler ended up failing; it appears to have been due to HTTPS. <br /> <br /> 2016-11-08: Continued to download Moroccan data in the background. Finished writing code for the embedded files on the Kuwait Site. Spent time debugging the frame errors due to the dynamically generated content. Never found an answer to the bug, and instead found a workaround that sacrificed run time for the ability to work. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-04: Continued to download Moroccan data in the background. Finished writing initial Kuwait Web Crawler/Driver for the Middle East Studies Department. Middle East Studies Department asked for additional embedded files in the Kuwait website. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-03: Continued to download Moroccan data in the background. Dr. Egan fixed systems requirements to run the GovTrack Web Crawler. Made significant progress on the Kuwait Web Crawler/Driver for the Middle East Studies Department. <br /> <br /> 2016-11-01: Continued to download Moroccan data in the background. Went over code for GovTracker Web Crawler, continued learning Perl. [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] Began Kuwait Web Crawler/Driver.<br /> <br /> 2016-10-21: Continued to download data for the Moroccan Parliament Written and Oral Questions. Looked over [http://mcnair.bakerinstitute.org/wiki/Christy_Warden_(Twitter_Crawler_Application_1) Christy's Twitter Crawler] to see how I can be helpful. Dr. Egan asked me to think about how to potentially make multiple tools to get cohorts and other sorts of data from accelerator sites. See [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator List] He also asked me to look at the [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] for potential ideas on how to bring this project to fruition.<br /> <br /> 2016-10-20: Continued to download data for the Moroccan Parliament Written and Oral Questions. Updated Wiki page. Started working on Twitter project with Christy. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-18: Finished code for Oral Questions web driver and Written Questions web driver using selenium. Now, the data for the dates of questions can be found using the crawler, and the pdfs of the questions will be downloaded using selenium. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-14: Finished Oral Questions crawler. Finished Written Questions crawler. Waiting for further details on whether that data needs to be tweaked in any way. Updated the Moroccan Web Driver/Web Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-13: Completed download of Moroccan Bills. Working on either a web driver screenshot approach or a webcrawler approach to download the Moroccan oral and written questions data. Began building Web Crawler for Oral and Written Questions site. Edited Moroccan Web Driver/Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-11: Fixed arabic bug, files can now be saved with arabic titles. Monarchy bills downloaded and ready for shipment. House of Representatives Bill mostly downloaded, ratified bills prepared for download. Started learning scrapy library in python for web scraping. Discussed idea of screenshot-ing questions instead of scraping. <br /> <br /> 2016-10-07: Learned unicode and utf8 encoding and decoding in arabic. Still working on transforming an ascii url into printable unicode.<br /> <br /> 2016-10-06: Discussed with Dr. Elbadawy about the desired file names for Moroccan data download. The consensus was that the bill programs are ready to launch once the files can be named properly, and the questions data must be retrieved using a web crawler which I need to learn how to implement. The naming of files is currently drawing errors in going from arabic, to url, to download, to filename. Debugging in process. Also built a demo selenium program for Dr. Egan that drives the McNair blog site on an infinite loop.<br /> <br /> 2016-10-03: Moroccan Web Driver projects completed for driving of the Monarchy proposed bills, the House of Representatives proposed bills, and the Ratified bills sites. Begun process of devising a naming system for the files that does not require scraping. Tinkered with naming through regular expression parsing of the URL. Structure for the oral questions and written questions drivers is set up, but need fixes due to the differences in the sites. Fixed bug on McNair wiki for women's biz team where email was plain text instead of an email link. Took a glimpse at Kuwait Parliament website, and it appears to be very different from the Moroccan setup.<br /> <br /> 2016-09-30: Selenium program selects view pdf option from the website, and goes to the pdf webpage. Program then switches handle to the new page. CTRL S is sent to the page to launch save dialog window. Text cannot be sent to this window. Brainstorm ways around this issue. Explored Chrome Options for saving automatically without a dialog window. Looking into other libraries besides selenium that may help.<br /> <br /> 2016-09-29: Re-enroll in Microsoft Remote Desktop with proper authentication, set up Selenium environment and Komodo IDE on Remote Desktop, wrote program using Selenium that goes to a link and opens up the print dialog box. Developed computational recipe for a different approach to the problem.<br /> <br /> 2016-09-26: Set up Staff wiki page, work log page; registered for Slack, Microsoft Remote Desktop; downloaded Selenium on personal computer, read Selenium docs. Created wiki page for Moroccan Web Driver Project.<br /> <br /> '''Notes'''<br /> <br /> *Ed moved the Morocco Data to E:\McNair\Projects from C:\Users\PeterJ\Documents<br /> * C Drive files moved to E:\McNair\Users\PeterJ<br /> <br /> [[Category:Work Log]]</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Peter_Jalbert_(Work_Log)&diff=22358 2017-12-14T15:58:55Z

<p>Peterjalbert: </p> <hr /> <div>===Fall 2017===<br /> &lt;onlyinclude&gt; <br /> <br /> [[Peter Jalbert]] [[Work Logs]] [[Peter Jalbert (Work Log)|(log page)]] <br /> <br /> 2017-12-14: Uploading Morocco Parliament Written Questions. Creating script for next Morocco Parliament download. Begin writing Selenium documentation. Continuing to download TIGER data.<br /> <br /> 2017-12-06: Running Morocco Parliament Written Questions script. Analyzing Demo Day Crawler results. Continued downloading for TIGER geocoder.<br /> <br /> 2017-11-28: Debugging Morocco Parliament Crawler. Running Demo Day Crawler for all accelerators and 10 pages per accelerator. TIGER geocoder is back to Forbidden Error.<br /> <br /> 2017-11-27: Rerunning Morocco Parliament Crawler. Fixed KeyTerms.py and running it again. Continued downloading for TIGER geocoder.<br /> <br /> 2017-11-20: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-16: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-15: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote a script to extract counts that were greater than 2 from Keyword Matcher. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder]. Finished re-formatting work logs.<br /> <br /> 2017-11-14: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote an HTML to Text parser. See Parser Demo Day Page for file location. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder].<br /> <br /> 2017-11-13: Built [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser].<br /> <br /> 2017-11-09: Running demo version of Demo Day crawler (Accelerator Google Crawler). Fixing work log format. <br /> <br /> 2017-11-07: Created file with 0s and 1s detailing whether crunchbase has the founder information for an accelerator. Details posted as a TODO on [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List] page. Still waiting for feedback on the PostGIS installation from [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder]. Continued working on Accelerator Google Crawler.<br /> <br /> 2017-11-06: Contacted Geography Center for the US Census Bureau, [https://www.census.gov/geo/about/contact.html here], and began email exchange on PostGIS installation problems. Began working on the [http://mcnair.bakerinstitute.org/wiki/Selenium_Documentation Selenium Documentation]. Also began working on an Accelerator Google Crawler that will be used with Yang and ML to find Demo Days for cohort companies.<br /> <br /> 2017-11-01: Attempted to continue downloading, however ran into HTTP Forbidden errors. Listed the errors on the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder Page].<br /> <br /> 2017-10-31: Began downloading blocks of data for individual states for the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder] project. Wrote out the new wiki page for installation, and beginning to write documentation on usage.<br /> <br /> 2017-10-30: With Ed's help, was able to get the national data from Tiger installed onto a database server. The process required much jumping around and changing users, and all the things we learned are outlined in [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation#Editing_Users the database server documentation] under &quot;Editing Users&quot;.<br /> <br /> 2017-10-25: Continued working on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation TigerCoder Installation].<br /> <br /> 2017-10-24: Throw some addresses into a database, use address normalizer and geocoder. May need to install things. Details on the installation process can be found on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation page].<br /> <br /> 2017-10-23: Finished Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-19: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-18: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_DistrictHouston Innovation District Project].<br /> <br /> 2017-10-17: Constructed ArcGIS maps for the agglomeration project. Finished maps of points for every year in the state of California. Finished maps of Route 128. Began working on selenium Yelp crawler to get cafe locations within the 610-loop.<br /> <br /> 2017-10-16: Assisted Harrison on the USITC project. Looked for natural language processing tools to extract complaintants and defendants along with their location from case files. Experimented with pulling based on parts of speech tags, as well as using geotext or geograpy to pull locations from a case segment.<br /> <br /> 2017-10-13: Updated various project wiki pages.<br /> <br /> 2017-10-12: Continued work on Patent Thicket project, awaiting further project specs.<br /> <br /> 2017-10-05: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-04: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-02: Worked on ArcGIS data. See Harrison's Work Log for the details.<br /> <br /> 2017-09-28: Added collaborative editing feature to PyCharm. <br /> <br /> 2017-09-27: Worked on big database file.<br /> <br /> 2017-09-25: New task -- Create text file with company, description, and company type.<br /> #[http://mcnair.bakerinstitute.org/wiki/VC_Database_Rebuild VC Database Rebuild]<br /> #psql vcdb2<br /> #table name, sdccompanybasecore2<br /> #Combine with Crunchbasebulk<br /> <br /> #TODO: Write wiki on linkedin crawler, write wiki on creating accounts.<br /> <br /> 2017-09-21: Wrote wiki on Linkedin crawler, met with Laura about patents project. <br /> <br /> 2017-09-20: Finished running linkedin crawler. Transferred data to RDP. Will write wikis next.<br /> <br /> 2017-09-19: Began running linkedin crawler. Helped Yang create RDP account, get permissions, and get wiki setup.<br /> <br /> 2017-09-18: Finished implementation of Experience Crawler, continued working on Education Crawler for LinkedIn.<br /> <br /> 2017-09-14: Continued implementing LinkedIn Crawler for profiles.<br /> <br /> 2017-09-13: Implemented LinkedIn Crawler for main portion of profiles. Began working on crawling Experience section of profiles.<br /> <br /> 2017-09-12: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. Added to the wiki on this topic.<br /> <br /> 2017-09-11: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. <br /> <br /> 2017-09-06: Combined founders data retrieved with the Crunchbase API with the crunchbasebulk data to get linkedin urls for different accelerator founders. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-09-05: Post Harvey. Finished retrieving names from the Crunchbase API on founders. Next step is to query crunchbase bulk database to get linkedin urls. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-08-24: Began using the Crunchbase API to retrieve founder information for accelerators. Halfway through compiling a dictionary that translates accelerator names into proper Crunchbase API URLs.<br /> <br /> 2017-08-23: Decided with Ed to abandon LinkedIn crawling to retrieve accelerator founder data, and instead use crunchbase. Spent the day navigating the crunchbasebulk database, and seeing what useful information was contained in it.<br /> <br /> 2017-08-22: Discovered that LinkedIn Profiles cannot be viewed through LinkedIn if the target is 3rd degree or further. However, if entering LinkedIn through a Google search, the profile can still be viewed if the user has previously logged into LinkedIn. Devising a workaround crawler that utilizes Google search. Continued blog post [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) here] under Section 4.<br /> <br /> 2017-08-21: Began work on extracting founders for accelerators through LinkedIn Crawler. Discovered that Python3 is not installed on RDP, so the virtual environment for the project cannot be fired up. Continued working on Ubuntu machine.<br /> &lt;/onlyinclude&gt; <br /> <br /> ===Spring 2017===<br /> <br /> 2017-05-01: Continued work on HTML Parser. Uploaded all semester projects to git server.<br /> <br /> 2017-04-20: Finished the HTML Parser for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Ran HTML parser on accelerator founders. Data is stored in projects/accelerators/LinkedIn Founder Data.<br /> <br /> 2017-04-19: Made updates to the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler] Wikipage. Ran LinkedIn Crawler on accelerator data. Working on an html parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-18: Ran LinkedIn Crawler on matches between Crunchbase Snapshot and the accelerator data.<br /> <br /> 2017-04-17: Worked on ways to get correct search results from the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Worked on an HTML Parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-13: Worked on debugging the logout procedure for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Began formulation of process to search for founders of startups using a combination of the LinkedIn Crawler with the data resources from the [http://mcnair.bakerinstitute.org/wiki/Crunchbase_2013_Snapshot CrunchBase Snapshot].<br /> <br /> 2017-04-12: Work on bugs with the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler].<br /> <br /> 2017-04-11: Completed functional [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) crawler of LinkedIn Recruiter Pro]. Basic search functions work and download profile information for a given person. <br /> <br /> 2017-04-10: Began writing functioning crawler of LinkedIn. <br /> <br /> 2017-04-06: Continued working on debugging and documenting the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Wrote a test program that logs in, searches for a query, navigates through search pages, and logs out. Recruiter program can now login and search.<br /> <br /> 2017-04-05: Began work on the LinkedIn Crawler. Researched on launching Python Virtual Environment.<br /> <br /> 2017-04-03: Finished debugging points for the Enclosing Circle Algorithm. Added Command Line functionality to the Industry Classifier.<br /> <br /> 2017-03-29: Worked on debugging points for the Enclosing Circle Algorithm.<br /> <br /> 2017-03-28: Finished running the Enclosing Circle Algorithm. Worked on removing incorrect points from the data set(see above). <br /> <br /> 2017-03-27: Worked on debugging the Enclosing Circle Algorithm. Implemented a way to remove interior circles, and determined that translation to latitude and longitude coordinates resulted in slightly off center circles.<br /> <br /> 2017-03-23: Finished debugging the brute force algorithm for [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm]. Implemented a method to plot the points and circles on a graph. Analyzed runtime of the brute force algorithm.<br /> <br /> 2017-03-21: Coded a brute force algorithm for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-20: Worked on debugging the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-09: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Finished script to draw Enclosing Circles on a Google Map.<br /> <br /> 2017-03-08: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Created script to draw outcome of the Enclosing Circle Algorithm on Google Maps.<br /> <br /> 2017-03-07: Redetermined the top 50 cities which Enclosing Circle should be run on. Data on the [http://mcnair.bakerinstitute.org/wiki/Top_Cities_for_VC_Backed_Companies Top 50 Cities for VC Backed Companies can be found here.] Ran [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities.<br /> <br /> 2017-03-06: Ran script to determine the top 50 cities which Enclosing Circle should be run on. Fixed the VC Circles script to take in a new data format.<br /> <br /> 2017-03-02: Cleaned up data for the VC Circles Project. Created histogram of data in Excel. See [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Began work on the [http://mcnair.bakerinstitute.org/wiki/LinkedInCrawlerPython LinkedIn Crawler].<br /> <br /> 2017-03-01: Created statistics for the VC Circles Project.<br /> <br /> 2017-02-28: Finished downloading geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Found bug in Enclosing Circle Algorithm.<br /> <br /> 2017-02-27: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-23: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Installed C++ Compiler for Python. Ran tests on difference between Python and C wrapped Python.<br /> <br /> 2017-02-22: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Helped out with [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-21: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Researched into C++ Compilers for Python so that the Enclosing Circle Algorithm could be wrapped in C. Found a recommended one [https://www.microsoft.com/en-us/download/details.aspx?id=44266 here].<br /> <br /> 2017-02-20: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-16: Reworked [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] to create a file of geocoded data. Began work on wrapping the algorithm in C to improve speed.<br /> <br /> 2017-02-15: Finished [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] applied to the VC study. Enclosing Circle algorithm still needs adjustment, but the program runs with the temporary fixes.<br /> <br /> 2017-02-14: Worked on the application of the Enclosing Circle algorithm to the VC study. Working on bug fixes in the Enclosing Circle algorithm. Created wiki page for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-02-13: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-08: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-07: Fixed bugs in parse_cohort_data.py, the script for parsing the cohort data from the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Added descriptive statistics to cohort data excel file.<br /> <br /> 2017-02-02: Out sick, independent research and work from RDP. Brief research into the [http://jorgeg.scripts.mit.edu/homepage/wp-content/uploads/2016/03/Guzman-Stern-State-of-American-Entrepreneurship-FINAL.pdf Stern-Guzman algorithm]. Research into [http://mcnair.bakerinstitute.org/wiki/interactive_maps Interactive Maps]. No helpful additions to map embedding problem.<br /> <br /> 2017-02-01: Notes from Session with Ed: Project on US university patenting and entrepreneurship programs (writing code to identify universities in assignees), search Wikipedia (XML then bulk download), student pop, faculty pop, etc.<br /> Circle project for VC data will end up being a joint project to join accelerator data. <br /> Pull descriptions for VC. Founders of accelerators in linkedin. LinkedIn cannot be caught(pretend to not be a bot). Can eventually get academic backgrounds through linkedin. <br /> Pull business registration data, Stern/Guzman Algorithm. <br /> GIS ontop of geocoded data.<br /> Maps that works on wiki or blog (CartoDB), Maps API and R.<br /> NLP Projects, Description Classifier.<br /> <br /> 2017-01-31: Built WayBack Machine Crawler. Updated documentation for coordinates script. Updated profile page to include locations of code.<br /> <br /> <br /> 2017-01-30: Optimized enclosing circle algorithm through memoization. Developed script to read addresses from accelerator data and return latitude and longitude coordinates.<br /> <br /> 2017-01-26: Continued working on Google sitesearch project. Discovered crunchbase, changed project priority. Priority 1, split accelerator data up by flag, priority 2, use crunchbase to get web urls for cohorts, priority 3, make internet archive wayback machine driver. Located [http://mcnair.bakerinstitute.org/wiki/Whois_Parser Whois Parser].<br /> <br /> <br /> 2017-01-25: Finished parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Some data files still need proofreading as they are not in an acceptable format. Began working on Google sitesearch project.<br /> <br /> 2017-01-24: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Cohort data file created, debugging is almost complete. Will begin work on the google accelerator search soon.<br /> <br /> 2017-01-23: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Preliminary code is written, working on debugging.<br /> <br /> 2017-01-19: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project]. Created parser for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project], completed creation of final data set(yay!). Began working on cohort parser.<br /> <br /> 2017-01-18: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-13: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-12: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-11: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-10: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> <br /> ===Fall 2016===<br /> <br /> 2016-12-08: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-07: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-06: Learned how to use git. Committed software projects from the semester to the McNair git repository. Projects can be found at; [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Executive Order Crawler], [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Foreign Government Web Crawlers], [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) F6S Crawler and Parser].<br /> <br /> 2016-12-02: Built and ran web crawler for Center for Middle East Studies on Kuwait. Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-01: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Built tool for the [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report E&amp;I Governance Report Project] with Christy. Adds a column of data that shows whether or not the bill has been passed.<br /> <br /> 2016-11-29: Began pulling data from the accelerators listed [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here]. Made text files for about 18 accelerators.<br /> <br /> 2016-11-22: Transferred downloaded Morocco Written Bills to provided SeaGate Drive. Made a &quot;gentle&quot; F6S crawler to retrieve HTMLs of possible accelerator pages documented [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here].<br /> <br /> 2016-11-18: Converted Executive Order PDFs to text files using adobe acrobat DC. See [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Wikipage] for details.<br /> <br /> 2016-11-17: Wrote a crawler to retrieve information about executive orders, and their corresponding pdfs. They can be found [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report here.] Next step is to run code to convert the pdfs to text files, then use the parser fixed by Christy. <br /> <br /> 2016-11-15: Finished download of Moroccan Written Question pdfs. Wrote a parser with Christy to be used for parsing bills from Congress and eventually executive orders. Found bug in the system Python that was worked out and rebooted.<br /> <br /> 2016-11-11: Continued to download Moroccan data in the background. Attempted to find bug fixes for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] crawler.<br /> <br /> 2016-11-10: Continued to download Moroccan data and Kuwait data in the background. Began work on [http://mcnair.bakerinstitute.org/wiki/Google_Scholar_Crawler Google Scholar Crawler]. Wrote a crawler for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] to get the HTML files of hundreds of accelerators. The crawler ended up failing; it appears to have been due to HTTPS. <br /> <br /> 2016-11-08: Continued to download Moroccan data in the background. Finished writing code for the embedded files on the Kuwait Site. Spent time debugging the frame errors due to the dynamically generated content. Never found an answer to the bug, and instead found a workaround that sacrificed run time for the ability to work. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-04: Continued to download Moroccan data in the background. Finished writing initial Kuwait Web Crawler/Driver for the Middle East Studies Department. Middle East Studies Department asked for additional embedded files in the Kuwait website. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-03: Continued to download Moroccan data in the background. Dr. Egan fixed systems requirements to run the GovTrack Web Crawler. Made significant progress on the Kuwait Web Crawler/Driver for the Middle East Studies Department. <br /> <br /> 2016-11-01: Continued to download Moroccan data in the background. Went over code for GovTracker Web Crawler, continued learning Perl. [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] Began Kuwait Web Crawler/Driver.<br /> <br /> 2016-10-21: Continued to download data for the Moroccan Parliament Written and Oral Questions. Looked over [http://mcnair.bakerinstitute.org/wiki/Christy_Warden_(Twitter_Crawler_Application_1) Christy's Twitter Crawler] to see how I can be helpful. Dr. Egan asked me to think about how to potentially make multiple tools to get cohorts and other sorts of data from accelerator sites. See [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator List] He also asked me to look at the [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] for potential ideas on how to bring this project to fruition.<br /> <br /> 2016-10-20: Continued to download data for the Moroccan Parliament Written and Oral Questions. Updated Wiki page. Started working on Twitter project with Christy. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-18: Finished code for Oral Questions web driver and Written Questions web driver using selenium. Now, the data for the dates of questions can be found using the crawler, and the pdfs of the questions will be downloaded using selenium. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-14: Finished Oral Questions crawler. Finished Written Questions crawler. Waiting for further details on whether that data needs to be tweaked in any way. Updated the Moroccan Web Driver/Web Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-13: Completed download of Moroccan Bills. Working on either a web driver screenshot approach or a webcrawler approach to download the Moroccan oral and written questions data. Began building Web Crawler for Oral and Written Questions site. Edited Moroccan Web Driver/Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-11: Fixed arabic bug, files can now be saved with arabic titles. Monarchy bills downloaded and ready for shipment. House of Representatives Bill mostly downloaded, ratified bills prepared for download. Started learning scrapy library in python for web scraping. Discussed idea of screenshot-ing questions instead of scraping. <br /> <br /> 2016-10-07: Learned unicode and utf8 encoding and decoding in arabic. Still working on transforming an ascii url into printable unicode.<br /> <br /> 2016-10-06: Discussed with Dr. Elbadawy about the desired file names for Moroccan data download. The consensus was that the bill programs are ready to launch once the files can be named properly, and the questions data must be retrieved using a web crawler which I need to learn how to implement. The naming of files is currently drawing errors in going from arabic, to url, to download, to filename. Debugging in process. Also built a demo selenium program for Dr. Egan that drives the McNair blog site on an infinite loop.<br /> <br /> 2016-10-03: Moroccan Web Driver projects completed for driving of the Monarchy proposed bills, the House of Representatives proposed bills, and the Ratified bills sites. Begun process of devising a naming system for the files that does not require scraping. Tinkered with naming through regular expression parsing of the URL. Structure for the oral questions and written questions drivers is set up, but need fixes due to the differences in the sites. Fixed bug on McNair wiki for women's biz team where email was plain text instead of an email link. Took a glimpse at Kuwait Parliament website, and it appears to be very different from the Moroccan setup.<br /> <br /> 2016-09-30: Selenium program selects view pdf option from the website, and goes to the pdf webpage. Program then switches handle to the new page. CTRL S is sent to the page to launch save dialog window. Text cannot be sent to this window. Brainstorm ways around this issue. Explored Chrome Options for saving automatically without a dialog window. Looking into other libraries besides selenium that may help.<br /> <br /> 2016-09-29: Re-enroll in Microsoft Remote Desktop with proper authentication, set up Selenium environment and Komodo IDE on Remote Desktop, wrote program using Selenium that goes to a link and opens up the print dialog box. Developed computational recipe for a different approach to the problem.<br /> <br /> 2016-09-26: Set up Staff wiki page, work log page; registered for Slack, Microsoft Remote Desktop; downloaded Selenium on personal computer, read Selenium docs. Created wiki page for Moroccan Web Driver Project.<br /> <br /> '''Notes'''<br /> <br /> *Ed moved the Morocco Data to E:\McNair\Projects from C:\Users\PeterJ\Documents<br /> * C Drive files moved to E:\McNair\Users\PeterJ<br /> <br /> [[Category:Work Log]]</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Tiger_Geocoder&diff=22307 2017-12-06T21:59:49Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Tiger Geocoder<br /> |Has owner=Peter Jalbert,<br /> |Has start date=Fall 2017<br /> |Has keywords=Tiger, Geocoder, Database<br /> |Has project status=Active<br /> |Has Image=Tiger.jpg<br /> }}<br /> <br /> Note: The Address Ranges National Geodatabase is available from https://www.census.gov/geo/maps-data/data/tiger-geodatabases.html<br /> <br /> This page serves as documentation for using the Tiger Geocoder on Postgres SQL, as part of the PostGIS extension. The following wiki pages may also be of use to you:<br /> <br /> [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation]<br /> [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation Database Server Documentation]<br /> <br /> The official documentation for using and installing the Tiger Geocoder can be found in the following.<br /> <br /> [https://postgis.net/docs/Extras.html#Tiger_Geocoder General Instructions]<br /> [https://postgis.net/docs/postgis_installation.html#install_tiger_geocoder_extension Installation Instructions]<br /> [http://postgis.net/docs/Geocode.html Geocoder Documentation]<br /> <br /> ==Location==<br /> The data is currently loaded into a psql database called geocoder. The tables contain the geocoding information, and there is a test table called &quot;coffeeshops&quot; that contains addresses of Houston coffeeshops according to yelp. To access the database, first login to the McNair DB Sever. Then, <br /> <br /> psql geocoder<br /> <br /> <br /> <br /> ==Installation==<br /> <br /> ===Install and Nation Data===<br /> I began by adding the extension listed above. First, enter into Postgres by using the psql command. Then:<br /> --Add Extensions to database<br /> CREATE EXTENSION postgis;<br /> CREATE EXTENSION fuzzystrmatch;<br /> CREATE EXTENSION postgis_tiger_geocoder;<br /> CREATE EXTENSION address_standardizer;<br /> <br /> You can test that the installation worked by running the following query: <br /> SELECT na.address, na.streetname,na.streettypeabbrev, na.zip<br /> FROM normalize_address('1 Devonshire Place, Boston, MA 02109') AS na;<br /> <br /> This should return the following:<br /> address | streetname | streettypeabbrev | zip<br /> ---------+------------+------------------+-------<br /> 1 | Devonshire | Pl | 02109<br /> <br /> Next, a new profile needs to be created by using the following command.<br /> INSERT INTO tiger.loader_platform(os, declare_sect, pgbin, wget, unzip_command, psql, path_sep, <br /> loader, environ_set_command, county_process_command)<br /> SELECT 'test', declare_sect, pgbin, wget, unzip_command, psql, path_sep,<br /> loader, environ_set_command, county_process_command<br /> FROM tiger.loader_platform<br /> WHERE os = 'sh';<br /> <br /> The installation instructions also provide the following note:<br /> <br /> As of PostGIS 2.4.1 the Zip code-5 digit tabulation area zcta5 load step was revised to load current zcta5 data and is part of the Loader_Generate_Nation_Script when enabled. It is turned off by default because it takes quite a bit of time to load (20 to 60 minutes), takes up quite a bit of disk space, and is not used that often.<br /> <br /> If you would like this feature, you can enable it by using the following command. This should be done before loading the script.<br /> <br /> UPDATE tiger.loader_lookuptables SET load = true WHERE table_name = 'zcta510';<br /> <br /> The paths in declare_sect need to be edited so they match our server locations. One option is to edit the declare_sect column in the tiger.loader_platform table. If so, the declare_sect looks like the following:<br /> export PGHOST=localhost +<br /> export PGUSER=postgres +<br /> export PGPASSWORD=yourpasswordhere +<br /> export PGDATABASE=geocoder +<br /> PSQL=${PGBIN}/psql +<br /> SHP2PGSQL=shp2pgsql +<br /> cd ${staging_fold} +<br /> <br /> TMPDIR=&quot;${staging_fold}/temp/&quot; +<br /> UNZIPTOOL=unzip +<br /> WGETTOOL=&quot;/usr/bin/wget&quot; +<br /> export PGBIN=/usr/lib/postgresql/9.6/bin +<br /> export PGPORT=5432 +<br /> export PGHOST=localhost +<br /> export PGUSER=postgres +<br /> export PGPASSWORD=yourpasswordhere +<br /> export PGDATABASE=geocoder +<br /> PSQL=${PGBIN}/psql +<br /> SHP2PGSQL=shp2pgsql +<br /> cd ${staging_fold}<br /> <br /> Another option is to edit the sh file before running the script. We will do this option until further notice. Simply use your favorite command line editor to change the fields to their correct values. The downloaded script is located in the following directory:<br /> /gisdata<br /> <br /> There needs to be a directory called &quot;temp&quot; in the gisdata directory. To make the script, use the following from the command line:<br /> psql -c &quot;SELECT Loader_Generate_Nation_Script('test')&quot; -d databasename -tA &gt; /gisdata/nation_script_load.sh<br /> <br /> This will create a script in the /gisdata directory. Change to that directory. If you did not edit the paths in the declare_sect table in psql, then you will need to edit this file to contain the correct paths. <br /> <br /> Change directories:<br /> cd /gisdata<br /> <br /> Edit the script using your favorite command line text editor. Specifically, edit the following fields.<br /> PGUSER=postgres<br /> PGPASSWORD=(Ask Anne for this password)!<br /> Everything else remains the same.<br /> <br /> Run the script by using: <br /> sh nation_script_load.sh<br /> <br /> Now, there is a barebones table in the database that will hold the information for the nation. Next is to download data on each state.<br /> <br /> ===State Data===<br /> <br /> The state scripts are generated in much the same way that the nation script was generated. Use the following command, substituting MA for your desired state abbreviation, and substituting a unique filename at the end.<br /> psql -c &quot;SELECT Loader_Generate_Script(ARRAY['MA'], 'test')&quot; -d geocoder -tA &gt; /gisdata/ma_load.sh<br /> <br /> CURRENT PROGRESS:<br /> The following states have been downloaded into the geocoder database.<br /> AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, IA, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN<br /> <br /> ===Current Errors===<br /> The state scripts stopped working on 11/1/2017 while they were working on 10/31/2017. Now, when a retrieval script is run, it draws the error <br /> HTTP request sent, awaiting response... 403 Forbidden<br /> Possible thoughts:<br /> <br /> Maybe our IP has been blacklisted for downloading data from a government website quickly? If so, PostGIS should really choose a different installation method. The current one is dumb.<br /> <br /> Maybe the nation downloader script never worked properly. Not sure how to check if it is correct or not; seems right.<br /> <br /> [https://trac.osgeo.org/postgis/ticket/3699 This] is the only online forum I could find with others who have faced a similar issue.<br /> <br /> NOTE: Contacted the US Census Bureau and never got a response. However, the scripts magically started working again about a week later. Second however, the scripts stopped working again the next day.<br /> <br /> ==Geocode Function==<br /> <br /> The official arguments for the function are the following:<br /> setof record geocode(varchar address, integer max_results=10, geometry restrict_region=NULL, norm_addy OUT addy, geometry OUT geomout, integer OUT rating);<br /> The arguments of interest are address, where you simply submit a string, and max_results, which restricts the number of geocoding attempts per address. The geocoder makes multiple guesses to the location of an address, and returns the best guesses in order. If you want multiple guesses to a specific address, then specify max_results to be more than 1.<br /> <br /> ===Single Address===<br /> <br /> An example query for a single address is:<br /> SELECT g.rating, ST_X(g.geomout) As lon, ST_Y(g.geomout) As lat,<br /> (addy).address As stno, (addy).streetname As street,<br /> (addy).streettypeabbrev As styp, (addy).location As city, (addy).stateabbrev As st,(addy).zip<br /> FROM geocode('75 State Street, Boston MA 02109', 1) As g;<br /> <br /> rating -- This is an integer that determines the confidence in the resulting geocode. The closer to 0, the more confident the guess.<br /> <br /> ST_X(g.geomout) -- This retrieves the longitude coordinate of the point.<br /> <br /> ST_Y(g.geomout) -- This retrieves the latitude corodinate of the point.<br /> <br /> addy -- In general, addy is a normalized address resulting from the input address.<br /> <br /> (addy).address -- The number of the address (Ex: &quot;75&quot; Blabla rd.)<br /> <br /> (addy).streetname -- The name of the street (Ex: 75 &quot;Blabla&quot; rd.)<br /> <br /> (addy).streettypeabbrev -- The abbreviation of the street (Ex: 75 blabla &quot;rd&quot;)<br /> <br /> (addy).location -- The city location of the address.<br /> <br /> (addy).stateabbrev -- The state abbreviation.<br /> <br /> (addy).zip -- The zipcode of the address.<br /> <br /> The output of the query above would be:<br /> rating | lon | lat | stno | street | styp | city | st | zip<br /> --------+-------------------+----------------+------+--------+------+--------+----+-------<br /> 0 | -71.0557505845646 | 42.35897920691 | 75 | State | St | Boston | MA | 02109<br /> <br /> ==Current Errors==<br /> Currently, we are getting a 403 Forbidden Error when trying to download state data. We are in contact with the US Census Bureau. Their contact information can be found [https://www.census.gov/geo/about/contact.html here].<br /> <br /> The email exchanges are recorded below.<br /> <br /> --------------------------------------<br /> --------------------------------------<br /> <br /> I am a student researcher at the McNair Center for Entrepreneurship and Innovation at Rice University, and I am in the process of installing a Postgres Extension that relies on the TIGER data. <br /> <br /> When I began installing the extension, things were working fine. Now however, I am getting a 403 Forbidden error when the script tries to download the TIGER files. Do you have any idea why this might be happening? <br /> <br /> The extension I'm trying to install is below:<br /> <br /> http://postgis.net/docs/postgis_installation.html#install_tiger_geocoder_extension<br /> <br /> When I run any of the scripts that require data from TIGER, I am receiving the following error:<br /> <br /> <br /> --2017-11-06 14:15:29-- http://www2.census.gov/geo/tiger/TIGER2016/FEATNAMES/tl_2016_06065_featnames.zip<br /> Resolving www2.census.gov (www2.census.gov)... 104.84.241.90, 2600:1404:a:382::208c, 2600:1404:a:39c::208c<br /> Connecting to www2.census.gov (www2.census.gov)|104.84.241.90|:80... connected.<br /> HTTP request sent, awaiting response... 403 Forbidden<br /> 2017-11-06 14:15:29 ERROR 403: Forbidden.<br /> <br /> <br /> ----------------------------------------<br /> <br /> Thank you for reaching out to us. The postgis docs is not from the Geography Division, so I cannot comment on that. A couple things that come to mind.<br /> <br /> We just released our new 2017 Shapefiles, so it's possible the scripts may be written for a previous version of our Shapefiles.<br /> You may need to clean out your cookies, restart your browser, and then attempt to reinstall.<br /> Were you able to download our Shapefiles successfully?<br /> <br /> ----------------------------------------<br /> <br /> The Shapefiles are part of the files being blocked by the 403 Forbidden error. <br /> <br /> The script is using a wget protocol to bulk download the data, so there were no cookies involved. Also, the download had been working in previous days; it was only recently that the same scripts stopped working. I am worried that our IP address somehow ended up on a blacklist for the TIGER data. Is there a blacklist for addresses that access the TIGER data?<br /> <br /> Our IP address is 128.42.44.181. <br /> <br /> ---------------------------------------<br /> <br /> I have forwarded your question on to our IT folks. Since I work on the subject matter side, I am unable to answer your questions. Once I hear back from them, I will forward their response to you. Hopefully they will provide to you what you need in order to download our Shapefiles. My apologies for your inconvenience.<br /> <br /> <br /> ----------------------------------------</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Peter_Jalbert&diff=22306 2017-12-06T21:42:06Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Staff<br /> |position=Tech Team<br /> |name=Peter Jalbert,<br /> |user_image=peter_headshot.jpg<br /> |degree=BA<br /> |major=Computer Science; Music Performance<br /> |class=2019,<br /> |join_date=09/27/2016,<br /> |skills=Python, Selenium, Javascript, Java, SQL,<br /> |interests=Music, Movies, Travel,<br /> |email=pwj1@rice.edu<br /> |status=Active<br /> }}<br /> ==Education==<br /> Peter is currently a junior at Rice University, pursuing a double major in Computer Science and Music. Peter graduated Salutatorian from the High School for the Performing and Visual Arts in Houston, TX in 2014. <br /> <br /> ==Contributing Projects==<br /> *[http://mcnair.bakerinstitute.org/wiki/Selenium_Documentation Selenium Documentation]<br /> *[http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Crawler]<br /> *[http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder#State_Data Tiger Geocoder]<br /> *[http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District]<br /> * [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data Crunchbase Data]<br /> * [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]<br /> * [http://mcnair.bakerinstitute.org/wiki/Urban_Start-up_Agglomeration Urban Start-up Agglomeration]<br /> * [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm]<br /> * [http://mcnair.bakerinstitute.org/wiki/Top_Cities_for_VC_Backed_Companies Top Cities for VC Backed Companies]<br /> * [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Parliament Web Crawler]<br /> * [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report E&amp;I Governance Policy Report]<br /> * [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler]<br /> * [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List]<br /> * [http://mcnair.bakerinstitute.org/wiki/Google_Scholar_Crawler Google Scholar Crawler]<br /> <br /> ==Looking for Code?==<br /> <br /> ===Demo Day Crawler===<br /> Term: Fall 2017<br /> <br /> Crawls Google search to find candidate web pages for accelerator companies' demo days.<br /> E:\McNair\Software\Accelerators\DemoDayCrawler.py<br /> <br /> ===Demo Day Hits===<br /> Term: Fall 2017<br /> <br /> Analyzes the results of a demo day crawl for hits of keywords.<br /> E:\McNair\Software\Accelerators\DemoDayHits.py<br /> <br /> ===HTML to Text===<br /> Term: Fall 2017<br /> <br /> Converts a folder of HTML files to a folder of TXT files.<br /> E:\McNair\Software\Accelerators\htmlToText.py<br /> <br /> ===Tiger Geocoder===<br /> Term: Fall 2017<br /> <br /> Installed a psql extension that allows for internal geocoding of addresses.<br /> psql geocoder<br /> <br /> ===Yelp Crawler===<br /> Term: Fall 2017<br /> <br /> Usage: Crawls for data on restaurants and coffeeshops within the 610 Loop. Part of the Houston Innovation District Project.<br /> E:\McNair\Software\YelpCrawler\yelp_crawl.py<br /> <br /> ===Accelerator Founders===<br /> Term: Fall 2017<br /> <br /> Usage: Uses the LinkedIn Crawler along with the Crunchbase founders data to retrieve information on accelerator founders.<br /> E:\McNair\Projects\LinkedIn Crawler\LinkedIn_Crawler\linkedin\linkedin_founders.py<br /> <br /> === Crunchbase Founders ===<br /> Term: Fall 2017<br /> <br /> Usage: Queries the Crunchbase API to get names of accelerator founders.<br /> E:\McNair\Projects\Accelerators\crunchbase_founders.py<br /> <br /> ===LinkedIn Crawler===<br /> Term: Spring 2017<br /> <br /> Usage: Crawls LinkedIn to obtain relevant information.<br /> E:\McNair\Projects\LinkedIn Crawler\web_crawler\linkedin\run_linkedin_recruiter.py<br /> <br /> ===Draw Enclosing Circles===<br /> Term: Spring 2017<br /> <br /> Usage: Draws the outcome of the Enclosing Circle Algorithm on a particular city to a google map HTML output.<br /> E:\McNair\Projects\Accelerators\Enclosing_Circle\draw_vc_circles.py<br /> <br /> ===Enclosing Circle for VCs ===<br /> Term: Spring 2017<br /> <br /> Usage: Uses the Enclosing Circle algorithm to find concentrations of VCs.<br /> E:\McNair\Projects\Accelerators\Enclosing_Circle\vc_circles.py<br /> <br /> === Industry Classifier ===<br /> Term: Spring 2017<br /> <br /> Usage: Neural Net that predicts a companies industry classification.<br /> E:\McNair\Projects\Accelerators\Code+Final_Data\ChristyCode\IndustryClassifier.py<br /> <br /> ===WayBack Machine Parser===<br /> Term: Spring 2017<br /> <br /> Usage: Uses the WayBack Machine API to retrieve timestamps for URLs.<br /> E:\McNair\Projects\Accelerators\Spring 2017\Code+Final_Data\wayback_machine.py<br /> <br /> ===Accelerator Address Geolocation===<br /> Term: Spring 2017<br /> <br /> Usage: Used to find latitude and longitude points for all accelerators in the accelerator data files.<br /> E:\McNair\Projects\Accelerators\Code+Final_Data\process_locations.py<br /> <br /> ===Accelerator Data Parser===<br /> Term: Spring 2017<br /> <br /> Usage: Used to parse the data for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List Accelerator Seed List Project].<br /> E:\McNair\Projects\Accelerators\Code+Final_Data\parse_accelerator_data.py<br /> <br /> ===Cohort Data Parser===<br /> Term: Spring 2017<br /> <br /> Usage: Used to parse cohort data for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List Accelerator Seed List Project].<br /> E:\McNair\Projects\Accelerators\Code+Final_Data\parse_cohort_data.py<br /> <br /> ===Google SiteSearch===<br /> Term: Spring 2017<br /> <br /> Usage: Preliminary stage project intended to find an accurate web site for an unlisted company web address by using Google Search.<br /> <br /> E:\McNair\Projects\Accelerators\Google_SiteSearch\sitesearch.py<br /> <br /> ===F6S Crawler===<br /> Term: Fall 2016<br /> <br /> Usage: Used to download html files containing accelerator information from the F6S website.<br /> <br /> E:\McNair\Projects\Accelerators\F6S Accelerator HTMLs\F6S_Crawler\f6s_crawler_gentle.py<br /> <br /> ===F6S Parser===<br /> Term: Fall 2016<br /> <br /> Usage: Used to parse the html files downloaded by the F6S crawler to create a list of accelerators.<br /> <br /> E:\McNair\Projects\Accelerators\F6S Accelerator HTMLs\F6S_Crawler\f6s_parser.py<br /> <br /> ===Executive Order Crawler===<br /> Term: Fall 2016<br /> <br /> Usage: Used to download executive orders. NOTE: uses scrapy format, run differently from regular python programs.<br /> <br /> E:\McNair\Projects\Executive_order_crawler\executive<br /> <br /> <br /> ===Kuwait Web Driver===<br /> Term: Fall 2016<br /> <br /> Usage: Used to download csvs of bills and questions from the Kuwait Government Website. Uses Selenium. All scripts in the folder do similar things.<br /> <br /> E:\McNair\Projects\Middle East Studies Web Drivers\Kuwait<br /> <br /> ===Moroccan Web Driver===<br /> Term: Fall 2016<br /> <br /> Usage: Used to download pdfs of bills and questions from the Moroccan Government Website. Uses Selenium. All scripts in the folder do similar things.<br /> <br /> E:\McNair\Projects\Middle East Studies Web Drivers\Morocco\Moroccan Bills<br /> <br /> <br /> ==Time at McNair==<br /> [[Peter Jalbert (Work Log)]]<br /> [[Category:McNair Staff]]</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Peter_Jalbert_(Work_Log)&diff=22305 2017-12-06T21:37:16Z

<p>Peterjalbert: </p> <hr /> <div>===Fall 2017===<br /> &lt;onlyinclude&gt; <br /> <br /> [[Peter Jalbert]] [[Work Logs]] [[Peter Jalbert (Work Log)|(log page)]] <br /> <br /> 2017-12-06: Running Morocco Parliament Written Questions script. Analyzing Demo Day Crawler results. Continued downloading for TIGER geocoder.<br /> <br /> 2017-11-28: Debugging Morocco Parliament Crawler. Running Demo Day Crawler for all accelerators and 10 pages per accelerator. TIGER geocoder is back to Forbidden Error.<br /> <br /> 2017-11-27: Rerunning Morocco Parliament Crawler. Fixed KeyTerms.py and running it again. Continued downloading for TIGER geocoder.<br /> <br /> 2017-11-20: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-16: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-15: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote a script to extract counts that were greater than 2 from Keyword Matcher. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder]. Finished re-formatting work logs.<br /> <br /> 2017-11-14: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote an HTML to Text parser. See Parser Demo Day Page for file location. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder].<br /> <br /> 2017-11-13: Built [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser].<br /> <br /> 2017-11-09: Running demo version of Demo Day crawler (Accelerator Google Crawler). Fixing work log format. <br /> <br /> 2017-11-07: Created file with 0s and 1s detailing whether crunchbase has the founder information for an accelerator. Details posted as a TODO on [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List] page. Still waiting for feedback on the PostGIS installation from [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder]. Continued working on Accelerator Google Crawler.<br /> <br /> 2017-11-06: Contacted Geography Center for the US Census Bureau, [https://www.census.gov/geo/about/contact.html here], and began email exchange on PostGIS installation problems. Began working on the [http://mcnair.bakerinstitute.org/wiki/Selenium_Documentation Selenium Documentation]. Also began working on an Accelerator Google Crawler that will be used with Yang and ML to find Demo Days for cohort companies.<br /> <br /> 2017-11-01: Attempted to continue downloading, however ran into HTTP Forbidden errors. Listed the errors on the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder Page].<br /> <br /> 2017-10-31: Began downloading blocks of data for individual states for the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder] project. Wrote out the new wiki page for installation, and beginning to write documentation on usage.<br /> <br /> 2017-10-30: With Ed's help, was able to get the national data from Tiger installed onto a database server. The process required much jumping around and changing users, and all the things we learned are outlined in [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation#Editing_Users the database server documentation] under &quot;Editing Users&quot;.<br /> <br /> 2017-10-25: Continued working on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation TigerCoder Installation].<br /> <br /> 2017-10-24: Throw some addresses into a database, use address normalizer and geocoder. May need to install things. Details on the installation process can be found on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation page].<br /> <br /> 2017-10-23: Finished Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-19: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-18: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_DistrictHouston Innovation District Project].<br /> <br /> 2017-10-17: Constructed ArcGIS maps for the agglomeration project. Finished maps of points for every year in the state of California. Finished maps of Route 128. Began working on selenium Yelp crawler to get cafe locations within the 610-loop.<br /> <br /> 2017-10-16: Assisted Harrison on the USITC project. Looked for natural language processing tools to extract complaintants and defendants along with their location from case files. Experimented with pulling based on parts of speech tags, as well as using geotext or geograpy to pull locations from a case segment.<br /> <br /> 2017-10-13: Updated various project wiki pages.<br /> <br /> 2017-10-12: Continued work on Patent Thicket project, awaiting further project specs.<br /> <br /> 2017-10-05: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-04: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-02: Worked on ArcGIS data. See Harrison's Work Log for the details.<br /> <br /> 2017-09-28: Added collaborative editing feature to PyCharm. <br /> <br /> 2017-09-27: Worked on big database file.<br /> <br /> 2017-09-25: New task -- Create text file with company, description, and company type.<br /> #[http://mcnair.bakerinstitute.org/wiki/VC_Database_Rebuild VC Database Rebuild]<br /> #psql vcdb2<br /> #table name, sdccompanybasecore2<br /> #Combine with Crunchbasebulk<br /> <br /> #TODO: Write wiki on linkedin crawler, write wiki on creating accounts.<br /> <br /> 2017-09-21: Wrote wiki on Linkedin crawler, met with Laura about patents project. <br /> <br /> 2017-09-20: Finished running linkedin crawler. Transferred data to RDP. Will write wikis next.<br /> <br /> 2017-09-19: Began running linkedin crawler. Helped Yang create RDP account, get permissions, and get wiki setup.<br /> <br /> 2017-09-18: Finished implementation of Experience Crawler, continued working on Education Crawler for LinkedIn.<br /> <br /> 2017-09-14: Continued implementing LinkedIn Crawler for profiles.<br /> <br /> 2017-09-13: Implemented LinkedIn Crawler for main portion of profiles. Began working on crawling Experience section of profiles.<br /> <br /> 2017-09-12: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. Added to the wiki on this topic.<br /> <br /> 2017-09-11: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. <br /> <br /> 2017-09-06: Combined founders data retrieved with the Crunchbase API with the crunchbasebulk data to get linkedin urls for different accelerator founders. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-09-05: Post Harvey. Finished retrieving names from the Crunchbase API on founders. Next step is to query crunchbase bulk database to get linkedin urls. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-08-24: Began using the Crunchbase API to retrieve founder information for accelerators. Halfway through compiling a dictionary that translates accelerator names into proper Crunchbase API URLs.<br /> <br /> 2017-08-23: Decided with Ed to abandon LinkedIn crawling to retrieve accelerator founder data, and instead use crunchbase. Spent the day navigating the crunchbasebulk database, and seeing what useful information was contained in it.<br /> <br /> 2017-08-22: Discovered that LinkedIn Profiles cannot be viewed through LinkedIn if the target is 3rd degree or further. However, if entering LinkedIn through a Google search, the profile can still be viewed if the user has previously logged into LinkedIn. Devising a workaround crawler that utilizes Google search. Continued blog post [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) here] under Section 4.<br /> <br /> 2017-08-21: Began work on extracting founders for accelerators through LinkedIn Crawler. Discovered that Python3 is not installed on RDP, so the virtual environment for the project cannot be fired up. Continued working on Ubuntu machine.<br /> &lt;/onlyinclude&gt; <br /> <br /> ===Spring 2017===<br /> <br /> 2017-05-01: Continued work on HTML Parser. Uploaded all semester projects to git server.<br /> <br /> 2017-04-20: Finished the HTML Parser for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Ran HTML parser on accelerator founders. Data is stored in projects/accelerators/LinkedIn Founder Data.<br /> <br /> 2017-04-19: Made updates to the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler] Wikipage. Ran LinkedIn Crawler on accelerator data. Working on an html parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-18: Ran LinkedIn Crawler on matches between Crunchbase Snapshot and the accelerator data.<br /> <br /> 2017-04-17: Worked on ways to get correct search results from the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Worked on an HTML Parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-13: Worked on debugging the logout procedure for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Began formulation of process to search for founders of startups using a combination of the LinkedIn Crawler with the data resources from the [http://mcnair.bakerinstitute.org/wiki/Crunchbase_2013_Snapshot CrunchBase Snapshot].<br /> <br /> 2017-04-12: Work on bugs with the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler].<br /> <br /> 2017-04-11: Completed functional [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) crawler of LinkedIn Recruiter Pro]. Basic search functions work and download profile information for a given person. <br /> <br /> 2017-04-10: Began writing functioning crawler of LinkedIn. <br /> <br /> 2017-04-06: Continued working on debugging and documenting the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Wrote a test program that logs in, searches for a query, navigates through search pages, and logs out. Recruiter program can now login and search.<br /> <br /> 2017-04-05: Began work on the LinkedIn Crawler. Researched on launching Python Virtual Environment.<br /> <br /> 2017-04-03: Finished debugging points for the Enclosing Circle Algorithm. Added Command Line functionality to the Industry Classifier.<br /> <br /> 2017-03-29: Worked on debugging points for the Enclosing Circle Algorithm.<br /> <br /> 2017-03-28: Finished running the Enclosing Circle Algorithm. Worked on removing incorrect points from the data set(see above). <br /> <br /> 2017-03-27: Worked on debugging the Enclosing Circle Algorithm. Implemented a way to remove interior circles, and determined that translation to latitude and longitude coordinates resulted in slightly off center circles.<br /> <br /> 2017-03-23: Finished debugging the brute force algorithm for [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm]. Implemented a method to plot the points and circles on a graph. Analyzed runtime of the brute force algorithm.<br /> <br /> 2017-03-21: Coded a brute force algorithm for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-20: Worked on debugging the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-09: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Finished script to draw Enclosing Circles on a Google Map.<br /> <br /> 2017-03-08: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Created script to draw outcome of the Enclosing Circle Algorithm on Google Maps.<br /> <br /> 2017-03-07: Redetermined the top 50 cities which Enclosing Circle should be run on. Data on the [http://mcnair.bakerinstitute.org/wiki/Top_Cities_for_VC_Backed_Companies Top 50 Cities for VC Backed Companies can be found here.] Ran [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities.<br /> <br /> 2017-03-06: Ran script to determine the top 50 cities which Enclosing Circle should be run on. Fixed the VC Circles script to take in a new data format.<br /> <br /> 2017-03-02: Cleaned up data for the VC Circles Project. Created histogram of data in Excel. See [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Began work on the [http://mcnair.bakerinstitute.org/wiki/LinkedInCrawlerPython LinkedIn Crawler].<br /> <br /> 2017-03-01: Created statistics for the VC Circles Project.<br /> <br /> 2017-02-28: Finished downloading geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Found bug in Enclosing Circle Algorithm.<br /> <br /> 2017-02-27: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-23: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Installed C++ Compiler for Python. Ran tests on difference between Python and C wrapped Python.<br /> <br /> 2017-02-22: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Helped out with [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-21: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Researched into C++ Compilers for Python so that the Enclosing Circle Algorithm could be wrapped in C. Found a recommended one [https://www.microsoft.com/en-us/download/details.aspx?id=44266 here].<br /> <br /> 2017-02-20: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-16: Reworked [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] to create a file of geocoded data. Began work on wrapping the algorithm in C to improve speed.<br /> <br /> 2017-02-15: Finished [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] applied to the VC study. Enclosing Circle algorithm still needs adjustment, but the program runs with the temporary fixes.<br /> <br /> 2017-02-14: Worked on the application of the Enclosing Circle algorithm to the VC study. Working on bug fixes in the Enclosing Circle algorithm. Created wiki page for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-02-13: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-08: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-07: Fixed bugs in parse_cohort_data.py, the script for parsing the cohort data from the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Added descriptive statistics to cohort data excel file.<br /> <br /> 2017-02-02: Out sick, independent research and work from RDP. Brief research into the [http://jorgeg.scripts.mit.edu/homepage/wp-content/uploads/2016/03/Guzman-Stern-State-of-American-Entrepreneurship-FINAL.pdf Stern-Guzman algorithm]. Research into [http://mcnair.bakerinstitute.org/wiki/interactive_maps Interactive Maps]. No helpful additions to map embedding problem.<br /> <br /> 2017-02-01: Notes from Session with Ed: Project on US university patenting and entrepreneurship programs (writing code to identify universities in assignees), search Wikipedia (XML then bulk download), student pop, faculty pop, etc.<br /> Circle project for VC data will end up being a joint project to join accelerator data. <br /> Pull descriptions for VC. Founders of accelerators in linkedin. LinkedIn cannot be caught(pretend to not be a bot). Can eventually get academic backgrounds through linkedin. <br /> Pull business registration data, Stern/Guzman Algorithm. <br /> GIS ontop of geocoded data.<br /> Maps that works on wiki or blog (CartoDB), Maps API and R.<br /> NLP Projects, Description Classifier.<br /> <br /> 2017-01-31: Built WayBack Machine Crawler. Updated documentation for coordinates script. Updated profile page to include locations of code.<br /> <br /> <br /> 2017-01-30: Optimized enclosing circle algorithm through memoization. Developed script to read addresses from accelerator data and return latitude and longitude coordinates.<br /> <br /> 2017-01-26: Continued working on Google sitesearch project. Discovered crunchbase, changed project priority. Priority 1, split accelerator data up by flag, priority 2, use crunchbase to get web urls for cohorts, priority 3, make internet archive wayback machine driver. Located [http://mcnair.bakerinstitute.org/wiki/Whois_Parser Whois Parser].<br /> <br /> <br /> 2017-01-25: Finished parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Some data files still need proofreading as they are not in an acceptable format. Began working on Google sitesearch project.<br /> <br /> 2017-01-24: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Cohort data file created, debugging is almost complete. Will begin work on the google accelerator search soon.<br /> <br /> 2017-01-23: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Preliminary code is written, working on debugging.<br /> <br /> 2017-01-19: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project]. Created parser for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project], completed creation of final data set(yay!). Began working on cohort parser.<br /> <br /> 2017-01-18: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-13: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-12: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-11: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-10: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> <br /> ===Fall 2016===<br /> <br /> 2016-12-08: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-07: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-06: Learned how to use git. Committed software projects from the semester to the McNair git repository. Projects can be found at; [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Executive Order Crawler], [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Foreign Government Web Crawlers], [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) F6S Crawler and Parser].<br /> <br /> 2016-12-02: Built and ran web crawler for Center for Middle East Studies on Kuwait. Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-01: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Built tool for the [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report E&amp;I Governance Report Project] with Christy. Adds a column of data that shows whether or not the bill has been passed.<br /> <br /> 2016-11-29: Began pulling data from the accelerators listed [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here]. Made text files for about 18 accelerators.<br /> <br /> 2016-11-22: Transferred downloaded Morocco Written Bills to provided SeaGate Drive. Made a &quot;gentle&quot; F6S crawler to retrieve HTMLs of possible accelerator pages documented [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here].<br /> <br /> 2016-11-18: Converted Executive Order PDFs to text files using adobe acrobat DC. See [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Wikipage] for details.<br /> <br /> 2016-11-17: Wrote a crawler to retrieve information about executive orders, and their corresponding pdfs. They can be found [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report here.] Next step is to run code to convert the pdfs to text files, then use the parser fixed by Christy. <br /> <br /> 2016-11-15: Finished download of Moroccan Written Question pdfs. Wrote a parser with Christy to be used for parsing bills from Congress and eventually executive orders. Found bug in the system Python that was worked out and rebooted.<br /> <br /> 2016-11-11: Continued to download Moroccan data in the background. Attempted to find bug fixes for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] crawler.<br /> <br /> 2016-11-10: Continued to download Moroccan data and Kuwait data in the background. Began work on [http://mcnair.bakerinstitute.org/wiki/Google_Scholar_Crawler Google Scholar Crawler]. Wrote a crawler for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] to get the HTML files of hundreds of accelerators. The crawler ended up failing; it appears to have been due to HTTPS. <br /> <br /> 2016-11-08: Continued to download Moroccan data in the background. Finished writing code for the embedded files on the Kuwait Site. Spent time debugging the frame errors due to the dynamically generated content. Never found an answer to the bug, and instead found a workaround that sacrificed run time for the ability to work. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-04: Continued to download Moroccan data in the background. Finished writing initial Kuwait Web Crawler/Driver for the Middle East Studies Department. Middle East Studies Department asked for additional embedded files in the Kuwait website. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-03: Continued to download Moroccan data in the background. Dr. Egan fixed systems requirements to run the GovTrack Web Crawler. Made significant progress on the Kuwait Web Crawler/Driver for the Middle East Studies Department. <br /> <br /> 2016-11-01: Continued to download Moroccan data in the background. Went over code for GovTracker Web Crawler, continued learning Perl. [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] Began Kuwait Web Crawler/Driver.<br /> <br /> 2016-10-21: Continued to download data for the Moroccan Parliament Written and Oral Questions. Looked over [http://mcnair.bakerinstitute.org/wiki/Christy_Warden_(Twitter_Crawler_Application_1) Christy's Twitter Crawler] to see how I can be helpful. Dr. Egan asked me to think about how to potentially make multiple tools to get cohorts and other sorts of data from accelerator sites. See [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator List] He also asked me to look at the [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] for potential ideas on how to bring this project to fruition.<br /> <br /> 2016-10-20: Continued to download data for the Moroccan Parliament Written and Oral Questions. Updated Wiki page. Started working on Twitter project with Christy. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-18: Finished code for Oral Questions web driver and Written Questions web driver using selenium. Now, the data for the dates of questions can be found using the crawler, and the pdfs of the questions will be downloaded using selenium. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-14: Finished Oral Questions crawler. Finished Written Questions crawler. Waiting for further details on whether that data needs to be tweaked in any way. Updated the Moroccan Web Driver/Web Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-13: Completed download of Moroccan Bills. Working on either a web driver screenshot approach or a webcrawler approach to download the Moroccan oral and written questions data. Began building Web Crawler for Oral and Written Questions site. Edited Moroccan Web Driver/Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-11: Fixed arabic bug, files can now be saved with arabic titles. Monarchy bills downloaded and ready for shipment. House of Representatives Bill mostly downloaded, ratified bills prepared for download. Started learning scrapy library in python for web scraping. Discussed idea of screenshot-ing questions instead of scraping. <br /> <br /> 2016-10-07: Learned unicode and utf8 encoding and decoding in arabic. Still working on transforming an ascii url into printable unicode.<br /> <br /> 2016-10-06: Discussed with Dr. Elbadawy about the desired file names for Moroccan data download. The consensus was that the bill programs are ready to launch once the files can be named properly, and the questions data must be retrieved using a web crawler which I need to learn how to implement. The naming of files is currently drawing errors in going from arabic, to url, to download, to filename. Debugging in process. Also built a demo selenium program for Dr. Egan that drives the McNair blog site on an infinite loop.<br /> <br /> 2016-10-03: Moroccan Web Driver projects completed for driving of the Monarchy proposed bills, the House of Representatives proposed bills, and the Ratified bills sites. Begun process of devising a naming system for the files that does not require scraping. Tinkered with naming through regular expression parsing of the URL. Structure for the oral questions and written questions drivers is set up, but need fixes due to the differences in the sites. Fixed bug on McNair wiki for women's biz team where email was plain text instead of an email link. Took a glimpse at Kuwait Parliament website, and it appears to be very different from the Moroccan setup.<br /> <br /> 2016-09-30: Selenium program selects view pdf option from the website, and goes to the pdf webpage. Program then switches handle to the new page. CTRL S is sent to the page to launch save dialog window. Text cannot be sent to this window. Brainstorm ways around this issue. Explored Chrome Options for saving automatically without a dialog window. Looking into other libraries besides selenium that may help.<br /> <br /> 2016-09-29: Re-enroll in Microsoft Remote Desktop with proper authentication, set up Selenium environment and Komodo IDE on Remote Desktop, wrote program using Selenium that goes to a link and opens up the print dialog box. Developed computational recipe for a different approach to the problem.<br /> <br /> 2016-09-26: Set up Staff wiki page, work log page; registered for Slack, Microsoft Remote Desktop; downloaded Selenium on personal computer, read Selenium docs. Created wiki page for Moroccan Web Driver Project.<br /> <br /> '''Notes'''<br /> <br /> *Ed moved the Morocco Data to E:\McNair\Projects from C:\Users\PeterJ\Documents<br /> * C Drive files moved to E:\McNair\Users\PeterJ<br /> <br /> [[Category:Work Log]]</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Tiger_Geocoder&diff=22304 2017-12-06T21:36:07Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Tiger Geocoder<br /> |Has owner=Peter Jalbert,<br /> |Has start date=Fall 2017<br /> |Has keywords=Tiger, Geocoder, Database<br /> |Has project status=Active<br /> |Has Image=Tiger.jpg<br /> }}<br /> <br /> Note: The Address Ranges National Geodatabase is available from https://www.census.gov/geo/maps-data/data/tiger-geodatabases.html<br /> <br /> This page serves as documentation for using the Tiger Geocoder on Postgres SQL, as part of the PostGIS extension. The following wiki pages may also be of use to you:<br /> <br /> [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation]<br /> [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation Database Server Documentation]<br /> <br /> The official documentation for using and installing the Tiger Geocoder can be found in the following.<br /> <br /> [https://postgis.net/docs/Extras.html#Tiger_Geocoder General Instructions]<br /> [https://postgis.net/docs/postgis_installation.html#install_tiger_geocoder_extension Installation Instructions]<br /> [http://postgis.net/docs/Geocode.html Geocoder Documentation]<br /> <br /> ==Location==<br /> The data is currently loaded into a psql database called geocoder. The tables contain the geocoding information, and there is a test table called &quot;coffeeshops&quot; that contains addresses of Houston coffeeshops according to yelp. To access the database, first login to the McNair DB Sever. Then, <br /> <br /> psql geocoder<br /> <br /> <br /> <br /> ==Installation==<br /> <br /> ===Install and Nation Data===<br /> I began by adding the extension listed above. First, enter into Postgres by using the psql command. Then:<br /> --Add Extensions to database<br /> CREATE EXTENSION postgis;<br /> CREATE EXTENSION fuzzystrmatch;<br /> CREATE EXTENSION postgis_tiger_geocoder;<br /> CREATE EXTENSION address_standardizer;<br /> <br /> You can test that the installation worked by running the following query: <br /> SELECT na.address, na.streetname,na.streettypeabbrev, na.zip<br /> FROM normalize_address('1 Devonshire Place, Boston, MA 02109') AS na;<br /> <br /> This should return the following:<br /> address | streetname | streettypeabbrev | zip<br /> ---------+------------+------------------+-------<br /> 1 | Devonshire | Pl | 02109<br /> <br /> Next, a new profile needs to be created by using the following command.<br /> INSERT INTO tiger.loader_platform(os, declare_sect, pgbin, wget, unzip_command, psql, path_sep, <br /> loader, environ_set_command, county_process_command)<br /> SELECT 'test', declare_sect, pgbin, wget, unzip_command, psql, path_sep,<br /> loader, environ_set_command, county_process_command<br /> FROM tiger.loader_platform<br /> WHERE os = 'sh';<br /> <br /> The installation instructions also provide the following note:<br /> <br /> As of PostGIS 2.4.1 the Zip code-5 digit tabulation area zcta5 load step was revised to load current zcta5 data and is part of the Loader_Generate_Nation_Script when enabled. It is turned off by default because it takes quite a bit of time to load (20 to 60 minutes), takes up quite a bit of disk space, and is not used that often.<br /> <br /> If you would like this feature, you can enable it by using the following command. This should be done before loading the script.<br /> <br /> UPDATE tiger.loader_lookuptables SET load = true WHERE table_name = 'zcta510';<br /> <br /> The paths in declare_sect need to be edited so they match our server locations. One option is to edit the declare_sect column in the tiger.loader_platform table. If so, the declare_sect looks like the following:<br /> export PGHOST=localhost +<br /> export PGUSER=postgres +<br /> export PGPASSWORD=yourpasswordhere +<br /> export PGDATABASE=geocoder +<br /> PSQL=${PGBIN}/psql +<br /> SHP2PGSQL=shp2pgsql +<br /> cd ${staging_fold} +<br /> <br /> TMPDIR=&quot;${staging_fold}/temp/&quot; +<br /> UNZIPTOOL=unzip +<br /> WGETTOOL=&quot;/usr/bin/wget&quot; +<br /> export PGBIN=/usr/lib/postgresql/9.6/bin +<br /> export PGPORT=5432 +<br /> export PGHOST=localhost +<br /> export PGUSER=postgres +<br /> export PGPASSWORD=yourpasswordhere +<br /> export PGDATABASE=geocoder +<br /> PSQL=${PGBIN}/psql +<br /> SHP2PGSQL=shp2pgsql +<br /> cd ${staging_fold}<br /> <br /> Another option is to edit the sh file before running the script. We will do this option until further notice. Simply use your favorite command line editor to change the fields to their correct values. The downloaded script is located in the following directory:<br /> /gisdata<br /> <br /> There needs to be a directory called &quot;temp&quot; in the gisdata directory. To make the script, use the following from the command line:<br /> psql -c &quot;SELECT Loader_Generate_Nation_Script('test')&quot; -d databasename -tA &gt; /gisdata/nation_script_load.sh<br /> <br /> This will create a script in the /gisdata directory. Change to that directory. If you did not edit the paths in the declare_sect table in psql, then you will need to edit this file to contain the correct paths. <br /> <br /> Change directories:<br /> cd /gisdata<br /> <br /> Edit the script using your favorite command line text editor. Specifically, edit the following fields.<br /> PGUSER=postgres<br /> PGPASSWORD=(Ask Anne for this password)!<br /> Everything else remains the same.<br /> <br /> Run the script by using: <br /> sh nation_script_load.sh<br /> <br /> Now, there is a barebones table in the database that will hold the information for the nation. Next is to download data on each state.<br /> <br /> ===State Data===<br /> <br /> The state scripts are generated in much the same way that the nation script was generated. Use the following command, substituting MA for your desired state abbreviation, and substituting a unique filename at the end.<br /> psql -c &quot;SELECT Loader_Generate_Script(ARRAY['MA'], 'test')&quot; -d geocoder -tA &gt; /gisdata/ma_load.sh<br /> <br /> CURRENT PROGRESS:<br /> The following states have been downloaded into the geocoder database.<br /> AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, IA, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI<br /> <br /> ===Current Errors===<br /> The state scripts stopped working on 11/1/2017 while they were working on 10/31/2017. Now, when a retrieval script is run, it draws the error <br /> HTTP request sent, awaiting response... 403 Forbidden<br /> Possible thoughts:<br /> <br /> Maybe our IP has been blacklisted for downloading data from a government website quickly? If so, PostGIS should really choose a different installation method. The current one is dumb.<br /> <br /> Maybe the nation downloader script never worked properly. Not sure how to check if it is correct or not; seems right.<br /> <br /> [https://trac.osgeo.org/postgis/ticket/3699 This] is the only online forum I could find with others who have faced a similar issue.<br /> <br /> NOTE: Contacted the US Census Bureau and never got a response. However, the scripts magically started working again about a week later. Second however, the scripts stopped working again the next day.<br /> <br /> ==Geocode Function==<br /> <br /> The official arguments for the function are the following:<br /> setof record geocode(varchar address, integer max_results=10, geometry restrict_region=NULL, norm_addy OUT addy, geometry OUT geomout, integer OUT rating);<br /> The arguments of interest are address, where you simply submit a string, and max_results, which restricts the number of geocoding attempts per address. The geocoder makes multiple guesses to the location of an address, and returns the best guesses in order. If you want multiple guesses to a specific address, then specify max_results to be more than 1.<br /> <br /> ===Single Address===<br /> <br /> An example query for a single address is:<br /> SELECT g.rating, ST_X(g.geomout) As lon, ST_Y(g.geomout) As lat,<br /> (addy).address As stno, (addy).streetname As street,<br /> (addy).streettypeabbrev As styp, (addy).location As city, (addy).stateabbrev As st,(addy).zip<br /> FROM geocode('75 State Street, Boston MA 02109', 1) As g;<br /> <br /> rating -- This is an integer that determines the confidence in the resulting geocode. The closer to 0, the more confident the guess.<br /> <br /> ST_X(g.geomout) -- This retrieves the longitude coordinate of the point.<br /> <br /> ST_Y(g.geomout) -- This retrieves the latitude corodinate of the point.<br /> <br /> addy -- In general, addy is a normalized address resulting from the input address.<br /> <br /> (addy).address -- The number of the address (Ex: &quot;75&quot; Blabla rd.)<br /> <br /> (addy).streetname -- The name of the street (Ex: 75 &quot;Blabla&quot; rd.)<br /> <br /> (addy).streettypeabbrev -- The abbreviation of the street (Ex: 75 blabla &quot;rd&quot;)<br /> <br /> (addy).location -- The city location of the address.<br /> <br /> (addy).stateabbrev -- The state abbreviation.<br /> <br /> (addy).zip -- The zipcode of the address.<br /> <br /> The output of the query above would be:<br /> rating | lon | lat | stno | street | styp | city | st | zip<br /> --------+-------------------+----------------+------+--------+------+--------+----+-------<br /> 0 | -71.0557505845646 | 42.35897920691 | 75 | State | St | Boston | MA | 02109<br /> <br /> ==Current Errors==<br /> Currently, we are getting a 403 Forbidden Error when trying to download state data. We are in contact with the US Census Bureau. Their contact information can be found [https://www.census.gov/geo/about/contact.html here].<br /> <br /> The email exchanges are recorded below.<br /> <br /> --------------------------------------<br /> --------------------------------------<br /> <br /> I am a student researcher at the McNair Center for Entrepreneurship and Innovation at Rice University, and I am in the process of installing a Postgres Extension that relies on the TIGER data. <br /> <br /> When I began installing the extension, things were working fine. Now however, I am getting a 403 Forbidden error when the script tries to download the TIGER files. Do you have any idea why this might be happening? <br /> <br /> The extension I'm trying to install is below:<br /> <br /> http://postgis.net/docs/postgis_installation.html#install_tiger_geocoder_extension<br /> <br /> When I run any of the scripts that require data from TIGER, I am receiving the following error:<br /> <br /> <br /> --2017-11-06 14:15:29-- http://www2.census.gov/geo/tiger/TIGER2016/FEATNAMES/tl_2016_06065_featnames.zip<br /> Resolving www2.census.gov (www2.census.gov)... 104.84.241.90, 2600:1404:a:382::208c, 2600:1404:a:39c::208c<br /> Connecting to www2.census.gov (www2.census.gov)|104.84.241.90|:80... connected.<br /> HTTP request sent, awaiting response... 403 Forbidden<br /> 2017-11-06 14:15:29 ERROR 403: Forbidden.<br /> <br /> <br /> ----------------------------------------<br /> <br /> Thank you for reaching out to us. The postgis docs is not from the Geography Division, so I cannot comment on that. A couple things that come to mind.<br /> <br /> We just released our new 2017 Shapefiles, so it's possible the scripts may be written for a previous version of our Shapefiles.<br /> You may need to clean out your cookies, restart your browser, and then attempt to reinstall.<br /> Were you able to download our Shapefiles successfully?<br /> <br /> ----------------------------------------<br /> <br /> The Shapefiles are part of the files being blocked by the 403 Forbidden error. <br /> <br /> The script is using a wget protocol to bulk download the data, so there were no cookies involved. Also, the download had been working in previous days; it was only recently that the same scripts stopped working. I am worried that our IP address somehow ended up on a blacklist for the TIGER data. Is there a blacklist for addresses that access the TIGER data?<br /> <br /> Our IP address is 128.42.44.181. <br /> <br /> ---------------------------------------<br /> <br /> I have forwarded your question on to our IT folks. Since I work on the subject matter side, I am unable to answer your questions. Once I hear back from them, I will forward their response to you. Hopefully they will provide to you what you need in order to download our Shapefiles. My apologies for your inconvenience.<br /> <br /> <br /> ----------------------------------------</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Demo_Day_Page_Parser&diff=22131 2017-11-28T21:49:52Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Demo Day Page Parser<br /> |Has owner=Peter Jalbert,<br /> |Has project status=Active<br /> }}<br /> <br /> ==Project Specs==<br /> The goal of this project is to leverage data mining with Selenium and Machine Learning to get good candidate web pages for Demo Days for accelerators. Relevant information on the project can be found on the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Data Accelerator Data] page.<br /> <br /> ==Code Location==<br /> The code directory for this project can be found:<br /> E:\McNair\Software\Accelerators<br /> <br /> The Selenium-based crawler can be found in the file below. This script runs a google search on accelerator names and keywords, and saves the urls and html pages for future use:<br /> DemoDayCrawler.py<br /> <br /> <br /> A script to rip from HTML to TXT can be found below. This script reads HTML files from the DemoDayHTML directory, and writes them to the DemoDayTxt directory:<br /> htmlToText.py<br /> <br /> A script to match Keywords (Accelerator and Cohort names) against the resulting text pages can be found in KeyTerms.py. The script takes the Keywords located in CohortAndAcceleratorsFullList.txt, and the text files in DemoDayTxt, and creates a file with the number of matches of each keyword against each text file.<br /> <br /> The script can be found:<br /> KeyTerms.py<br /> <br /> The Keyword matches text file can be found:<br /> DemoDayTxt\KeyTermFile\KeyTerms.txt<br /> <br /> A script to determine the text files of webpages that have at least one hit of these key words can be found:<br /> DemoDayHits.py<br /> <br /> ==Downloading HTML Files with Selenium==<br /> The code for utilizing Selenium to download HTML files can be found in the DemoDayCrawler.py file.<br /> <br /> The initial observation set over the data scraped 100 links for each of 20 sample accelerators from the list of overall accelerators. These sample pages were turned to text, and scored to remove web pages with no mention of relevant accelerators or companies.<br /> <br /> Once the process was tweaked in response to the initial sample testing, the process ran again over all accelerators. The test determined that we needed take no more than 10 links for each accelerator, and that 'Demo Day' was a suitable search term.<br /> <br /> COMPLETE FILES<br /> -----------------<br /> These files hold data for all the accelerators: not just the test set.<br /> <br /> The full list of accelerators:<br /> ListOfAccs.txt<br /> <br /> The full list of potential keywords (used for throwing out irrelevant results):<br /> Keywords.txt<br /> <br /> A list of accelerators, queries, and urls:<br /> demoday_crawl_full.txt<br /> <br /> A directory with HTML files for all accelerator demo day results:<br /> DemoDayHTMLFull<br /> <br /> A directory with TXT files for all accelerator demo day results:<br /> DemoDayTxtFull<br /> <br /> A file with the name of the results that passed keyword matching:<br /> DemoDayHitsFull.txt</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Demo_Day_Page_Parser&diff=22130 2017-11-28T21:24:21Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Demo Day Page Parser<br /> |Has owner=Peter Jalbert,<br /> |Has project status=Active<br /> }}<br /> <br /> ==Project Specs==<br /> The goal of this project is to leverage data mining with Selenium and Machine Learning to get good candidate web pages for Demo Days for accelerators. Relevant information on the project can be found on the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Data Accelerator Data] page.<br /> <br /> ==Code Location==<br /> The code directory for this project can be found:<br /> E:\McNair\Software\Accelerators<br /> <br /> The Selenium-based crawler can be found in the file below. This script runs a google search on accelerator names and keywords, and saves the urls and html pages for future use:<br /> DemoDayCrawler.py<br /> <br /> <br /> A script to rip from HTML to TXT can be found below. This script reads HTML files from the DemoDayHTML directory, and writes them to the DemoDayTxt directory:<br /> htmlToText.py<br /> <br /> A script to match Keywords (Accelerator and Cohort names) against the resulting text pages can be found in KeyTerms.py. The script takes the Keywords located in CohortAndAcceleratorsFullList.txt, and the text files in DemoDayTxt, and creates a file with the number of matches of each keyword against each text file.<br /> <br /> The script can be found:<br /> KeyTerms.py<br /> <br /> The Keyword matches text file can be found:<br /> DemoDayTxt\KeyTermFile\KeyTerms.txt<br /> <br /> A script to determine the text files of webpages that have at least one hit of these key words can be found:<br /> DemoDayHits.py<br /> <br /> ==Downloading HTML Files with Selenium==<br /> The code for utilizing Selenium to download HTML files can be found in the DemoDayCrawler.py file.<br /> <br /> The initial observation set over the data scraped 100 links for each of 20 sample accelerators from the list of overall accelerators. These sample pages were turned to text, and scored to remove web pages with no mention of relevant accelerators or companies.<br /> <br /> Once the process was tweaked in response to the initial sample testing, the process ran again over all accelerators. The test determined that we needed take no more than 10 links for each accelerator, and that 'Demo Day' was a suitable search term.<br /> <br /> COMPLETE FILES<br /> ============<br /> These files hold data for all the accelerators: not just the test set.<br /> <br /> The full list of accelerators:<br /> ListOfAccs.txt<br /> <br /> The full list of potential keywords (used for throwing out irrelevant results):<br /> Keywords.txt<br /> <br /> A list of accelerators, queries, and urls:<br /> demoday_crawl_full.txt<br /> <br /> A directory with HTML files for all accelerator demo day results:<br /> DemoDayHTMLFull<br /> <br /> A directory with TXT files for all accelerator demo day results:<br /> DemoDayTxtFull<br /> <br /> A file with the name of the results that passed keyword matching:<br /> DemoDayHitsFull.txt</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Demo_Day_Page_Parser&diff=22129 2017-11-28T21:23:42Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Demo Day Page Parser<br /> |Has owner=Peter Jalbert,<br /> |Has project status=Active<br /> }}<br /> <br /> ==Project Specs==<br /> The goal of this project is to leverage data mining with Selenium and Machine Learning to get good candidate web pages for Demo Days for accelerators. Relevant information on the project can be found on the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Data Accelerator Data] page.<br /> <br /> ==Code Location==<br /> The code directory for this project can be found:<br /> E:\McNair\Software\Accelerators<br /> <br /> The Selenium-based crawler can be found in the file below. This script runs a google search on accelerator names and keywords, and saves the urls and html pages for future use:<br /> DemoDayCrawler.py<br /> <br /> <br /> A script to rip from HTML to TXT can be found below. This script reads HTML files from the DemoDayHTML directory, and writes them to the DemoDayTxt directory:<br /> htmlToText.py<br /> <br /> A script to match Keywords (Accelerator and Cohort names) against the resulting text pages can be found in KeyTerms.py. The script takes the Keywords located in CohortAndAcceleratorsFullList.txt, and the text files in DemoDayTxt, and creates a file with the number of matches of each keyword against each text file.<br /> <br /> The script can be found:<br /> KeyTerms.py<br /> <br /> The Keyword matches text file can be found:<br /> DemoDayTxt\KeyTermFile\KeyTerms.txt<br /> <br /> A script to determine the text files of webpages that have at least one hit of these key words can be found:<br /> DemoDayHits.py<br /> <br /> ==Downloading HTML Files with Selenium==<br /> The code for utilizing Selenium to download HTML files can be found in the DemoDayCrawler.py file.<br /> <br /> The initial observation set over the data scraped 100 links for each of 20 sample accelerators from the list of overall accelerators. These sample pages were turned to text, and scored to remove web pages with no mention of relevant accelerators or companies.<br /> <br /> Once the process was tweaked in response to the initial sample testing, the process ran again over all accelerators. The test determined that we needed take no more than 10 links for each accelerator, and that 'Demo Day' was a suitable search term.<br /> <br /> COMPLETE FILES<br /> ============<br /> These files hold data for all the accelerators: not just the test set.<br /> <br /> The full list of accelerators:<br /> ListOfAccs.txt<br /> <br /> The full list of potential keywords (used for throwing out irrelevant results):<br /> Keywords.txt<br /> <br /> A list of accelerators, queries, and urls:<br /> demoday_crawl_full.txt<br /> <br /> A directory with HTML files for all accelerator demo day results:<br /> DemoDayHTMLFull<br /> <br /> A directory with TXT files for all accelerator demo day results:<br /> DemoDayTxtFull</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Peter_Jalbert_(Work_Log)&diff=22128 2017-11-28T20:55:12Z

<p>Peterjalbert: </p> <hr /> <div>===Fall 2017===<br /> &lt;onlyinclude&gt; <br /> <br /> [[Peter Jalbert]] [[Work Logs]] [[Peter Jalbert (Work Log)|(log page)]] <br /> <br /> 2017-11-28: Debugging Morocco Parliament Crawler. Running Demo Day Crawler for all accelerators and 10 pages per accelerator. TIGER geocoder is back to Forbidden Error.<br /> <br /> 2017-11-27: Rerunning Morocco Parliament Crawler. Fixed KeyTerms.py and running it again. Continued downloading for TIGER geocoder.<br /> <br /> 2017-11-20: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-16: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-15: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote a script to extract counts that were greater than 2 from Keyword Matcher. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder]. Finished re-formatting work logs.<br /> <br /> 2017-11-14: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote an HTML to Text parser. See Parser Demo Day Page for file location. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder].<br /> <br /> 2017-11-13: Built [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser].<br /> <br /> 2017-11-09: Running demo version of Demo Day crawler (Accelerator Google Crawler). Fixing work log format. <br /> <br /> 2017-11-07: Created file with 0s and 1s detailing whether crunchbase has the founder information for an accelerator. Details posted as a TODO on [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List] page. Still waiting for feedback on the PostGIS installation from [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder]. Continued working on Accelerator Google Crawler.<br /> <br /> 2017-11-06: Contacted Geography Center for the US Census Bureau, [https://www.census.gov/geo/about/contact.html here], and began email exchange on PostGIS installation problems. Began working on the [http://mcnair.bakerinstitute.org/wiki/Selenium_Documentation Selenium Documentation]. Also began working on an Accelerator Google Crawler that will be used with Yang and ML to find Demo Days for cohort companies.<br /> <br /> 2017-11-01: Attempted to continue downloading, however ran into HTTP Forbidden errors. Listed the errors on the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder Page].<br /> <br /> 2017-10-31: Began downloading blocks of data for individual states for the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder] project. Wrote out the new wiki page for installation, and beginning to write documentation on usage.<br /> <br /> 2017-10-30: With Ed's help, was able to get the national data from Tiger installed onto a database server. The process required much jumping around and changing users, and all the things we learned are outlined in [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation#Editing_Users the database server documentation] under &quot;Editing Users&quot;.<br /> <br /> 2017-10-25: Continued working on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation TigerCoder Installation].<br /> <br /> 2017-10-24: Throw some addresses into a database, use address normalizer and geocoder. May need to install things. Details on the installation process can be found on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation page].<br /> <br /> 2017-10-23: Finished Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-19: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-18: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_DistrictHouston Innovation District Project].<br /> <br /> 2017-10-17: Constructed ArcGIS maps for the agglomeration project. Finished maps of points for every year in the state of California. Finished maps of Route 128. Began working on selenium Yelp crawler to get cafe locations within the 610-loop.<br /> <br /> 2017-10-16: Assisted Harrison on the USITC project. Looked for natural language processing tools to extract complaintants and defendants along with their location from case files. Experimented with pulling based on parts of speech tags, as well as using geotext or geograpy to pull locations from a case segment.<br /> <br /> 2017-10-13: Updated various project wiki pages.<br /> <br /> 2017-10-12: Continued work on Patent Thicket project, awaiting further project specs.<br /> <br /> 2017-10-05: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-04: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-02: Worked on ArcGIS data. See Harrison's Work Log for the details.<br /> <br /> 2017-09-28: Added collaborative editing feature to PyCharm. <br /> <br /> 2017-09-27: Worked on big database file.<br /> <br /> 2017-09-25: New task -- Create text file with company, description, and company type.<br /> #[http://mcnair.bakerinstitute.org/wiki/VC_Database_Rebuild VC Database Rebuild]<br /> #psql vcdb2<br /> #table name, sdccompanybasecore2<br /> #Combine with Crunchbasebulk<br /> <br /> #TODO: Write wiki on linkedin crawler, write wiki on creating accounts.<br /> <br /> 2017-09-21: Wrote wiki on Linkedin crawler, met with Laura about patents project. <br /> <br /> 2017-09-20: Finished running linkedin crawler. Transferred data to RDP. Will write wikis next.<br /> <br /> 2017-09-19: Began running linkedin crawler. Helped Yang create RDP account, get permissions, and get wiki setup.<br /> <br /> 2017-09-18: Finished implementation of Experience Crawler, continued working on Education Crawler for LinkedIn.<br /> <br /> 2017-09-14: Continued implementing LinkedIn Crawler for profiles.<br /> <br /> 2017-09-13: Implemented LinkedIn Crawler for main portion of profiles. Began working on crawling Experience section of profiles.<br /> <br /> 2017-09-12: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. Added to the wiki on this topic.<br /> <br /> 2017-09-11: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. <br /> <br /> 2017-09-06: Combined founders data retrieved with the Crunchbase API with the crunchbasebulk data to get linkedin urls for different accelerator founders. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-09-05: Post Harvey. Finished retrieving names from the Crunchbase API on founders. Next step is to query crunchbase bulk database to get linkedin urls. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-08-24: Began using the Crunchbase API to retrieve founder information for accelerators. Halfway through compiling a dictionary that translates accelerator names into proper Crunchbase API URLs.<br /> <br /> 2017-08-23: Decided with Ed to abandon LinkedIn crawling to retrieve accelerator founder data, and instead use crunchbase. Spent the day navigating the crunchbasebulk database, and seeing what useful information was contained in it.<br /> <br /> 2017-08-22: Discovered that LinkedIn Profiles cannot be viewed through LinkedIn if the target is 3rd degree or further. However, if entering LinkedIn through a Google search, the profile can still be viewed if the user has previously logged into LinkedIn. Devising a workaround crawler that utilizes Google search. Continued blog post [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) here] under Section 4.<br /> <br /> 2017-08-21: Began work on extracting founders for accelerators through LinkedIn Crawler. Discovered that Python3 is not installed on RDP, so the virtual environment for the project cannot be fired up. Continued working on Ubuntu machine.<br /> &lt;/onlyinclude&gt; <br /> <br /> ===Spring 2017===<br /> <br /> 2017-05-01: Continued work on HTML Parser. Uploaded all semester projects to git server.<br /> <br /> 2017-04-20: Finished the HTML Parser for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Ran HTML parser on accelerator founders. Data is stored in projects/accelerators/LinkedIn Founder Data.<br /> <br /> 2017-04-19: Made updates to the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler] Wikipage. Ran LinkedIn Crawler on accelerator data. Working on an html parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-18: Ran LinkedIn Crawler on matches between Crunchbase Snapshot and the accelerator data.<br /> <br /> 2017-04-17: Worked on ways to get correct search results from the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Worked on an HTML Parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-13: Worked on debugging the logout procedure for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Began formulation of process to search for founders of startups using a combination of the LinkedIn Crawler with the data resources from the [http://mcnair.bakerinstitute.org/wiki/Crunchbase_2013_Snapshot CrunchBase Snapshot].<br /> <br /> 2017-04-12: Work on bugs with the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler].<br /> <br /> 2017-04-11: Completed functional [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) crawler of LinkedIn Recruiter Pro]. Basic search functions work and download profile information for a given person. <br /> <br /> 2017-04-10: Began writing functioning crawler of LinkedIn. <br /> <br /> 2017-04-06: Continued working on debugging and documenting the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Wrote a test program that logs in, searches for a query, navigates through search pages, and logs out. Recruiter program can now login and search.<br /> <br /> 2017-04-05: Began work on the LinkedIn Crawler. Researched on launching Python Virtual Environment.<br /> <br /> 2017-04-03: Finished debugging points for the Enclosing Circle Algorithm. Added Command Line functionality to the Industry Classifier.<br /> <br /> 2017-03-29: Worked on debugging points for the Enclosing Circle Algorithm.<br /> <br /> 2017-03-28: Finished running the Enclosing Circle Algorithm. Worked on removing incorrect points from the data set(see above). <br /> <br /> 2017-03-27: Worked on debugging the Enclosing Circle Algorithm. Implemented a way to remove interior circles, and determined that translation to latitude and longitude coordinates resulted in slightly off center circles.<br /> <br /> 2017-03-23: Finished debugging the brute force algorithm for [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm]. Implemented a method to plot the points and circles on a graph. Analyzed runtime of the brute force algorithm.<br /> <br /> 2017-03-21: Coded a brute force algorithm for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-20: Worked on debugging the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-09: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Finished script to draw Enclosing Circles on a Google Map.<br /> <br /> 2017-03-08: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Created script to draw outcome of the Enclosing Circle Algorithm on Google Maps.<br /> <br /> 2017-03-07: Redetermined the top 50 cities which Enclosing Circle should be run on. Data on the [http://mcnair.bakerinstitute.org/wiki/Top_Cities_for_VC_Backed_Companies Top 50 Cities for VC Backed Companies can be found here.] Ran [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities.<br /> <br /> 2017-03-06: Ran script to determine the top 50 cities which Enclosing Circle should be run on. Fixed the VC Circles script to take in a new data format.<br /> <br /> 2017-03-02: Cleaned up data for the VC Circles Project. Created histogram of data in Excel. See [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Began work on the [http://mcnair.bakerinstitute.org/wiki/LinkedInCrawlerPython LinkedIn Crawler].<br /> <br /> 2017-03-01: Created statistics for the VC Circles Project.<br /> <br /> 2017-02-28: Finished downloading geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Found bug in Enclosing Circle Algorithm.<br /> <br /> 2017-02-27: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-23: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Installed C++ Compiler for Python. Ran tests on difference between Python and C wrapped Python.<br /> <br /> 2017-02-22: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Helped out with [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-21: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Researched into C++ Compilers for Python so that the Enclosing Circle Algorithm could be wrapped in C. Found a recommended one [https://www.microsoft.com/en-us/download/details.aspx?id=44266 here].<br /> <br /> 2017-02-20: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-16: Reworked [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] to create a file of geocoded data. Began work on wrapping the algorithm in C to improve speed.<br /> <br /> 2017-02-15: Finished [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] applied to the VC study. Enclosing Circle algorithm still needs adjustment, but the program runs with the temporary fixes.<br /> <br /> 2017-02-14: Worked on the application of the Enclosing Circle algorithm to the VC study. Working on bug fixes in the Enclosing Circle algorithm. Created wiki page for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-02-13: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-08: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-07: Fixed bugs in parse_cohort_data.py, the script for parsing the cohort data from the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Added descriptive statistics to cohort data excel file.<br /> <br /> 2017-02-02: Out sick, independent research and work from RDP. Brief research into the [http://jorgeg.scripts.mit.edu/homepage/wp-content/uploads/2016/03/Guzman-Stern-State-of-American-Entrepreneurship-FINAL.pdf Stern-Guzman algorithm]. Research into [http://mcnair.bakerinstitute.org/wiki/interactive_maps Interactive Maps]. No helpful additions to map embedding problem.<br /> <br /> 2017-02-01: Notes from Session with Ed: Project on US university patenting and entrepreneurship programs (writing code to identify universities in assignees), search Wikipedia (XML then bulk download), student pop, faculty pop, etc.<br /> Circle project for VC data will end up being a joint project to join accelerator data. <br /> Pull descriptions for VC. Founders of accelerators in linkedin. LinkedIn cannot be caught(pretend to not be a bot). Can eventually get academic backgrounds through linkedin. <br /> Pull business registration data, Stern/Guzman Algorithm. <br /> GIS ontop of geocoded data.<br /> Maps that works on wiki or blog (CartoDB), Maps API and R.<br /> NLP Projects, Description Classifier.<br /> <br /> 2017-01-31: Built WayBack Machine Crawler. Updated documentation for coordinates script. Updated profile page to include locations of code.<br /> <br /> <br /> 2017-01-30: Optimized enclosing circle algorithm through memoization. Developed script to read addresses from accelerator data and return latitude and longitude coordinates.<br /> <br /> 2017-01-26: Continued working on Google sitesearch project. Discovered crunchbase, changed project priority. Priority 1, split accelerator data up by flag, priority 2, use crunchbase to get web urls for cohorts, priority 3, make internet archive wayback machine driver. Located [http://mcnair.bakerinstitute.org/wiki/Whois_Parser Whois Parser].<br /> <br /> <br /> 2017-01-25: Finished parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Some data files still need proofreading as they are not in an acceptable format. Began working on Google sitesearch project.<br /> <br /> 2017-01-24: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Cohort data file created, debugging is almost complete. Will begin work on the google accelerator search soon.<br /> <br /> 2017-01-23: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Preliminary code is written, working on debugging.<br /> <br /> 2017-01-19: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project]. Created parser for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project], completed creation of final data set(yay!). Began working on cohort parser.<br /> <br /> 2017-01-18: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-13: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-12: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-11: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-10: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> <br /> ===Fall 2016===<br /> <br /> 2016-12-08: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-07: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-06: Learned how to use git. Committed software projects from the semester to the McNair git repository. Projects can be found at; [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Executive Order Crawler], [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Foreign Government Web Crawlers], [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) F6S Crawler and Parser].<br /> <br /> 2016-12-02: Built and ran web crawler for Center for Middle East Studies on Kuwait. Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-01: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Built tool for the [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report E&amp;I Governance Report Project] with Christy. Adds a column of data that shows whether or not the bill has been passed.<br /> <br /> 2016-11-29: Began pulling data from the accelerators listed [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here]. Made text files for about 18 accelerators.<br /> <br /> 2016-11-22: Transferred downloaded Morocco Written Bills to provided SeaGate Drive. Made a &quot;gentle&quot; F6S crawler to retrieve HTMLs of possible accelerator pages documented [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here].<br /> <br /> 2016-11-18: Converted Executive Order PDFs to text files using adobe acrobat DC. See [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Wikipage] for details.<br /> <br /> 2016-11-17: Wrote a crawler to retrieve information about executive orders, and their corresponding pdfs. They can be found [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report here.] Next step is to run code to convert the pdfs to text files, then use the parser fixed by Christy. <br /> <br /> 2016-11-15: Finished download of Moroccan Written Question pdfs. Wrote a parser with Christy to be used for parsing bills from Congress and eventually executive orders. Found bug in the system Python that was worked out and rebooted.<br /> <br /> 2016-11-11: Continued to download Moroccan data in the background. Attempted to find bug fixes for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] crawler.<br /> <br /> 2016-11-10: Continued to download Moroccan data and Kuwait data in the background. Began work on [http://mcnair.bakerinstitute.org/wiki/Google_Scholar_Crawler Google Scholar Crawler]. Wrote a crawler for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] to get the HTML files of hundreds of accelerators. The crawler ended up failing; it appears to have been due to HTTPS. <br /> <br /> 2016-11-08: Continued to download Moroccan data in the background. Finished writing code for the embedded files on the Kuwait Site. Spent time debugging the frame errors due to the dynamically generated content. Never found an answer to the bug, and instead found a workaround that sacrificed run time for the ability to work. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-04: Continued to download Moroccan data in the background. Finished writing initial Kuwait Web Crawler/Driver for the Middle East Studies Department. Middle East Studies Department asked for additional embedded files in the Kuwait website. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-03: Continued to download Moroccan data in the background. Dr. Egan fixed systems requirements to run the GovTrack Web Crawler. Made significant progress on the Kuwait Web Crawler/Driver for the Middle East Studies Department. <br /> <br /> 2016-11-01: Continued to download Moroccan data in the background. Went over code for GovTracker Web Crawler, continued learning Perl. [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] Began Kuwait Web Crawler/Driver.<br /> <br /> 2016-10-21: Continued to download data for the Moroccan Parliament Written and Oral Questions. Looked over [http://mcnair.bakerinstitute.org/wiki/Christy_Warden_(Twitter_Crawler_Application_1) Christy's Twitter Crawler] to see how I can be helpful. Dr. Egan asked me to think about how to potentially make multiple tools to get cohorts and other sorts of data from accelerator sites. See [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator List] He also asked me to look at the [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] for potential ideas on how to bring this project to fruition.<br /> <br /> 2016-10-20: Continued to download data for the Moroccan Parliament Written and Oral Questions. Updated Wiki page. Started working on Twitter project with Christy. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-18: Finished code for Oral Questions web driver and Written Questions web driver using selenium. Now, the data for the dates of questions can be found using the crawler, and the pdfs of the questions will be downloaded using selenium. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-14: Finished Oral Questions crawler. Finished Written Questions crawler. Waiting for further details on whether that data needs to be tweaked in any way. Updated the Moroccan Web Driver/Web Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-13: Completed download of Moroccan Bills. Working on either a web driver screenshot approach or a webcrawler approach to download the Moroccan oral and written questions data. Began building Web Crawler for Oral and Written Questions site. Edited Moroccan Web Driver/Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-11: Fixed arabic bug, files can now be saved with arabic titles. Monarchy bills downloaded and ready for shipment. House of Representatives Bill mostly downloaded, ratified bills prepared for download. Started learning scrapy library in python for web scraping. Discussed idea of screenshot-ing questions instead of scraping. <br /> <br /> 2016-10-07: Learned unicode and utf8 encoding and decoding in arabic. Still working on transforming an ascii url into printable unicode.<br /> <br /> 2016-10-06: Discussed with Dr. Elbadawy about the desired file names for Moroccan data download. The consensus was that the bill programs are ready to launch once the files can be named properly, and the questions data must be retrieved using a web crawler which I need to learn how to implement. The naming of files is currently drawing errors in going from arabic, to url, to download, to filename. Debugging in process. Also built a demo selenium program for Dr. Egan that drives the McNair blog site on an infinite loop.<br /> <br /> 2016-10-03: Moroccan Web Driver projects completed for driving of the Monarchy proposed bills, the House of Representatives proposed bills, and the Ratified bills sites. Begun process of devising a naming system for the files that does not require scraping. Tinkered with naming through regular expression parsing of the URL. Structure for the oral questions and written questions drivers is set up, but need fixes due to the differences in the sites. Fixed bug on McNair wiki for women's biz team where email was plain text instead of an email link. Took a glimpse at Kuwait Parliament website, and it appears to be very different from the Moroccan setup.<br /> <br /> 2016-09-30: Selenium program selects view pdf option from the website, and goes to the pdf webpage. Program then switches handle to the new page. CTRL S is sent to the page to launch save dialog window. Text cannot be sent to this window. Brainstorm ways around this issue. Explored Chrome Options for saving automatically without a dialog window. Looking into other libraries besides selenium that may help.<br /> <br /> 2016-09-29: Re-enroll in Microsoft Remote Desktop with proper authentication, set up Selenium environment and Komodo IDE on Remote Desktop, wrote program using Selenium that goes to a link and opens up the print dialog box. Developed computational recipe for a different approach to the problem.<br /> <br /> 2016-09-26: Set up Staff wiki page, work log page; registered for Slack, Microsoft Remote Desktop; downloaded Selenium on personal computer, read Selenium docs. Created wiki page for Moroccan Web Driver Project.<br /> <br /> '''Notes'''<br /> <br /> *Ed moved the Morocco Data to E:\McNair\Projects from C:\Users\PeterJ\Documents<br /> * C Drive files moved to E:\McNair\Users\PeterJ<br /> <br /> [[Category:Work Log]]</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Tiger_Geocoder&diff=22079 2017-11-27T22:31:45Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Tiger Geocoder<br /> |Has owner=Peter Jalbert,<br /> |Has start date=Fall 2017<br /> |Has keywords=Tiger, Geocoder, Database<br /> |Has project status=Active<br /> |Has Image=Tiger.jpg<br /> }}<br /> <br /> Note: The Address Ranges National Geodatabase is available from https://www.census.gov/geo/maps-data/data/tiger-geodatabases.html<br /> <br /> This page serves as documentation for using the Tiger Geocoder on Postgres SQL, as part of the PostGIS extension. The following wiki pages may also be of use to you:<br /> <br /> [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation]<br /> [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation Database Server Documentation]<br /> <br /> The official documentation for using and installing the Tiger Geocoder can be found in the following.<br /> <br /> [https://postgis.net/docs/Extras.html#Tiger_Geocoder General Instructions]<br /> [https://postgis.net/docs/postgis_installation.html#install_tiger_geocoder_extension Installation Instructions]<br /> [http://postgis.net/docs/Geocode.html Geocoder Documentation]<br /> <br /> ==Location==<br /> The data is currently loaded into a psql database called geocoder. The tables contain the geocoding information, and there is a test table called &quot;coffeeshops&quot; that contains addresses of Houston coffeeshops according to yelp. To access the database, first login to the McNair DB Sever. Then, <br /> <br /> psql geocoder<br /> <br /> <br /> <br /> ==Installation==<br /> <br /> ===Install and Nation Data===<br /> I began by adding the extension listed above. First, enter into Postgres by using the psql command. Then:<br /> --Add Extensions to database<br /> CREATE EXTENSION postgis;<br /> CREATE EXTENSION fuzzystrmatch;<br /> CREATE EXTENSION postgis_tiger_geocoder;<br /> CREATE EXTENSION address_standardizer;<br /> <br /> You can test that the installation worked by running the following query: <br /> SELECT na.address, na.streetname,na.streettypeabbrev, na.zip<br /> FROM normalize_address('1 Devonshire Place, Boston, MA 02109') AS na;<br /> <br /> This should return the following:<br /> address | streetname | streettypeabbrev | zip<br /> ---------+------------+------------------+-------<br /> 1 | Devonshire | Pl | 02109<br /> <br /> Next, a new profile needs to be created by using the following command.<br /> INSERT INTO tiger.loader_platform(os, declare_sect, pgbin, wget, unzip_command, psql, path_sep, <br /> loader, environ_set_command, county_process_command)<br /> SELECT 'test', declare_sect, pgbin, wget, unzip_command, psql, path_sep,<br /> loader, environ_set_command, county_process_command<br /> FROM tiger.loader_platform<br /> WHERE os = 'sh';<br /> <br /> The installation instructions also provide the following note:<br /> <br /> As of PostGIS 2.4.1 the Zip code-5 digit tabulation area zcta5 load step was revised to load current zcta5 data and is part of the Loader_Generate_Nation_Script when enabled. It is turned off by default because it takes quite a bit of time to load (20 to 60 minutes), takes up quite a bit of disk space, and is not used that often.<br /> <br /> If you would like this feature, you can enable it by using the following command. This should be done before loading the script.<br /> <br /> UPDATE tiger.loader_lookuptables SET load = true WHERE table_name = 'zcta510';<br /> <br /> The paths in declare_sect need to be edited so they match our server locations. One option is to edit the declare_sect column in the tiger.loader_platform table. If so, the declare_sect looks like the following:<br /> export PGHOST=localhost +<br /> export PGUSER=postgres +<br /> export PGPASSWORD=yourpasswordhere +<br /> export PGDATABASE=geocoder +<br /> PSQL=${PGBIN}/psql +<br /> SHP2PGSQL=shp2pgsql +<br /> cd ${staging_fold} +<br /> <br /> TMPDIR=&quot;${staging_fold}/temp/&quot; +<br /> UNZIPTOOL=unzip +<br /> WGETTOOL=&quot;/usr/bin/wget&quot; +<br /> export PGBIN=/usr/lib/postgresql/9.6/bin +<br /> export PGPORT=5432 +<br /> export PGHOST=localhost +<br /> export PGUSER=postgres +<br /> export PGPASSWORD=yourpasswordhere +<br /> export PGDATABASE=geocoder +<br /> PSQL=${PGBIN}/psql +<br /> SHP2PGSQL=shp2pgsql +<br /> cd ${staging_fold}<br /> <br /> Another option is to edit the sh file before running the script. We will do this option until further notice. Simply use your favorite command line editor to change the fields to their correct values. The downloaded script is located in the following directory:<br /> /gisdata<br /> <br /> There needs to be a directory called &quot;temp&quot; in the gisdata directory. To make the script, use the following from the command line:<br /> psql -c &quot;SELECT Loader_Generate_Nation_Script('test')&quot; -d databasename -tA &gt; /gisdata/nation_script_load.sh<br /> <br /> This will create a script in the /gisdata directory. Change to that directory. If you did not edit the paths in the declare_sect table in psql, then you will need to edit this file to contain the correct paths. <br /> <br /> Change directories:<br /> cd /gisdata<br /> <br /> Edit the script using your favorite command line text editor. Specifically, edit the following fields.<br /> PGUSER=postgres<br /> PGPASSWORD=(Ask Anne for this password)!<br /> Everything else remains the same.<br /> <br /> Run the script by using: <br /> sh nation_script_load.sh<br /> <br /> Now, there is a barebones table in the database that will hold the information for the nation. Next is to download data on each state.<br /> <br /> ===State Data===<br /> <br /> The state scripts are generated in much the same way that the nation script was generated. Use the following command, substituting MA for your desired state abbreviation, and substituting a unique filename at the end.<br /> psql -c &quot;SELECT Loader_Generate_Script(ARRAY['MA'], 'test')&quot; -d geocoder -tA &gt; /gisdata/ma_load.sh<br /> <br /> CURRENT PROGRESS:<br /> The following states have been downloaded into the geocoder database.<br /> AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, IA, ID, IL, MA<br /> <br /> ===Current Errors===<br /> The state scripts stopped working on 11/1/2017 while they were working on 10/31/2017. Now, when a retrieval script is run, it draws the error <br /> HTTP request sent, awaiting response... 403 Forbidden<br /> Possible thoughts:<br /> <br /> Maybe our IP has been blacklisted for downloading data from a government website quickly? If so, PostGIS should really choose a different installation method. The current one is dumb.<br /> <br /> Maybe the nation downloader script never worked properly. Not sure how to check if it is correct or not; seems right.<br /> <br /> [https://trac.osgeo.org/postgis/ticket/3699 This] is the only online forum I could find with others who have faced a similar issue.<br /> <br /> NOTE: Contacted the US Census Bureau and never got a response. However, the scripts magically started working again about a week later. Second however, the scripts stopped working again the next day.<br /> <br /> ==Geocode Function==<br /> <br /> The official arguments for the function are the following:<br /> setof record geocode(varchar address, integer max_results=10, geometry restrict_region=NULL, norm_addy OUT addy, geometry OUT geomout, integer OUT rating);<br /> The arguments of interest are address, where you simply submit a string, and max_results, which restricts the number of geocoding attempts per address. The geocoder makes multiple guesses to the location of an address, and returns the best guesses in order. If you want multiple guesses to a specific address, then specify max_results to be more than 1.<br /> <br /> ===Single Address===<br /> <br /> An example query for a single address is:<br /> SELECT g.rating, ST_X(g.geomout) As lon, ST_Y(g.geomout) As lat,<br /> (addy).address As stno, (addy).streetname As street,<br /> (addy).streettypeabbrev As styp, (addy).location As city, (addy).stateabbrev As st,(addy).zip<br /> FROM geocode('75 State Street, Boston MA 02109', 1) As g;<br /> <br /> rating -- This is an integer that determines the confidence in the resulting geocode. The closer to 0, the more confident the guess.<br /> <br /> ST_X(g.geomout) -- This retrieves the longitude coordinate of the point.<br /> <br /> ST_Y(g.geomout) -- This retrieves the latitude corodinate of the point.<br /> <br /> addy -- In general, addy is a normalized address resulting from the input address.<br /> <br /> (addy).address -- The number of the address (Ex: &quot;75&quot; Blabla rd.)<br /> <br /> (addy).streetname -- The name of the street (Ex: 75 &quot;Blabla&quot; rd.)<br /> <br /> (addy).streettypeabbrev -- The abbreviation of the street (Ex: 75 blabla &quot;rd&quot;)<br /> <br /> (addy).location -- The city location of the address.<br /> <br /> (addy).stateabbrev -- The state abbreviation.<br /> <br /> (addy).zip -- The zipcode of the address.<br /> <br /> The output of the query above would be:<br /> rating | lon | lat | stno | street | styp | city | st | zip<br /> --------+-------------------+----------------+------+--------+------+--------+----+-------<br /> 0 | -71.0557505845646 | 42.35897920691 | 75 | State | St | Boston | MA | 02109<br /> <br /> ==Current Errors==<br /> Currently, we are getting a 403 Forbidden Error when trying to download state data. We are in contact with the US Census Bureau. Their contact information can be found [https://www.census.gov/geo/about/contact.html here].<br /> <br /> The email exchanges are recorded below.<br /> <br /> --------------------------------------<br /> --------------------------------------<br /> <br /> I am a student researcher at the McNair Center for Entrepreneurship and Innovation at Rice University, and I am in the process of installing a Postgres Extension that relies on the TIGER data. <br /> <br /> When I began installing the extension, things were working fine. Now however, I am getting a 403 Forbidden error when the script tries to download the TIGER files. Do you have any idea why this might be happening? <br /> <br /> The extension I'm trying to install is below:<br /> <br /> http://postgis.net/docs/postgis_installation.html#install_tiger_geocoder_extension<br /> <br /> When I run any of the scripts that require data from TIGER, I am receiving the following error:<br /> <br /> <br /> --2017-11-06 14:15:29-- http://www2.census.gov/geo/tiger/TIGER2016/FEATNAMES/tl_2016_06065_featnames.zip<br /> Resolving www2.census.gov (www2.census.gov)... 104.84.241.90, 2600:1404:a:382::208c, 2600:1404:a:39c::208c<br /> Connecting to www2.census.gov (www2.census.gov)|104.84.241.90|:80... connected.<br /> HTTP request sent, awaiting response... 403 Forbidden<br /> 2017-11-06 14:15:29 ERROR 403: Forbidden.<br /> <br /> <br /> ----------------------------------------<br /> <br /> Thank you for reaching out to us. The postgis docs is not from the Geography Division, so I cannot comment on that. A couple things that come to mind.<br /> <br /> We just released our new 2017 Shapefiles, so it's possible the scripts may be written for a previous version of our Shapefiles.<br /> You may need to clean out your cookies, restart your browser, and then attempt to reinstall.<br /> Were you able to download our Shapefiles successfully?<br /> <br /> ----------------------------------------<br /> <br /> The Shapefiles are part of the files being blocked by the 403 Forbidden error. <br /> <br /> The script is using a wget protocol to bulk download the data, so there were no cookies involved. Also, the download had been working in previous days; it was only recently that the same scripts stopped working. I am worried that our IP address somehow ended up on a blacklist for the TIGER data. Is there a blacklist for addresses that access the TIGER data?<br /> <br /> Our IP address is 128.42.44.181. <br /> <br /> ---------------------------------------<br /> <br /> I have forwarded your question on to our IT folks. Since I work on the subject matter side, I am unable to answer your questions. Once I hear back from them, I will forward their response to you. Hopefully they will provide to you what you need in order to download our Shapefiles. My apologies for your inconvenience.<br /> <br /> <br /> ----------------------------------------</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Peter_Jalbert_(Work_Log)&diff=22078 2017-11-27T21:26:32Z

<p>Peterjalbert: </p> <hr /> <div>===Fall 2017===<br /> &lt;onlyinclude&gt; <br /> <br /> [[Peter Jalbert]] [[Work Logs]] [[Peter Jalbert (Work Log)|(log page)]] <br /> <br /> 2017-11-27: Rerunning Morocco Parliament Crawler. Fixed KeyTerms.py and running it again. Continued downloading for TIGER geocoder.<br /> <br /> 2017-11-20: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-16: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-15: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote a script to extract counts that were greater than 2 from Keyword Matcher. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder]. Finished re-formatting work logs.<br /> <br /> 2017-11-14: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote an HTML to Text parser. See Parser Demo Day Page for file location. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder].<br /> <br /> 2017-11-13: Built [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser].<br /> <br /> 2017-11-09: Running demo version of Demo Day crawler (Accelerator Google Crawler). Fixing work log format. <br /> <br /> 2017-11-07: Created file with 0s and 1s detailing whether crunchbase has the founder information for an accelerator. Details posted as a TODO on [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List] page. Still waiting for feedback on the PostGIS installation from [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder]. Continued working on Accelerator Google Crawler.<br /> <br /> 2017-11-06: Contacted Geography Center for the US Census Bureau, [https://www.census.gov/geo/about/contact.html here], and began email exchange on PostGIS installation problems. Began working on the [http://mcnair.bakerinstitute.org/wiki/Selenium_Documentation Selenium Documentation]. Also began working on an Accelerator Google Crawler that will be used with Yang and ML to find Demo Days for cohort companies.<br /> <br /> 2017-11-01: Attempted to continue downloading, however ran into HTTP Forbidden errors. Listed the errors on the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder Page].<br /> <br /> 2017-10-31: Began downloading blocks of data for individual states for the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder] project. Wrote out the new wiki page for installation, and beginning to write documentation on usage.<br /> <br /> 2017-10-30: With Ed's help, was able to get the national data from Tiger installed onto a database server. The process required much jumping around and changing users, and all the things we learned are outlined in [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation#Editing_Users the database server documentation] under &quot;Editing Users&quot;.<br /> <br /> 2017-10-25: Continued working on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation TigerCoder Installation].<br /> <br /> 2017-10-24: Throw some addresses into a database, use address normalizer and geocoder. May need to install things. Details on the installation process can be found on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation page].<br /> <br /> 2017-10-23: Finished Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-19: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-18: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_DistrictHouston Innovation District Project].<br /> <br /> 2017-10-17: Constructed ArcGIS maps for the agglomeration project. Finished maps of points for every year in the state of California. Finished maps of Route 128. Began working on selenium Yelp crawler to get cafe locations within the 610-loop.<br /> <br /> 2017-10-16: Assisted Harrison on the USITC project. Looked for natural language processing tools to extract complaintants and defendants along with their location from case files. Experimented with pulling based on parts of speech tags, as well as using geotext or geograpy to pull locations from a case segment.<br /> <br /> 2017-10-13: Updated various project wiki pages.<br /> <br /> 2017-10-12: Continued work on Patent Thicket project, awaiting further project specs.<br /> <br /> 2017-10-05: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-04: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-02: Worked on ArcGIS data. See Harrison's Work Log for the details.<br /> <br /> 2017-09-28: Added collaborative editing feature to PyCharm. <br /> <br /> 2017-09-27: Worked on big database file.<br /> <br /> 2017-09-25: New task -- Create text file with company, description, and company type.<br /> #[http://mcnair.bakerinstitute.org/wiki/VC_Database_Rebuild VC Database Rebuild]<br /> #psql vcdb2<br /> #table name, sdccompanybasecore2<br /> #Combine with Crunchbasebulk<br /> <br /> #TODO: Write wiki on linkedin crawler, write wiki on creating accounts.<br /> <br /> 2017-09-21: Wrote wiki on Linkedin crawler, met with Laura about patents project. <br /> <br /> 2017-09-20: Finished running linkedin crawler. Transferred data to RDP. Will write wikis next.<br /> <br /> 2017-09-19: Began running linkedin crawler. Helped Yang create RDP account, get permissions, and get wiki setup.<br /> <br /> 2017-09-18: Finished implementation of Experience Crawler, continued working on Education Crawler for LinkedIn.<br /> <br /> 2017-09-14: Continued implementing LinkedIn Crawler for profiles.<br /> <br /> 2017-09-13: Implemented LinkedIn Crawler for main portion of profiles. Began working on crawling Experience section of profiles.<br /> <br /> 2017-09-12: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. Added to the wiki on this topic.<br /> <br /> 2017-09-11: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. <br /> <br /> 2017-09-06: Combined founders data retrieved with the Crunchbase API with the crunchbasebulk data to get linkedin urls for different accelerator founders. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-09-05: Post Harvey. Finished retrieving names from the Crunchbase API on founders. Next step is to query crunchbase bulk database to get linkedin urls. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-08-24: Began using the Crunchbase API to retrieve founder information for accelerators. Halfway through compiling a dictionary that translates accelerator names into proper Crunchbase API URLs.<br /> <br /> 2017-08-23: Decided with Ed to abandon LinkedIn crawling to retrieve accelerator founder data, and instead use crunchbase. Spent the day navigating the crunchbasebulk database, and seeing what useful information was contained in it.<br /> <br /> 2017-08-22: Discovered that LinkedIn Profiles cannot be viewed through LinkedIn if the target is 3rd degree or further. However, if entering LinkedIn through a Google search, the profile can still be viewed if the user has previously logged into LinkedIn. Devising a workaround crawler that utilizes Google search. Continued blog post [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) here] under Section 4.<br /> <br /> 2017-08-21: Began work on extracting founders for accelerators through LinkedIn Crawler. Discovered that Python3 is not installed on RDP, so the virtual environment for the project cannot be fired up. Continued working on Ubuntu machine.<br /> &lt;/onlyinclude&gt; <br /> <br /> ===Spring 2017===<br /> <br /> 2017-05-01: Continued work on HTML Parser. Uploaded all semester projects to git server.<br /> <br /> 2017-04-20: Finished the HTML Parser for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Ran HTML parser on accelerator founders. Data is stored in projects/accelerators/LinkedIn Founder Data.<br /> <br /> 2017-04-19: Made updates to the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler] Wikipage. Ran LinkedIn Crawler on accelerator data. Working on an html parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-18: Ran LinkedIn Crawler on matches between Crunchbase Snapshot and the accelerator data.<br /> <br /> 2017-04-17: Worked on ways to get correct search results from the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Worked on an HTML Parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-13: Worked on debugging the logout procedure for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Began formulation of process to search for founders of startups using a combination of the LinkedIn Crawler with the data resources from the [http://mcnair.bakerinstitute.org/wiki/Crunchbase_2013_Snapshot CrunchBase Snapshot].<br /> <br /> 2017-04-12: Work on bugs with the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler].<br /> <br /> 2017-04-11: Completed functional [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) crawler of LinkedIn Recruiter Pro]. Basic search functions work and download profile information for a given person. <br /> <br /> 2017-04-10: Began writing functioning crawler of LinkedIn. <br /> <br /> 2017-04-06: Continued working on debugging and documenting the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Wrote a test program that logs in, searches for a query, navigates through search pages, and logs out. Recruiter program can now login and search.<br /> <br /> 2017-04-05: Began work on the LinkedIn Crawler. Researched on launching Python Virtual Environment.<br /> <br /> 2017-04-03: Finished debugging points for the Enclosing Circle Algorithm. Added Command Line functionality to the Industry Classifier.<br /> <br /> 2017-03-29: Worked on debugging points for the Enclosing Circle Algorithm.<br /> <br /> 2017-03-28: Finished running the Enclosing Circle Algorithm. Worked on removing incorrect points from the data set(see above). <br /> <br /> 2017-03-27: Worked on debugging the Enclosing Circle Algorithm. Implemented a way to remove interior circles, and determined that translation to latitude and longitude coordinates resulted in slightly off center circles.<br /> <br /> 2017-03-23: Finished debugging the brute force algorithm for [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm]. Implemented a method to plot the points and circles on a graph. Analyzed runtime of the brute force algorithm.<br /> <br /> 2017-03-21: Coded a brute force algorithm for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-20: Worked on debugging the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-09: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Finished script to draw Enclosing Circles on a Google Map.<br /> <br /> 2017-03-08: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Created script to draw outcome of the Enclosing Circle Algorithm on Google Maps.<br /> <br /> 2017-03-07: Redetermined the top 50 cities which Enclosing Circle should be run on. Data on the [http://mcnair.bakerinstitute.org/wiki/Top_Cities_for_VC_Backed_Companies Top 50 Cities for VC Backed Companies can be found here.] Ran [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities.<br /> <br /> 2017-03-06: Ran script to determine the top 50 cities which Enclosing Circle should be run on. Fixed the VC Circles script to take in a new data format.<br /> <br /> 2017-03-02: Cleaned up data for the VC Circles Project. Created histogram of data in Excel. See [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Began work on the [http://mcnair.bakerinstitute.org/wiki/LinkedInCrawlerPython LinkedIn Crawler].<br /> <br /> 2017-03-01: Created statistics for the VC Circles Project.<br /> <br /> 2017-02-28: Finished downloading geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Found bug in Enclosing Circle Algorithm.<br /> <br /> 2017-02-27: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-23: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Installed C++ Compiler for Python. Ran tests on difference between Python and C wrapped Python.<br /> <br /> 2017-02-22: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Helped out with [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-21: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Researched into C++ Compilers for Python so that the Enclosing Circle Algorithm could be wrapped in C. Found a recommended one [https://www.microsoft.com/en-us/download/details.aspx?id=44266 here].<br /> <br /> 2017-02-20: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-16: Reworked [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] to create a file of geocoded data. Began work on wrapping the algorithm in C to improve speed.<br /> <br /> 2017-02-15: Finished [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] applied to the VC study. Enclosing Circle algorithm still needs adjustment, but the program runs with the temporary fixes.<br /> <br /> 2017-02-14: Worked on the application of the Enclosing Circle algorithm to the VC study. Working on bug fixes in the Enclosing Circle algorithm. Created wiki page for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-02-13: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-08: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-07: Fixed bugs in parse_cohort_data.py, the script for parsing the cohort data from the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Added descriptive statistics to cohort data excel file.<br /> <br /> 2017-02-02: Out sick, independent research and work from RDP. Brief research into the [http://jorgeg.scripts.mit.edu/homepage/wp-content/uploads/2016/03/Guzman-Stern-State-of-American-Entrepreneurship-FINAL.pdf Stern-Guzman algorithm]. Research into [http://mcnair.bakerinstitute.org/wiki/interactive_maps Interactive Maps]. No helpful additions to map embedding problem.<br /> <br /> 2017-02-01: Notes from Session with Ed: Project on US university patenting and entrepreneurship programs (writing code to identify universities in assignees), search Wikipedia (XML then bulk download), student pop, faculty pop, etc.<br /> Circle project for VC data will end up being a joint project to join accelerator data. <br /> Pull descriptions for VC. Founders of accelerators in linkedin. LinkedIn cannot be caught(pretend to not be a bot). Can eventually get academic backgrounds through linkedin. <br /> Pull business registration data, Stern/Guzman Algorithm. <br /> GIS ontop of geocoded data.<br /> Maps that works on wiki or blog (CartoDB), Maps API and R.<br /> NLP Projects, Description Classifier.<br /> <br /> 2017-01-31: Built WayBack Machine Crawler. Updated documentation for coordinates script. Updated profile page to include locations of code.<br /> <br /> <br /> 2017-01-30: Optimized enclosing circle algorithm through memoization. Developed script to read addresses from accelerator data and return latitude and longitude coordinates.<br /> <br /> 2017-01-26: Continued working on Google sitesearch project. Discovered crunchbase, changed project priority. Priority 1, split accelerator data up by flag, priority 2, use crunchbase to get web urls for cohorts, priority 3, make internet archive wayback machine driver. Located [http://mcnair.bakerinstitute.org/wiki/Whois_Parser Whois Parser].<br /> <br /> <br /> 2017-01-25: Finished parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Some data files still need proofreading as they are not in an acceptable format. Began working on Google sitesearch project.<br /> <br /> 2017-01-24: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Cohort data file created, debugging is almost complete. Will begin work on the google accelerator search soon.<br /> <br /> 2017-01-23: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Preliminary code is written, working on debugging.<br /> <br /> 2017-01-19: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project]. Created parser for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project], completed creation of final data set(yay!). Began working on cohort parser.<br /> <br /> 2017-01-18: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-13: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-12: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-11: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-10: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> <br /> ===Fall 2016===<br /> <br /> 2016-12-08: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-07: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-06: Learned how to use git. Committed software projects from the semester to the McNair git repository. Projects can be found at; [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Executive Order Crawler], [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Foreign Government Web Crawlers], [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) F6S Crawler and Parser].<br /> <br /> 2016-12-02: Built and ran web crawler for Center for Middle East Studies on Kuwait. Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-01: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Built tool for the [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report E&amp;I Governance Report Project] with Christy. Adds a column of data that shows whether or not the bill has been passed.<br /> <br /> 2016-11-29: Began pulling data from the accelerators listed [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here]. Made text files for about 18 accelerators.<br /> <br /> 2016-11-22: Transferred downloaded Morocco Written Bills to provided SeaGate Drive. Made a &quot;gentle&quot; F6S crawler to retrieve HTMLs of possible accelerator pages documented [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here].<br /> <br /> 2016-11-18: Converted Executive Order PDFs to text files using adobe acrobat DC. See [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Wikipage] for details.<br /> <br /> 2016-11-17: Wrote a crawler to retrieve information about executive orders, and their corresponding pdfs. They can be found [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report here.] Next step is to run code to convert the pdfs to text files, then use the parser fixed by Christy. <br /> <br /> 2016-11-15: Finished download of Moroccan Written Question pdfs. Wrote a parser with Christy to be used for parsing bills from Congress and eventually executive orders. Found bug in the system Python that was worked out and rebooted.<br /> <br /> 2016-11-11: Continued to download Moroccan data in the background. Attempted to find bug fixes for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] crawler.<br /> <br /> 2016-11-10: Continued to download Moroccan data and Kuwait data in the background. Began work on [http://mcnair.bakerinstitute.org/wiki/Google_Scholar_Crawler Google Scholar Crawler]. Wrote a crawler for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] to get the HTML files of hundreds of accelerators. The crawler ended up failing; it appears to have been due to HTTPS. <br /> <br /> 2016-11-08: Continued to download Moroccan data in the background. Finished writing code for the embedded files on the Kuwait Site. Spent time debugging the frame errors due to the dynamically generated content. Never found an answer to the bug, and instead found a workaround that sacrificed run time for the ability to work. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-04: Continued to download Moroccan data in the background. Finished writing initial Kuwait Web Crawler/Driver for the Middle East Studies Department. Middle East Studies Department asked for additional embedded files in the Kuwait website. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-03: Continued to download Moroccan data in the background. Dr. Egan fixed systems requirements to run the GovTrack Web Crawler. Made significant progress on the Kuwait Web Crawler/Driver for the Middle East Studies Department. <br /> <br /> 2016-11-01: Continued to download Moroccan data in the background. Went over code for GovTracker Web Crawler, continued learning Perl. [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] Began Kuwait Web Crawler/Driver.<br /> <br /> 2016-10-21: Continued to download data for the Moroccan Parliament Written and Oral Questions. Looked over [http://mcnair.bakerinstitute.org/wiki/Christy_Warden_(Twitter_Crawler_Application_1) Christy's Twitter Crawler] to see how I can be helpful. Dr. Egan asked me to think about how to potentially make multiple tools to get cohorts and other sorts of data from accelerator sites. See [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator List] He also asked me to look at the [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] for potential ideas on how to bring this project to fruition.<br /> <br /> 2016-10-20: Continued to download data for the Moroccan Parliament Written and Oral Questions. Updated Wiki page. Started working on Twitter project with Christy. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-18: Finished code for Oral Questions web driver and Written Questions web driver using selenium. Now, the data for the dates of questions can be found using the crawler, and the pdfs of the questions will be downloaded using selenium. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-14: Finished Oral Questions crawler. Finished Written Questions crawler. Waiting for further details on whether that data needs to be tweaked in any way. Updated the Moroccan Web Driver/Web Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-13: Completed download of Moroccan Bills. Working on either a web driver screenshot approach or a webcrawler approach to download the Moroccan oral and written questions data. Began building Web Crawler for Oral and Written Questions site. Edited Moroccan Web Driver/Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-11: Fixed arabic bug, files can now be saved with arabic titles. Monarchy bills downloaded and ready for shipment. House of Representatives Bill mostly downloaded, ratified bills prepared for download. Started learning scrapy library in python for web scraping. Discussed idea of screenshot-ing questions instead of scraping. <br /> <br /> 2016-10-07: Learned unicode and utf8 encoding and decoding in arabic. Still working on transforming an ascii url into printable unicode.<br /> <br /> 2016-10-06: Discussed with Dr. Elbadawy about the desired file names for Moroccan data download. The consensus was that the bill programs are ready to launch once the files can be named properly, and the questions data must be retrieved using a web crawler which I need to learn how to implement. The naming of files is currently drawing errors in going from arabic, to url, to download, to filename. Debugging in process. Also built a demo selenium program for Dr. Egan that drives the McNair blog site on an infinite loop.<br /> <br /> 2016-10-03: Moroccan Web Driver projects completed for driving of the Monarchy proposed bills, the House of Representatives proposed bills, and the Ratified bills sites. Begun process of devising a naming system for the files that does not require scraping. Tinkered with naming through regular expression parsing of the URL. Structure for the oral questions and written questions drivers is set up, but need fixes due to the differences in the sites. Fixed bug on McNair wiki for women's biz team where email was plain text instead of an email link. Took a glimpse at Kuwait Parliament website, and it appears to be very different from the Moroccan setup.<br /> <br /> 2016-09-30: Selenium program selects view pdf option from the website, and goes to the pdf webpage. Program then switches handle to the new page. CTRL S is sent to the page to launch save dialog window. Text cannot be sent to this window. Brainstorm ways around this issue. Explored Chrome Options for saving automatically without a dialog window. Looking into other libraries besides selenium that may help.<br /> <br /> 2016-09-29: Re-enroll in Microsoft Remote Desktop with proper authentication, set up Selenium environment and Komodo IDE on Remote Desktop, wrote program using Selenium that goes to a link and opens up the print dialog box. Developed computational recipe for a different approach to the problem.<br /> <br /> 2016-09-26: Set up Staff wiki page, work log page; registered for Slack, Microsoft Remote Desktop; downloaded Selenium on personal computer, read Selenium docs. Created wiki page for Moroccan Web Driver Project.<br /> <br /> '''Notes'''<br /> <br /> *Ed moved the Morocco Data to E:\McNair\Projects from C:\Users\PeterJ\Documents<br /> * C Drive files moved to E:\McNair\Users\PeterJ<br /> <br /> [[Category:Work Log]]</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Peter_Jalbert_(Work_Log)&diff=22077 2017-11-27T21:16:14Z

<p>Peterjalbert: </p> <hr /> <div>===Fall 2017===<br /> &lt;onlyinclude&gt; <br /> <br /> [[Peter Jalbert]] [[Work Logs]] [[Peter Jalbert (Work Log)|(log page)]] <br /> <br /> 2017-11-27: Rerunning Morocco Parliament Crawler. Fixed KeyTerms.py and running it again. <br /> <br /> 2017-11-20: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-16: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-15: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote a script to extract counts that were greater than 2 from Keyword Matcher. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder]. Finished re-formatting work logs.<br /> <br /> 2017-11-14: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote an HTML to Text parser. See Parser Demo Day Page for file location. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder].<br /> <br /> 2017-11-13: Built [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser].<br /> <br /> 2017-11-09: Running demo version of Demo Day crawler (Accelerator Google Crawler). Fixing work log format. <br /> <br /> 2017-11-07: Created file with 0s and 1s detailing whether crunchbase has the founder information for an accelerator. Details posted as a TODO on [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List] page. Still waiting for feedback on the PostGIS installation from [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder]. Continued working on Accelerator Google Crawler.<br /> <br /> 2017-11-06: Contacted Geography Center for the US Census Bureau, [https://www.census.gov/geo/about/contact.html here], and began email exchange on PostGIS installation problems. Began working on the [http://mcnair.bakerinstitute.org/wiki/Selenium_Documentation Selenium Documentation]. Also began working on an Accelerator Google Crawler that will be used with Yang and ML to find Demo Days for cohort companies.<br /> <br /> 2017-11-01: Attempted to continue downloading, however ran into HTTP Forbidden errors. Listed the errors on the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder Page].<br /> <br /> 2017-10-31: Began downloading blocks of data for individual states for the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder] project. Wrote out the new wiki page for installation, and beginning to write documentation on usage.<br /> <br /> 2017-10-30: With Ed's help, was able to get the national data from Tiger installed onto a database server. The process required much jumping around and changing users, and all the things we learned are outlined in [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation#Editing_Users the database server documentation] under &quot;Editing Users&quot;.<br /> <br /> 2017-10-25: Continued working on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation TigerCoder Installation].<br /> <br /> 2017-10-24: Throw some addresses into a database, use address normalizer and geocoder. May need to install things. Details on the installation process can be found on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation page].<br /> <br /> 2017-10-23: Finished Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-19: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-18: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_DistrictHouston Innovation District Project].<br /> <br /> 2017-10-17: Constructed ArcGIS maps for the agglomeration project. Finished maps of points for every year in the state of California. Finished maps of Route 128. Began working on selenium Yelp crawler to get cafe locations within the 610-loop.<br /> <br /> 2017-10-16: Assisted Harrison on the USITC project. Looked for natural language processing tools to extract complaintants and defendants along with their location from case files. Experimented with pulling based on parts of speech tags, as well as using geotext or geograpy to pull locations from a case segment.<br /> <br /> 2017-10-13: Updated various project wiki pages.<br /> <br /> 2017-10-12: Continued work on Patent Thicket project, awaiting further project specs.<br /> <br /> 2017-10-05: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-04: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-02: Worked on ArcGIS data. See Harrison's Work Log for the details.<br /> <br /> 2017-09-28: Added collaborative editing feature to PyCharm. <br /> <br /> 2017-09-27: Worked on big database file.<br /> <br /> 2017-09-25: New task -- Create text file with company, description, and company type.<br /> #[http://mcnair.bakerinstitute.org/wiki/VC_Database_Rebuild VC Database Rebuild]<br /> #psql vcdb2<br /> #table name, sdccompanybasecore2<br /> #Combine with Crunchbasebulk<br /> <br /> #TODO: Write wiki on linkedin crawler, write wiki on creating accounts.<br /> <br /> 2017-09-21: Wrote wiki on Linkedin crawler, met with Laura about patents project. <br /> <br /> 2017-09-20: Finished running linkedin crawler. Transferred data to RDP. Will write wikis next.<br /> <br /> 2017-09-19: Began running linkedin crawler. Helped Yang create RDP account, get permissions, and get wiki setup.<br /> <br /> 2017-09-18: Finished implementation of Experience Crawler, continued working on Education Crawler for LinkedIn.<br /> <br /> 2017-09-14: Continued implementing LinkedIn Crawler for profiles.<br /> <br /> 2017-09-13: Implemented LinkedIn Crawler for main portion of profiles. Began working on crawling Experience section of profiles.<br /> <br /> 2017-09-12: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. Added to the wiki on this topic.<br /> <br /> 2017-09-11: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. <br /> <br /> 2017-09-06: Combined founders data retrieved with the Crunchbase API with the crunchbasebulk data to get linkedin urls for different accelerator founders. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-09-05: Post Harvey. Finished retrieving names from the Crunchbase API on founders. Next step is to query crunchbase bulk database to get linkedin urls. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-08-24: Began using the Crunchbase API to retrieve founder information for accelerators. Halfway through compiling a dictionary that translates accelerator names into proper Crunchbase API URLs.<br /> <br /> 2017-08-23: Decided with Ed to abandon LinkedIn crawling to retrieve accelerator founder data, and instead use crunchbase. Spent the day navigating the crunchbasebulk database, and seeing what useful information was contained in it.<br /> <br /> 2017-08-22: Discovered that LinkedIn Profiles cannot be viewed through LinkedIn if the target is 3rd degree or further. However, if entering LinkedIn through a Google search, the profile can still be viewed if the user has previously logged into LinkedIn. Devising a workaround crawler that utilizes Google search. Continued blog post [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) here] under Section 4.<br /> <br /> 2017-08-21: Began work on extracting founders for accelerators through LinkedIn Crawler. Discovered that Python3 is not installed on RDP, so the virtual environment for the project cannot be fired up. Continued working on Ubuntu machine.<br /> &lt;/onlyinclude&gt; <br /> <br /> ===Spring 2017===<br /> <br /> 2017-05-01: Continued work on HTML Parser. Uploaded all semester projects to git server.<br /> <br /> 2017-04-20: Finished the HTML Parser for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Ran HTML parser on accelerator founders. Data is stored in projects/accelerators/LinkedIn Founder Data.<br /> <br /> 2017-04-19: Made updates to the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler] Wikipage. Ran LinkedIn Crawler on accelerator data. Working on an html parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-18: Ran LinkedIn Crawler on matches between Crunchbase Snapshot and the accelerator data.<br /> <br /> 2017-04-17: Worked on ways to get correct search results from the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Worked on an HTML Parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-13: Worked on debugging the logout procedure for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Began formulation of process to search for founders of startups using a combination of the LinkedIn Crawler with the data resources from the [http://mcnair.bakerinstitute.org/wiki/Crunchbase_2013_Snapshot CrunchBase Snapshot].<br /> <br /> 2017-04-12: Work on bugs with the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler].<br /> <br /> 2017-04-11: Completed functional [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) crawler of LinkedIn Recruiter Pro]. Basic search functions work and download profile information for a given person. <br /> <br /> 2017-04-10: Began writing functioning crawler of LinkedIn. <br /> <br /> 2017-04-06: Continued working on debugging and documenting the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Wrote a test program that logs in, searches for a query, navigates through search pages, and logs out. Recruiter program can now login and search.<br /> <br /> 2017-04-05: Began work on the LinkedIn Crawler. Researched on launching Python Virtual Environment.<br /> <br /> 2017-04-03: Finished debugging points for the Enclosing Circle Algorithm. Added Command Line functionality to the Industry Classifier.<br /> <br /> 2017-03-29: Worked on debugging points for the Enclosing Circle Algorithm.<br /> <br /> 2017-03-28: Finished running the Enclosing Circle Algorithm. Worked on removing incorrect points from the data set(see above). <br /> <br /> 2017-03-27: Worked on debugging the Enclosing Circle Algorithm. Implemented a way to remove interior circles, and determined that translation to latitude and longitude coordinates resulted in slightly off center circles.<br /> <br /> 2017-03-23: Finished debugging the brute force algorithm for [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm]. Implemented a method to plot the points and circles on a graph. Analyzed runtime of the brute force algorithm.<br /> <br /> 2017-03-21: Coded a brute force algorithm for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-20: Worked on debugging the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-09: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Finished script to draw Enclosing Circles on a Google Map.<br /> <br /> 2017-03-08: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Created script to draw outcome of the Enclosing Circle Algorithm on Google Maps.<br /> <br /> 2017-03-07: Redetermined the top 50 cities which Enclosing Circle should be run on. Data on the [http://mcnair.bakerinstitute.org/wiki/Top_Cities_for_VC_Backed_Companies Top 50 Cities for VC Backed Companies can be found here.] Ran [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities.<br /> <br /> 2017-03-06: Ran script to determine the top 50 cities which Enclosing Circle should be run on. Fixed the VC Circles script to take in a new data format.<br /> <br /> 2017-03-02: Cleaned up data for the VC Circles Project. Created histogram of data in Excel. See [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Began work on the [http://mcnair.bakerinstitute.org/wiki/LinkedInCrawlerPython LinkedIn Crawler].<br /> <br /> 2017-03-01: Created statistics for the VC Circles Project.<br /> <br /> 2017-02-28: Finished downloading geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Found bug in Enclosing Circle Algorithm.<br /> <br /> 2017-02-27: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-23: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Installed C++ Compiler for Python. Ran tests on difference between Python and C wrapped Python.<br /> <br /> 2017-02-22: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Helped out with [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-21: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Researched into C++ Compilers for Python so that the Enclosing Circle Algorithm could be wrapped in C. Found a recommended one [https://www.microsoft.com/en-us/download/details.aspx?id=44266 here].<br /> <br /> 2017-02-20: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-16: Reworked [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] to create a file of geocoded data. Began work on wrapping the algorithm in C to improve speed.<br /> <br /> 2017-02-15: Finished [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] applied to the VC study. Enclosing Circle algorithm still needs adjustment, but the program runs with the temporary fixes.<br /> <br /> 2017-02-14: Worked on the application of the Enclosing Circle algorithm to the VC study. Working on bug fixes in the Enclosing Circle algorithm. Created wiki page for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-02-13: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-08: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-07: Fixed bugs in parse_cohort_data.py, the script for parsing the cohort data from the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Added descriptive statistics to cohort data excel file.<br /> <br /> 2017-02-02: Out sick, independent research and work from RDP. Brief research into the [http://jorgeg.scripts.mit.edu/homepage/wp-content/uploads/2016/03/Guzman-Stern-State-of-American-Entrepreneurship-FINAL.pdf Stern-Guzman algorithm]. Research into [http://mcnair.bakerinstitute.org/wiki/interactive_maps Interactive Maps]. No helpful additions to map embedding problem.<br /> <br /> 2017-02-01: Notes from Session with Ed: Project on US university patenting and entrepreneurship programs (writing code to identify universities in assignees), search Wikipedia (XML then bulk download), student pop, faculty pop, etc.<br /> Circle project for VC data will end up being a joint project to join accelerator data. <br /> Pull descriptions for VC. Founders of accelerators in linkedin. LinkedIn cannot be caught(pretend to not be a bot). Can eventually get academic backgrounds through linkedin. <br /> Pull business registration data, Stern/Guzman Algorithm. <br /> GIS ontop of geocoded data.<br /> Maps that works on wiki or blog (CartoDB), Maps API and R.<br /> NLP Projects, Description Classifier.<br /> <br /> 2017-01-31: Built WayBack Machine Crawler. Updated documentation for coordinates script. Updated profile page to include locations of code.<br /> <br /> <br /> 2017-01-30: Optimized enclosing circle algorithm through memoization. Developed script to read addresses from accelerator data and return latitude and longitude coordinates.<br /> <br /> 2017-01-26: Continued working on Google sitesearch project. Discovered crunchbase, changed project priority. Priority 1, split accelerator data up by flag, priority 2, use crunchbase to get web urls for cohorts, priority 3, make internet archive wayback machine driver. Located [http://mcnair.bakerinstitute.org/wiki/Whois_Parser Whois Parser].<br /> <br /> <br /> 2017-01-25: Finished parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Some data files still need proofreading as they are not in an acceptable format. Began working on Google sitesearch project.<br /> <br /> 2017-01-24: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Cohort data file created, debugging is almost complete. Will begin work on the google accelerator search soon.<br /> <br /> 2017-01-23: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Preliminary code is written, working on debugging.<br /> <br /> 2017-01-19: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project]. Created parser for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project], completed creation of final data set(yay!). Began working on cohort parser.<br /> <br /> 2017-01-18: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-13: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-12: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-11: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-10: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> <br /> ===Fall 2016===<br /> <br /> 2016-12-08: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-07: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-06: Learned how to use git. Committed software projects from the semester to the McNair git repository. Projects can be found at; [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Executive Order Crawler], [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Foreign Government Web Crawlers], [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) F6S Crawler and Parser].<br /> <br /> 2016-12-02: Built and ran web crawler for Center for Middle East Studies on Kuwait. Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-01: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Built tool for the [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report E&amp;I Governance Report Project] with Christy. Adds a column of data that shows whether or not the bill has been passed.<br /> <br /> 2016-11-29: Began pulling data from the accelerators listed [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here]. Made text files for about 18 accelerators.<br /> <br /> 2016-11-22: Transferred downloaded Morocco Written Bills to provided SeaGate Drive. Made a &quot;gentle&quot; F6S crawler to retrieve HTMLs of possible accelerator pages documented [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here].<br /> <br /> 2016-11-18: Converted Executive Order PDFs to text files using adobe acrobat DC. See [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Wikipage] for details.<br /> <br /> 2016-11-17: Wrote a crawler to retrieve information about executive orders, and their corresponding pdfs. They can be found [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report here.] Next step is to run code to convert the pdfs to text files, then use the parser fixed by Christy. <br /> <br /> 2016-11-15: Finished download of Moroccan Written Question pdfs. Wrote a parser with Christy to be used for parsing bills from Congress and eventually executive orders. Found bug in the system Python that was worked out and rebooted.<br /> <br /> 2016-11-11: Continued to download Moroccan data in the background. Attempted to find bug fixes for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] crawler.<br /> <br /> 2016-11-10: Continued to download Moroccan data and Kuwait data in the background. Began work on [http://mcnair.bakerinstitute.org/wiki/Google_Scholar_Crawler Google Scholar Crawler]. Wrote a crawler for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] to get the HTML files of hundreds of accelerators. The crawler ended up failing; it appears to have been due to HTTPS. <br /> <br /> 2016-11-08: Continued to download Moroccan data in the background. Finished writing code for the embedded files on the Kuwait Site. Spent time debugging the frame errors due to the dynamically generated content. Never found an answer to the bug, and instead found a workaround that sacrificed run time for the ability to work. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-04: Continued to download Moroccan data in the background. Finished writing initial Kuwait Web Crawler/Driver for the Middle East Studies Department. Middle East Studies Department asked for additional embedded files in the Kuwait website. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-03: Continued to download Moroccan data in the background. Dr. Egan fixed systems requirements to run the GovTrack Web Crawler. Made significant progress on the Kuwait Web Crawler/Driver for the Middle East Studies Department. <br /> <br /> 2016-11-01: Continued to download Moroccan data in the background. Went over code for GovTracker Web Crawler, continued learning Perl. [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] Began Kuwait Web Crawler/Driver.<br /> <br /> 2016-10-21: Continued to download data for the Moroccan Parliament Written and Oral Questions. Looked over [http://mcnair.bakerinstitute.org/wiki/Christy_Warden_(Twitter_Crawler_Application_1) Christy's Twitter Crawler] to see how I can be helpful. Dr. Egan asked me to think about how to potentially make multiple tools to get cohorts and other sorts of data from accelerator sites. See [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator List] He also asked me to look at the [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] for potential ideas on how to bring this project to fruition.<br /> <br /> 2016-10-20: Continued to download data for the Moroccan Parliament Written and Oral Questions. Updated Wiki page. Started working on Twitter project with Christy. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-18: Finished code for Oral Questions web driver and Written Questions web driver using selenium. Now, the data for the dates of questions can be found using the crawler, and the pdfs of the questions will be downloaded using selenium. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-14: Finished Oral Questions crawler. Finished Written Questions crawler. Waiting for further details on whether that data needs to be tweaked in any way. Updated the Moroccan Web Driver/Web Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-13: Completed download of Moroccan Bills. Working on either a web driver screenshot approach or a webcrawler approach to download the Moroccan oral and written questions data. Began building Web Crawler for Oral and Written Questions site. Edited Moroccan Web Driver/Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-11: Fixed arabic bug, files can now be saved with arabic titles. Monarchy bills downloaded and ready for shipment. House of Representatives Bill mostly downloaded, ratified bills prepared for download. Started learning scrapy library in python for web scraping. Discussed idea of screenshot-ing questions instead of scraping. <br /> <br /> 2016-10-07: Learned unicode and utf8 encoding and decoding in arabic. Still working on transforming an ascii url into printable unicode.<br /> <br /> 2016-10-06: Discussed with Dr. Elbadawy about the desired file names for Moroccan data download. The consensus was that the bill programs are ready to launch once the files can be named properly, and the questions data must be retrieved using a web crawler which I need to learn how to implement. The naming of files is currently drawing errors in going from arabic, to url, to download, to filename. Debugging in process. Also built a demo selenium program for Dr. Egan that drives the McNair blog site on an infinite loop.<br /> <br /> 2016-10-03: Moroccan Web Driver projects completed for driving of the Monarchy proposed bills, the House of Representatives proposed bills, and the Ratified bills sites. Begun process of devising a naming system for the files that does not require scraping. Tinkered with naming through regular expression parsing of the URL. Structure for the oral questions and written questions drivers is set up, but need fixes due to the differences in the sites. Fixed bug on McNair wiki for women's biz team where email was plain text instead of an email link. Took a glimpse at Kuwait Parliament website, and it appears to be very different from the Moroccan setup.<br /> <br /> 2016-09-30: Selenium program selects view pdf option from the website, and goes to the pdf webpage. Program then switches handle to the new page. CTRL S is sent to the page to launch save dialog window. Text cannot be sent to this window. Brainstorm ways around this issue. Explored Chrome Options for saving automatically without a dialog window. Looking into other libraries besides selenium that may help.<br /> <br /> 2016-09-29: Re-enroll in Microsoft Remote Desktop with proper authentication, set up Selenium environment and Komodo IDE on Remote Desktop, wrote program using Selenium that goes to a link and opens up the print dialog box. Developed computational recipe for a different approach to the problem.<br /> <br /> 2016-09-26: Set up Staff wiki page, work log page; registered for Slack, Microsoft Remote Desktop; downloaded Selenium on personal computer, read Selenium docs. Created wiki page for Moroccan Web Driver Project.<br /> <br /> '''Notes'''<br /> <br /> *Ed moved the Morocco Data to E:\McNair\Projects from C:\Users\PeterJ\Documents<br /> * C Drive files moved to E:\McNair\Users\PeterJ<br /> <br /> [[Category:Work Log]]</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Peter_Jalbert_(Work_Log)&diff=21996 2017-11-20T23:00:51Z

<p>Peterjalbert: </p> <hr /> <div>===Fall 2017===<br /> &lt;onlyinclude&gt; <br /> <br /> [[Peter Jalbert]] [[Work Logs]] [[Peter Jalbert (Work Log)|(log page)]] <br /> <br /> 2017-11-17: <br /> <br /> 2017-11-16: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-15: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote a script to extract counts that were greater than 2 from Keyword Matcher. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder]. Finished re-formatting work logs.<br /> <br /> 2017-11-14: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote an HTML to Text parser. See Parser Demo Day Page for file location. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder].<br /> <br /> 2017-11-13: Built [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser].<br /> <br /> 2017-11-09: Running demo version of Demo Day crawler (Accelerator Google Crawler). Fixing work log format. <br /> <br /> 2017-11-07: Created file with 0s and 1s detailing whether crunchbase has the founder information for an accelerator. Details posted as a TODO on [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List] page. Still waiting for feedback on the PostGIS installation from [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder]. Continued working on Accelerator Google Crawler.<br /> <br /> 2017-11-06: Contacted Geography Center for the US Census Bureau, [https://www.census.gov/geo/about/contact.html here], and began email exchange on PostGIS installation problems. Began working on the [http://mcnair.bakerinstitute.org/wiki/Selenium_Documentation Selenium Documentation]. Also began working on an Accelerator Google Crawler that will be used with Yang and ML to find Demo Days for cohort companies.<br /> <br /> 2017-11-01: Attempted to continue downloading, however ran into HTTP Forbidden errors. Listed the errors on the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder Page].<br /> <br /> 2017-10-31: Began downloading blocks of data for individual states for the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder] project. Wrote out the new wiki page for installation, and beginning to write documentation on usage.<br /> <br /> 2017-10-30: With Ed's help, was able to get the national data from Tiger installed onto a database server. The process required much jumping around and changing users, and all the things we learned are outlined in [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation#Editing_Users the database server documentation] under &quot;Editing Users&quot;.<br /> <br /> 2017-10-25: Continued working on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation TigerCoder Installation].<br /> <br /> 2017-10-24: Throw some addresses into a database, use address normalizer and geocoder. May need to install things. Details on the installation process can be found on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation page].<br /> <br /> 2017-10-23: Finished Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-19: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-18: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_DistrictHouston Innovation District Project].<br /> <br /> 2017-10-17: Constructed ArcGIS maps for the agglomeration project. Finished maps of points for every year in the state of California. Finished maps of Route 128. Began working on selenium Yelp crawler to get cafe locations within the 610-loop.<br /> <br /> 2017-10-16: Assisted Harrison on the USITC project. Looked for natural language processing tools to extract complaintants and defendants along with their location from case files. Experimented with pulling based on parts of speech tags, as well as using geotext or geograpy to pull locations from a case segment.<br /> <br /> 2017-10-13: Updated various project wiki pages.<br /> <br /> 2017-10-12: Continued work on Patent Thicket project, awaiting further project specs.<br /> <br /> 2017-10-05: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-04: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-02: Worked on ArcGIS data. See Harrison's Work Log for the details.<br /> <br /> 2017-09-28: Added collaborative editing feature to PyCharm. <br /> <br /> 2017-09-27: Worked on big database file.<br /> <br /> 2017-09-25: New task -- Create text file with company, description, and company type.<br /> #[http://mcnair.bakerinstitute.org/wiki/VC_Database_Rebuild VC Database Rebuild]<br /> #psql vcdb2<br /> #table name, sdccompanybasecore2<br /> #Combine with Crunchbasebulk<br /> <br /> #TODO: Write wiki on linkedin crawler, write wiki on creating accounts.<br /> <br /> 2017-09-21: Wrote wiki on Linkedin crawler, met with Laura about patents project. <br /> <br /> 2017-09-20: Finished running linkedin crawler. Transferred data to RDP. Will write wikis next.<br /> <br /> 2017-09-19: Began running linkedin crawler. Helped Yang create RDP account, get permissions, and get wiki setup.<br /> <br /> 2017-09-18: Finished implementation of Experience Crawler, continued working on Education Crawler for LinkedIn.<br /> <br /> 2017-09-14: Continued implementing LinkedIn Crawler for profiles.<br /> <br /> 2017-09-13: Implemented LinkedIn Crawler for main portion of profiles. Began working on crawling Experience section of profiles.<br /> <br /> 2017-09-12: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. Added to the wiki on this topic.<br /> <br /> 2017-09-11: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. <br /> <br /> 2017-09-06: Combined founders data retrieved with the Crunchbase API with the crunchbasebulk data to get linkedin urls for different accelerator founders. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-09-05: Post Harvey. Finished retrieving names from the Crunchbase API on founders. Next step is to query crunchbase bulk database to get linkedin urls. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-08-24: Began using the Crunchbase API to retrieve founder information for accelerators. Halfway through compiling a dictionary that translates accelerator names into proper Crunchbase API URLs.<br /> <br /> 2017-08-23: Decided with Ed to abandon LinkedIn crawling to retrieve accelerator founder data, and instead use crunchbase. Spent the day navigating the crunchbasebulk database, and seeing what useful information was contained in it.<br /> <br /> 2017-08-22: Discovered that LinkedIn Profiles cannot be viewed through LinkedIn if the target is 3rd degree or further. However, if entering LinkedIn through a Google search, the profile can still be viewed if the user has previously logged into LinkedIn. Devising a workaround crawler that utilizes Google search. Continued blog post [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) here] under Section 4.<br /> <br /> 2017-08-21: Began work on extracting founders for accelerators through LinkedIn Crawler. Discovered that Python3 is not installed on RDP, so the virtual environment for the project cannot be fired up. Continued working on Ubuntu machine.<br /> &lt;/onlyinclude&gt; <br /> <br /> ===Spring 2017===<br /> <br /> 2017-05-01: Continued work on HTML Parser. Uploaded all semester projects to git server.<br /> <br /> 2017-04-20: Finished the HTML Parser for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Ran HTML parser on accelerator founders. Data is stored in projects/accelerators/LinkedIn Founder Data.<br /> <br /> 2017-04-19: Made updates to the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler] Wikipage. Ran LinkedIn Crawler on accelerator data. Working on an html parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-18: Ran LinkedIn Crawler on matches between Crunchbase Snapshot and the accelerator data.<br /> <br /> 2017-04-17: Worked on ways to get correct search results from the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Worked on an HTML Parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-13: Worked on debugging the logout procedure for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Began formulation of process to search for founders of startups using a combination of the LinkedIn Crawler with the data resources from the [http://mcnair.bakerinstitute.org/wiki/Crunchbase_2013_Snapshot CrunchBase Snapshot].<br /> <br /> 2017-04-12: Work on bugs with the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler].<br /> <br /> 2017-04-11: Completed functional [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) crawler of LinkedIn Recruiter Pro]. Basic search functions work and download profile information for a given person. <br /> <br /> 2017-04-10: Began writing functioning crawler of LinkedIn. <br /> <br /> 2017-04-06: Continued working on debugging and documenting the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Wrote a test program that logs in, searches for a query, navigates through search pages, and logs out. Recruiter program can now login and search.<br /> <br /> 2017-04-05: Began work on the LinkedIn Crawler. Researched on launching Python Virtual Environment.<br /> <br /> 2017-04-03: Finished debugging points for the Enclosing Circle Algorithm. Added Command Line functionality to the Industry Classifier.<br /> <br /> 2017-03-29: Worked on debugging points for the Enclosing Circle Algorithm.<br /> <br /> 2017-03-28: Finished running the Enclosing Circle Algorithm. Worked on removing incorrect points from the data set(see above). <br /> <br /> 2017-03-27: Worked on debugging the Enclosing Circle Algorithm. Implemented a way to remove interior circles, and determined that translation to latitude and longitude coordinates resulted in slightly off center circles.<br /> <br /> 2017-03-23: Finished debugging the brute force algorithm for [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm]. Implemented a method to plot the points and circles on a graph. Analyzed runtime of the brute force algorithm.<br /> <br /> 2017-03-21: Coded a brute force algorithm for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-20: Worked on debugging the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-09: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Finished script to draw Enclosing Circles on a Google Map.<br /> <br /> 2017-03-08: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Created script to draw outcome of the Enclosing Circle Algorithm on Google Maps.<br /> <br /> 2017-03-07: Redetermined the top 50 cities which Enclosing Circle should be run on. Data on the [http://mcnair.bakerinstitute.org/wiki/Top_Cities_for_VC_Backed_Companies Top 50 Cities for VC Backed Companies can be found here.] Ran [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities.<br /> <br /> 2017-03-06: Ran script to determine the top 50 cities which Enclosing Circle should be run on. Fixed the VC Circles script to take in a new data format.<br /> <br /> 2017-03-02: Cleaned up data for the VC Circles Project. Created histogram of data in Excel. See [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Began work on the [http://mcnair.bakerinstitute.org/wiki/LinkedInCrawlerPython LinkedIn Crawler].<br /> <br /> 2017-03-01: Created statistics for the VC Circles Project.<br /> <br /> 2017-02-28: Finished downloading geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Found bug in Enclosing Circle Algorithm.<br /> <br /> 2017-02-27: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-23: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Installed C++ Compiler for Python. Ran tests on difference between Python and C wrapped Python.<br /> <br /> 2017-02-22: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Helped out with [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-21: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Researched into C++ Compilers for Python so that the Enclosing Circle Algorithm could be wrapped in C. Found a recommended one [https://www.microsoft.com/en-us/download/details.aspx?id=44266 here].<br /> <br /> 2017-02-20: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-16: Reworked [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] to create a file of geocoded data. Began work on wrapping the algorithm in C to improve speed.<br /> <br /> 2017-02-15: Finished [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] applied to the VC study. Enclosing Circle algorithm still needs adjustment, but the program runs with the temporary fixes.<br /> <br /> 2017-02-14: Worked on the application of the Enclosing Circle algorithm to the VC study. Working on bug fixes in the Enclosing Circle algorithm. Created wiki page for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-02-13: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-08: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-07: Fixed bugs in parse_cohort_data.py, the script for parsing the cohort data from the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Added descriptive statistics to cohort data excel file.<br /> <br /> 2017-02-02: Out sick, independent research and work from RDP. Brief research into the [http://jorgeg.scripts.mit.edu/homepage/wp-content/uploads/2016/03/Guzman-Stern-State-of-American-Entrepreneurship-FINAL.pdf Stern-Guzman algorithm]. Research into [http://mcnair.bakerinstitute.org/wiki/interactive_maps Interactive Maps]. No helpful additions to map embedding problem.<br /> <br /> 2017-02-01: Notes from Session with Ed: Project on US university patenting and entrepreneurship programs (writing code to identify universities in assignees), search Wikipedia (XML then bulk download), student pop, faculty pop, etc.<br /> Circle project for VC data will end up being a joint project to join accelerator data. <br /> Pull descriptions for VC. Founders of accelerators in linkedin. LinkedIn cannot be caught(pretend to not be a bot). Can eventually get academic backgrounds through linkedin. <br /> Pull business registration data, Stern/Guzman Algorithm. <br /> GIS ontop of geocoded data.<br /> Maps that works on wiki or blog (CartoDB), Maps API and R.<br /> NLP Projects, Description Classifier.<br /> <br /> 2017-01-31: Built WayBack Machine Crawler. Updated documentation for coordinates script. Updated profile page to include locations of code.<br /> <br /> <br /> 2017-01-30: Optimized enclosing circle algorithm through memoization. Developed script to read addresses from accelerator data and return latitude and longitude coordinates.<br /> <br /> 2017-01-26: Continued working on Google sitesearch project. Discovered crunchbase, changed project priority. Priority 1, split accelerator data up by flag, priority 2, use crunchbase to get web urls for cohorts, priority 3, make internet archive wayback machine driver. Located [http://mcnair.bakerinstitute.org/wiki/Whois_Parser Whois Parser].<br /> <br /> <br /> 2017-01-25: Finished parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Some data files still need proofreading as they are not in an acceptable format. Began working on Google sitesearch project.<br /> <br /> 2017-01-24: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Cohort data file created, debugging is almost complete. Will begin work on the google accelerator search soon.<br /> <br /> 2017-01-23: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Preliminary code is written, working on debugging.<br /> <br /> 2017-01-19: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project]. Created parser for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project], completed creation of final data set(yay!). Began working on cohort parser.<br /> <br /> 2017-01-18: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-13: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-12: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-11: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-10: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> <br /> ===Fall 2016===<br /> <br /> 2016-12-08: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-07: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-06: Learned how to use git. Committed software projects from the semester to the McNair git repository. Projects can be found at; [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Executive Order Crawler], [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Foreign Government Web Crawlers], [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) F6S Crawler and Parser].<br /> <br /> 2016-12-02: Built and ran web crawler for Center for Middle East Studies on Kuwait. Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-01: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Built tool for the [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report E&amp;I Governance Report Project] with Christy. Adds a column of data that shows whether or not the bill has been passed.<br /> <br /> 2016-11-29: Began pulling data from the accelerators listed [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here]. Made text files for about 18 accelerators.<br /> <br /> 2016-11-22: Transferred downloaded Morocco Written Bills to provided SeaGate Drive. Made a &quot;gentle&quot; F6S crawler to retrieve HTMLs of possible accelerator pages documented [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here].<br /> <br /> 2016-11-18: Converted Executive Order PDFs to text files using adobe acrobat DC. See [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Wikipage] for details.<br /> <br /> 2016-11-17: Wrote a crawler to retrieve information about executive orders, and their corresponding pdfs. They can be found [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report here.] Next step is to run code to convert the pdfs to text files, then use the parser fixed by Christy. <br /> <br /> 2016-11-15: Finished download of Moroccan Written Question pdfs. Wrote a parser with Christy to be used for parsing bills from Congress and eventually executive orders. Found bug in the system Python that was worked out and rebooted.<br /> <br /> 2016-11-11: Continued to download Moroccan data in the background. Attempted to find bug fixes for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] crawler.<br /> <br /> 2016-11-10: Continued to download Moroccan data and Kuwait data in the background. Began work on [http://mcnair.bakerinstitute.org/wiki/Google_Scholar_Crawler Google Scholar Crawler]. Wrote a crawler for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] to get the HTML files of hundreds of accelerators. The crawler ended up failing; it appears to have been due to HTTPS. <br /> <br /> 2016-11-08: Continued to download Moroccan data in the background. Finished writing code for the embedded files on the Kuwait Site. Spent time debugging the frame errors due to the dynamically generated content. Never found an answer to the bug, and instead found a workaround that sacrificed run time for the ability to work. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-04: Continued to download Moroccan data in the background. Finished writing initial Kuwait Web Crawler/Driver for the Middle East Studies Department. Middle East Studies Department asked for additional embedded files in the Kuwait website. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-03: Continued to download Moroccan data in the background. Dr. Egan fixed systems requirements to run the GovTrack Web Crawler. Made significant progress on the Kuwait Web Crawler/Driver for the Middle East Studies Department. <br /> <br /> 2016-11-01: Continued to download Moroccan data in the background. Went over code for GovTracker Web Crawler, continued learning Perl. [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] Began Kuwait Web Crawler/Driver.<br /> <br /> 2016-10-21: Continued to download data for the Moroccan Parliament Written and Oral Questions. Looked over [http://mcnair.bakerinstitute.org/wiki/Christy_Warden_(Twitter_Crawler_Application_1) Christy's Twitter Crawler] to see how I can be helpful. Dr. Egan asked me to think about how to potentially make multiple tools to get cohorts and other sorts of data from accelerator sites. See [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator List] He also asked me to look at the [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] for potential ideas on how to bring this project to fruition.<br /> <br /> 2016-10-20: Continued to download data for the Moroccan Parliament Written and Oral Questions. Updated Wiki page. Started working on Twitter project with Christy. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-18: Finished code for Oral Questions web driver and Written Questions web driver using selenium. Now, the data for the dates of questions can be found using the crawler, and the pdfs of the questions will be downloaded using selenium. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-14: Finished Oral Questions crawler. Finished Written Questions crawler. Waiting for further details on whether that data needs to be tweaked in any way. Updated the Moroccan Web Driver/Web Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-13: Completed download of Moroccan Bills. Working on either a web driver screenshot approach or a webcrawler approach to download the Moroccan oral and written questions data. Began building Web Crawler for Oral and Written Questions site. Edited Moroccan Web Driver/Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-11: Fixed arabic bug, files can now be saved with arabic titles. Monarchy bills downloaded and ready for shipment. House of Representatives Bill mostly downloaded, ratified bills prepared for download. Started learning scrapy library in python for web scraping. Discussed idea of screenshot-ing questions instead of scraping. <br /> <br /> 2016-10-07: Learned unicode and utf8 encoding and decoding in arabic. Still working on transforming an ascii url into printable unicode.<br /> <br /> 2016-10-06: Discussed with Dr. Elbadawy about the desired file names for Moroccan data download. The consensus was that the bill programs are ready to launch once the files can be named properly, and the questions data must be retrieved using a web crawler which I need to learn how to implement. The naming of files is currently drawing errors in going from arabic, to url, to download, to filename. Debugging in process. Also built a demo selenium program for Dr. Egan that drives the McNair blog site on an infinite loop.<br /> <br /> 2016-10-03: Moroccan Web Driver projects completed for driving of the Monarchy proposed bills, the House of Representatives proposed bills, and the Ratified bills sites. Begun process of devising a naming system for the files that does not require scraping. Tinkered with naming through regular expression parsing of the URL. Structure for the oral questions and written questions drivers is set up, but need fixes due to the differences in the sites. Fixed bug on McNair wiki for women's biz team where email was plain text instead of an email link. Took a glimpse at Kuwait Parliament website, and it appears to be very different from the Moroccan setup.<br /> <br /> 2016-09-30: Selenium program selects view pdf option from the website, and goes to the pdf webpage. Program then switches handle to the new page. CTRL S is sent to the page to launch save dialog window. Text cannot be sent to this window. Brainstorm ways around this issue. Explored Chrome Options for saving automatically without a dialog window. Looking into other libraries besides selenium that may help.<br /> <br /> 2016-09-29: Re-enroll in Microsoft Remote Desktop with proper authentication, set up Selenium environment and Komodo IDE on Remote Desktop, wrote program using Selenium that goes to a link and opens up the print dialog box. Developed computational recipe for a different approach to the problem.<br /> <br /> 2016-09-26: Set up Staff wiki page, work log page; registered for Slack, Microsoft Remote Desktop; downloaded Selenium on personal computer, read Selenium docs. Created wiki page for Moroccan Web Driver Project.<br /> <br /> '''Notes'''<br /> <br /> *Ed moved the Morocco Data to E:\McNair\Projects from C:\Users\PeterJ\Documents<br /> * C Drive files moved to E:\McNair\Users\PeterJ<br /> <br /> [[Category:Work Log]]</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Tiger_Geocoder&diff=21990 2017-11-20T21:52:51Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Tiger Geocoder<br /> |Has owner=Peter Jalbert,<br /> |Has start date=Fall 2017<br /> |Has keywords=Tiger, Geocoder, Database<br /> |Has project status=Active<br /> |Has Image=Tiger.jpg<br /> }}<br /> This page serves as documentation for using the Tiger Geocoder on Postgres SQL, as part of the PostGIS extension. The following wiki pages may also be of use to you:<br /> <br /> [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation]<br /> [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation Database Server Documentation]<br /> <br /> The official documentation for using and installing the Tiger Geocoder can be found in the following.<br /> <br /> [https://postgis.net/docs/Extras.html#Tiger_Geocoder General Instructions]<br /> [https://postgis.net/docs/postgis_installation.html#install_tiger_geocoder_extension Installation Instructions]<br /> [http://postgis.net/docs/Geocode.html Geocoder Documentation]<br /> <br /> ==Location==<br /> The data is currently loaded into a psql database called geocoder. The tables contain the geocoding information, and there is a test table called &quot;coffeeshops&quot; that contains addresses of Houston coffeeshops according to yelp. To access the database, first login to the McNair DB Sever. Then, <br /> <br /> psql geocoder<br /> <br /> <br /> <br /> ==Installation==<br /> <br /> ===Install and Nation Data===<br /> I began by adding the extension listed above. First, enter into Postgres by using the psql command. Then:<br /> --Add Extensions to database<br /> CREATE EXTENSION postgis;<br /> CREATE EXTENSION fuzzystrmatch;<br /> CREATE EXTENSION postgis_tiger_geocoder;<br /> CREATE EXTENSION address_standardizer;<br /> <br /> You can test that the installation worked by running the following query: <br /> SELECT na.address, na.streetname,na.streettypeabbrev, na.zip<br /> FROM normalize_address('1 Devonshire Place, Boston, MA 02109') AS na;<br /> <br /> This should return the following:<br /> address | streetname | streettypeabbrev | zip<br /> ---------+------------+------------------+-------<br /> 1 | Devonshire | Pl | 02109<br /> <br /> Next, a new profile needs to be created by using the following command.<br /> INSERT INTO tiger.loader_platform(os, declare_sect, pgbin, wget, unzip_command, psql, path_sep, <br /> loader, environ_set_command, county_process_command)<br /> SELECT 'test', declare_sect, pgbin, wget, unzip_command, psql, path_sep,<br /> loader, environ_set_command, county_process_command<br /> FROM tiger.loader_platform<br /> WHERE os = 'sh';<br /> <br /> The installation instructions also provide the following note:<br /> <br /> As of PostGIS 2.4.1 the Zip code-5 digit tabulation area zcta5 load step was revised to load current zcta5 data and is part of the Loader_Generate_Nation_Script when enabled. It is turned off by default because it takes quite a bit of time to load (20 to 60 minutes), takes up quite a bit of disk space, and is not used that often.<br /> <br /> If you would like this feature, you can enable it by using the following command. This should be done before loading the script.<br /> <br /> UPDATE tiger.loader_lookuptables SET load = true WHERE table_name = 'zcta510';<br /> <br /> The paths in declare_sect need to be edited so they match our server locations. One option is to edit the declare_sect column in the tiger.loader_platform table. If so, the declare_sect looks like the following:<br /> export PGHOST=localhost +<br /> export PGUSER=postgres +<br /> export PGPASSWORD=yourpasswordhere +<br /> export PGDATABASE=geocoder +<br /> PSQL=${PGBIN}/psql +<br /> SHP2PGSQL=shp2pgsql +<br /> cd ${staging_fold} +<br /> <br /> TMPDIR=&quot;${staging_fold}/temp/&quot; +<br /> UNZIPTOOL=unzip +<br /> WGETTOOL=&quot;/usr/bin/wget&quot; +<br /> export PGBIN=/usr/lib/postgresql/9.6/bin +<br /> export PGPORT=5432 +<br /> export PGHOST=localhost +<br /> export PGUSER=postgres +<br /> export PGPASSWORD=yourpasswordhere +<br /> export PGDATABASE=geocoder +<br /> PSQL=${PGBIN}/psql +<br /> SHP2PGSQL=shp2pgsql +<br /> cd ${staging_fold}<br /> <br /> Another option is to edit the sh file before running the script. We will do this option until further notice. Simply use your favorite command line editor to change the fields to their correct values. The downloaded script is located in the following directory:<br /> /gisdata<br /> <br /> There needs to be a directory called &quot;temp&quot; in the gisdata directory. To make the script, use the following from the command line:<br /> psql -c &quot;SELECT Loader_Generate_Nation_Script('test')&quot; -d databasename -tA &gt; /gisdata/nation_script_load.sh<br /> <br /> This will create a script in the /gisdata directory. Change to that directory. If you did not edit the paths in the declare_sect table in psql, then you will need to edit this file to contain the correct paths. <br /> <br /> Change directories:<br /> cd /gisdata<br /> <br /> Edit the script using your favorite command line text editor. Specifically, edit the following fields.<br /> PGUSER=postgres<br /> PGPASSWORD=(Ask Anne for this password)!<br /> Everything else remains the same.<br /> <br /> Run the script by using: <br /> sh nation_script_load.sh<br /> <br /> Now, there is a barebones table in the database that will hold the information for the nation. Next is to download data on each state.<br /> <br /> ===State Data===<br /> <br /> The state scripts are generated in much the same way that the nation script was generated. Use the following command, substituting MA for your desired state abbreviation, and substituting a unique filename at the end.<br /> psql -c &quot;SELECT Loader_Generate_Script(ARRAY['MA'], 'test')&quot; -d geocoder -tA &gt; /gisdata/ma_load.sh<br /> <br /> CURRENT PROGRESS:<br /> The following states have been downloaded into the geocoder database.<br /> AL, AK, AZ, AR, CA, CO, CT, DE, FL, MA<br /> <br /> ===Current Errors===<br /> The state scripts stopped working on 11/1/2017 while they were working on 10/31/2017. Now, when a retrieval script is run, it draws the error <br /> HTTP request sent, awaiting response... 403 Forbidden<br /> Possible thoughts:<br /> <br /> Maybe our IP has been blacklisted for downloading data from a government website quickly? If so, PostGIS should really choose a different installation method. The current one is dumb.<br /> <br /> Maybe the nation downloader script never worked properly. Not sure how to check if it is correct or not; seems right.<br /> <br /> [https://trac.osgeo.org/postgis/ticket/3699 This] is the only online forum I could find with others who have faced a similar issue.<br /> <br /> NOTE: Contacted the US Census Bureau and never got a response. However, the scripts magically started working again about a week later. Second however, the scripts stopped working again the next day.<br /> <br /> ==Geocode Function==<br /> <br /> The official arguments for the function are the following:<br /> setof record geocode(varchar address, integer max_results=10, geometry restrict_region=NULL, norm_addy OUT addy, geometry OUT geomout, integer OUT rating);<br /> The arguments of interest are address, where you simply submit a string, and max_results, which restricts the number of geocoding attempts per address. The geocoder makes multiple guesses to the location of an address, and returns the best guesses in order. If you want multiple guesses to a specific address, then specify max_results to be more than 1.<br /> <br /> ===Single Address===<br /> <br /> An example query for a single address is:<br /> SELECT g.rating, ST_X(g.geomout) As lon, ST_Y(g.geomout) As lat,<br /> (addy).address As stno, (addy).streetname As street,<br /> (addy).streettypeabbrev As styp, (addy).location As city, (addy).stateabbrev As st,(addy).zip<br /> FROM geocode('75 State Street, Boston MA 02109', 1) As g;<br /> <br /> rating -- This is an integer that determines the confidence in the resulting geocode. The closer to 0, the more confident the guess.<br /> <br /> ST_X(g.geomout) -- This retrieves the longitude coordinate of the point.<br /> <br /> ST_Y(g.geomout) -- This retrieves the latitude corodinate of the point.<br /> <br /> addy -- In general, addy is a normalized address resulting from the input address.<br /> <br /> (addy).address -- The number of the address (Ex: &quot;75&quot; Blabla rd.)<br /> <br /> (addy).streetname -- The name of the street (Ex: 75 &quot;Blabla&quot; rd.)<br /> <br /> (addy).streettypeabbrev -- The abbreviation of the street (Ex: 75 blabla &quot;rd&quot;)<br /> <br /> (addy).location -- The city location of the address.<br /> <br /> (addy).stateabbrev -- The state abbreviation.<br /> <br /> (addy).zip -- The zipcode of the address.<br /> <br /> The output of the query above would be:<br /> rating | lon | lat | stno | street | styp | city | st | zip<br /> --------+-------------------+----------------+------+--------+------+--------+----+-------<br /> 0 | -71.0557505845646 | 42.35897920691 | 75 | State | St | Boston | MA | 02109<br /> <br /> ==Current Errors==<br /> Currently, we are getting a 403 Forbidden Error when trying to download state data. We are in contact with the US Census Bureau. Their contact information can be found [https://www.census.gov/geo/about/contact.html here].<br /> <br /> The email exchanges are recorded below.<br /> <br /> --------------------------------------<br /> --------------------------------------<br /> <br /> I am a student researcher at the McNair Center for Entrepreneurship and Innovation at Rice University, and I am in the process of installing a Postgres Extension that relies on the TIGER data. <br /> <br /> When I began installing the extension, things were working fine. Now however, I am getting a 403 Forbidden error when the script tries to download the TIGER files. Do you have any idea why this might be happening? <br /> <br /> The extension I'm trying to install is below:<br /> <br /> http://postgis.net/docs/postgis_installation.html#install_tiger_geocoder_extension<br /> <br /> When I run any of the scripts that require data from TIGER, I am receiving the following error:<br /> <br /> <br /> --2017-11-06 14:15:29-- http://www2.census.gov/geo/tiger/TIGER2016/FEATNAMES/tl_2016_06065_featnames.zip<br /> Resolving www2.census.gov (www2.census.gov)... 104.84.241.90, 2600:1404:a:382::208c, 2600:1404:a:39c::208c<br /> Connecting to www2.census.gov (www2.census.gov)|104.84.241.90|:80... connected.<br /> HTTP request sent, awaiting response... 403 Forbidden<br /> 2017-11-06 14:15:29 ERROR 403: Forbidden.<br /> <br /> <br /> ----------------------------------------<br /> <br /> Thank you for reaching out to us. The postgis docs is not from the Geography Division, so I cannot comment on that. A couple things that come to mind.<br /> <br /> We just released our new 2017 Shapefiles, so it's possible the scripts may be written for a previous version of our Shapefiles.<br /> You may need to clean out your cookies, restart your browser, and then attempt to reinstall.<br /> Were you able to download our Shapefiles successfully?<br /> <br /> ----------------------------------------<br /> <br /> The Shapefiles are part of the files being blocked by the 403 Forbidden error. <br /> <br /> The script is using a wget protocol to bulk download the data, so there were no cookies involved. Also, the download had been working in previous days; it was only recently that the same scripts stopped working. I am worried that our IP address somehow ended up on a blacklist for the TIGER data. Is there a blacklist for addresses that access the TIGER data?<br /> <br /> Our IP address is 128.42.44.181. <br /> <br /> ---------------------------------------<br /> <br /> I have forwarded your question on to our IT folks. Since I work on the subject matter side, I am unable to answer your questions. Once I hear back from them, I will forward their response to you. Hopefully they will provide to you what you need in order to download our Shapefiles. My apologies for your inconvenience.<br /> <br /> <br /> ----------------------------------------</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Peter_Jalbert_(Work_Log)&diff=21957 2017-11-16T21:36:37Z

<p>Peterjalbert: </p> <hr /> <div>===Fall 2017===<br /> &lt;onlyinclude&gt; <br /> <br /> [[Peter Jalbert]] [[Work Logs]] [[Peter Jalbert (Work Log)|(log page)]] <br /> <br /> 2017-11-16: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Fixed KeyTerms.py and trying to run it again. Forbidden Error continues with the TIGER Geocoder. Began Image download for Image Classification on cohort pages. Clarifying specs for Morocco Parliament crawler.<br /> <br /> 2017-11-15: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote a script to extract counts that were greater than 2 from Keyword Matcher. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder]. Finished re-formatting work logs.<br /> <br /> 2017-11-14: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote an HTML to Text parser. See Parser Demo Day Page for file location. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder].<br /> <br /> 2017-11-13: Built [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser].<br /> <br /> 2017-11-09: Running demo version of Demo Day crawler (Accelerator Google Crawler). Fixing work log format. <br /> <br /> 2017-11-07: Created file with 0s and 1s detailing whether crunchbase has the founder information for an accelerator. Details posted as a TODO on [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List] page. Still waiting for feedback on the PostGIS installation from [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder]. Continued working on Accelerator Google Crawler.<br /> <br /> 2017-11-06: Contacted Geography Center for the US Census Bureau, [https://www.census.gov/geo/about/contact.html here], and began email exchange on PostGIS installation problems. Began working on the [http://mcnair.bakerinstitute.org/wiki/Selenium_Documentation Selenium Documentation]. Also began working on an Accelerator Google Crawler that will be used with Yang and ML to find Demo Days for cohort companies.<br /> <br /> 2017-11-01: Attempted to continue downloading, however ran into HTTP Forbidden errors. Listed the errors on the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder Page].<br /> <br /> 2017-10-31: Began downloading blocks of data for individual states for the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder] project. Wrote out the new wiki page for installation, and beginning to write documentation on usage.<br /> <br /> 2017-10-30: With Ed's help, was able to get the national data from Tiger installed onto a database server. The process required much jumping around and changing users, and all the things we learned are outlined in [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation#Editing_Users the database server documentation] under &quot;Editing Users&quot;.<br /> <br /> 2017-10-25: Continued working on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation TigerCoder Installation].<br /> <br /> 2017-10-24: Throw some addresses into a database, use address normalizer and geocoder. May need to install things. Details on the installation process can be found on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation page].<br /> <br /> 2017-10-23: Finished Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-19: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-18: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_DistrictHouston Innovation District Project].<br /> <br /> 2017-10-17: Constructed ArcGIS maps for the agglomeration project. Finished maps of points for every year in the state of California. Finished maps of Route 128. Began working on selenium Yelp crawler to get cafe locations within the 610-loop.<br /> <br /> 2017-10-16: Assisted Harrison on the USITC project. Looked for natural language processing tools to extract complaintants and defendants along with their location from case files. Experimented with pulling based on parts of speech tags, as well as using geotext or geograpy to pull locations from a case segment.<br /> <br /> 2017-10-13: Updated various project wiki pages.<br /> <br /> 2017-10-12: Continued work on Patent Thicket project, awaiting further project specs.<br /> <br /> 2017-10-05: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-04: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-02: Worked on ArcGIS data. See Harrison's Work Log for the details.<br /> <br /> 2017-09-28: Added collaborative editing feature to PyCharm. <br /> <br /> 2017-09-27: Worked on big database file.<br /> <br /> 2017-09-25: New task -- Create text file with company, description, and company type.<br /> #[http://mcnair.bakerinstitute.org/wiki/VC_Database_Rebuild VC Database Rebuild]<br /> #psql vcdb2<br /> #table name, sdccompanybasecore2<br /> #Combine with Crunchbasebulk<br /> <br /> #TODO: Write wiki on linkedin crawler, write wiki on creating accounts.<br /> <br /> 2017-09-21: Wrote wiki on Linkedin crawler, met with Laura about patents project. <br /> <br /> 2017-09-20: Finished running linkedin crawler. Transferred data to RDP. Will write wikis next.<br /> <br /> 2017-09-19: Began running linkedin crawler. Helped Yang create RDP account, get permissions, and get wiki setup.<br /> <br /> 2017-09-18: Finished implementation of Experience Crawler, continued working on Education Crawler for LinkedIn.<br /> <br /> 2017-09-14: Continued implementing LinkedIn Crawler for profiles.<br /> <br /> 2017-09-13: Implemented LinkedIn Crawler for main portion of profiles. Began working on crawling Experience section of profiles.<br /> <br /> 2017-09-12: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. Added to the wiki on this topic.<br /> <br /> 2017-09-11: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. <br /> <br /> 2017-09-06: Combined founders data retrieved with the Crunchbase API with the crunchbasebulk data to get linkedin urls for different accelerator founders. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-09-05: Post Harvey. Finished retrieving names from the Crunchbase API on founders. Next step is to query crunchbase bulk database to get linkedin urls. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-08-24: Began using the Crunchbase API to retrieve founder information for accelerators. Halfway through compiling a dictionary that translates accelerator names into proper Crunchbase API URLs.<br /> <br /> 2017-08-23: Decided with Ed to abandon LinkedIn crawling to retrieve accelerator founder data, and instead use crunchbase. Spent the day navigating the crunchbasebulk database, and seeing what useful information was contained in it.<br /> <br /> 2017-08-22: Discovered that LinkedIn Profiles cannot be viewed through LinkedIn if the target is 3rd degree or further. However, if entering LinkedIn through a Google search, the profile can still be viewed if the user has previously logged into LinkedIn. Devising a workaround crawler that utilizes Google search. Continued blog post [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) here] under Section 4.<br /> <br /> 2017-08-21: Began work on extracting founders for accelerators through LinkedIn Crawler. Discovered that Python3 is not installed on RDP, so the virtual environment for the project cannot be fired up. Continued working on Ubuntu machine.<br /> &lt;/onlyinclude&gt; <br /> <br /> ===Spring 2017===<br /> <br /> 2017-05-01: Continued work on HTML Parser. Uploaded all semester projects to git server.<br /> <br /> 2017-04-20: Finished the HTML Parser for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Ran HTML parser on accelerator founders. Data is stored in projects/accelerators/LinkedIn Founder Data.<br /> <br /> 2017-04-19: Made updates to the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler] Wikipage. Ran LinkedIn Crawler on accelerator data. Working on an html parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-18: Ran LinkedIn Crawler on matches between Crunchbase Snapshot and the accelerator data.<br /> <br /> 2017-04-17: Worked on ways to get correct search results from the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Worked on an HTML Parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-13: Worked on debugging the logout procedure for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Began formulation of process to search for founders of startups using a combination of the LinkedIn Crawler with the data resources from the [http://mcnair.bakerinstitute.org/wiki/Crunchbase_2013_Snapshot CrunchBase Snapshot].<br /> <br /> 2017-04-12: Work on bugs with the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler].<br /> <br /> 2017-04-11: Completed functional [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) crawler of LinkedIn Recruiter Pro]. Basic search functions work and download profile information for a given person. <br /> <br /> 2017-04-10: Began writing functioning crawler of LinkedIn. <br /> <br /> 2017-04-06: Continued working on debugging and documenting the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Wrote a test program that logs in, searches for a query, navigates through search pages, and logs out. Recruiter program can now login and search.<br /> <br /> 2017-04-05: Began work on the LinkedIn Crawler. Researched on launching Python Virtual Environment.<br /> <br /> 2017-04-03: Finished debugging points for the Enclosing Circle Algorithm. Added Command Line functionality to the Industry Classifier.<br /> <br /> 2017-03-29: Worked on debugging points for the Enclosing Circle Algorithm.<br /> <br /> 2017-03-28: Finished running the Enclosing Circle Algorithm. Worked on removing incorrect points from the data set(see above). <br /> <br /> 2017-03-27: Worked on debugging the Enclosing Circle Algorithm. Implemented a way to remove interior circles, and determined that translation to latitude and longitude coordinates resulted in slightly off center circles.<br /> <br /> 2017-03-23: Finished debugging the brute force algorithm for [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm]. Implemented a method to plot the points and circles on a graph. Analyzed runtime of the brute force algorithm.<br /> <br /> 2017-03-21: Coded a brute force algorithm for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-20: Worked on debugging the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-09: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Finished script to draw Enclosing Circles on a Google Map.<br /> <br /> 2017-03-08: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Created script to draw outcome of the Enclosing Circle Algorithm on Google Maps.<br /> <br /> 2017-03-07: Redetermined the top 50 cities which Enclosing Circle should be run on. Data on the [http://mcnair.bakerinstitute.org/wiki/Top_Cities_for_VC_Backed_Companies Top 50 Cities for VC Backed Companies can be found here.] Ran [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities.<br /> <br /> 2017-03-06: Ran script to determine the top 50 cities which Enclosing Circle should be run on. Fixed the VC Circles script to take in a new data format.<br /> <br /> 2017-03-02: Cleaned up data for the VC Circles Project. Created histogram of data in Excel. See [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Began work on the [http://mcnair.bakerinstitute.org/wiki/LinkedInCrawlerPython LinkedIn Crawler].<br /> <br /> 2017-03-01: Created statistics for the VC Circles Project.<br /> <br /> 2017-02-28: Finished downloading geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Found bug in Enclosing Circle Algorithm.<br /> <br /> 2017-02-27: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-23: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Installed C++ Compiler for Python. Ran tests on difference between Python and C wrapped Python.<br /> <br /> 2017-02-22: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Helped out with [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-21: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Researched into C++ Compilers for Python so that the Enclosing Circle Algorithm could be wrapped in C. Found a recommended one [https://www.microsoft.com/en-us/download/details.aspx?id=44266 here].<br /> <br /> 2017-02-20: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-16: Reworked [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] to create a file of geocoded data. Began work on wrapping the algorithm in C to improve speed.<br /> <br /> 2017-02-15: Finished [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] applied to the VC study. Enclosing Circle algorithm still needs adjustment, but the program runs with the temporary fixes.<br /> <br /> 2017-02-14: Worked on the application of the Enclosing Circle algorithm to the VC study. Working on bug fixes in the Enclosing Circle algorithm. Created wiki page for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-02-13: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-08: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-07: Fixed bugs in parse_cohort_data.py, the script for parsing the cohort data from the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Added descriptive statistics to cohort data excel file.<br /> <br /> 2017-02-02: Out sick, independent research and work from RDP. Brief research into the [http://jorgeg.scripts.mit.edu/homepage/wp-content/uploads/2016/03/Guzman-Stern-State-of-American-Entrepreneurship-FINAL.pdf Stern-Guzman algorithm]. Research into [http://mcnair.bakerinstitute.org/wiki/interactive_maps Interactive Maps]. No helpful additions to map embedding problem.<br /> <br /> 2017-02-01: Notes from Session with Ed: Project on US university patenting and entrepreneurship programs (writing code to identify universities in assignees), search Wikipedia (XML then bulk download), student pop, faculty pop, etc.<br /> Circle project for VC data will end up being a joint project to join accelerator data. <br /> Pull descriptions for VC. Founders of accelerators in linkedin. LinkedIn cannot be caught(pretend to not be a bot). Can eventually get academic backgrounds through linkedin. <br /> Pull business registration data, Stern/Guzman Algorithm. <br /> GIS ontop of geocoded data.<br /> Maps that works on wiki or blog (CartoDB), Maps API and R.<br /> NLP Projects, Description Classifier.<br /> <br /> 2017-01-31: Built WayBack Machine Crawler. Updated documentation for coordinates script. Updated profile page to include locations of code.<br /> <br /> <br /> 2017-01-30: Optimized enclosing circle algorithm through memoization. Developed script to read addresses from accelerator data and return latitude and longitude coordinates.<br /> <br /> 2017-01-26: Continued working on Google sitesearch project. Discovered crunchbase, changed project priority. Priority 1, split accelerator data up by flag, priority 2, use crunchbase to get web urls for cohorts, priority 3, make internet archive wayback machine driver. Located [http://mcnair.bakerinstitute.org/wiki/Whois_Parser Whois Parser].<br /> <br /> <br /> 2017-01-25: Finished parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Some data files still need proofreading as they are not in an acceptable format. Began working on Google sitesearch project.<br /> <br /> 2017-01-24: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Cohort data file created, debugging is almost complete. Will begin work on the google accelerator search soon.<br /> <br /> 2017-01-23: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Preliminary code is written, working on debugging.<br /> <br /> 2017-01-19: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project]. Created parser for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project], completed creation of final data set(yay!). Began working on cohort parser.<br /> <br /> 2017-01-18: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-13: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-12: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-11: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-10: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> <br /> ===Fall 2016===<br /> <br /> 2016-12-08: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-07: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-06: Learned how to use git. Committed software projects from the semester to the McNair git repository. Projects can be found at; [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Executive Order Crawler], [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Foreign Government Web Crawlers], [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) F6S Crawler and Parser].<br /> <br /> 2016-12-02: Built and ran web crawler for Center for Middle East Studies on Kuwait. Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-01: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Built tool for the [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report E&amp;I Governance Report Project] with Christy. Adds a column of data that shows whether or not the bill has been passed.<br /> <br /> 2016-11-29: Began pulling data from the accelerators listed [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here]. Made text files for about 18 accelerators.<br /> <br /> 2016-11-22: Transferred downloaded Morocco Written Bills to provided SeaGate Drive. Made a &quot;gentle&quot; F6S crawler to retrieve HTMLs of possible accelerator pages documented [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here].<br /> <br /> 2016-11-18: Converted Executive Order PDFs to text files using adobe acrobat DC. See [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Wikipage] for details.<br /> <br /> 2016-11-17: Wrote a crawler to retrieve information about executive orders, and their corresponding pdfs. They can be found [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report here.] Next step is to run code to convert the pdfs to text files, then use the parser fixed by Christy. <br /> <br /> 2016-11-15: Finished download of Moroccan Written Question pdfs. Wrote a parser with Christy to be used for parsing bills from Congress and eventually executive orders. Found bug in the system Python that was worked out and rebooted.<br /> <br /> 2016-11-11: Continued to download Moroccan data in the background. Attempted to find bug fixes for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] crawler.<br /> <br /> 2016-11-10: Continued to download Moroccan data and Kuwait data in the background. Began work on [http://mcnair.bakerinstitute.org/wiki/Google_Scholar_Crawler Google Scholar Crawler]. Wrote a crawler for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] to get the HTML files of hundreds of accelerators. The crawler ended up failing; it appears to have been due to HTTPS. <br /> <br /> 2016-11-08: Continued to download Moroccan data in the background. Finished writing code for the embedded files on the Kuwait Site. Spent time debugging the frame errors due to the dynamically generated content. Never found an answer to the bug, and instead found a workaround that sacrificed run time for the ability to work. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-04: Continued to download Moroccan data in the background. Finished writing initial Kuwait Web Crawler/Driver for the Middle East Studies Department. Middle East Studies Department asked for additional embedded files in the Kuwait website. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-03: Continued to download Moroccan data in the background. Dr. Egan fixed systems requirements to run the GovTrack Web Crawler. Made significant progress on the Kuwait Web Crawler/Driver for the Middle East Studies Department. <br /> <br /> 2016-11-01: Continued to download Moroccan data in the background. Went over code for GovTracker Web Crawler, continued learning Perl. [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] Began Kuwait Web Crawler/Driver.<br /> <br /> 2016-10-21: Continued to download data for the Moroccan Parliament Written and Oral Questions. Looked over [http://mcnair.bakerinstitute.org/wiki/Christy_Warden_(Twitter_Crawler_Application_1) Christy's Twitter Crawler] to see how I can be helpful. Dr. Egan asked me to think about how to potentially make multiple tools to get cohorts and other sorts of data from accelerator sites. See [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator List] He also asked me to look at the [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] for potential ideas on how to bring this project to fruition.<br /> <br /> 2016-10-20: Continued to download data for the Moroccan Parliament Written and Oral Questions. Updated Wiki page. Started working on Twitter project with Christy. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-18: Finished code for Oral Questions web driver and Written Questions web driver using selenium. Now, the data for the dates of questions can be found using the crawler, and the pdfs of the questions will be downloaded using selenium. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-14: Finished Oral Questions crawler. Finished Written Questions crawler. Waiting for further details on whether that data needs to be tweaked in any way. Updated the Moroccan Web Driver/Web Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-13: Completed download of Moroccan Bills. Working on either a web driver screenshot approach or a webcrawler approach to download the Moroccan oral and written questions data. Began building Web Crawler for Oral and Written Questions site. Edited Moroccan Web Driver/Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-11: Fixed arabic bug, files can now be saved with arabic titles. Monarchy bills downloaded and ready for shipment. House of Representatives Bill mostly downloaded, ratified bills prepared for download. Started learning scrapy library in python for web scraping. Discussed idea of screenshot-ing questions instead of scraping. <br /> <br /> 2016-10-07: Learned unicode and utf8 encoding and decoding in arabic. Still working on transforming an ascii url into printable unicode.<br /> <br /> 2016-10-06: Discussed with Dr. Elbadawy about the desired file names for Moroccan data download. The consensus was that the bill programs are ready to launch once the files can be named properly, and the questions data must be retrieved using a web crawler which I need to learn how to implement. The naming of files is currently drawing errors in going from arabic, to url, to download, to filename. Debugging in process. Also built a demo selenium program for Dr. Egan that drives the McNair blog site on an infinite loop.<br /> <br /> 2016-10-03: Moroccan Web Driver projects completed for driving of the Monarchy proposed bills, the House of Representatives proposed bills, and the Ratified bills sites. Begun process of devising a naming system for the files that does not require scraping. Tinkered with naming through regular expression parsing of the URL. Structure for the oral questions and written questions drivers is set up, but need fixes due to the differences in the sites. Fixed bug on McNair wiki for women's biz team where email was plain text instead of an email link. Took a glimpse at Kuwait Parliament website, and it appears to be very different from the Moroccan setup.<br /> <br /> 2016-09-30: Selenium program selects view pdf option from the website, and goes to the pdf webpage. Program then switches handle to the new page. CTRL S is sent to the page to launch save dialog window. Text cannot be sent to this window. Brainstorm ways around this issue. Explored Chrome Options for saving automatically without a dialog window. Looking into other libraries besides selenium that may help.<br /> <br /> 2016-09-29: Re-enroll in Microsoft Remote Desktop with proper authentication, set up Selenium environment and Komodo IDE on Remote Desktop, wrote program using Selenium that goes to a link and opens up the print dialog box. Developed computational recipe for a different approach to the problem.<br /> <br /> 2016-09-26: Set up Staff wiki page, work log page; registered for Slack, Microsoft Remote Desktop; downloaded Selenium on personal computer, read Selenium docs. Created wiki page for Moroccan Web Driver Project.<br /> <br /> '''Notes'''<br /> <br /> *Ed moved the Morocco Data to E:\McNair\Projects from C:\Users\PeterJ\Documents<br /> * C Drive files moved to E:\McNair\Users\PeterJ<br /> <br /> [[Category:Work Log]]</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Demo_Day_Page_Parser&diff=21916 2017-11-15T21:34:32Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Demo Day Page Parser<br /> |Has owner=Peter Jalbert,<br /> |Has project status=Active<br /> }}<br /> <br /> ==Project Specs==<br /> The goal of this project is to leverage data mining with Selenium and Machine Learning to get good candidate web pages for Demo Days for accelerators. Relevant information on the project can be found on the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Data Accelerator Data] page.<br /> <br /> ==Code Location==<br /> The code directory for this project can be found:<br /> E:\McNair\Software\Accelerators<br /> <br /> The Selenium-based crawler can be found in the file below. This script runs a google search on accelerator names and keywords, and saves the urls and html pages for future use:<br /> DemoDayCrawler.py<br /> <br /> <br /> A script to rip from HTML to TXT can be found below. This script reads HTML files from the DemoDayHTML directory, and writes them to the DemoDayTxt directory:<br /> htmlToText.py<br /> <br /> A script to match Keywords (Accelerator and Cohort names) against the resulting text pages can be found in KeyTerms.py. The script takes the Keywords located in CohortAndAcceleratorsFullList.txt, and the text files in DemoDayTxt, and creates a file with the number of matches of each keyword against each text file.<br /> <br /> The script can be found:<br /> KeyTerms.py<br /> <br /> The Keyword matches text file can be found:<br /> DemoDayTxt\KeyTermFile\KeyTerms.txt<br /> <br /> ==Downloading HTML Files with Selenium==</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Demo_Day_Page_Parser&diff=21915 2017-11-15T21:27:00Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Demo Day Page Parser<br /> |Has owner=Peter Jalbert,<br /> |Has project status=Active<br /> }}<br /> <br /> ==Project Specs==<br /> The goal of this project is to leverage data mining with Selenium and Machine Learning to get good candidate web pages for Demo Days for accelerators. Relevant information on the project can be found on the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Data Accelerator Data] page.<br /> <br /> ==Code Location==<br /> The code directory for this project can be found:<br /> E:\McNair\Software\Accelerators<br /> <br /> The Selenium-based crawler can be found in the file below. This script runs a google search on accelerator names and keywords, and saves the urls and html pages for future use:<br /> DemoDayCrawler.py<br /> <br /> <br /> A script to rip from HTML to TXT can be found below. This script reads HTML files from the DemoDayHTML directory, and writes them to the DemoDayTxt directory:<br /> htmlToText.py<br /> <br /> A script to match Keywords (Accelerator and Cohort names) against the resulting text pages can be found in KeyTerms.py. The script takes the Keywords located in CohortAndAcceleratorsFullList.txt, and the text files in DemoDayTxt, and creates a file with the number of matches of each keyword against each text file.<br /> <br /> The script can be found:<br /> KeyTerms.py<br /> <br /> The Keyword matches text file can be found:<br /> DemoDayTxt\KeyTermFile\KeyTerms.txt</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Demo_Day_Page_Parser&diff=21914 2017-11-15T21:20:43Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Demo Day Page Parser<br /> |Has owner=Peter Jalbert,<br /> |Has project status=Active<br /> }}<br /> <br /> ==Project Specs==<br /> The goal of this project is to leverage data mining with Selenium and Machine Learning to get good candidate web pages for Demo Days for accelerators. Relevant information on the project can be found on the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Data Accelerator Data] page.<br /> <br /> ==Code Location==<br /> The code directory for this project can be found:<br /> E:\McNair\Software\Accelerators<br /> <br /> The Selenium-based crawler can be found in the file below. This script runs a google search on accelerator names and keywords, and saves the urls and html pages for future use:<br /> DemoDayCrawler.py<br /> <br /> <br /> A script to rip from HTML to TXT can be found below. This script reads HTML files from a directory, and writes them to TXT in another directory:<br /> htmlToText.py</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Peter_Jalbert_(Work_Log)&diff=21913 2017-11-15T21:19:46Z

<p>Peterjalbert: </p> <hr /> <div>===Fall 2017===<br /> &lt;onlyinclude&gt; <br /> <br /> [[Peter Jalbert]] [[Work Logs]] [[Peter Jalbert (Work Log)|(log page)]] <br /> <br /> 2017-11-15: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote a script to extract counts that were greater than 2 from Keyword Matcher. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder]. Finished re-formatting work logs.<br /> <br /> 2017-11-14: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote an HTML to Text parser. See Parser Demo Day Page for file location. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder].<br /> <br /> 2017-11-13: Built [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser].<br /> <br /> 2017-11-09: Running demo version of Demo Day crawler (Accelerator Google Crawler). Fixing work log format. <br /> <br /> 2017-11-07: Created file with 0s and 1s detailing whether crunchbase has the founder information for an accelerator. Details posted as a TODO on [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List] page. Still waiting for feedback on the PostGIS installation from [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder]. Continued working on Accelerator Google Crawler.<br /> <br /> 2017-11-06: Contacted Geography Center for the US Census Bureau, [https://www.census.gov/geo/about/contact.html here], and began email exchange on PostGIS installation problems. Began working on the [http://mcnair.bakerinstitute.org/wiki/Selenium_Documentation Selenium Documentation]. Also began working on an Accelerator Google Crawler that will be used with Yang and ML to find Demo Days for cohort companies.<br /> <br /> 2017-11-01: Attempted to continue downloading, however ran into HTTP Forbidden errors. Listed the errors on the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder Page].<br /> <br /> 2017-10-31: Began downloading blocks of data for individual states for the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder] project. Wrote out the new wiki page for installation, and beginning to write documentation on usage.<br /> <br /> 2017-10-30: With Ed's help, was able to get the national data from Tiger installed onto a database server. The process required much jumping around and changing users, and all the things we learned are outlined in [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation#Editing_Users the database server documentation] under &quot;Editing Users&quot;.<br /> <br /> 2017-10-25: Continued working on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation TigerCoder Installation].<br /> <br /> 2017-10-24: Throw some addresses into a database, use address normalizer and geocoder. May need to install things. Details on the installation process can be found on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation page].<br /> <br /> 2017-10-23: Finished Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-19: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-18: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_DistrictHouston Innovation District Project].<br /> <br /> 2017-10-17: Constructed ArcGIS maps for the agglomeration project. Finished maps of points for every year in the state of California. Finished maps of Route 128. Began working on selenium Yelp crawler to get cafe locations within the 610-loop.<br /> <br /> 2017-10-16: Assisted Harrison on the USITC project. Looked for natural language processing tools to extract complaintants and defendants along with their location from case files. Experimented with pulling based on parts of speech tags, as well as using geotext or geograpy to pull locations from a case segment.<br /> <br /> 2017-10-13: Updated various project wiki pages.<br /> <br /> 2017-10-12: Continued work on Patent Thicket project, awaiting further project specs.<br /> <br /> 2017-10-05: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-04: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-02: Worked on ArcGIS data. See Harrison's Work Log for the details.<br /> <br /> 2017-09-28: Added collaborative editing feature to PyCharm. <br /> <br /> 2017-09-27: Worked on big database file.<br /> <br /> 2017-09-25: New task -- Create text file with company, description, and company type.<br /> #[http://mcnair.bakerinstitute.org/wiki/VC_Database_Rebuild VC Database Rebuild]<br /> #psql vcdb2<br /> #table name, sdccompanybasecore2<br /> #Combine with Crunchbasebulk<br /> <br /> #TODO: Write wiki on linkedin crawler, write wiki on creating accounts.<br /> <br /> 2017-09-21: Wrote wiki on Linkedin crawler, met with Laura about patents project. <br /> <br /> 2017-09-20: Finished running linkedin crawler. Transferred data to RDP. Will write wikis next.<br /> <br /> 2017-09-19: Began running linkedin crawler. Helped Yang create RDP account, get permissions, and get wiki setup.<br /> <br /> 2017-09-18: Finished implementation of Experience Crawler, continued working on Education Crawler for LinkedIn.<br /> <br /> 2017-09-14: Continued implementing LinkedIn Crawler for profiles.<br /> <br /> 2017-09-13: Implemented LinkedIn Crawler for main portion of profiles. Began working on crawling Experience section of profiles.<br /> <br /> 2017-09-12: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. Added to the wiki on this topic.<br /> <br /> 2017-09-11: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. <br /> <br /> 2017-09-06: Combined founders data retrieved with the Crunchbase API with the crunchbasebulk data to get linkedin urls for different accelerator founders. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-09-05: Post Harvey. Finished retrieving names from the Crunchbase API on founders. Next step is to query crunchbase bulk database to get linkedin urls. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-08-24: Began using the Crunchbase API to retrieve founder information for accelerators. Halfway through compiling a dictionary that translates accelerator names into proper Crunchbase API URLs.<br /> <br /> 2017-08-23: Decided with Ed to abandon LinkedIn crawling to retrieve accelerator founder data, and instead use crunchbase. Spent the day navigating the crunchbasebulk database, and seeing what useful information was contained in it.<br /> <br /> 2017-08-22: Discovered that LinkedIn Profiles cannot be viewed through LinkedIn if the target is 3rd degree or further. However, if entering LinkedIn through a Google search, the profile can still be viewed if the user has previously logged into LinkedIn. Devising a workaround crawler that utilizes Google search. Continued blog post [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) here] under Section 4.<br /> <br /> 2017-08-21: Began work on extracting founders for accelerators through LinkedIn Crawler. Discovered that Python3 is not installed on RDP, so the virtual environment for the project cannot be fired up. Continued working on Ubuntu machine.<br /> &lt;/onlyinclude&gt; <br /> <br /> ===Spring 2017===<br /> <br /> 2017-05-01: Continued work on HTML Parser. Uploaded all semester projects to git server.<br /> <br /> 2017-04-20: Finished the HTML Parser for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Ran HTML parser on accelerator founders. Data is stored in projects/accelerators/LinkedIn Founder Data.<br /> <br /> 2017-04-19: Made updates to the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler] Wikipage. Ran LinkedIn Crawler on accelerator data. Working on an html parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-18: Ran LinkedIn Crawler on matches between Crunchbase Snapshot and the accelerator data.<br /> <br /> 2017-04-17: Worked on ways to get correct search results from the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Worked on an HTML Parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-13: Worked on debugging the logout procedure for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Began formulation of process to search for founders of startups using a combination of the LinkedIn Crawler with the data resources from the [http://mcnair.bakerinstitute.org/wiki/Crunchbase_2013_Snapshot CrunchBase Snapshot].<br /> <br /> 2017-04-12: Work on bugs with the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler].<br /> <br /> 2017-04-11: Completed functional [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) crawler of LinkedIn Recruiter Pro]. Basic search functions work and download profile information for a given person. <br /> <br /> 2017-04-10: Began writing functioning crawler of LinkedIn. <br /> <br /> 2017-04-06: Continued working on debugging and documenting the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Wrote a test program that logs in, searches for a query, navigates through search pages, and logs out. Recruiter program can now login and search.<br /> <br /> 2017-04-05: Began work on the LinkedIn Crawler. Researched on launching Python Virtual Environment.<br /> <br /> 2017-04-03: Finished debugging points for the Enclosing Circle Algorithm. Added Command Line functionality to the Industry Classifier.<br /> <br /> 2017-03-29: Worked on debugging points for the Enclosing Circle Algorithm.<br /> <br /> 2017-03-28: Finished running the Enclosing Circle Algorithm. Worked on removing incorrect points from the data set(see above). <br /> <br /> 2017-03-27: Worked on debugging the Enclosing Circle Algorithm. Implemented a way to remove interior circles, and determined that translation to latitude and longitude coordinates resulted in slightly off center circles.<br /> <br /> 2017-03-23: Finished debugging the brute force algorithm for [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm]. Implemented a method to plot the points and circles on a graph. Analyzed runtime of the brute force algorithm.<br /> <br /> 2017-03-21: Coded a brute force algorithm for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-20: Worked on debugging the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-09: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Finished script to draw Enclosing Circles on a Google Map.<br /> <br /> 2017-03-08: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Created script to draw outcome of the Enclosing Circle Algorithm on Google Maps.<br /> <br /> 2017-03-07: Redetermined the top 50 cities which Enclosing Circle should be run on. Data on the [http://mcnair.bakerinstitute.org/wiki/Top_Cities_for_VC_Backed_Companies Top 50 Cities for VC Backed Companies can be found here.] Ran [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities.<br /> <br /> 2017-03-06: Ran script to determine the top 50 cities which Enclosing Circle should be run on. Fixed the VC Circles script to take in a new data format.<br /> <br /> 2017-03-02: Cleaned up data for the VC Circles Project. Created histogram of data in Excel. See [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Began work on the [http://mcnair.bakerinstitute.org/wiki/LinkedInCrawlerPython LinkedIn Crawler].<br /> <br /> 2017-03-01: Created statistics for the VC Circles Project.<br /> <br /> 2017-02-28: Finished downloading geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Found bug in Enclosing Circle Algorithm.<br /> <br /> 2017-02-27: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-23: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Installed C++ Compiler for Python. Ran tests on difference between Python and C wrapped Python.<br /> <br /> 2017-02-22: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Helped out with [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-21: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Researched into C++ Compilers for Python so that the Enclosing Circle Algorithm could be wrapped in C. Found a recommended one [https://www.microsoft.com/en-us/download/details.aspx?id=44266 here].<br /> <br /> 2017-02-20: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-16: Reworked [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] to create a file of geocoded data. Began work on wrapping the algorithm in C to improve speed.<br /> <br /> 2017-02-15: Finished [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] applied to the VC study. Enclosing Circle algorithm still needs adjustment, but the program runs with the temporary fixes.<br /> <br /> 2017-02-14: Worked on the application of the Enclosing Circle algorithm to the VC study. Working on bug fixes in the Enclosing Circle algorithm. Created wiki page for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-02-13: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-08: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-07: Fixed bugs in parse_cohort_data.py, the script for parsing the cohort data from the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Added descriptive statistics to cohort data excel file.<br /> <br /> 2017-02-02: Out sick, independent research and work from RDP. Brief research into the [http://jorgeg.scripts.mit.edu/homepage/wp-content/uploads/2016/03/Guzman-Stern-State-of-American-Entrepreneurship-FINAL.pdf Stern-Guzman algorithm]. Research into [http://mcnair.bakerinstitute.org/wiki/interactive_maps Interactive Maps]. No helpful additions to map embedding problem.<br /> <br /> 2017-02-01: Notes from Session with Ed: Project on US university patenting and entrepreneurship programs (writing code to identify universities in assignees), search Wikipedia (XML then bulk download), student pop, faculty pop, etc.<br /> Circle project for VC data will end up being a joint project to join accelerator data. <br /> Pull descriptions for VC. Founders of accelerators in linkedin. LinkedIn cannot be caught(pretend to not be a bot). Can eventually get academic backgrounds through linkedin. <br /> Pull business registration data, Stern/Guzman Algorithm. <br /> GIS ontop of geocoded data.<br /> Maps that works on wiki or blog (CartoDB), Maps API and R.<br /> NLP Projects, Description Classifier.<br /> <br /> 2017-01-31: Built WayBack Machine Crawler. Updated documentation for coordinates script. Updated profile page to include locations of code.<br /> <br /> <br /> 2017-01-30: Optimized enclosing circle algorithm through memoization. Developed script to read addresses from accelerator data and return latitude and longitude coordinates.<br /> <br /> 2017-01-26: Continued working on Google sitesearch project. Discovered crunchbase, changed project priority. Priority 1, split accelerator data up by flag, priority 2, use crunchbase to get web urls for cohorts, priority 3, make internet archive wayback machine driver. Located [http://mcnair.bakerinstitute.org/wiki/Whois_Parser Whois Parser].<br /> <br /> <br /> 2017-01-25: Finished parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Some data files still need proofreading as they are not in an acceptable format. Began working on Google sitesearch project.<br /> <br /> 2017-01-24: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Cohort data file created, debugging is almost complete. Will begin work on the google accelerator search soon.<br /> <br /> 2017-01-23: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Preliminary code is written, working on debugging.<br /> <br /> 2017-01-19: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project]. Created parser for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project], completed creation of final data set(yay!). Began working on cohort parser.<br /> <br /> 2017-01-18: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-13: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-12: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-11: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-10: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> <br /> ===Fall 2016===<br /> <br /> 2016-12-08: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-07: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-06: Learned how to use git. Committed software projects from the semester to the McNair git repository. Projects can be found at; [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Executive Order Crawler], [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Foreign Government Web Crawlers], [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) F6S Crawler and Parser].<br /> <br /> 2016-12-02: Built and ran web crawler for Center for Middle East Studies on Kuwait. Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-01: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Built tool for the [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report E&amp;I Governance Report Project] with Christy. Adds a column of data that shows whether or not the bill has been passed.<br /> <br /> 2016-11-29: Began pulling data from the accelerators listed [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here]. Made text files for about 18 accelerators.<br /> <br /> 2016-11-22: Transferred downloaded Morocco Written Bills to provided SeaGate Drive. Made a &quot;gentle&quot; F6S crawler to retrieve HTMLs of possible accelerator pages documented [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here].<br /> <br /> 2016-11-18: Converted Executive Order PDFs to text files using adobe acrobat DC. See [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Wikipage] for details.<br /> <br /> 2016-11-17: Wrote a crawler to retrieve information about executive orders, and their corresponding pdfs. They can be found [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report here.] Next step is to run code to convert the pdfs to text files, then use the parser fixed by Christy. <br /> <br /> 2016-11-15: Finished download of Moroccan Written Question pdfs. Wrote a parser with Christy to be used for parsing bills from Congress and eventually executive orders. Found bug in the system Python that was worked out and rebooted.<br /> <br /> 2016-11-11: Continued to download Moroccan data in the background. Attempted to find bug fixes for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] crawler.<br /> <br /> 2016-11-10: Continued to download Moroccan data and Kuwait data in the background. Began work on [http://mcnair.bakerinstitute.org/wiki/Google_Scholar_Crawler Google Scholar Crawler]. Wrote a crawler for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] to get the HTML files of hundreds of accelerators. The crawler ended up failing; it appears to have been due to HTTPS. <br /> <br /> 2016-11-08: Continued to download Moroccan data in the background. Finished writing code for the embedded files on the Kuwait Site. Spent time debugging the frame errors due to the dynamically generated content. Never found an answer to the bug, and instead found a workaround that sacrificed run time for the ability to work. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-04: Continued to download Moroccan data in the background. Finished writing initial Kuwait Web Crawler/Driver for the Middle East Studies Department. Middle East Studies Department asked for additional embedded files in the Kuwait website. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-03: Continued to download Moroccan data in the background. Dr. Egan fixed systems requirements to run the GovTrack Web Crawler. Made significant progress on the Kuwait Web Crawler/Driver for the Middle East Studies Department. <br /> <br /> 2016-11-01: Continued to download Moroccan data in the background. Went over code for GovTracker Web Crawler, continued learning Perl. [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] Began Kuwait Web Crawler/Driver.<br /> <br /> 2016-10-21: Continued to download data for the Moroccan Parliament Written and Oral Questions. Looked over [http://mcnair.bakerinstitute.org/wiki/Christy_Warden_(Twitter_Crawler_Application_1) Christy's Twitter Crawler] to see how I can be helpful. Dr. Egan asked me to think about how to potentially make multiple tools to get cohorts and other sorts of data from accelerator sites. See [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator List] He also asked me to look at the [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] for potential ideas on how to bring this project to fruition.<br /> <br /> 2016-10-20: Continued to download data for the Moroccan Parliament Written and Oral Questions. Updated Wiki page. Started working on Twitter project with Christy. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-18: Finished code for Oral Questions web driver and Written Questions web driver using selenium. Now, the data for the dates of questions can be found using the crawler, and the pdfs of the questions will be downloaded using selenium. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-14: Finished Oral Questions crawler. Finished Written Questions crawler. Waiting for further details on whether that data needs to be tweaked in any way. Updated the Moroccan Web Driver/Web Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-13: Completed download of Moroccan Bills. Working on either a web driver screenshot approach or a webcrawler approach to download the Moroccan oral and written questions data. Began building Web Crawler for Oral and Written Questions site. Edited Moroccan Web Driver/Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-11: Fixed arabic bug, files can now be saved with arabic titles. Monarchy bills downloaded and ready for shipment. House of Representatives Bill mostly downloaded, ratified bills prepared for download. Started learning scrapy library in python for web scraping. Discussed idea of screenshot-ing questions instead of scraping. <br /> <br /> 2016-10-07: Learned unicode and utf8 encoding and decoding in arabic. Still working on transforming an ascii url into printable unicode.<br /> <br /> 2016-10-06: Discussed with Dr. Elbadawy about the desired file names for Moroccan data download. The consensus was that the bill programs are ready to launch once the files can be named properly, and the questions data must be retrieved using a web crawler which I need to learn how to implement. The naming of files is currently drawing errors in going from arabic, to url, to download, to filename. Debugging in process. Also built a demo selenium program for Dr. Egan that drives the McNair blog site on an infinite loop.<br /> <br /> 2016-10-03: Moroccan Web Driver projects completed for driving of the Monarchy proposed bills, the House of Representatives proposed bills, and the Ratified bills sites. Begun process of devising a naming system for the files that does not require scraping. Tinkered with naming through regular expression parsing of the URL. Structure for the oral questions and written questions drivers is set up, but need fixes due to the differences in the sites. Fixed bug on McNair wiki for women's biz team where email was plain text instead of an email link. Took a glimpse at Kuwait Parliament website, and it appears to be very different from the Moroccan setup.<br /> <br /> 2016-09-30: Selenium program selects view pdf option from the website, and goes to the pdf webpage. Program then switches handle to the new page. CTRL S is sent to the page to launch save dialog window. Text cannot be sent to this window. Brainstorm ways around this issue. Explored Chrome Options for saving automatically without a dialog window. Looking into other libraries besides selenium that may help.<br /> <br /> 2016-09-29: Re-enroll in Microsoft Remote Desktop with proper authentication, set up Selenium environment and Komodo IDE on Remote Desktop, wrote program using Selenium that goes to a link and opens up the print dialog box. Developed computational recipe for a different approach to the problem.<br /> <br /> 2016-09-26: Set up Staff wiki page, work log page; registered for Slack, Microsoft Remote Desktop; downloaded Selenium on personal computer, read Selenium docs. Created wiki page for Moroccan Web Driver Project.<br /> <br /> '''Notes'''<br /> <br /> *Ed moved the Morocco Data to E:\McNair\Projects from C:\Users\PeterJ\Documents<br /> * C Drive files moved to E:\McNair\Users\PeterJ<br /> <br /> [[Category:Work Log]]</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Peter_Jalbert_(Work_Log)&diff=21912 2017-11-15T21:19:23Z

<p>Peterjalbert: </p> <hr /> <div>===Fall 2017===<br /> &lt;onlyinclude&gt; <br /> <br /> [[Peter Jalbert]] [[Work Logs]] [[Peter Jalbert (Work Log)|(log page)]] <br /> <br /> 2017-11-15: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote a script to extract counts that were greater than 2 from Keyword Matcher. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder].<br /> <br /> 2017-11-14: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote an HTML to Text parser. See Parser Demo Day Page for file location. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder].<br /> <br /> 2017-11-13: Built [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser].<br /> <br /> 2017-11-09: Running demo version of Demo Day crawler (Accelerator Google Crawler). Fixing work log format. <br /> <br /> 2017-11-07: Created file with 0s and 1s detailing whether crunchbase has the founder information for an accelerator. Details posted as a TODO on [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List] page. Still waiting for feedback on the PostGIS installation from [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder]. Continued working on Accelerator Google Crawler.<br /> <br /> 2017-11-06: Contacted Geography Center for the US Census Bureau, [https://www.census.gov/geo/about/contact.html here], and began email exchange on PostGIS installation problems. Began working on the [http://mcnair.bakerinstitute.org/wiki/Selenium_Documentation Selenium Documentation]. Also began working on an Accelerator Google Crawler that will be used with Yang and ML to find Demo Days for cohort companies.<br /> <br /> 2017-11-01: Attempted to continue downloading, however ran into HTTP Forbidden errors. Listed the errors on the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder Page].<br /> <br /> 2017-10-31: Began downloading blocks of data for individual states for the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder] project. Wrote out the new wiki page for installation, and beginning to write documentation on usage.<br /> <br /> 2017-10-30: With Ed's help, was able to get the national data from Tiger installed onto a database server. The process required much jumping around and changing users, and all the things we learned are outlined in [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation#Editing_Users the database server documentation] under &quot;Editing Users&quot;.<br /> <br /> 2017-10-25: Continued working on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation TigerCoder Installation].<br /> <br /> 2017-10-24: Throw some addresses into a database, use address normalizer and geocoder. May need to install things. Details on the installation process can be found on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation page].<br /> <br /> 2017-10-23: Finished Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-19: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-18: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_DistrictHouston Innovation District Project].<br /> <br /> 2017-10-17: Constructed ArcGIS maps for the agglomeration project. Finished maps of points for every year in the state of California. Finished maps of Route 128. Began working on selenium Yelp crawler to get cafe locations within the 610-loop.<br /> <br /> 2017-10-16: Assisted Harrison on the USITC project. Looked for natural language processing tools to extract complaintants and defendants along with their location from case files. Experimented with pulling based on parts of speech tags, as well as using geotext or geograpy to pull locations from a case segment.<br /> <br /> 2017-10-13: Updated various project wiki pages.<br /> <br /> 2017-10-12: Continued work on Patent Thicket project, awaiting further project specs.<br /> <br /> 2017-10-05: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-04: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-02: Worked on ArcGIS data. See Harrison's Work Log for the details.<br /> <br /> 2017-09-28: Added collaborative editing feature to PyCharm. <br /> <br /> 2017-09-27: Worked on big database file.<br /> <br /> 2017-09-25: New task -- Create text file with company, description, and company type.<br /> #[http://mcnair.bakerinstitute.org/wiki/VC_Database_Rebuild VC Database Rebuild]<br /> #psql vcdb2<br /> #table name, sdccompanybasecore2<br /> #Combine with Crunchbasebulk<br /> <br /> #TODO: Write wiki on linkedin crawler, write wiki on creating accounts.<br /> <br /> 2017-09-21: Wrote wiki on Linkedin crawler, met with Laura about patents project. <br /> <br /> 2017-09-20: Finished running linkedin crawler. Transferred data to RDP. Will write wikis next.<br /> <br /> 2017-09-19: Began running linkedin crawler. Helped Yang create RDP account, get permissions, and get wiki setup.<br /> <br /> 2017-09-18: Finished implementation of Experience Crawler, continued working on Education Crawler for LinkedIn.<br /> <br /> 2017-09-14: Continued implementing LinkedIn Crawler for profiles.<br /> <br /> 2017-09-13: Implemented LinkedIn Crawler for main portion of profiles. Began working on crawling Experience section of profiles.<br /> <br /> 2017-09-12: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. Added to the wiki on this topic.<br /> <br /> 2017-09-11: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. <br /> <br /> 2017-09-06: Combined founders data retrieved with the Crunchbase API with the crunchbasebulk data to get linkedin urls for different accelerator founders. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-09-05: Post Harvey. Finished retrieving names from the Crunchbase API on founders. Next step is to query crunchbase bulk database to get linkedin urls. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-08-24: Began using the Crunchbase API to retrieve founder information for accelerators. Halfway through compiling a dictionary that translates accelerator names into proper Crunchbase API URLs.<br /> <br /> 2017-08-23: Decided with Ed to abandon LinkedIn crawling to retrieve accelerator founder data, and instead use crunchbase. Spent the day navigating the crunchbasebulk database, and seeing what useful information was contained in it.<br /> <br /> 2017-08-22: Discovered that LinkedIn Profiles cannot be viewed through LinkedIn if the target is 3rd degree or further. However, if entering LinkedIn through a Google search, the profile can still be viewed if the user has previously logged into LinkedIn. Devising a workaround crawler that utilizes Google search. Continued blog post [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) here] under Section 4.<br /> <br /> 2017-08-21: Began work on extracting founders for accelerators through LinkedIn Crawler. Discovered that Python3 is not installed on RDP, so the virtual environment for the project cannot be fired up. Continued working on Ubuntu machine.<br /> &lt;/onlyinclude&gt; <br /> <br /> ===Spring 2017===<br /> <br /> 2017-05-01: Continued work on HTML Parser. Uploaded all semester projects to git server.<br /> <br /> 2017-04-20: Finished the HTML Parser for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Ran HTML parser on accelerator founders. Data is stored in projects/accelerators/LinkedIn Founder Data.<br /> <br /> 2017-04-19: Made updates to the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler] Wikipage. Ran LinkedIn Crawler on accelerator data. Working on an html parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-18: Ran LinkedIn Crawler on matches between Crunchbase Snapshot and the accelerator data.<br /> <br /> 2017-04-17: Worked on ways to get correct search results from the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Worked on an HTML Parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-13: Worked on debugging the logout procedure for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Began formulation of process to search for founders of startups using a combination of the LinkedIn Crawler with the data resources from the [http://mcnair.bakerinstitute.org/wiki/Crunchbase_2013_Snapshot CrunchBase Snapshot].<br /> <br /> 2017-04-12: Work on bugs with the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler].<br /> <br /> 2017-04-11: Completed functional [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) crawler of LinkedIn Recruiter Pro]. Basic search functions work and download profile information for a given person. <br /> <br /> 2017-04-10: Began writing functioning crawler of LinkedIn. <br /> <br /> 2017-04-06: Continued working on debugging and documenting the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Wrote a test program that logs in, searches for a query, navigates through search pages, and logs out. Recruiter program can now login and search.<br /> <br /> 2017-04-05: Began work on the LinkedIn Crawler. Researched on launching Python Virtual Environment.<br /> <br /> 2017-04-03: Finished debugging points for the Enclosing Circle Algorithm. Added Command Line functionality to the Industry Classifier.<br /> <br /> 2017-03-29: Worked on debugging points for the Enclosing Circle Algorithm.<br /> <br /> 2017-03-28: Finished running the Enclosing Circle Algorithm. Worked on removing incorrect points from the data set(see above). <br /> <br /> 2017-03-27: Worked on debugging the Enclosing Circle Algorithm. Implemented a way to remove interior circles, and determined that translation to latitude and longitude coordinates resulted in slightly off center circles.<br /> <br /> 2017-03-23: Finished debugging the brute force algorithm for [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm]. Implemented a method to plot the points and circles on a graph. Analyzed runtime of the brute force algorithm.<br /> <br /> 2017-03-21: Coded a brute force algorithm for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-20: Worked on debugging the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-09: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Finished script to draw Enclosing Circles on a Google Map.<br /> <br /> 2017-03-08: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Created script to draw outcome of the Enclosing Circle Algorithm on Google Maps.<br /> <br /> 2017-03-07: Redetermined the top 50 cities which Enclosing Circle should be run on. Data on the [http://mcnair.bakerinstitute.org/wiki/Top_Cities_for_VC_Backed_Companies Top 50 Cities for VC Backed Companies can be found here.] Ran [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities.<br /> <br /> 2017-03-06: Ran script to determine the top 50 cities which Enclosing Circle should be run on. Fixed the VC Circles script to take in a new data format.<br /> <br /> 2017-03-02: Cleaned up data for the VC Circles Project. Created histogram of data in Excel. See [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Began work on the [http://mcnair.bakerinstitute.org/wiki/LinkedInCrawlerPython LinkedIn Crawler].<br /> <br /> 2017-03-01: Created statistics for the VC Circles Project.<br /> <br /> 2017-02-28: Finished downloading geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Found bug in Enclosing Circle Algorithm.<br /> <br /> 2017-02-27: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-23: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Installed C++ Compiler for Python. Ran tests on difference between Python and C wrapped Python.<br /> <br /> 2017-02-22: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Helped out with [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-21: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Researched into C++ Compilers for Python so that the Enclosing Circle Algorithm could be wrapped in C. Found a recommended one [https://www.microsoft.com/en-us/download/details.aspx?id=44266 here].<br /> <br /> 2017-02-20: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-16: Reworked [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] to create a file of geocoded data. Began work on wrapping the algorithm in C to improve speed.<br /> <br /> 2017-02-15: Finished [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] applied to the VC study. Enclosing Circle algorithm still needs adjustment, but the program runs with the temporary fixes.<br /> <br /> 2017-02-14: Worked on the application of the Enclosing Circle algorithm to the VC study. Working on bug fixes in the Enclosing Circle algorithm. Created wiki page for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-02-13: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-08: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-07: Fixed bugs in parse_cohort_data.py, the script for parsing the cohort data from the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Added descriptive statistics to cohort data excel file.<br /> <br /> 2017-02-02: Out sick, independent research and work from RDP. Brief research into the [http://jorgeg.scripts.mit.edu/homepage/wp-content/uploads/2016/03/Guzman-Stern-State-of-American-Entrepreneurship-FINAL.pdf Stern-Guzman algorithm]. Research into [http://mcnair.bakerinstitute.org/wiki/interactive_maps Interactive Maps]. No helpful additions to map embedding problem.<br /> <br /> 2017-02-01: Notes from Session with Ed: Project on US university patenting and entrepreneurship programs (writing code to identify universities in assignees), search Wikipedia (XML then bulk download), student pop, faculty pop, etc.<br /> Circle project for VC data will end up being a joint project to join accelerator data. <br /> Pull descriptions for VC. Founders of accelerators in linkedin. LinkedIn cannot be caught(pretend to not be a bot). Can eventually get academic backgrounds through linkedin. <br /> Pull business registration data, Stern/Guzman Algorithm. <br /> GIS ontop of geocoded data.<br /> Maps that works on wiki or blog (CartoDB), Maps API and R.<br /> NLP Projects, Description Classifier.<br /> <br /> 2017-01-31: Built WayBack Machine Crawler. Updated documentation for coordinates script. Updated profile page to include locations of code.<br /> <br /> <br /> 2017-01-30: Optimized enclosing circle algorithm through memoization. Developed script to read addresses from accelerator data and return latitude and longitude coordinates.<br /> <br /> 2017-01-26: Continued working on Google sitesearch project. Discovered crunchbase, changed project priority. Priority 1, split accelerator data up by flag, priority 2, use crunchbase to get web urls for cohorts, priority 3, make internet archive wayback machine driver. Located [http://mcnair.bakerinstitute.org/wiki/Whois_Parser Whois Parser].<br /> <br /> <br /> 2017-01-25: Finished parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Some data files still need proofreading as they are not in an acceptable format. Began working on Google sitesearch project.<br /> <br /> 2017-01-24: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Cohort data file created, debugging is almost complete. Will begin work on the google accelerator search soon.<br /> <br /> 2017-01-23: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Preliminary code is written, working on debugging.<br /> <br /> 2017-01-19: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project]. Created parser for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project], completed creation of final data set(yay!). Began working on cohort parser.<br /> <br /> 2017-01-18: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-13: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-12: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-11: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-10: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> <br /> ===Fall 2016===<br /> <br /> 2016-12-08: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-07: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-06: Learned how to use git. Committed software projects from the semester to the McNair git repository. Projects can be found at; [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Executive Order Crawler], [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Foreign Government Web Crawlers], [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) F6S Crawler and Parser].<br /> <br /> 2016-12-02: Built and ran web crawler for Center for Middle East Studies on Kuwait. Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-01: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Built tool for the [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report E&amp;I Governance Report Project] with Christy. Adds a column of data that shows whether or not the bill has been passed.<br /> <br /> 2016-11-29: Began pulling data from the accelerators listed [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here]. Made text files for about 18 accelerators.<br /> <br /> 2016-11-22: Transferred downloaded Morocco Written Bills to provided SeaGate Drive. Made a &quot;gentle&quot; F6S crawler to retrieve HTMLs of possible accelerator pages documented [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here].<br /> <br /> 2016-11-18: Converted Executive Order PDFs to text files using adobe acrobat DC. See [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Wikipage] for details.<br /> <br /> 2016-11-17: Wrote a crawler to retrieve information about executive orders, and their corresponding pdfs. They can be found [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report here.] Next step is to run code to convert the pdfs to text files, then use the parser fixed by Christy. <br /> <br /> 2016-11-15: Finished download of Moroccan Written Question pdfs. Wrote a parser with Christy to be used for parsing bills from Congress and eventually executive orders. Found bug in the system Python that was worked out and rebooted.<br /> <br /> 2016-11-11: Continued to download Moroccan data in the background. Attempted to find bug fixes for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] crawler.<br /> <br /> 2016-11-10: Continued to download Moroccan data and Kuwait data in the background. Began work on [http://mcnair.bakerinstitute.org/wiki/Google_Scholar_Crawler Google Scholar Crawler]. Wrote a crawler for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] to get the HTML files of hundreds of accelerators. The crawler ended up failing; it appears to have been due to HTTPS. <br /> <br /> 2016-11-08: Continued to download Moroccan data in the background. Finished writing code for the embedded files on the Kuwait Site. Spent time debugging the frame errors due to the dynamically generated content. Never found an answer to the bug, and instead found a workaround that sacrificed run time for the ability to work. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-04: Continued to download Moroccan data in the background. Finished writing initial Kuwait Web Crawler/Driver for the Middle East Studies Department. Middle East Studies Department asked for additional embedded files in the Kuwait website. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-03: Continued to download Moroccan data in the background. Dr. Egan fixed systems requirements to run the GovTrack Web Crawler. Made significant progress on the Kuwait Web Crawler/Driver for the Middle East Studies Department. <br /> <br /> 2016-11-01: Continued to download Moroccan data in the background. Went over code for GovTracker Web Crawler, continued learning Perl. [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] Began Kuwait Web Crawler/Driver.<br /> <br /> 2016-10-21: Continued to download data for the Moroccan Parliament Written and Oral Questions. Looked over [http://mcnair.bakerinstitute.org/wiki/Christy_Warden_(Twitter_Crawler_Application_1) Christy's Twitter Crawler] to see how I can be helpful. Dr. Egan asked me to think about how to potentially make multiple tools to get cohorts and other sorts of data from accelerator sites. See [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator List] He also asked me to look at the [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] for potential ideas on how to bring this project to fruition.<br /> <br /> 2016-10-20: Continued to download data for the Moroccan Parliament Written and Oral Questions. Updated Wiki page. Started working on Twitter project with Christy. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-18: Finished code for Oral Questions web driver and Written Questions web driver using selenium. Now, the data for the dates of questions can be found using the crawler, and the pdfs of the questions will be downloaded using selenium. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-14: Finished Oral Questions crawler. Finished Written Questions crawler. Waiting for further details on whether that data needs to be tweaked in any way. Updated the Moroccan Web Driver/Web Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-13: Completed download of Moroccan Bills. Working on either a web driver screenshot approach or a webcrawler approach to download the Moroccan oral and written questions data. Began building Web Crawler for Oral and Written Questions site. Edited Moroccan Web Driver/Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-11: Fixed arabic bug, files can now be saved with arabic titles. Monarchy bills downloaded and ready for shipment. House of Representatives Bill mostly downloaded, ratified bills prepared for download. Started learning scrapy library in python for web scraping. Discussed idea of screenshot-ing questions instead of scraping. <br /> <br /> 2016-10-07: Learned unicode and utf8 encoding and decoding in arabic. Still working on transforming an ascii url into printable unicode.<br /> <br /> 2016-10-06: Discussed with Dr. Elbadawy about the desired file names for Moroccan data download. The consensus was that the bill programs are ready to launch once the files can be named properly, and the questions data must be retrieved using a web crawler which I need to learn how to implement. The naming of files is currently drawing errors in going from arabic, to url, to download, to filename. Debugging in process. Also built a demo selenium program for Dr. Egan that drives the McNair blog site on an infinite loop.<br /> <br /> 2016-10-03: Moroccan Web Driver projects completed for driving of the Monarchy proposed bills, the House of Representatives proposed bills, and the Ratified bills sites. Begun process of devising a naming system for the files that does not require scraping. Tinkered with naming through regular expression parsing of the URL. Structure for the oral questions and written questions drivers is set up, but need fixes due to the differences in the sites. Fixed bug on McNair wiki for women's biz team where email was plain text instead of an email link. Took a glimpse at Kuwait Parliament website, and it appears to be very different from the Moroccan setup.<br /> <br /> 2016-09-30: Selenium program selects view pdf option from the website, and goes to the pdf webpage. Program then switches handle to the new page. CTRL S is sent to the page to launch save dialog window. Text cannot be sent to this window. Brainstorm ways around this issue. Explored Chrome Options for saving automatically without a dialog window. Looking into other libraries besides selenium that may help.<br /> <br /> 2016-09-29: Re-enroll in Microsoft Remote Desktop with proper authentication, set up Selenium environment and Komodo IDE on Remote Desktop, wrote program using Selenium that goes to a link and opens up the print dialog box. Developed computational recipe for a different approach to the problem.<br /> <br /> 2016-09-26: Set up Staff wiki page, work log page; registered for Slack, Microsoft Remote Desktop; downloaded Selenium on personal computer, read Selenium docs. Created wiki page for Moroccan Web Driver Project.<br /> <br /> '''Notes'''<br /> <br /> *Ed moved the Morocco Data to E:\McNair\Projects from C:\Users\PeterJ\Documents<br /> * C Drive files moved to E:\McNair\Users\PeterJ<br /> <br /> [[Category:Work Log]]</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Tiger_Geocoder&diff=21911 2017-11-15T21:14:12Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Tiger Geocoder<br /> |Has owner=Peter Jalbert,<br /> |Has start date=Fall 2017<br /> |Has keywords=Tiger, Geocoder, Database<br /> |Has project status=Active<br /> |Has Image=Tiger.jpg<br /> }}<br /> This page serves as documentation for using the Tiger Geocoder on Postgres SQL, as part of the PostGIS extension. The following wiki pages may also be of use to you:<br /> <br /> [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation]<br /> [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation Database Server Documentation]<br /> <br /> The official documentation for using and installing the Tiger Geocoder can be found in the following.<br /> <br /> [https://postgis.net/docs/Extras.html#Tiger_Geocoder General Instructions]<br /> [https://postgis.net/docs/postgis_installation.html#install_tiger_geocoder_extension Installation Instructions]<br /> [http://postgis.net/docs/Geocode.html Geocoder Documentation]<br /> <br /> ==Location==<br /> The data is currently loaded into a psql database called geocoder. The tables contain the geocoding information, and there is a test table called &quot;coffeeshops&quot; that contains addresses of Houston coffeeshops according to yelp. To access the database, first login to the McNair DB Sever. Then, <br /> <br /> psql geocoder<br /> <br /> <br /> <br /> ==Installation==<br /> <br /> ===Install and Nation Data===<br /> I began by adding the extension listed above. First, enter into Postgres by using the psql command. Then:<br /> --Add Extensions to database<br /> CREATE EXTENSION postgis;<br /> CREATE EXTENSION fuzzystrmatch;<br /> CREATE EXTENSION postgis_tiger_geocoder;<br /> CREATE EXTENSION address_standardizer;<br /> <br /> You can test that the installation worked by running the following query: <br /> SELECT na.address, na.streetname,na.streettypeabbrev, na.zip<br /> FROM normalize_address('1 Devonshire Place, Boston, MA 02109') AS na;<br /> <br /> This should return the following:<br /> address | streetname | streettypeabbrev | zip<br /> ---------+------------+------------------+-------<br /> 1 | Devonshire | Pl | 02109<br /> <br /> Next, a new profile needs to be created by using the following command.<br /> INSERT INTO tiger.loader_platform(os, declare_sect, pgbin, wget, unzip_command, psql, path_sep, <br /> loader, environ_set_command, county_process_command)<br /> SELECT 'test', declare_sect, pgbin, wget, unzip_command, psql, path_sep,<br /> loader, environ_set_command, county_process_command<br /> FROM tiger.loader_platform<br /> WHERE os = 'sh';<br /> <br /> The installation instructions also provide the following note:<br /> <br /> As of PostGIS 2.4.1 the Zip code-5 digit tabulation area zcta5 load step was revised to load current zcta5 data and is part of the Loader_Generate_Nation_Script when enabled. It is turned off by default because it takes quite a bit of time to load (20 to 60 minutes), takes up quite a bit of disk space, and is not used that often.<br /> <br /> If you would like this feature, you can enable it by using the following command. This should be done before loading the script.<br /> <br /> UPDATE tiger.loader_lookuptables SET load = true WHERE table_name = 'zcta510';<br /> <br /> The paths in declare_sect need to be edited so they match our server locations. One option is to edit the declare_sect column in the tiger.loader_platform table. If so, the declare_sect looks like the following:<br /> export PGHOST=localhost +<br /> export PGUSER=postgres +<br /> export PGPASSWORD=yourpasswordhere +<br /> export PGDATABASE=geocoder +<br /> PSQL=${PGBIN}/psql +<br /> SHP2PGSQL=shp2pgsql +<br /> cd ${staging_fold} +<br /> <br /> TMPDIR=&quot;${staging_fold}/temp/&quot; +<br /> UNZIPTOOL=unzip +<br /> WGETTOOL=&quot;/usr/bin/wget&quot; +<br /> export PGBIN=/usr/lib/postgresql/9.6/bin +<br /> export PGPORT=5432 +<br /> export PGHOST=localhost +<br /> export PGUSER=postgres +<br /> export PGPASSWORD=yourpasswordhere +<br /> export PGDATABASE=geocoder +<br /> PSQL=${PGBIN}/psql +<br /> SHP2PGSQL=shp2pgsql +<br /> cd ${staging_fold}<br /> <br /> Another option is to edit the sh file before running the script. We will do this option until further notice. Simply use your favorite command line editor to change the fields to their correct values. The downloaded script is located in the following directory:<br /> /gisdata<br /> <br /> There needs to be a directory called &quot;temp&quot; in the gisdata directory. To make the script, use the following from the command line:<br /> psql -c &quot;SELECT Loader_Generate_Nation_Script('test')&quot; -d databasename -tA &gt; /gisdata/nation_script_load.sh<br /> <br /> This will create a script in the /gisdata directory. Change to that directory. If you did not edit the paths in the declare_sect table in psql, then you will need to edit this file to contain the correct paths. <br /> <br /> Change directories:<br /> cd /gisdata<br /> <br /> Edit the script using your favorite command line text editor. Specifically, edit the following fields.<br /> PGUSER=postgres<br /> PGPASSWORD=(Ask Anne for this password)!<br /> Everything else remains the same.<br /> <br /> Run the script by using: <br /> sh nation_script_load.sh<br /> <br /> Now, there is a barebones table in the database that will hold the information for the nation. Next is to download data on each state.<br /> <br /> ===State Data===<br /> <br /> The state scripts are generated in much the same way that the nation script was generated. Use the following command, substituting MA for your desired state abbreviation, and substituting a unique filename at the end.<br /> psql -c &quot;SELECT Loader_Generate_Script(ARRAY['MA'], 'test')&quot; -d geocoder -tA &gt; /gisdata/ma_load.sh<br /> <br /> CURRENT PROGRESS:<br /> The following states have been downloaded into the geocoder database.<br /> AL, AK, AZ, AR, CA, CO, CT, DE, FL, MA<br /> <br /> ===Current Errors===<br /> The state scripts stopped working on 11/1/2017 while they were working on 10/31/2017. Now, when a retrieval script is run, it draws the error <br /> HTTP request sent, awaiting response... 403 Forbidden<br /> Possible thoughts:<br /> <br /> Maybe our IP has been blacklisted for downloading data from a government website quickly? If so, PostGIS should really choose a different installation method. The current one is dumb.<br /> <br /> Maybe the nation downloader script never worked properly. Not sure how to check if it is correct or not; seems right.<br /> <br /> [https://trac.osgeo.org/postgis/ticket/3699 This] is the only online forum I could find with others who have faced a similar issue.<br /> <br /> ==Geocode Function==<br /> <br /> The official arguments for the function are the following:<br /> setof record geocode(varchar address, integer max_results=10, geometry restrict_region=NULL, norm_addy OUT addy, geometry OUT geomout, integer OUT rating);<br /> The arguments of interest are address, where you simply submit a string, and max_results, which restricts the number of geocoding attempts per address. The geocoder makes multiple guesses to the location of an address, and returns the best guesses in order. If you want multiple guesses to a specific address, then specify max_results to be more than 1.<br /> <br /> ===Single Address===<br /> <br /> An example query for a single address is:<br /> SELECT g.rating, ST_X(g.geomout) As lon, ST_Y(g.geomout) As lat,<br /> (addy).address As stno, (addy).streetname As street,<br /> (addy).streettypeabbrev As styp, (addy).location As city, (addy).stateabbrev As st,(addy).zip<br /> FROM geocode('75 State Street, Boston MA 02109', 1) As g;<br /> <br /> rating -- This is an integer that determines the confidence in the resulting geocode. The closer to 0, the more confident the guess.<br /> <br /> ST_X(g.geomout) -- This retrieves the longitude coordinate of the point.<br /> <br /> ST_Y(g.geomout) -- This retrieves the latitude corodinate of the point.<br /> <br /> addy -- In general, addy is a normalized address resulting from the input address.<br /> <br /> (addy).address -- The number of the address (Ex: &quot;75&quot; Blabla rd.)<br /> <br /> (addy).streetname -- The name of the street (Ex: 75 &quot;Blabla&quot; rd.)<br /> <br /> (addy).streettypeabbrev -- The abbreviation of the street (Ex: 75 blabla &quot;rd&quot;)<br /> <br /> (addy).location -- The city location of the address.<br /> <br /> (addy).stateabbrev -- The state abbreviation.<br /> <br /> (addy).zip -- The zipcode of the address.<br /> <br /> The output of the query above would be:<br /> rating | lon | lat | stno | street | styp | city | st | zip<br /> --------+-------------------+----------------+------+--------+------+--------+----+-------<br /> 0 | -71.0557505845646 | 42.35897920691 | 75 | State | St | Boston | MA | 02109<br /> <br /> ==Current Errors==<br /> Currently, we are getting a 403 Forbidden Error when trying to download state data. We are in contact with the US Census Bureau. Their contact information can be found [https://www.census.gov/geo/about/contact.html here].<br /> <br /> The email exchanges are recorded below.<br /> <br /> --------------------------------------<br /> --------------------------------------<br /> <br /> I am a student researcher at the McNair Center for Entrepreneurship and Innovation at Rice University, and I am in the process of installing a Postgres Extension that relies on the TIGER data. <br /> <br /> When I began installing the extension, things were working fine. Now however, I am getting a 403 Forbidden error when the script tries to download the TIGER files. Do you have any idea why this might be happening? <br /> <br /> The extension I'm trying to install is below:<br /> <br /> http://postgis.net/docs/postgis_installation.html#install_tiger_geocoder_extension<br /> <br /> When I run any of the scripts that require data from TIGER, I am receiving the following error:<br /> <br /> <br /> --2017-11-06 14:15:29-- http://www2.census.gov/geo/tiger/TIGER2016/FEATNAMES/tl_2016_06065_featnames.zip<br /> Resolving www2.census.gov (www2.census.gov)... 104.84.241.90, 2600:1404:a:382::208c, 2600:1404:a:39c::208c<br /> Connecting to www2.census.gov (www2.census.gov)|104.84.241.90|:80... connected.<br /> HTTP request sent, awaiting response... 403 Forbidden<br /> 2017-11-06 14:15:29 ERROR 403: Forbidden.<br /> <br /> <br /> ----------------------------------------<br /> <br /> Thank you for reaching out to us. The postgis docs is not from the Geography Division, so I cannot comment on that. A couple things that come to mind.<br /> <br /> We just released our new 2017 Shapefiles, so it's possible the scripts may be written for a previous version of our Shapefiles.<br /> You may need to clean out your cookies, restart your browser, and then attempt to reinstall.<br /> Were you able to download our Shapefiles successfully?<br /> <br /> ----------------------------------------<br /> <br /> The Shapefiles are part of the files being blocked by the 403 Forbidden error. <br /> <br /> The script is using a wget protocol to bulk download the data, so there were no cookies involved. Also, the download had been working in previous days; it was only recently that the same scripts stopped working. I am worried that our IP address somehow ended up on a blacklist for the TIGER data. Is there a blacklist for addresses that access the TIGER data?<br /> <br /> Our IP address is 128.42.44.181. <br /> <br /> ---------------------------------------<br /> <br /> I have forwarded your question on to our IT folks. Since I work on the subject matter side, I am unable to answer your questions. Once I hear back from them, I will forward their response to you. Hopefully they will provide to you what you need in order to download our Shapefiles. My apologies for your inconvenience.<br /> <br /> <br /> ----------------------------------------</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Tiger_Geocoder&diff=21910 2017-11-15T21:13:42Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Tiger Geocoder<br /> |Has owner=Peter Jalbert,<br /> |Has start date=Fall 2017<br /> |Has keywords=Tiger, Geocoder, Database<br /> |Has project status=Active<br /> |Has Image=Tiger.jpg<br /> }}<br /> This page serves as documentation for using the Tiger Geocoder on Postgres SQL, as part of the PostGIS extension. The following wiki pages may also be of use to you:<br /> <br /> [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation]<br /> [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation Database Server Documentation]<br /> <br /> The official documentation for using and installing the Tiger Geocoder can be found in the following.<br /> <br /> [https://postgis.net/docs/Extras.html#Tiger_Geocoder General Instructions]<br /> [https://postgis.net/docs/postgis_installation.html#install_tiger_geocoder_extension Installation Instructions]<br /> [http://postgis.net/docs/Geocode.html Geocoder Documentation]<br /> <br /> ==Location==<br /> The data is currently loaded into a psql database called geocoder. The tables contain the geocoding information, and there is a test table called &quot;coffeeshops&quot; that contains addresses of Houston coffeeshops according to yelp. To access the database, first login to the McNair DB Sever. Then, <br /> <br /> psql geocoder<br /> <br /> <br /> <br /> ==Installation==<br /> <br /> ===Install and Nation Data===<br /> I began by adding the extension listed above. First, enter into Postgres by using the psql command. Then:<br /> --Add Extensions to database<br /> CREATE EXTENSION postgis;<br /> CREATE EXTENSION fuzzystrmatch;<br /> CREATE EXTENSION postgis_tiger_geocoder;<br /> CREATE EXTENSION address_standardizer;<br /> <br /> You can test that the installation worked by running the following query: <br /> SELECT na.address, na.streetname,na.streettypeabbrev, na.zip<br /> FROM normalize_address('1 Devonshire Place, Boston, MA 02109') AS na;<br /> <br /> This should return the following:<br /> address | streetname | streettypeabbrev | zip<br /> ---------+------------+------------------+-------<br /> 1 | Devonshire | Pl | 02109<br /> <br /> Next, a new profile needs to be created by using the following command.<br /> INSERT INTO tiger.loader_platform(os, declare_sect, pgbin, wget, unzip_command, psql, path_sep, <br /> loader, environ_set_command, county_process_command)<br /> SELECT 'test', declare_sect, pgbin, wget, unzip_command, psql, path_sep,<br /> loader, environ_set_command, county_process_command<br /> FROM tiger.loader_platform<br /> WHERE os = 'sh';<br /> <br /> The installation instructions also provide the following note:<br /> <br /> As of PostGIS 2.4.1 the Zip code-5 digit tabulation area zcta5 load step was revised to load current zcta5 data and is part of the Loader_Generate_Nation_Script when enabled. It is turned off by default because it takes quite a bit of time to load (20 to 60 minutes), takes up quite a bit of disk space, and is not used that often.<br /> <br /> If you would like this feature, you can enable it by using the following command. This should be done before loading the script.<br /> <br /> UPDATE tiger.loader_lookuptables SET load = true WHERE table_name = 'zcta510';<br /> <br /> The paths in declare_sect need to be edited so they match our server locations. One option is to edit the declare_sect column in the tiger.loader_platform table. If so, the declare_sect looks like the following:<br /> export PGHOST=localhost +<br /> export PGUSER=postgres +<br /> export PGPASSWORD=yourpasswordhere +<br /> export PGDATABASE=geocoder +<br /> PSQL=${PGBIN}/psql +<br /> SHP2PGSQL=shp2pgsql +<br /> cd ${staging_fold} +<br /> <br /> TMPDIR=&quot;${staging_fold}/temp/&quot; +<br /> UNZIPTOOL=unzip +<br /> WGETTOOL=&quot;/usr/bin/wget&quot; +<br /> export PGBIN=/usr/lib/postgresql/9.6/bin +<br /> export PGPORT=5432 +<br /> export PGHOST=localhost +<br /> export PGUSER=postgres +<br /> export PGPASSWORD=yourpasswordhere +<br /> export PGDATABASE=geocoder +<br /> PSQL=${PGBIN}/psql +<br /> SHP2PGSQL=shp2pgsql +<br /> cd ${staging_fold}<br /> <br /> Another option is to edit the sh file before running the script. We will do this option until further notice. Simply use your favorite command line editor to change the fields to their correct values. The downloaded script is located in the following directory:<br /> /gisdata<br /> <br /> There needs to be a directory called &quot;temp&quot; in the gisdata directory. To make the script, use the following from the command line:<br /> psql -c &quot;SELECT Loader_Generate_Nation_Script('test')&quot; -d databasename -tA &gt; /gisdata/nation_script_load.sh<br /> <br /> This will create a script in the /gisdata directory. Change to that directory. If you did not edit the paths in the declare_sect table in psql, then you will need to edit this file to contain the correct paths. <br /> <br /> Change directories:<br /> cd /gisdata<br /> <br /> Edit the script using your favorite command line text editor. Specifically, edit the following fields.<br /> PGUSER=postgres<br /> PGPASSWORD=(Ask Anne for this password)!<br /> Everything else remains the same.<br /> <br /> Run the script by using: <br /> sh nation_script_load.sh<br /> <br /> Now, there is a barebones table in the database that will hold the information for the nation. Next is to download data on each state.<br /> <br /> ===State Data===<br /> <br /> The state scripts are generated in much the same way that the nation script was generated. Use the following command, substituting MA for your desired state abbreviation, and substituting a unique filename at the end.<br /> psql -c &quot;SELECT Loader_Generate_Script(ARRAY['MA'], 'test')&quot; -d geocoder -tA &gt; /gisdata/ma_load.sh<br /> <br /> CURRENT PROGRESS:<br /> The following states have been downloaded into the geocoder database.<br /> AL, AK, AZ, AR, CA, CO, CT, DE, FL, MA<br /> <br /> ===Current Errors===<br /> The state scripts stopped working on 11/1/2017 while they were working on 10/31/2017. Now, when a retrieval script is run, it draws the error <br /> HTTP request sent, awaiting response... 403 Forbidden<br /> Possible thoughts:<br /> <br /> Maybe our IP has been blacklisted for downloading data from a government website quickly? If so, PostGIS should really choose a different installation method. The current one is dumb.<br /> <br /> Maybe the nation downloader script never worked properly. Not sure how to check if it is correct or not; seems right.<br /> <br /> [https://trac.osgeo.org/postgis/ticket/3699 This] is the only online forum I could find with others who have faced a similar issue.<br /> <br /> ==Geocode Function==<br /> <br /> The official arguments for the function are the following:<br /> setof record geocode(varchar address, integer max_results=10, geometry restrict_region=NULL, norm_addy OUT addy, geometry OUT geomout, integer OUT rating);<br /> The arguments of interest are address, where you simply submit a string, and max_results, which restricts the number of geocoding attempts per address. The geocoder makes multiple guesses to the location of an address, and returns the best guesses in order. If you want multiple guesses to a specific address, then specify max_results to be more than 1.<br /> <br /> ===Single Address===<br /> <br /> An example query for a single address is:<br /> SELECT g.rating, ST_X(g.geomout) As lon, ST_Y(g.geomout) As lat,<br /> (addy).address As stno, (addy).streetname As street,<br /> (addy).streettypeabbrev As styp, (addy).location As city, (addy).stateabbrev As st,(addy).zip<br /> FROM geocode('75 State Street, Boston MA 02109', 1) As g;<br /> <br /> rating -- This is an integer that determines the confidence in the resulting geocode. The closer to 0, the more confident the guess.<br /> ST_X(g.geomout) -- This retrieves the longitude coordinate of the point.<br /> ST_Y(g.geomout) -- This retrieves the latitude corodinate of the point.<br /> addy -- In general, addy is a normalized address resulting from the input address.<br /> (addy).address -- The number of the address (Ex: &quot;75&quot; Blabla rd.)<br /> (addy).streetname -- The name of the street (Ex: 75 &quot;Blabla&quot; rd.)<br /> (addy).streettypeabbrev -- The abbreviation of the street (Ex: 75 blabla &quot;rd&quot;)<br /> (addy).location -- The city location of the address.<br /> (addy).stateabbrev -- The state abbreviation.<br /> (addy).zip -- The zipcode of the address.<br /> <br /> The output of the query above would be:<br /> rating | lon | lat | stno | street | styp | city | st | zip<br /> --------+-------------------+----------------+------+--------+------+--------+----+-------<br /> 0 | -71.0557505845646 | 42.35897920691 | 75 | State | St | Boston | MA | 02109<br /> <br /> ==Current Errors==<br /> Currently, we are getting a 403 Forbidden Error when trying to download state data. We are in contact with the US Census Bureau. Their contact information can be found [https://www.census.gov/geo/about/contact.html here].<br /> <br /> The email exchanges are recorded below.<br /> <br /> --------------------------------------<br /> --------------------------------------<br /> <br /> I am a student researcher at the McNair Center for Entrepreneurship and Innovation at Rice University, and I am in the process of installing a Postgres Extension that relies on the TIGER data. <br /> <br /> When I began installing the extension, things were working fine. Now however, I am getting a 403 Forbidden error when the script tries to download the TIGER files. Do you have any idea why this might be happening? <br /> <br /> The extension I'm trying to install is below:<br /> <br /> http://postgis.net/docs/postgis_installation.html#install_tiger_geocoder_extension<br /> <br /> When I run any of the scripts that require data from TIGER, I am receiving the following error:<br /> <br /> <br /> --2017-11-06 14:15:29-- http://www2.census.gov/geo/tiger/TIGER2016/FEATNAMES/tl_2016_06065_featnames.zip<br /> Resolving www2.census.gov (www2.census.gov)... 104.84.241.90, 2600:1404:a:382::208c, 2600:1404:a:39c::208c<br /> Connecting to www2.census.gov (www2.census.gov)|104.84.241.90|:80... connected.<br /> HTTP request sent, awaiting response... 403 Forbidden<br /> 2017-11-06 14:15:29 ERROR 403: Forbidden.<br /> <br /> <br /> ----------------------------------------<br /> <br /> Thank you for reaching out to us. The postgis docs is not from the Geography Division, so I cannot comment on that. A couple things that come to mind.<br /> <br /> We just released our new 2017 Shapefiles, so it's possible the scripts may be written for a previous version of our Shapefiles.<br /> You may need to clean out your cookies, restart your browser, and then attempt to reinstall.<br /> Were you able to download our Shapefiles successfully?<br /> <br /> ----------------------------------------<br /> <br /> The Shapefiles are part of the files being blocked by the 403 Forbidden error. <br /> <br /> The script is using a wget protocol to bulk download the data, so there were no cookies involved. Also, the download had been working in previous days; it was only recently that the same scripts stopped working. I am worried that our IP address somehow ended up on a blacklist for the TIGER data. Is there a blacklist for addresses that access the TIGER data?<br /> <br /> Our IP address is 128.42.44.181. <br /> <br /> ---------------------------------------<br /> <br /> I have forwarded your question on to our IT folks. Since I work on the subject matter side, I am unable to answer your questions. Once I hear back from them, I will forward their response to you. Hopefully they will provide to you what you need in order to download our Shapefiles. My apologies for your inconvenience.<br /> <br /> <br /> ----------------------------------------</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Tiger_Geocoder&diff=21908 2017-11-15T21:05:42Z

<p>Peterjalbert: </p> <hr /> <div>{{McNair Projects<br /> |Has title=Tiger Geocoder<br /> |Has Image=Tiger.jpg<br /> |Has owner=Peter Jalbert,<br /> |Has start date=Fall 2017<br /> |Has keywords=Tiger, Geocoder, Database<br /> }}<br /> <br /> This page serves as documentation for using the Tiger Geocoder on Postgres SQL, as part of the PostGIS extension. The following wiki pages may also be of use to you:<br /> <br /> [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation]<br /> [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation Database Server Documentation]<br /> <br /> The official documentation for using and installing the Tiger Geocoder can be found in the following.<br /> <br /> [https://postgis.net/docs/Extras.html#Tiger_Geocoder General Instructions]<br /> [https://postgis.net/docs/postgis_installation.html#install_tiger_geocoder_extension Installation Instructions]<br /> [http://postgis.net/docs/Geocode.html Geocoder Documentation]<br /> <br /> ==Location==<br /> The data is currently loaded into a psql database called geocoder. The tables contain the geocoding information, and there is a test table called &quot;coffeeshops&quot; that contains addresses of Houston coffeeshops according to yelp. To access the database, first login to the McNair DB Sever. Then, <br /> <br /> psql geocoder<br /> <br /> <br /> <br /> ==Installation==<br /> <br /> ===Install and Nation Data===<br /> I began by adding the extension listed above. First, enter into Postgres by using the psql command. Then:<br /> --Add Extensions to database<br /> CREATE EXTENSION postgis;<br /> CREATE EXTENSION fuzzystrmatch;<br /> CREATE EXTENSION postgis_tiger_geocoder;<br /> CREATE EXTENSION address_standardizer;<br /> <br /> You can test that the installation worked by running the following query: <br /> SELECT na.address, na.streetname,na.streettypeabbrev, na.zip<br /> FROM normalize_address('1 Devonshire Place, Boston, MA 02109') AS na;<br /> <br /> This should return the following:<br /> address | streetname | streettypeabbrev | zip<br /> ---------+------------+------------------+-------<br /> 1 | Devonshire | Pl | 02109<br /> <br /> Next, a new profile needs to be created by using the following command.<br /> INSERT INTO tiger.loader_platform(os, declare_sect, pgbin, wget, unzip_command, psql, path_sep, <br /> loader, environ_set_command, county_process_command)<br /> SELECT 'test', declare_sect, pgbin, wget, unzip_command, psql, path_sep,<br /> loader, environ_set_command, county_process_command<br /> FROM tiger.loader_platform<br /> WHERE os = 'sh';<br /> <br /> The installation instructions also provide the following note:<br /> <br /> As of PostGIS 2.4.1 the Zip code-5 digit tabulation area zcta5 load step was revised to load current zcta5 data and is part of the Loader_Generate_Nation_Script when enabled. It is turned off by default because it takes quite a bit of time to load (20 to 60 minutes), takes up quite a bit of disk space, and is not used that often.<br /> <br /> If you would like this feature, you can enable it by using the following command. This should be done before loading the script.<br /> <br /> UPDATE tiger.loader_lookuptables SET load = true WHERE table_name = 'zcta510';<br /> <br /> The paths in declare_sect need to be edited so they match our server locations. One option is to edit the declare_sect column in the tiger.loader_platform table. If so, the declare_sect looks like the following:<br /> export PGHOST=localhost +<br /> export PGUSER=postgres +<br /> export PGPASSWORD=yourpasswordhere +<br /> export PGDATABASE=geocoder +<br /> PSQL=${PGBIN}/psql +<br /> SHP2PGSQL=shp2pgsql +<br /> cd ${staging_fold} +<br /> <br /> TMPDIR=&quot;${staging_fold}/temp/&quot; +<br /> UNZIPTOOL=unzip +<br /> WGETTOOL=&quot;/usr/bin/wget&quot; +<br /> export PGBIN=/usr/lib/postgresql/9.6/bin +<br /> export PGPORT=5432 +<br /> export PGHOST=localhost +<br /> export PGUSER=postgres +<br /> export PGPASSWORD=yourpasswordhere +<br /> export PGDATABASE=geocoder +<br /> PSQL=${PGBIN}/psql +<br /> SHP2PGSQL=shp2pgsql +<br /> cd ${staging_fold}<br /> <br /> Another option is to edit the sh file before running the script. We will do this option until further notice. Simply use your favorite command line editor to change the fields to their correct values. The downloaded script is located in the following directory:<br /> /gisdata<br /> <br /> There needs to be a directory called &quot;temp&quot; in the gisdata directory. To make the script, use the following from the command line:<br /> psql -c &quot;SELECT Loader_Generate_Nation_Script('test')&quot; -d databasename -tA &gt; /gisdata/nation_script_load.sh<br /> <br /> This will create a script in the /gisdata directory. Change to that directory. If you did not edit the paths in the declare_sect table in psql, then you will need to edit this file to contain the correct paths. <br /> <br /> Change directories:<br /> cd /gisdata<br /> <br /> Edit the script using your favorite command line text editor. Specifically, edit the following fields.<br /> PGUSER=postgres<br /> PGPASSWORD=(Ask Anne for this password)!<br /> Everything else remains the same.<br /> <br /> Run the script by using: <br /> sh nation_script_load.sh<br /> <br /> Now, there is a barebones table in the database that will hold the information for the nation. Next is to download data on each state.<br /> <br /> ===State Data===<br /> <br /> The state scripts are generated in much the same way that the nation script was generated. Use the following command, substituting MA for your desired state abbreviation, and substituting a unique filename at the end.<br /> psql -c &quot;SELECT Loader_Generate_Script(ARRAY['MA'], 'test')&quot; -d geocoder -tA &gt; /gisdata/ma_load.sh<br /> <br /> CURRENT PROGRESS:<br /> The following states have been downloaded into the geocoder database.<br /> AL, AK, AZ, AR, CA, CO, CT, DE, FL, MA<br /> <br /> ===Current Errors===<br /> The state scripts stopped working on 11/1/2017 while they were working on 10/31/2017. Now, when a retrieval script is run, it draws the error <br /> HTTP request sent, awaiting response... 403 Forbidden<br /> Possible thoughts:<br /> <br /> Maybe our IP has been blacklisted for downloading data from a government website quickly? If so, PostGIS should really choose a different installation method. The current one is dumb.<br /> <br /> Maybe the nation downloader script never worked properly. Not sure how to check if it is correct or not; seems right.<br /> <br /> [https://trac.osgeo.org/postgis/ticket/3699 This] is the only online forum I could find with others who have faced a similar issue.<br /> <br /> ==Geocode Function==<br /> <br /> The official arguments for the function are the following:<br /> setof record geocode(varchar address, integer max_results=10, geometry restrict_region=NULL, norm_addy OUT addy, geometry OUT geomout, integer OUT rating);<br /> The arguments of interest are address, where you simply submit a string, and max_results, which restricts the number of geocoding attempts per address. The geocoder makes multiple guesses to the location of an address, and returns the best guesses in order. If you want multiple guesses to a specific address, then specify max_results to be more than 1.<br /> <br /> ===Single Address===<br /> <br /> An example query for a single address is:<br /> SELECT g.rating, ST_X(g.geomout) As lon, ST_Y(g.geomout) As lat,<br /> (addy).address As stno, (addy).streetname As street,<br /> (addy).streettypeabbrev As styp, (addy).location As city, (addy).stateabbrev As st,(addy).zip<br /> FROM geocode('75 State Street, Boston MA 02109', 1) As g;<br /> <br /> rating -- This is an integer that determines the confidence in the resulting geocode. The closer to 0, the more confident the guess.<br /> ST_X(g.geomout) -- This retrieves the longitude coordinate of the point.<br /> ST_Y(g.geomout) -- This retrieves the latitude corodinate of the point.<br /> addy -- In general, addy is a normalized address resulting from the input address.<br /> (addy).address -- The number of the address (Ex: &quot;75&quot; Blabla rd.)<br /> (addy).streetname -- The name of the street (Ex: 75 &quot;Blabla&quot; rd.)<br /> (addy).streettypeabbrev -- The abbreviation of the street (Ex: 75 blabla &quot;rd&quot;)<br /> (addy).location -- The city location of the address.<br /> (addy).stateabbrev -- The state abbreviation.<br /> (addy).zip -- The zipcode of the address.<br /> <br /> The output of the query above would be:<br /> rating | lon | lat | stno | street | styp | city | st | zip<br /> --------+-------------------+----------------+------+--------+------+--------+----+-------<br /> 0 | -71.0557505845646 | 42.35897920691 | 75 | State | St | Boston | MA | 02109<br /> <br /> ==Current Errors==<br /> Currently, we are getting a 403 Forbidden Error when trying to download state data. We are in contact with the US Census Bureau. Their contact information can be found [https://www.census.gov/geo/about/contact.html here].<br /> <br /> The email exchanges are recorded below.<br /> <br /> --------------------------------------<br /> --------------------------------------<br /> <br /> I am a student researcher at the McNair Center for Entrepreneurship and Innovation at Rice University, and I am in the process of installing a Postgres Extension that relies on the TIGER data. <br /> <br /> When I began installing the extension, things were working fine. Now however, I am getting a 403 Forbidden error when the script tries to download the TIGER files. Do you have any idea why this might be happening? <br /> <br /> The extension I'm trying to install is below:<br /> <br /> http://postgis.net/docs/postgis_installation.html#install_tiger_geocoder_extension<br /> <br /> When I run any of the scripts that require data from TIGER, I am receiving the following error:<br /> <br /> <br /> --2017-11-06 14:15:29-- http://www2.census.gov/geo/tiger/TIGER2016/FEATNAMES/tl_2016_06065_featnames.zip<br /> Resolving www2.census.gov (www2.census.gov)... 104.84.241.90, 2600:1404:a:382::208c, 2600:1404:a:39c::208c<br /> Connecting to www2.census.gov (www2.census.gov)|104.84.241.90|:80... connected.<br /> HTTP request sent, awaiting response... 403 Forbidden<br /> 2017-11-06 14:15:29 ERROR 403: Forbidden.<br /> <br /> <br /> ----------------------------------------<br /> <br /> Thank you for reaching out to us. The postgis docs is not from the Geography Division, so I cannot comment on that. A couple things that come to mind.<br /> <br /> We just released our new 2017 Shapefiles, so it's possible the scripts may be written for a previous version of our Shapefiles.<br /> You may need to clean out your cookies, restart your browser, and then attempt to reinstall.<br /> Were you able to download our Shapefiles successfully?<br /> <br /> ----------------------------------------<br /> <br /> The Shapefiles are part of the files being blocked by the 403 Forbidden error. <br /> <br /> The script is using a wget protocol to bulk download the data, so there were no cookies involved. Also, the download had been working in previous days; it was only recently that the same scripts stopped working. I am worried that our IP address somehow ended up on a blacklist for the TIGER data. Is there a blacklist for addresses that access the TIGER data?<br /> <br /> Our IP address is 128.42.44.181. <br /> <br /> ---------------------------------------<br /> <br /> I have forwarded your question on to our IT folks. Since I work on the subject matter side, I am unable to answer your questions. Once I hear back from them, I will forward their response to you. Hopefully they will provide to you what you need in order to download our Shapefiles. My apologies for your inconvenience.<br /> <br /> <br /> ----------------------------------------</div>

Peterjalbert http://www.edegan.com/mediawiki/index.php?title=Peter_Jalbert_(Work_Log)&diff=21907 2017-11-15T20:57:54Z

<p>Peterjalbert: </p> <hr /> <div>===Fall 2017===<br /> &lt;onlyinclude&gt; <br /> <br /> [[Peter Jalbert]] [[Work Logs]] [[Peter Jalbert (Work Log)|(log page)]] <br /> <br /> 2017-11-15: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote a script to extract counts that were greater than 2 from Keyword Matcher. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder].<br /> <br /> 2017-11-14: Continued running [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser]. Wrote an HTML to Text parser. See Parser Demo Day Page for file location. Continued downloading for [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder TIGER Geocoder].<br /> <br /> 2017-11-13: Built [http://mcnair.bakerinstitute.org/wiki/Demo_Day_Page_Parser Demo Day Page Parser].<br /> <br /> 2017-11-09: Running demo version of Demo Day crawler (Accelerator Google Crawler). Fixing work log format. <br /> <br /> 2017-11-07: Created file with 0s and 1s detailing whether crunchbase has the founder information for an accelerator. Details posted as a TODO on [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List] page. Still waiting for feedback on the PostGIS installation from [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder]. Continued working on Accelerator Google Crawler.<br /> <br /> 2017-11-06: Contacted Geography Center for the US Census Bureau, [https://www.census.gov/geo/about/contact.html here], and began email exchange on PostGIS installation problems. Began working on the [http://mcnair.bakerinstitute.org/wiki/Selenium_Documentation Selenium Documentation]. Also began working on an Accelerator Google Crawler that will be used with Yang and ML to find Demo Days for cohort companies.<br /> <br /> 2017-11-01: Attempted to continue downloading, however ran into HTTP Forbidden errors. Listed the errors on the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder Page].<br /> <br /> 2017-10-31: Began downloading blocks of data for individual states for the [http://mcnair.bakerinstitute.org/wiki/Tiger_Geocoder Tiger Geocoder] project. Wrote out the new wiki page for installation, and beginning to write documentation on usage.<br /> <br /> 2017-10-30: With Ed's help, was able to get the national data from Tiger installed onto a database server. The process required much jumping around and changing users, and all the things we learned are outlined in [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation#Editing_Users the database server documentation] under &quot;Editing Users&quot;.<br /> <br /> 2017-10-25: Continued working on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation TigerCoder Installation].<br /> <br /> 2017-10-24: Throw some addresses into a database, use address normalizer and geocoder. May need to install things. Details on the installation process can be found on the [http://mcnair.bakerinstitute.org/wiki/PostGIS_Installation PostGIS Installation page].<br /> <br /> 2017-10-23: Finished Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-19: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_District Houston Innovation District Project].<br /> <br /> 2017-10-18: Continued work on Yelp crawler for [http://mcnair.bakerinstitute.org/wiki/Houston_Innovation_DistrictHouston Innovation District Project].<br /> <br /> 2017-10-17: Constructed ArcGIS maps for the agglomeration project. Finished maps of points for every year in the state of California. Finished maps of Route 128. Began working on selenium Yelp crawler to get cafe locations within the 610-loop.<br /> <br /> 2017-10-16: Assisted Harrison on the USITC project. Looked for natural language processing tools to extract complaintants and defendants along with their location from case files. Experimented with pulling based on parts of speech tags, as well as using geotext or geograpy to pull locations from a case segment.<br /> <br /> 2017-10-13: Updated various project wiki pages.<br /> <br /> 2017-10-12: Continued work on Patent Thicket project, awaiting further project specs.<br /> <br /> 2017-10-05: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-04: Emergency ArcGIS creation for Agglomeration project.<br /> <br /> 2017-10-02: Worked on ArcGIS data. See Harrison's Work Log for the details.<br /> <br /> 2017-09-28: Added collaborative editing feature to PyCharm. <br /> <br /> 2017-09-27: Worked on big database file.<br /> <br /> 2017-09-25: New task -- Create text file with company, description, and company type.<br /> #[http://mcnair.bakerinstitute.org/wiki/VC_Database_Rebuild VC Database Rebuild]<br /> #psql vcdb2<br /> #table name, sdccompanybasecore2<br /> #Combine with Crunchbasebulk<br /> <br /> #TODO: Write wiki on linkedin crawler, write wiki on creating accounts.<br /> <br /> 2017-09-21: Wrote wiki on Linkedin crawler, met with Laura about patents project. <br /> <br /> 2017-09-20: Finished running linkedin crawler. Transferred data to RDP. Will write wikis next.<br /> <br /> 2017-09-19: Began running linkedin crawler. Helped Yang create RDP account, get permissions, and get wiki setup.<br /> <br /> 2017-09-18: Finished implementation of Experience Crawler, continued working on Education Crawler for LinkedIn.<br /> <br /> 2017-09-14: Continued implementing LinkedIn Crawler for profiles.<br /> <br /> 2017-09-13: Implemented LinkedIn Crawler for main portion of profiles. Began working on crawling Experience section of profiles.<br /> <br /> 2017-09-12: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. Added to the wiki on this topic.<br /> <br /> 2017-09-11: Continued working on the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler for Accelerator Founders Data]. <br /> <br /> 2017-09-06: Combined founders data retrieved with the Crunchbase API with the crunchbasebulk data to get linkedin urls for different accelerator founders. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-09-05: Post Harvey. Finished retrieving names from the Crunchbase API on founders. Next step is to query crunchbase bulk database to get linkedin urls. For more information, see [http://mcnair.bakerinstitute.org/wiki/Crunchbase_Data here].<br /> <br /> 2017-08-24: Began using the Crunchbase API to retrieve founder information for accelerators. Halfway through compiling a dictionary that translates accelerator names into proper Crunchbase API URLs.<br /> <br /> 2017-08-23: Decided with Ed to abandon LinkedIn crawling to retrieve accelerator founder data, and instead use crunchbase. Spent the day navigating the crunchbasebulk database, and seeing what useful information was contained in it.<br /> <br /> 2017-08-22: Discovered that LinkedIn Profiles cannot be viewed through LinkedIn if the target is 3rd degree or further. However, if entering LinkedIn through a Google search, the profile can still be viewed if the user has previously logged into LinkedIn. Devising a workaround crawler that utilizes Google search. Continued blog post [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) here] under Section 4.<br /> <br /> 2017-08-21: Began work on extracting founders for accelerators through LinkedIn Crawler. Discovered that Python3 is not installed on RDP, so the virtual environment for the project cannot be fired up. Continued working on Ubuntu machine.<br /> &lt;/onlyinclude&gt; <br /> <br /> ===Spring 2017===<br /> <br /> 2017-05-01: Continued work on HTML Parser. Uploaded all semester projects to git server.<br /> <br /> 2017-04-20: Finished the HTML Parser for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Ran HTML parser on accelerator founders. Data is stored in projects/accelerators/LinkedIn Founder Data.<br /> <br /> 2017-04-19: Made updates to the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler] Wikipage. Ran LinkedIn Crawler on accelerator data. Working on an html parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-18: Ran LinkedIn Crawler on matches between Crunchbase Snapshot and the accelerator data.<br /> <br /> 2017-04-17: Worked on ways to get correct search results from the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Worked on an HTML Parser for the results from the LinkedIn Crawler.<br /> <br /> 2017-04-13: Worked on debugging the logout procedure for the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Began formulation of process to search for founders of startups using a combination of the LinkedIn Crawler with the data resources from the [http://mcnair.bakerinstitute.org/wiki/Crunchbase_2013_Snapshot CrunchBase Snapshot].<br /> <br /> 2017-04-12: Work on bugs with the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler].<br /> <br /> 2017-04-11: Completed functional [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) crawler of LinkedIn Recruiter Pro]. Basic search functions work and download profile information for a given person. <br /> <br /> 2017-04-10: Began writing functioning crawler of LinkedIn. <br /> <br /> 2017-04-06: Continued working on debugging and documenting the [http://mcnair.bakerinstitute.org/wiki/LinkedIn_Crawler_(Python) LinkedIn Crawler]. Wrote a test program that logs in, searches for a query, navigates through search pages, and logs out. Recruiter program can now login and search.<br /> <br /> 2017-04-05: Began work on the LinkedIn Crawler. Researched on launching Python Virtual Environment.<br /> <br /> 2017-04-03: Finished debugging points for the Enclosing Circle Algorithm. Added Command Line functionality to the Industry Classifier.<br /> <br /> 2017-03-29: Worked on debugging points for the Enclosing Circle Algorithm.<br /> <br /> 2017-03-28: Finished running the Enclosing Circle Algorithm. Worked on removing incorrect points from the data set(see above). <br /> <br /> 2017-03-27: Worked on debugging the Enclosing Circle Algorithm. Implemented a way to remove interior circles, and determined that translation to latitude and longitude coordinates resulted in slightly off center circles.<br /> <br /> 2017-03-23: Finished debugging the brute force algorithm for [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm]. Implemented a method to plot the points and circles on a graph. Analyzed runtime of the brute force algorithm.<br /> <br /> 2017-03-21: Coded a brute force algorithm for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-20: Worked on debugging the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-03-09: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Finished script to draw Enclosing Circles on a Google Map.<br /> <br /> 2017-03-08: Continued running [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities. Created script to draw outcome of the Enclosing Circle Algorithm on Google Maps.<br /> <br /> 2017-03-07: Redetermined the top 50 cities which Enclosing Circle should be run on. Data on the [http://mcnair.bakerinstitute.org/wiki/Top_Cities_for_VC_Backed_Companies Top 50 Cities for VC Backed Companies can be found here.] Ran [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] on the Top 50 Cities.<br /> <br /> 2017-03-06: Ran script to determine the top 50 cities which Enclosing Circle should be run on. Fixed the VC Circles script to take in a new data format.<br /> <br /> 2017-03-02: Cleaned up data for the VC Circles Project. Created histogram of data in Excel. See [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Began work on the [http://mcnair.bakerinstitute.org/wiki/LinkedInCrawlerPython LinkedIn Crawler].<br /> <br /> 2017-03-01: Created statistics for the VC Circles Project.<br /> <br /> 2017-02-28: Finished downloading geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Found bug in Enclosing Circle Algorithm.<br /> <br /> 2017-02-27: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-23: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Installed C++ Compiler for Python. Ran tests on difference between Python and C wrapped Python.<br /> <br /> 2017-02-22: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Helped out with [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-21: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Researched into C++ Compilers for Python so that the Enclosing Circle Algorithm could be wrapped in C. Found a recommended one [https://www.microsoft.com/en-us/download/details.aspx?id=44266 here].<br /> <br /> 2017-02-20: Continued to download geocoded data for VC Data as part of the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] Project. Assisted work on the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier].<br /> <br /> 2017-02-16: Reworked [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] to create a file of geocoded data. Began work on wrapping the algorithm in C to improve speed.<br /> <br /> 2017-02-15: Finished [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm] applied to the VC study. Enclosing Circle algorithm still needs adjustment, but the program runs with the temporary fixes.<br /> <br /> 2017-02-14: Worked on the application of the Enclosing Circle algorithm to the VC study. Working on bug fixes in the Enclosing Circle algorithm. Created wiki page for the [http://mcnair.bakerinstitute.org/wiki/Enclosing_Circle_Algorithm Enclosing Circle Algorithm].<br /> <br /> 2017-02-13: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-08: Worked on Neural Net for the [http://mcnair.bakerinstitute.org/wiki/Industry_Classifier Industry Classifier Project].<br /> <br /> 2017-02-07: Fixed bugs in parse_cohort_data.py, the script for parsing the cohort data from the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Added descriptive statistics to cohort data excel file.<br /> <br /> 2017-02-02: Out sick, independent research and work from RDP. Brief research into the [http://jorgeg.scripts.mit.edu/homepage/wp-content/uploads/2016/03/Guzman-Stern-State-of-American-Entrepreneurship-FINAL.pdf Stern-Guzman algorithm]. Research into [http://mcnair.bakerinstitute.org/wiki/interactive_maps Interactive Maps]. No helpful additions to map embedding problem.<br /> <br /> 2017-02-01: Notes from Session with Ed: Project on US university patenting and entrepreneurship programs (writing code to identify universities in assignees), search Wikipedia (XML then bulk download), student pop, faculty pop, etc.<br /> Circle project for VC data will end up being a joint project to join accelerator data. <br /> Pull descriptions for VC. Founders of accelerators in linkedin. LinkedIn cannot be caught(pretend to not be a bot). Can eventually get academic backgrounds through linkedin. <br /> Pull business registration data, Stern/Guzman Algorithm. <br /> GIS ontop of geocoded data.<br /> Maps that works on wiki or blog (CartoDB), Maps API and R.<br /> NLP Projects, Description Classifier.<br /> <br /> 2017-01-31: Built WayBack Machine Crawler. Updated documentation for coordinates script. Updated profile page to include locations of code.<br /> <br /> <br /> 2017-01-30: Optimized enclosing circle algorithm through memoization. Developed script to read addresses from accelerator data and return latitude and longitude coordinates.<br /> <br /> 2017-01-26: Continued working on Google sitesearch project. Discovered crunchbase, changed project priority. Priority 1, split accelerator data up by flag, priority 2, use crunchbase to get web urls for cohorts, priority 3, make internet archive wayback machine driver. Located [http://mcnair.bakerinstitute.org/wiki/Whois_Parser Whois Parser].<br /> <br /> <br /> 2017-01-25: Finished parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Some data files still need proofreading as they are not in an acceptable format. Began working on Google sitesearch project.<br /> <br /> 2017-01-24: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Cohort data file created, debugging is almost complete. Will begin work on the google accelerator search soon.<br /> <br /> 2017-01-23: Worked on parser for cohort data of the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Preliminary code is written, working on debugging.<br /> <br /> 2017-01-19: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project]. Created parser for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project], completed creation of final data set(yay!). Began working on cohort parser.<br /> <br /> 2017-01-18: Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-13: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-12: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-11: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> 2017-01-10: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Downloaded pdfs in the background for the [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Government Crawler Project].<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> ===Fall 2016===<br /> <br /> 2016-09-26: Set up Staff wiki page, work log page; registered for Slack, Microsoft Remote Desktop; downloaded Selenium on personal computer, read Selenium docs. Created wiki page for Moroccan Web Driver Project.<br /> <br /> 2016-09-29: Re-enroll in Microsoft Remote Desktop with proper authentication, set up Selenium environment and Komodo IDE on Remote Desktop, wrote program using Selenium that goes to a link and opens up the print dialog box. Developed computational recipe for a different approach to the problem.<br /> <br /> 2016-09-30: Selenium program selects view pdf option from the website, and goes to the pdf webpage. Program then switches handle to the new page. CTRL S is sent to the page to launch save dialog window. Text cannot be sent to this window. Brainstorm ways around this issue. Explored Chrome Options for saving automatically without a dialog window. Looking into other libraries besides selenium that may help.<br /> <br /> 2016-10-03: Moroccan Web Driver projects completed for driving of the Monarchy proposed bills, the House of Representatives proposed bills, and the Ratified bills sites. Begun process of devising a naming system for the files that does not require scraping. Tinkered with naming through regular expression parsing of the URL. Structure for the oral questions and written questions drivers is set up, but need fixes due to the differences in the sites. Fixed bug on McNair wiki for women's biz team where email was plain text instead of an email link. Took a glimpse at Kuwait Parliament website, and it appears to be very different from the Moroccan setup.<br /> <br /> 2016-10-06: Discussed with Dr. Elbadawy about the desired file names for Moroccan data download. The consensus was that the bill programs are ready to launch once the files can be named properly, and the questions data must be retrieved using a web crawler which I need to learn how to implement. The naming of files is currently drawing errors in going from arabic, to url, to download, to filename. Debugging in process. Also built a demo selenium program for Dr. Egan that drives the McNair blog site on an infinite loop.<br /> <br /> 2016-10-07: Learned unicode and utf8 encoding and decoding in arabic. Still working on transforming an ascii url into printable unicode.<br /> <br /> 2016-10-11: Fixed arabic bug, files can now be saved with arabic titles. Monarchy bills downloaded and ready for shipment. House of Representatives Bill mostly downloaded, ratified bills prepared for download. Started learning scrapy library in python for web scraping. Discussed idea of screenshot-ing questions instead of scraping. <br /> <br /> 2016-10-13: Completed download of Moroccan Bills. Working on either a web driver screenshot approach or a webcrawler approach to download the Moroccan oral and written questions data. Began building Web Crawler for Oral and Written Questions site. Edited Moroccan Web Driver/Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-14: Finished Oral Questions crawler. Finished Written Questions crawler. Waiting for further details on whether that data needs to be tweaked in any way. Updated the Moroccan Web Driver/Web Crawler wiki page. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-18: Finished code for Oral Questions web driver and Written Questions web driver using selenium. Now, the data for the dates of questions can be found using the crawler, and the pdfs of the questions will be downloaded using selenium. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-20: Continued to download data for the Moroccan Parliament Written and Oral Questions. Updated Wiki page. Started working on Twitter project with Christy. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-10-21: Continued to download data for the Moroccan Parliament Written and Oral Questions. Looked over [http://mcnair.bakerinstitute.org/wiki/Christy_Warden_(Twitter_Crawler_Application_1) Christy's Twitter Crawler] to see how I can be helpful. Dr. Egan asked me to think about how to potentially make multiple tools to get cohorts and other sorts of data from accelerator sites. See [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator List] He also asked me to look at the [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] for potential ideas on how to bring this project to fruition.<br /> <br /> 2016-11-01: Continued to download Moroccan data in the background. Went over code for GovTracker Web Crawler, continued learning Perl. [http://mcnair.bakerinstitute.org/wiki/Govtrack_Webcrawler_(Wiki_Page) GovTrack Web Crawler] Began Kuwait Web Crawler/Driver.<br /> <br /> 2016-11-03: Continued to download Moroccan data in the background. Dr. Egan fixed systems requirements to run the GovTrack Web Crawler. Made significant progress on the Kuwait Web Crawler/Driver for the Middle East Studies Department. <br /> <br /> 2016-11-04: Continued to download Moroccan data in the background. Finished writing initial Kuwait Web Crawler/Driver for the Middle East Studies Department. Middle East Studies Department asked for additional embedded files in the Kuwait website. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-08: Continued to download Moroccan data in the background. Finished writing code for the embedded files on the Kuwait Site. Spent time debugging the frame errors due to the dynamically generated content. Never found an answer to the bug, and instead found a workaround that sacrificed run time for the ability to work. [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Moroccan Web Driver]<br /> <br /> 2016-11-10: Continued to download Moroccan data and Kuwait data in the background. Began work on [http://mcnair.bakerinstitute.org/wiki/Google_Scholar_Crawler Google Scholar Crawler]. Wrote a crawler for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] to get the HTML files of hundreds of accelerators. The crawler ended up failing; it appears to have been due to HTTPS. <br /> <br /> 2016-11-11: Continued to download Moroccan data in the background. Attempted to find bug fixes for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Project] crawler.<br /> <br /> 2016-11-15: Finished download of Moroccan Written Question pdfs. Wrote a parser with Christy to be used for parsing bills from Congress and eventually executive orders. Found bug in the system Python that was worked out and rebooted.<br /> <br /> 2016-11-17: Wrote a crawler to retrieve information about executive orders, and their corresponding pdfs. They can be found [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report here.] Next step is to run code to convert the pdfs to text files, then use the parser fixed by Christy. <br /> <br /> 2016-11-18: Converted Executive Order PDFs to text files using adobe acrobat DC. See [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Wikipage] for details.<br /> <br /> 2016-11-22: Transferred downloaded Morocco Written Bills to provided SeaGate Drive. Made a &quot;gentle&quot; F6S crawler to retrieve HTMLs of possible accelerator pages documented [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here].<br /> <br /> 2016-11-29: Began pulling data from the accelerators listed [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) here]. Made text files for about 18 accelerators.<br /> <br /> 2016-12-01: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project]. Built tool for the [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report E&amp;I Governance Report Project] with Christy. Adds a column of data that shows whether or not the bill has been passed.<br /> <br /> 2016-12-02: Built and ran web crawler for Center for Middle East Studies on Kuwait. Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-06: Learned how to use git. Committed software projects from the semester to the McNair git repository. Projects can be found at; [http://mcnair.bakerinstitute.org/wiki/E%26I_Governance_Policy_Report Executive Order Crawler], [http://mcnair.bakerinstitute.org/wiki/Moroccan_Parliament_Web_Crawler Foreign Government Web Crawlers], [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) F6S Crawler and Parser].<br /> <br /> 2016-12-07: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> 2016-12-08: Continued making text files for the [http://mcnair.bakerinstitute.org/wiki/Accelerator_Seed_List_(Data) Accelerator Seed List project].<br /> <br /> '''Notes'''<br /> <br /> *Ed moved the Morocco Data to E:\McNair\Projects from C:\Users\PeterJ\Documents<br /> * C Drive files moved to E:\McNair\Users\PeterJ<br /> <br /> [[Category:Work Log]]</div>

Peterjalbert