2,798
edits
Changes
Jump to navigation
Jump to search
Created page with "{{McNair Projects |Project Title=Twitter Webcrawler (Tool) |Topic Area=Resources and Tools |Owner=Gunny Liu |Start Term=Summer 2016 |Status=Active |Deliverable=Tool |Audience=..."
{{McNair Projects
|Project Title=Twitter Webcrawler (Tool)
|Topic Area=Resources and Tools
|Owner=Gunny Liu
|Start Term=Summer 2016
|Status=Active
|Deliverable=Tool
|Audience=McNair Staff
|Keywords=Webcrawler, Database, Twitter, API, Python
|Primary Billing=AccNBER01
}}
==Description==
'''Notes''':
The Twitter Webcrawler, in its alpha version, is an expedition project involving the Twittwer API in search of a sustainable and scale-able way to excavate retweet-retweeter, favorited-favoriter following-follower relationships in the entrepreneurship Tweet-o-sphere. On the same beat, we also seek to document tweeting activities/timelines of important twitters in the same Tweet-o-sphere.
'''Input''':
Twitter database
'''Output''':
Local database documenting important timelines and relationships in the entrepreneurship Tweet-o-sphere.
==Development Notes==
===7/11: Project start===
----
*Dan wanted:
[[File:Capture 15.PNG|400px|none]]
*First-take on Twitter API [https://dev.twitter.com/overview/api Overview]
**Cumbersome API that is not directly accessible/requires great deal of configuration if one chooses to leverage e.g. <code>import requests</code> library.
***Turns out Twitter has a long controversial history wrt third-party development. There is no clean canonical interface to access its database.
***'''DO NOT attempt to access Twitter API through canonical documented methods''' - huge waste of time
***Obsolete authentication process documented - do not be use canonical documentation for Oauth procedure
*Instead, '''DO USE''' third-party developed python interfaces such as [https://github.com/bear/python-twitter python-twitter by bear] - highly recommended in hindsight
**Follow python-twitter's documented methods for authentication
**The twitter account that I am using is <code>shortname: BIPPMcNair</code> and <code>password: amount</code>
***One can obtain the consumer key, consumer secret, access key and access secret through accessing the dev portal using the account and tapping <code>TOOLS > Manage Your Apps</code> in the footer bar of the portal.
**There is '''no''' direct access to Twitter database through http://, as before, so expect to do all processing in a py dev environment.
===7/12: Grasping API===
*The [http://python-twitter.readthedocs.io/en/latest/twitter.html python-twitter library] is extremely intricate and well-synchronized
**All queries are to be launched through a <code>twitter.api.Api</code> object, which is produced by the authentication process implemented yesterday
>>> import twitter
>>> api = twitter.Api(consumer_key='consumer_key',
consumer_secret='consumer_secret',
access_token_key='access_token',
access_token_secret='access_token_secret')
**Some potentially very useful query methods are:
***<code>Api.GetUserTimeline(user_id=None, screen_name=None)</code> which returns up to 200 recent tweets of input user. Really nice that twitter database operates on something as simple as <code>screen_name</code>, which is @shortname that is v public and familiar.
***<code>Api.GetRetweeters(status_id=None)</code> and <code>Api.GetRetweets(status_id=None)</code> which identifies a tweet as a status by its status_id and spits out all the retweets that this particular tweet has undergone.
***<code>Api.GetFavorites(user_id=None)</code> which seems to satisfy our need for tracking favorited tweets
***<code>Api.GetFollowers(user_id=None, screen_name=None)</code> and <code>Api.GetFollowerIDs(user_id=None, screen_name=None)</code> which seems to be a good relationship mapping mechanism for esp. the mothernodes tweeters we care about.
===7/5: Eventbrite API First-Take===
----
*Eventbrite developer account for McNair Center:
**first name: '''Anne''', last name: '''Dayton'''
**Login Email: '''admin@mcnaircenter.org'''
**Login Password: '''amount'''
*Eventbrite API is well-documented and its database readily accessible. In the python dev environment, I am using the http <code>requests</code> library to make queries to the database, to obtain json data containing event objects that in turn contain organizer objects, venue objects, start/end time values, longitude/latitude values specific to each event. The <code>requests</code> library has inbuilt <code>.json()</code> access methods, simplifying the json reading/writing process. Bang.
**In querying for events organized by techstar, one of the biggest startup programs organization in the U.S., I use the following. Note that the organizer ID of techstar is 2300226659.
import requests
response = requests.get(
"https://www.eventbriteapi.com/v3/organizers/2300226659/events/",
headers = {
"Authorization": "Bearer CRAQ5MAXEGHKEXSUSWXN",
},
verify = True,
)
**In querying for, instead, keywords such as "startup weekend," I use the following.
import requests
response = requests.get(
"https://www.eventbriteapi.com/v3/events/search/q="startup weekend"",
headers = {
"Authorization": "Bearer CRAQ5MAXEGHKEXSUSWXN",
},
verify = True,
)
**In querying for events parked under the category "science and technology", I use the following. However, this query also returns scientific seminars unrelated to entrepreneurship and is yet to be refined.
**Note that the category ID of science and technology is 102.
import requests
response = requests.get(
"https://www.eventbriteapi.com/v3/categories/102",
headers = {
"Authorization": "Bearer CRAQ5MAXEGHKEXSUSWXN",
},
verify = True,
)
**In each case, var <code>response</code> is a json object, that can be read/written in python using the requests method <code>response.json()</code>. Each endpoint used above are instances of e.g. <code>GET events/search/</code> or <code>GET categories/:id</code> EventBrite API methods. There are different parameters each GET function can harness to get more specific results. To populate a comprehensive local database, the '''dream''' is to systematic queries from different endpoints and collecting all results, without repetition, in a centralized database. In order to do this, I'll have to familarize further with these GET functions and develop a systematic approach to automate queries to the eventbrite server. One way to do this is to import entrepreneurship buzzword libraries that are available on the web, and make queries by iterating through these search strings systematically.
*Eventbrite event objects in json are well-organized and consistent. There are many interesting fields such as the longitude/latitude decimals, apart from name/location/organizer/start-time/end-time data which are data we want to amass initially.
**For instance, the upcoming startup weekend event in Seville looks like the following.
[[File:Capture 12.PNG|400px|none]]
**In the events object, organizer and venue are represented as ID's and have to be queried separately since they contain a multitude of string-value pairs such as "description", "logo", and "url" in the case of organizer data. Huge opportunity here for more data extraction. Kudos to eventbrite for documenting their stuff meticulously. Can you tell I'm impressed?
**To produce a local database, I'm using the <code>import pandas as pd</code> library, the <code>pandas.DataFrame</code> object and the <code>pandas.DataFrame.to_csv()</code> method. Currently, I initialize a dataframe with columns of variables that I seek to extract, and iterate through event objects and venue/organizer objects within to populate the dataframe with rows of event data.
**'''Still debugging/writing at the moment'''.
**RDP went down, major sadness.
===7/6: Alpha Development===
----
*Eventbrite stipulates a system of ID-numbering for all organizers and venues objects, for instance.
**For the endpoint <code>GET /venues/:id/</code>, replace <code>:id</code> with the venue_id associated with desired organizer
**For the endpoint <code>GET /organizers/:id</code>, replace <code>:id</code> with the organizer_id associated with desired organizer
**Where are these ID numbers located, you ask? Any query for an event will return them as values the the strings "venue_id" and "organizer_id"
*Script development slowed considerably by lack of modularity and debugging functionality
**Modules to generate query url strings from input GET
**Module to create empty <code>pandas.DataFrame</code> table based on input rows and columns
**Modules to retrieve information from venues and organizer data from their respective ID numbers
**To learn and operate komodo debugger and write appropriate tests for each modules detached from main driver function
**To learn pandas.DataFrame and appropriate methods to update it
*'''Notes and Ideas'''
**Develop smart iteration to query for all events sought
:::To create intelligent searches:
:::Note that eventbrite is esp good for free events
:::Note that past events may extend only to a certain point
:::Note that eventbrite was launched in 2006, but is the first major player in online event ticketing
:::Category is always science and tech
:::Organiser is impt; some entrepreneurship events are organised by known collectives
:::Organiser description also has many impt keywords
:::keywords from SEO material on [[marketing artfully]] is very good
:::Event series, dates and venues endpoints are secondarily important
===7/7 Alpha Development #2===
----
*Full swing: pseudo-code, modularity, docstrings, tests, naming style
*Komodo debugger works
*Alpha development complete. All tests passed. Complete code as below.
https://github.com/scroungemyvibe/mcnair_center_builds/blob/master/EventBrite_Webcrawler_Build.py
*'''Notes'''
**Current query (without input parameters) by organizer ID returns only active events listed under organizer. For instance, techstars has 45 upcoming events and I am pulling 45 json event objects from the database.
**Current build should be applied systematically to lists of organizer_id's
**Further build ideas/notes documented in code proper on the git
|Project Title=Twitter Webcrawler (Tool)
|Topic Area=Resources and Tools
|Owner=Gunny Liu
|Start Term=Summer 2016
|Status=Active
|Deliverable=Tool
|Audience=McNair Staff
|Keywords=Webcrawler, Database, Twitter, API, Python
|Primary Billing=AccNBER01
}}
==Description==
'''Notes''':
The Twitter Webcrawler, in its alpha version, is an expedition project involving the Twittwer API in search of a sustainable and scale-able way to excavate retweet-retweeter, favorited-favoriter following-follower relationships in the entrepreneurship Tweet-o-sphere. On the same beat, we also seek to document tweeting activities/timelines of important twitters in the same Tweet-o-sphere.
'''Input''':
Twitter database
'''Output''':
Local database documenting important timelines and relationships in the entrepreneurship Tweet-o-sphere.
==Development Notes==
===7/11: Project start===
----
*Dan wanted:
[[File:Capture 15.PNG|400px|none]]
*First-take on Twitter API [https://dev.twitter.com/overview/api Overview]
**Cumbersome API that is not directly accessible/requires great deal of configuration if one chooses to leverage e.g. <code>import requests</code> library.
***Turns out Twitter has a long controversial history wrt third-party development. There is no clean canonical interface to access its database.
***'''DO NOT attempt to access Twitter API through canonical documented methods''' - huge waste of time
***Obsolete authentication process documented - do not be use canonical documentation for Oauth procedure
*Instead, '''DO USE''' third-party developed python interfaces such as [https://github.com/bear/python-twitter python-twitter by bear] - highly recommended in hindsight
**Follow python-twitter's documented methods for authentication
**The twitter account that I am using is <code>shortname: BIPPMcNair</code> and <code>password: amount</code>
***One can obtain the consumer key, consumer secret, access key and access secret through accessing the dev portal using the account and tapping <code>TOOLS > Manage Your Apps</code> in the footer bar of the portal.
**There is '''no''' direct access to Twitter database through http://, as before, so expect to do all processing in a py dev environment.
===7/12: Grasping API===
*The [http://python-twitter.readthedocs.io/en/latest/twitter.html python-twitter library] is extremely intricate and well-synchronized
**All queries are to be launched through a <code>twitter.api.Api</code> object, which is produced by the authentication process implemented yesterday
>>> import twitter
>>> api = twitter.Api(consumer_key='consumer_key',
consumer_secret='consumer_secret',
access_token_key='access_token',
access_token_secret='access_token_secret')
**Some potentially very useful query methods are:
***<code>Api.GetUserTimeline(user_id=None, screen_name=None)</code> which returns up to 200 recent tweets of input user. Really nice that twitter database operates on something as simple as <code>screen_name</code>, which is @shortname that is v public and familiar.
***<code>Api.GetRetweeters(status_id=None)</code> and <code>Api.GetRetweets(status_id=None)</code> which identifies a tweet as a status by its status_id and spits out all the retweets that this particular tweet has undergone.
***<code>Api.GetFavorites(user_id=None)</code> which seems to satisfy our need for tracking favorited tweets
***<code>Api.GetFollowers(user_id=None, screen_name=None)</code> and <code>Api.GetFollowerIDs(user_id=None, screen_name=None)</code> which seems to be a good relationship mapping mechanism for esp. the mothernodes tweeters we care about.
===7/5: Eventbrite API First-Take===
----
*Eventbrite developer account for McNair Center:
**first name: '''Anne''', last name: '''Dayton'''
**Login Email: '''admin@mcnaircenter.org'''
**Login Password: '''amount'''
*Eventbrite API is well-documented and its database readily accessible. In the python dev environment, I am using the http <code>requests</code> library to make queries to the database, to obtain json data containing event objects that in turn contain organizer objects, venue objects, start/end time values, longitude/latitude values specific to each event. The <code>requests</code> library has inbuilt <code>.json()</code> access methods, simplifying the json reading/writing process. Bang.
**In querying for events organized by techstar, one of the biggest startup programs organization in the U.S., I use the following. Note that the organizer ID of techstar is 2300226659.
import requests
response = requests.get(
"https://www.eventbriteapi.com/v3/organizers/2300226659/events/",
headers = {
"Authorization": "Bearer CRAQ5MAXEGHKEXSUSWXN",
},
verify = True,
)
**In querying for, instead, keywords such as "startup weekend," I use the following.
import requests
response = requests.get(
"https://www.eventbriteapi.com/v3/events/search/q="startup weekend"",
headers = {
"Authorization": "Bearer CRAQ5MAXEGHKEXSUSWXN",
},
verify = True,
)
**In querying for events parked under the category "science and technology", I use the following. However, this query also returns scientific seminars unrelated to entrepreneurship and is yet to be refined.
**Note that the category ID of science and technology is 102.
import requests
response = requests.get(
"https://www.eventbriteapi.com/v3/categories/102",
headers = {
"Authorization": "Bearer CRAQ5MAXEGHKEXSUSWXN",
},
verify = True,
)
**In each case, var <code>response</code> is a json object, that can be read/written in python using the requests method <code>response.json()</code>. Each endpoint used above are instances of e.g. <code>GET events/search/</code> or <code>GET categories/:id</code> EventBrite API methods. There are different parameters each GET function can harness to get more specific results. To populate a comprehensive local database, the '''dream''' is to systematic queries from different endpoints and collecting all results, without repetition, in a centralized database. In order to do this, I'll have to familarize further with these GET functions and develop a systematic approach to automate queries to the eventbrite server. One way to do this is to import entrepreneurship buzzword libraries that are available on the web, and make queries by iterating through these search strings systematically.
*Eventbrite event objects in json are well-organized and consistent. There are many interesting fields such as the longitude/latitude decimals, apart from name/location/organizer/start-time/end-time data which are data we want to amass initially.
**For instance, the upcoming startup weekend event in Seville looks like the following.
[[File:Capture 12.PNG|400px|none]]
**In the events object, organizer and venue are represented as ID's and have to be queried separately since they contain a multitude of string-value pairs such as "description", "logo", and "url" in the case of organizer data. Huge opportunity here for more data extraction. Kudos to eventbrite for documenting their stuff meticulously. Can you tell I'm impressed?
**To produce a local database, I'm using the <code>import pandas as pd</code> library, the <code>pandas.DataFrame</code> object and the <code>pandas.DataFrame.to_csv()</code> method. Currently, I initialize a dataframe with columns of variables that I seek to extract, and iterate through event objects and venue/organizer objects within to populate the dataframe with rows of event data.
**'''Still debugging/writing at the moment'''.
**RDP went down, major sadness.
===7/6: Alpha Development===
----
*Eventbrite stipulates a system of ID-numbering for all organizers and venues objects, for instance.
**For the endpoint <code>GET /venues/:id/</code>, replace <code>:id</code> with the venue_id associated with desired organizer
**For the endpoint <code>GET /organizers/:id</code>, replace <code>:id</code> with the organizer_id associated with desired organizer
**Where are these ID numbers located, you ask? Any query for an event will return them as values the the strings "venue_id" and "organizer_id"
*Script development slowed considerably by lack of modularity and debugging functionality
**Modules to generate query url strings from input GET
**Module to create empty <code>pandas.DataFrame</code> table based on input rows and columns
**Modules to retrieve information from venues and organizer data from their respective ID numbers
**To learn and operate komodo debugger and write appropriate tests for each modules detached from main driver function
**To learn pandas.DataFrame and appropriate methods to update it
*'''Notes and Ideas'''
**Develop smart iteration to query for all events sought
:::To create intelligent searches:
:::Note that eventbrite is esp good for free events
:::Note that past events may extend only to a certain point
:::Note that eventbrite was launched in 2006, but is the first major player in online event ticketing
:::Category is always science and tech
:::Organiser is impt; some entrepreneurship events are organised by known collectives
:::Organiser description also has many impt keywords
:::keywords from SEO material on [[marketing artfully]] is very good
:::Event series, dates and venues endpoints are secondarily important
===7/7 Alpha Development #2===
----
*Full swing: pseudo-code, modularity, docstrings, tests, naming style
*Komodo debugger works
*Alpha development complete. All tests passed. Complete code as below.
https://github.com/scroungemyvibe/mcnair_center_builds/blob/master/EventBrite_Webcrawler_Build.py
*'''Notes'''
**Current query (without input parameters) by organizer ID returns only active events listed under organizer. For instance, techstars has 45 upcoming events and I am pulling 45 json event objects from the database.
**Current build should be applied systematically to lists of organizer_id's
**Further build ideas/notes documented in code proper on the git
