Govtrack Webcrawler - Revision history

imported>Sahil: Finished up documentation on how webcrawler was made

2016-02-01T22:26:21Z

Finished up documentation on how webcrawler was made

imported>Sahil at 21:47, 25 January 2016

2016-01-25T21:47:03Z

imported>Sahil: New page: The goal of the Govtrack Webcrawler is to create and automated system in perl by which bills relevant to a certian topic can be pulled from the Govtrack API which can be found [https://www...

2016-01-25T21:45:59Z

New page: The goal of the Govtrack Webcrawler is to create and automated system in perl by which bills relevant to a certian topic can be pulled from the Govtrack API which can be found [https://www...

New page

The goal of the Govtrack Webcrawler is to create and automated system in perl by which bills relevant to a certian topic can be pulled from the Govtrack API which can be found [https://www.govtrack.us/api/v2/bill?congress=114&order_by=-current_status_date here].

==Process==

In order to perform this task several libraries are used most of these libraries come with [http://www.activestate.com/activeperl ActivePerl] but we are also using
[http://search.cpan.org/~mlehmann/JSON-XS-3.01/XS.pm JSON::XS] in order to make parsing the JSON data simpler. The LWP::UserAgent and HTTP::Request libraries are used to pull data from the API.
use strict;
use [http://search.cpan.org/~ether/libwww-perl-6.15/lib/LWP/UserAgent.pm LWP::UserAgent];
use [http://search.cpan.org/~ether/HTTP-Message-6.11/lib/HTTP/Request.pm HTTP::Request];
use JSON;
Next the useragent object is created.
my $ua = new LWP::UserAgent;
Now the parameters used to search the api are decided. Currently we are searching for the 107 bills related to entrepreneurship during the 114th congress.
my $queryName = "Entrepreneurship";
my $congressNo = "114";
my $limit = "107";
Using these parameters the url can be constructed.
my $genUrl = "https://www.govtrack.us/api/v2/bill?order_by=-current_status_date&congress=". $congressNo."&q=".$queryName."&limit=".$limit;
The useragent object can now retrieve and decode the text from the url into JSON data.
my $genResponse = $ua->get($genUrl);
my $genContent=$genResponse->decoded_content;
After getting the resulting JSON Data bill id's can now be located. Using the ids another page can be found which contains more specific information about each bill. From this bill specific page the tags of each bill can be used to determine whether or not the bill is relevant and should be reviewed by Mcnair Center staff. Currently tags that are considered relevant:

*Commerce: ID 5914
*Business Investment and Capital: ID 5918
*Small Business: ID 5935
*Small Business Administration: ID 6769

← Older revision		Revision as of 21:47, 25 January 2016
Line 1:		Line 1:
−	The goal of the Govtrack Webcrawler is to create and automated system in ~~perl~~ by which bills relevant to a certian topic can be pulled from the Govtrack API which can be found [https://www.govtrack.us/api/v2/bill?congress=114&order_by=-current_status_date here].	+	The goal of the Govtrack Webcrawler is to create and automated system in ActivePerl by which bills relevant to a certian topic can be pulled from the Govtrack API which can be found [https://www.govtrack.us/api/v2/bill?congress=114&order_by=-current_status_date here].

	==Process==		==Process==

@@ Line 11: / Line 11: @@
 Next the useragent object is created.
   my $ua = new LWP::UserAgent;
-Now the parameters used to search the api are decided. Currently we are searching for the 107 bills related to entrepreneurship during the 114th congress.
+Now the parameters used to search the api are decided.
-  my $queryName = "Entrepreneurship";
+  my $queryName = "''insert query here''";
-  my $congressNo = "114";
+  my $congressNo = "''insert congress num here''";
-  my $limit = "107";
+  my $limit = "''insert maximum number of bills to search here''";
 Using these parameters the url can be constructed.
   my $genUrl = "https://www.govtrack.us/api/v2/bill?order_by=-current_status_date&congress=". $congressNo."&q=".$queryName."&limit=".$limit;
-The useragent object can now retrieve and decode the text from the url into JSON data.
+The useragent object can now retrieve and decode the text from the url into a JSON string.
   my $genResponse = $ua->get($genUrl);
   my $genContent=$genResponse->decoded_content;
-After getting the resulting JSON Data bill id's can now be located. Using the ids another page can be found which contains more specific information about each bill. From this bill specific page the tags of each bill can be used to determine whether or not the bill is relevant and should be reviewed by Mcnair Center staff. Currently tags that are considered relevant:
+Using the JSON module the JSON string can be converted into a data structure that can be searched through for relevant data.
 *Commerce: ID 5914
 *Business Investment and Capital: ID 5918
 *Small Business: ID 5935
 *Small Business Administration: ID 6769