Govtrack Webcrawler (Tool)

Revision as of 13:47, 21 September 2020 by Ed (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Govtrack Webcrawler (Tool)
Project logo 02.png
Project Information
Has title Govtrack Webcrawler (Tool)
Has start date Spring 2016
Has deadline date
Has keywords Webcrawler, database, Tool
Has project status Complete
Has sponsor McNair Center
Has project output Tool
Copyright © 2019 All Rights Reserved.


The goal of the Govtrack Webcrawler is to create and automated system in ActivePerl by which bills relevant to a certian topic can be pulled from the Govtrack API which can be found here.


In order to perform this task several libraries are used most of these libraries come with ActivePerl but we are also using JSON::XS in order to make parsing the JSON data simpler. The LWP::UserAgent and HTTP::Request libraries are used to pull data from the API.

use strict;
use LWP::UserAgent;
use HTTP::Request;
use JSON;

Next the useragent object is created.

my $ua = new LWP::UserAgent;

Now the parameters used to search the api are decided.

my $queryName = "insert query here";
my $congressNo = "insert congress num here";
my $limit = "insert maximum number of bills to search here";

Using these parameters the url can be constructed.

my $genUrl = "". $congressNo."&q=".$queryName."&limit=".$limit;

The useragent object can now retrieve and decode the text from the url into a JSON string.

my $genResponse = $ua->get($genUrl);
my $genContent=$genResponse->decoded_content;

Using the JSON module the JSON string can be converted into a data structure that can be searched through for relevant data.

my $JSONcontent = JSON::XS::decode_json($genResponse->decoded_content);

After getting the resulting JSON Data an array of all the bills matching the parameters bill can be located at


Iterating over this array more specific bill information can be found by constructing a url using the bill's ID number as such,

my $billurl = "" . $bill->{id};

After finding this page the data can once again be parsed from a string into a data structure.

my $billresponse = $ua->get($billurl);
my $billcontent = JSON::XS::decode_json($billresponse->decoded_content);

From this bill specific page the tags of each bill can be used to determine whether or not the bill is relevant and should be reviewed by Mcnair Center staff. An array of these tags can be found at


The data retrieved from this search is then placed into several tab delimited text files containing sets of useful information about bills deemed relevant. Currently tags that are considered relevant:

  • Commerce: ID 5914
  • Business Investment and Capital: ID 5918
  • Intellectual Property: ID 5927
  • Small Business: ID 5935
  • Advanced Technology and Technological Innovations: ID 6294
  • Computers and Information Tech: ID 6300
  • Small Business Administration: ID 6769

how to download bulk data from govtrack website

rsync -avz --delete --delete-excluded --include='document.txt' --include='*/' --exclude='*' 

See also