7,612
edits
Changes
Jump to navigation
Jump to search
== Example Usage ==
Parallel Enclosing Circle Algorithm (view source)
Revision as of 13:47, 21 September 2020
, 13:47, 21 September 2020no edit summary
{{Project|Has project output=Tool|Has sponsor=McNair ProjectsCenter
|Has title=Parallel Enclosing Circle Algorithm
|Has owner=Oliver Chang,
A thin-wrapper around [[Enclosing_Circle_Algorithm_(Rework)|the enclosing circle algorithm]] which allows for instance-level parallelization.
This project consists of the python files in <code>E:\McNair\Projects\OliverLovesCircles\src\python</code>. There is another version of the project with plotting functionality that uses a slightly different approach (removes duplicate points and uses their counts before running the algorithm) in <code>E:\McNair\Projects\KyranLovesCircles\src\python</code>.
Parallelization is implemented via Python2's [https://docs.python.org/2/library/subprocess.html#subprocess.Popen <code>subprocess.open()</code>] which is non-blocking and available in the standard library.
* in <code>circles.py</code>:
** <code>PATH_SEPARATOR</code>: the string that separates parts of the filename for both input and output files. For example, an input could look like "St. Louis#MO#2017#0.tsv" for PATH_SEPARATOR = '#'
** <code>ITERATIONS</code>: the number of iterations to attempt for each <code>k</code> to find minimum for that <code>k</code>
** <code>MIN_POINTS_PER_CIRCLE</code> (AKA <code>n</code>): the minimum number of data points that must be included in a circle
** <code>TIMEOUT_MINUTES</code>: maximum running time of a parallel instance of the algorithm
** <code>SPLIT_THRESHOLD</code>: if a dataset has more than this threshold of data points, it will be split via k-means
** <code>EXECUTABLE_INSTANCE_PATH</code>: the path to circles.py
** <code>OUTJOINER_INSTANCE_PATH</code>: the path to outjoiner.py
** <code>DATA_DIRECTORY</code>: the input directory
** <code>OUTPUT_DIRECTORY</code>: the directory to write the outputs of circle.py to
** <code>GENERATE_REPORTS</code>: whether or not to call outjoiner.py (writes reports on the output of circles.py)
** <code>REPORT_DIRECTORY</code>: the directory to write reports to
== Structure and Usage ==
==== vc_circles.py ====
*What it does
**If given a "master file" through argument infile, splits it into constituent data files, and stores them in DATA_DIRECTORY
**Takes data files in DATA_DIRECTORY and calls circles.py in parallel for each of these data files, which writes its output files to OUTPUT_DIRECTORY
**Takes output files in OUTPUT_DIRECTORY and calls outjoiner.py, which writes its report files to REPORT_DIRECTORY
*Command Line Arguments
**--sweep-time overwrites <code>SWEEP_CYCLE_SECONDS</code>
**--instances overwrites <code>NUMBER_INSTANCES</code>
**--min_points overwrites <code>MIN_POINTS_PER_CIRCLE</code>
**--infile: Path to large master file, e.g. CirclesTestData.txt
**--split-out overwrites <code>DATA_DIRECTORY</code>
**--out overwrites <code>OUTPUT_DIRECTORY</code>
**--report overwrites <code>REPORT_DIRECTORY</code>
==== circles.py ====
*What it does
**Called with two command line arguments, the input path and the output path
**Calculates points and circles for input and writes it to output
==== outjoiner.py ====
*What it does
**Using a given output directory, generates three files: circles.tsv, points.tsv, and summary.tsv, and stores them in a given reports directory
==== DATA_DIRECTORY ====
*The format of the filenames in this directory are <code>{city}{sep}{state}{sep}{year}{sep}{num}.tsv</code> where <code>num</code> is a 0-indexed integer of a split city/state/year <code>infile</code> that has greater than <code>SPLIT_THRESHOLD</code>.
*These are files created when vc_circles.py splits up a master file.
==== OUTPUT_DIRECTORY====
*The format of the filenames in this directory are <code>{city}{sep}{state}{sep}{year}{sep}{num}.tsv</code> where <code>num</code> is a 0-indexed integer of a split city/state/year <code>infile</code> that has greater than <code>SPLIT_THRESHOLD</code>.
*These are files created when circles.py processes a file from DATA_DIRECTORY.
==== REPORT_DIRECTORY====
*There are three files in this directory: circles.tsv, points.tsv, and summary.tsv.
=== Example Usage ===
==== Splitting a master file and running ====
<nowiki>
$ python vc_circles.py --infile E:/McNair/Projects/OliverLovesCircles/CoLevelForCirclesNotRunGTE200.txt</nowiki>
<code>placestate, place, statecode, year, latitude, longitude, coname, datefirstinv, placens, geoid, city</code>
This command will populate (and overwrite) any files in <code>data/</code> and , <code>reportsout/</code>. The format of the filenames in this directory are , and <code>{city}{sep}{state}{sep}{year}{sep}{num}.tsvreports/</code> where . ==== Running on already split files ==== <codenowiki>num$ python vc_circles.py</codenowiki> is a 0-indexed integer of a split city/state/year This command will populate (and overwrite) any files in <code>infileout/</code> that has greater than and <code>SPLIT_THRESHOLDreports/</code>.
== Bugs/Issues ==
# Some place/state/year combinations do not run to completion regardless of how tractable the number of points
# How to merge small enclosing circles? This is a better measure of agglomeration regardless
# How to separate outliers? # Sometimes circles with 0 radius are created# enclosingcirclealg() returns None sometimes == Makeshift way to plot circles == # Connect to database with command <code>psql -U postgres arc</code># password is tabspaceenter I think# <code>\d</code> lists tables# Now run SQL script LoadCircles.sql in OliverLovesCircles# Open ArcMap# Add data -> Top of file tree -> Database connection -> localhost for instance, database arc -> connect to localhost and table testcirclegeom# Add points from local files, make sure they are txt or tab files, not tsv, or they won't be found# Points -> Properties -> Source -> Set data source -> x field: long, y field: lat == St. Louis bug ==[[File:st_louis_bug.png]] This image shows a rendering of the results of running St. Louis. There are four circles (the centers of circles are green dots), but two have radii of 0.0. === Progress on the bug ===# Removing duplicate points from the data actually removes all of the errors, but this doesn't give you the solution with the smallest area.# I tried removing duplicates but keeping track of a "count." # I narrowed down the bug to the constrained_kmeans method in ckmeans.py [https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-2000-65.pdf (paper here)]## For some reason, this returns clusters with smaller numbers of points than n## [http://adared.ch/constrained-k-means-implementation-in-python/ This is a good overview of the algorithm]# I wrote a plotter, the plot method in circles.py
== Related Pages ==
