7,612
edits
Changes
Jump to navigation
Jump to search
== Structure of Program ==
* TODO
Parallel Enclosing Circle Algorithm (view source)
Revision as of 13:47, 21 September 2020
, 13:47, 21 September 2020no edit summary
{{Project|Has project output=Tool|Has sponsor=McNair ProjectsCenter
|Has title=Parallel Enclosing Circle Algorithm
|Has owner=Oliver Chang,
A thin-wrapper around [[Enclosing_Circle_Algorithm_(Rework)|the enclosing circle algorithm]] which allows for instance-level parallelization.
This project consists of the python files in <code>E:\McNair\Projects\OliverLovesCircles\src\python</code>. There is another version of the project with plotting functionality that uses a slightly different approach (removes duplicate points and uses their counts before running the algorithm) in <code>E:\McNair\Projects\KyranLovesCircles\src\python</code>.
Parallelization is implemented via Python2's [https://docs.python.org/2/library/subprocess.html#subprocess.Popen <code>subprocess.open()</code>] which is non-blocking and available in the standard library.
This algorithm has terrible time-performance characteristics, so we make the assumption that we can divide a large number of points with k-means and then solve those subproblems.
In other words, we make the simplifying assumption that the Enclosing Circle Algorithm has [https://en.wikipedia.org/wiki/Optimal_substructure Optimal Substructure].
== Parameters ==
== Structure and Usage == ==== vc_circles.py ==== *What it does**If given a "master file" through argument infile, splits it into constituent data files, and stores them in DATA_DIRECTORY**Takes data files in DATA_DIRECTORY and calls circles.py in parallel for each of these data files, which writes its output files to OUTPUT_DIRECTORY**Takes output files in OUTPUT_DIRECTORY and calls outjoiner.py, which writes its report files to REPORT_DIRECTORY *Command Line Arguments**--sweep-time overwrites <code>SWEEP_CYCLE_SECONDS</code>**--instances overwrites <code>NUMBER_INSTANCES</code>**--min_points overwrites <code>MIN_POINTS_PER_CIRCLE</code>**--infile: Path to large master file, e.g. CirclesTestData.txt**--split-out overwrites <code>DATA_DIRECTORY</code>**--out overwrites <code>OUTPUT_DIRECTORY</code>**--report overwrites <code>REPORT_DIRECTORY</code> ==== circles.py ==== *What it does **Called with two command line arguments, the input path and the output path**Calculates points and circles for input and writes it to output ==== outjoiner.py ==== *What it does **Using a given output directory, generates three files: circles.tsv, points.tsv, and summary.tsv, and stores them in a given reports directory ==== DATA_DIRECTORY ====*The format of the filenames in this directory are <code>{city}{sep}{state}{sep}{year}{sep}{num}.tsv</code> where <code>num</code> is a 0-indexed integer of a split city/state/year <code>infile</code> that has greater than <code>SPLIT_THRESHOLD</code>. *These are files created when vc_circles.py splits up a master file.==== OUTPUT_DIRECTORY====*The format of the filenames in this directory are <code>{city}{sep}{state}{sep}{year}{sep}{num}.tsv</code> where <code>num</code> is a 0-indexed integer of a split city/state/year <code>infile</code> that has greater than <code>SPLIT_THRESHOLD</code>. *These are files created when circles.py processes a file from DATA_DIRECTORY.==== REPORT_DIRECTORY====*There are three files in this directory: circles.tsv, points.tsv, and summary.tsv. === Example Usage ===
==== Splitting a master file and running ====
<nowiki>
$ python vc_circles.py --infile E:/McNair/Projects/OliverLovesCircles/CoLevelForCirclesNotRunGTE200.txt</nowiki>
<code>placestate, place, statecode, year, latitude, longitude, coname, datefirstinv, placens, geoid, city</code>
This command will populate (and overwrite) any files in <code>data/</code> and , <code>reportsout/</code>. The format of the filenames in this directory are , and <code>{city}{sep}{state}{sep}{year}{sep}{num}.tsvreports/</code> where . ==== Running on already split files ==== <codenowiki>num$ python vc_circles.py</codenowiki> is a 0-indexed integer of a split city/state/year This command will populate (and overwrite) any files in <code>infileout/</code> that has greater than and <code>SPLIT_THRESHOLDreports/</code>.
== Bugs/Issues ==
# Some place/state/year combinations do not run to completion regardless of how tractable the number of points
# How to merge small enclosing circles? This is a better measure of agglomeration regardless
# How to separate outliers? # Sometimes circles with 0 radius are created# enclosingcirclealg() returns None sometimes
== Plotting Circles Makeshift way to plot circles ==
# Connect to database with command <code>psql -U postgres arc</code>
# password is tabspaceenter I think
# Open ArcMap
# Add data -> Top of file tree -> Database connection -> localhost for instance, database arc -> connect to localhost and table testcirclegeom
# Add points from local files, make sure they are txt or tab files, not tsv, or they won't be found# Points -> Properties -> Source -> Set data source -> x field: long, y field: lat == St. Louis bug ==[[File:st_louis_bug.png]] This image shows a rendering of the results of running St. Louis. There are four circles (the centers of circles are green dots), but two have radii of 0.0. === Progress on the bug ===# Removing duplicate points from the data actually removes all of the errors, but this doesn't give you the solution with the smallest area.# I tried removing duplicates but keeping track of a "count." # I narrowed down the bug to the constrained_kmeans method in ckmeans.py [https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-2000-65.pdf (paper here)]## For some reason, this returns clusters with smaller numbers of points than n## [http://adared.ch/constrained-k-means-implementation-in-python/ This is a good overview of the algorithm]# I wrote a plotter, the plot method in circles.py
== Related Pages ==
