http://www.edegan.com/mediawiki/api.php?action=feedcontributions&user=WeiWu&feedformat=atomedegan.com - User contributions [en]2024-03-29T11:23:32ZUser contributionsMediaWiki 1.34.2http://www.edegan.com/mediawiki/index.php?title=Wei_Wu(Work_log)&diff=23913Wei Wu(Work log)2018-08-03T21:08:10Z<p>WeiWu: /* Summer 2018 */</p>
<hr />
<div>==Notes from Ed==<br />
<br />
Detail about the install/config/basic use of software on the db server is on the [[Database Server Documentation]] documentation page.<br />
<br />
Please build and link to project pages that describe what you have done to date!<br />
<br />
==Summer 2018==<br />
<br />
<onlyinclude><br />
[[Wei Wu]] [[Work Logs]] [[Wei Wu(Work log)|(log page)]] <br><br />
<br />
2018-06-11 <br><br />
*Set up wiki page and RDP for work. Installed CUDA on dbserver. Waiting for Matlab and Gurobi to be <br />
installed on the dbserver (or I will do it myself later this week). <br />
*Started looking at the paper in progress [http://mcnair.bakerinstitute.org/wiki/Estimating_Unobserved_Complementarities_between_Entrepreneurs_and_Venture_Capitalists ''Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists''] and its related Matlab code. <br />
*Began looking for ways to incorporate CUDA parallel computation with Matlab and Julia. <br />
<br />
2018-06-12 <br><br />
*Installed tightVNC on dbserver by following steps 1-3 from this [https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-vnc-on-ubuntu-14-04 tightVNC tutorial]<br />
<br />
*Tested connection via localhost port. <br />
<br />
*Matlab matching code with Ed, James, and Chenyu via Skype.<br />
<br />
2018-06-13 <br><br />
*Continued searching for a method to set up vnc for dbserver without ssh. <br />
*Started moving the selenium box, monitors, keyboards, etc, from Room 310. <br />
*Matlab matching code with Ed, James, and Chenyu via Skype. <br />
<br />
2018-06-18 <br><br />
Trained on using PostgreSQL for DBServer. <br />
<br />
2018-06-19 <br><br />
Further training with SQL and SDC Platinum. Job assignment among team members. <br />
<br />
2018-06-20 <br> <br />
*Started reading a short tutorial on GMM and its implementation[https://www.kevinsheppard.com/images/5/55/Chapter6.pdf][https://ocw.mit.edu/courses/economics/14-385-nonlinear-econometric-analysis-fall-2007/lecture-notes/lec13_gmm.pdf]. Should have a good grasp before the end of the week. <br />
*[http://www.gurobi.com/documentation/8.0/quickstart_windows.pdf Gurobi interface guide]<br />
*It seems that Gurobi does not support GPGPU computation here in [http://www.gurobi.com/pdfs/webinar-parallel-and-distributed-optimization-english.pdf page 36], and here is [https://groups.google.com/forum/?fromgroups#!searchin/gurobi/gpu/gurobi/KTP6zDvodII/oPPQT4-mofMJ a slightly more elaborate communication] between the engineering director of Gurobi and the community regarding GPGPU computation support. Need to figure out how to do parallel computation in Matlab[https://www.mathworks.com/discovery/matlab-gpu.html][https://www.mathworks.com/help/distcomp/getting-started-with-parallel-computing-toolbox.html], and where we need it in the Startup-VC Code. <br />
</onlyinclude><br />
<br />
2018-06-21 <br><br />
*Huge problem with code '''gmm_2stage_estimated.m'''. In line 80, we compute ''W'' by taking the inverse of matrix ''Om''. We kept getting ''W'' as an ill-conditioned matrix, whose entries are infinitely large. There might be a bug in the readjusted code. I will try to catch it by comparing with the original code. If I can't, will try to set up another skype phone call with Chenyu.<br />
'''Update''': This might be related to the bug reported in the [http://mcnair.bakerinstitute.org/wiki/Estimating_Unobserved_Complementarities_between_Entrepreneurs_and_Venture_Capitalists_Matlab_Code#Bugs Matlab Code page]. I also don't think the fix was correct. I will look into that.<br />
<br />
2018-06-22 <br><br />
*Want to test Matlab and its parallel computing toolbox on DBServer. Cannot use the Matlab GUI remotely. This is possible due to the environment variable setting for remote access. '''Update''': now we have Matlab GUI. Nvidia CUDA is configured correctly as well. Today is a good day for a Linux user. <br />
*Probably it's the right time to further configure the VNC server on DBServer. [https://www.tightvnc.com/vncserver.1.php Documentation for TightVNC configuration]. '''Done'''. Documented in the [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation Database Server Documentation] <br />
<br />
2018-06-25/26 <br><br />
Looking at quick tutorials for C/C++ and CUDA, in case that I will need to read CUDA code in the future. <br />
[[Category:Work Log]]<br />
<br />
2018-06-27 <br><br />
Sick. Working from home.<br />
<br />
2018-06-28 <br><br />
Emailed Chenyu about the bug in '''gmm_2stage_estimated.m'''.<br />
<br />
2018-06-29 <br><br />
Set up a meeting with Jeremy to talk about the Matlab code and paper.<br />
<br />
2018-07-02 <br><br />
*Meeting with Jeremy. <br />
*Here is a [http://hpc.fs.uni-lj.si/sites/default/files/HPC_for_dummies.pdf gentle introduction to HPC] for myself (and whoever might benefit from it), even though this was published in 2009 and by AMD (hence it is less likely to have recent information on GPU computing).<br />
*This recent paper [https://www.researchgate.net/publication/308967833_GPU_Computing_Applied_to_Linear_and_Mixed_Integer_Programming GPU Computing Applied to Linear and Mixed Integer Programming] should be helpful. It summarizes recent advancement in GPU computing in the OR community.<br />
<br />
2018-07-03 <br><br />
Finally learned from Chenyu that the "bug" reported on June 21 was not a bug at all. We are getting singular matrices because we are using too small R and monte_M.<br />
<br />
2018-07-05 <br><br />
*There is still problems with W being singular. I have changed R and monte_M to be as big as in the original code. Either I neglected something, or there is still a bug. Or perhaps it's just normal. See [[File:a copy of warning messages from Matlab command windows.pdf]]. When W is singular, this leads to fitness function for the second stage ga being minimized to negative infinity. <br />
*I am trying to put gurobi into a parfor in Matlab. So far, not good. Wanted to figure out how to do CPU-based parallel computing with Gurobi. I cannot find a way to run Gurobi solvers inside a parfor. I believe Matlab's linprog can, but linprog is much slower than Gurobi. There will be some trade off. I need to test this.<br />
<br />
2018-07-06<br />
I really need to understand the code better. Also we probably can run Gurobi inside parfor, but I need to wrap it around inside a function.<br />
<br />
2018-07-09<br />
I have run profiling on the Matlab code several times. It seems that moments.m takes up as much time as calling Gurobi to solve LPs. Probably we should optimize moments.m instead.<br />
<br />
[[File:profiling.png]]<br />
<br />
2018-07-10<br />
Ran profiling again. With big enough R, the parallel code is much faster. Documented in the project [[Matlab, CUDA, and GPU Computing]]<br><br />
<br />
2018-07-11<br />
'''Croatia 2-1 England!!!!!!!!!!!!!!!!!!!!!'''<br />
<br />
2018-07-12<br />
*Helped Minh install Tensorflow on DB Server.<br />
*Learned to use NOTS.<br />
<br />
2018-07-13<br />
*Further parallelize Matlab code (msmf_corr_coeff.m). Now on our 12 cores server, one call to msmf_corr_coeff takes about 35 seconds for R=200, monte_M = 70, mktsize = 30. <br />
*Will try to parallelize moments.m. Currently it takes 10 seconds per call. This in included in the 35 seconds runtime of msmf_corr_coeff.m.<br />
*Reverted back to using Matlab's linprog rather than Gurobi. In a parfor, gurobi takes much longer than the native linprog to solve our LPs. I do not fully understand why this is happening. It might be the way that Gurobi was called inside a function that increases the overhead, or it might be that Gurobi couldn't utilize the full power of our CPU since all 12 cores have been scheduled to work on 12 different LPs (some LP algorithm of Gurobi has parallelism). Note that creating a model for Gurobi takes time. <br />
[[FILE:msmf35seconds.png]]<br />
<br><br />
2018-07-16<br />
*Run monte_data mode with R=200, monte_M = 70, mktsize = 30. The msmf was computed fairly fast.<br />
[[FILE:msmf_monte_data.png]]<br />
*Run data mode with R=200, monte_M=70, mktsize = 30.<br />
[[FILE:msmf_data.png]]<br />
<br><br />
2018-07-18<br />
*Run monte mode with R=200, monte_M = 200, mktsize = 30.<br />
*Helped Maxine with industrial classifier.<br />
*Worked on documentations for NOTS and parallelization <br />
*Worked on running matlab code on NOTS<br />
<br />
2018-07-19 ~ 30<br />
Ran diagnostics requested by Chenyu<br />
<br />
2018-08<br />
Help Marcus get familiar with the Matlab code for matching VCs to startups.</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Wei_Wu(Work_log)&diff=23912Wei Wu(Work log)2018-08-03T21:07:11Z<p>WeiWu: /* Summer 2018 */</p>
<hr />
<div>==Notes from Ed==<br />
<br />
Detail about the install/config/basic use of software on the db server is on the [[Database Server Documentation]] documentation page.<br />
<br />
Please build and link to project pages that describe what you have done to date!<br />
<br />
==Summer 2018==<br />
<br />
<onlyinclude><br />
[[Wei Wu]] [[Work Logs]] [[Wei Wu(Work log)|(log page)]] <br><br />
<br />
2018-06-11 <br><br />
*Set up wiki page and RDP for work. Installed CUDA on dbserver. Waiting for Matlab and Gurobi to be <br />
installed on the dbserver (or I will do it myself later this week). <br />
*Started looking at the paper in progress [http://mcnair.bakerinstitute.org/wiki/Estimating_Unobserved_Complementarities_between_Entrepreneurs_and_Venture_Capitalists ''Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists''] and its related Matlab code. <br />
*Began looking for ways to incorporate CUDA parallel computation with Matlab and Julia. <br />
<br />
2018-06-12 <br><br />
*Installed tightVNC on dbserver by following steps 1-3 from this [https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-vnc-on-ubuntu-14-04 tightVNC tutorial]<br />
<br />
*Tested connection via localhost port. <br />
<br />
*Matlab matching code with Ed, James, and Chenyu via Skype.<br />
<br />
2018-06-13 <br><br />
*Continued searching for a method to set up vnc for dbserver without ssh. <br />
*Started moving the selenium box, monitors, keyboards, etc, from Room 310. <br />
*Matlab matching code with Ed, James, and Chenyu via Skype. <br />
<br />
2018-06-18 <br><br />
Trained on using PostgreSQL for DBServer. <br />
<br />
2018-06-19 <br><br />
Further training with SQL and SDC Platinum. Job assignment among team members. <br />
<br />
2018-06-20 <br> <br />
*Started reading a short tutorial on GMM and its implementation[https://www.kevinsheppard.com/images/5/55/Chapter6.pdf][https://ocw.mit.edu/courses/economics/14-385-nonlinear-econometric-analysis-fall-2007/lecture-notes/lec13_gmm.pdf]. Should have a good grasp before the end of the week. <br />
*[http://www.gurobi.com/documentation/8.0/quickstart_windows.pdf Gurobi interface guide]<br />
*It seems that Gurobi does not support GPGPU computation here in [http://www.gurobi.com/pdfs/webinar-parallel-and-distributed-optimization-english.pdf page 36], and here is [https://groups.google.com/forum/?fromgroups#!searchin/gurobi/gpu/gurobi/KTP6zDvodII/oPPQT4-mofMJ a slightly more elaborate communication] between the engineering director of Gurobi and the community regarding GPGPU computation support. Need to figure out how to do parallel computation in Matlab[https://www.mathworks.com/discovery/matlab-gpu.html][https://www.mathworks.com/help/distcomp/getting-started-with-parallel-computing-toolbox.html], and where we need it in the Startup-VC Code. <br />
</onlyinclude><br />
<br />
2018-06-21 <br><br />
*Huge problem with code '''gmm_2stage_estimated.m'''. In line 80, we compute ''W'' by taking the inverse of matrix ''Om''. We kept getting ''W'' as an ill-conditioned matrix, whose entries are infinitely large. There might be a bug in the readjusted code. I will try to catch it by comparing with the original code. If I can't, will try to set up another skype phone call with Chenyu.<br />
'''Update''': This might be related to the bug reported in the [http://mcnair.bakerinstitute.org/wiki/Estimating_Unobserved_Complementarities_between_Entrepreneurs_and_Venture_Capitalists_Matlab_Code#Bugs Matlab Code page]. I also don't think the fix was correct. I will look into that.<br />
<br />
2018-06-22 <br><br />
*Want to test Matlab and its parallel computing toolbox on DBServer. Cannot use the Matlab GUI remotely. This is possible due to the environment variable setting for remote access. '''Update''': now we have Matlab GUI. Nvidia CUDA is configured correctly as well. Today is a good day for a Linux user. <br />
*Probably it's the right time to further configure the VNC server on DBServer. [https://www.tightvnc.com/vncserver.1.php Documentation for TightVNC configuration]. '''Done'''. Documented in the [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation Database Server Documentation] <br />
<br />
2018-06-25/26 <br><br />
Looking at quick tutorials for C/C++ and CUDA, in case that I will need to read CUDA code in the future. <br />
[[Category:Work Log]]<br />
<br />
2018-06-27 <br><br />
Sick. Working from home.<br />
<br />
2018-06-28 <br><br />
Emailed Chenyu about the bug in '''gmm_2stage_estimated.m'''.<br />
<br />
2018-06-29 <br><br />
Set up a meeting with Jeremy to talk about the Matlab code and paper.<br />
<br />
2018-07-02 <br><br />
*Meeting with Jeremy. <br />
*Here is a [http://hpc.fs.uni-lj.si/sites/default/files/HPC_for_dummies.pdf gentle introduction to HPC] for myself (and whoever might benefit from it), even though this was published in 2009 and by AMD (hence it is less likely to have recent information on GPU computing).<br />
*This recent paper [https://www.researchgate.net/publication/308967833_GPU_Computing_Applied_to_Linear_and_Mixed_Integer_Programming GPU Computing Applied to Linear and Mixed Integer Programming] should be helpful. It summarizes recent advancement in GPU computing in the OR community.<br />
<br />
2018-07-03 <br><br />
Finally learned from Chenyu that the "bug" reported on June 21 was not a bug at all. We are getting singular matrices because we are using too small R and monte_M.<br />
<br />
2018-07-05 <br><br />
*There is still problems with W being singular. I have changed R and monte_M to be as big as in the original code. Either I neglected something, or there is still a bug. Or perhaps it's just normal. See [[File:a copy of warning messages from Matlab command windows.pdf]]. When W is singular, this leads to fitness function for the second stage ga being minimized to negative infinity. <br />
*I am trying to put gurobi into a parfor in Matlab. So far, not good. Wanted to figure out how to do CPU-based parallel computing with Gurobi. I cannot find a way to run Gurobi solvers inside a parfor. I believe Matlab's linprog can, but linprog is much slower than Gurobi. There will be some trade off. I need to test this.<br />
<br />
2018-07-06<br />
I really need to understand the code better. Also we probably can run Gurobi inside parfor, but I need to wrap it around inside a function.<br />
<br />
2018-07-09<br />
I have run profiling on the Matlab code several times. It seems that moments.m takes up as much time as calling Gurobi to solve LPs. Probably we should optimize moments.m instead.<br />
<br />
[[File:profiling.png]]<br />
<br />
2018-07-10<br />
Ran profiling again. With big enough R, the parallel code is much faster. Documented in the project [[Matlab, CUDA, and GPU Computing]]<br><br />
<br />
2018-07-11<br />
'''Croatia 2-1 England!!!!!!!!!!!!!!!!!!!!!'''<br />
<br />
2018-07-12<br />
*Helped Minh install Tensorflow on DB Server.<br />
*Learned to use NOTS.<br />
<br />
2018-07-13<br />
*Further parallelize Matlab code (msmf_corr_coeff.m). Now on our 12 cores server, one call to msmf_corr_coeff takes about 35 seconds for R=200, monte_M = 70, mktsize = 30. <br />
*Will try to parallelize moments.m. Currently it takes 10 seconds per call. This in included in the 35 seconds runtime of msmf_corr_coeff.m.<br />
*Reverted back to using Matlab's linprog rather than Gurobi. In a parfor, gurobi takes much longer than the native linprog to solve our LPs. I do not fully understand why this is happening. It might be the way that Gurobi was called inside a function that increases the overhead, or it might be that Gurobi couldn't utilize the full power of our CPU since all 12 cores have been scheduled to work on 12 different LPs (some LP algorithm of Gurobi has parallelism). Note that creating a model for Gurobi takes time. <br />
[[FILE:msmf35seconds.png]]<br />
<br><br />
2018-07-16<br />
*Run monte_data mode with R=200, monte_M = 70, mktsize = 30. The msmf was computed fairly fast.<br />
[[FILE:msmf_monte_data.png]]<br />
*Run data mode with R=200, monte_M=70, mktsize = 30.<br />
[[FILE:msmf_data.png]]<br />
<br><br />
2018-07-18<br />
*Run monte mode with R=200, monte_M = 200, mktsize = 30.<br />
*Helped Maxine with industrial classifier.<br />
*Worked on documentations for NOTS and parallelization <br />
*Worked on running matlab code on NOTS<br />
<br />
2018-07-19 ~ 30<br />
Ran diagnostics requested by Chenyu<br />
<br />
2018-08<br />
Help Marcus with getting familiar with the Matlab code for matching VCs to startups.</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Wei_Wu(Work_log)&diff=23910Wei Wu(Work log)2018-08-03T20:52:20Z<p>WeiWu: /* Summer 2018 */</p>
<hr />
<div>==Notes from Ed==<br />
<br />
Detail about the install/config/basic use of software on the db server is on the [[Database Server Documentation]] documentation page.<br />
<br />
Please build and link to project pages that describe what you have done to date!<br />
<br />
==Summer 2018==<br />
<br />
<onlyinclude><br />
[[Wei Wu]] [[Work Logs]] [[Wei Wu(Work log)|(log page)]] <br><br />
<br />
2018-06-11 <br><br />
*Set up wiki page and RDP for work. Installed CUDA on dbserver. Waiting for Matlab and Gurobi to be <br />
installed on the dbserver (or I will do it myself later this week). <br />
*Started looking at the paper in progress [http://mcnair.bakerinstitute.org/wiki/Estimating_Unobserved_Complementarities_between_Entrepreneurs_and_Venture_Capitalists ''Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists''] and its related Matlab code. <br />
*Began looking for ways to incorporate CUDA parallel computation with Matlab and Julia. <br />
<br />
2018-06-12 <br><br />
*Installed tightVNC on dbserver by following steps 1-3 from this [https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-vnc-on-ubuntu-14-04 tightVNC tutorial]<br />
<br />
*Tested connection via localhost port. <br />
<br />
*Matlab matching code with Ed, James, and Chenyu via Skype.<br />
<br />
2018-06-13 <br><br />
*Continued searching for a method to set up vnc for dbserver without ssh. <br />
*Started moving the selenium box, monitors, keyboards, etc, from Room 310. <br />
*Matlab matching code with Ed, James, and Chenyu via Skype. <br />
<br />
2018-06-18 <br><br />
Trained on using PostgreSQL for DBServer. <br />
<br />
2018-06-19 <br><br />
Further training with SQL and SDC Platinum. Job assignment among team members. <br />
<br />
2018-06-20 <br> <br />
*Started reading a short tutorial on GMM and its implementation[https://www.kevinsheppard.com/images/5/55/Chapter6.pdf][https://ocw.mit.edu/courses/economics/14-385-nonlinear-econometric-analysis-fall-2007/lecture-notes/lec13_gmm.pdf]. Should have a good grasp before the end of the week. <br />
*[http://www.gurobi.com/documentation/8.0/quickstart_windows.pdf Gurobi interface guide]<br />
*It seems that Gurobi does not support GPGPU computation here in [http://www.gurobi.com/pdfs/webinar-parallel-and-distributed-optimization-english.pdf page 36], and here is [https://groups.google.com/forum/?fromgroups#!searchin/gurobi/gpu/gurobi/KTP6zDvodII/oPPQT4-mofMJ a slightly more elaborate communication] between the engineering director of Gurobi and the community regarding GPGPU computation support. Need to figure out how to do parallel computation in Matlab[https://www.mathworks.com/discovery/matlab-gpu.html][https://www.mathworks.com/help/distcomp/getting-started-with-parallel-computing-toolbox.html], and where we need it in the Startup-VC Code. <br />
</onlyinclude><br />
<br />
2018-06-21 <br><br />
*Huge problem with code '''gmm_2stage_estimated.m'''. In line 80, we compute ''W'' by taking the inverse of matrix ''Om''. We kept getting ''W'' as an ill-conditioned matrix, whose entries are infinitely large. There might be a bug in the readjusted code. I will try to catch it by comparing with the original code. If I can't, will try to set up another skype phone call with Chenyu.<br />
'''Update''': This might be related to the bug reported in the [http://mcnair.bakerinstitute.org/wiki/Estimating_Unobserved_Complementarities_between_Entrepreneurs_and_Venture_Capitalists_Matlab_Code#Bugs Matlab Code page]. I also don't think the fix was correct. I will look into that.<br />
<br />
2018-06-22 <br><br />
*Want to test Matlab and its parallel computing toolbox on DBServer. Cannot use the Matlab GUI remotely. This is possible due to the environment variable setting for remote access. '''Update''': now we have Matlab GUI. Nvidia CUDA is configured correctly as well. Today is a good day for a Linux user. <br />
*Probably it's the right time to further configure the VNC server on DBServer. [https://www.tightvnc.com/vncserver.1.php Documentation for TightVNC configuration]. '''Done'''. Documented in the [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation Database Server Documentation] <br />
<br />
2018-06-25/26 <br><br />
Looking at quick tutorials for C/C++ and CUDA, in case that I will need to read CUDA code in the future. <br />
[[Category:Work Log]]<br />
<br />
2018-06-27 <br><br />
Sick. Working from home.<br />
<br />
2018-06-28 <br><br />
Emailed Chenyu about the bug in '''gmm_2stage_estimated.m'''.<br />
<br />
2018-06-29 <br><br />
Set up a meeting with Jeremy to talk about the Matlab code and paper.<br />
<br />
2018-07-02 <br><br />
*Meeting with Jeremy. <br />
*Here is a [http://hpc.fs.uni-lj.si/sites/default/files/HPC_for_dummies.pdf gentle introduction to HPC] for myself (and whoever might benefit from it), even though this was published in 2009 and by AMD (hence it is less likely to have recent information on GPU computing).<br />
*This recent paper [https://www.researchgate.net/publication/308967833_GPU_Computing_Applied_to_Linear_and_Mixed_Integer_Programming GPU Computing Applied to Linear and Mixed Integer Programming] should be helpful. It summarizes recent advancement in GPU computing in the OR community.<br />
<br />
2018-07-03 <br><br />
Finally learned from Chenyu that the "bug" reported on June 21 was not a bug at all. We are getting singular matrices because we are using too small R and monte_M.<br />
<br />
2018-07-05 <br><br />
*There is still problems with W being singular. I have changed R and monte_M to be as big as in the original code. Either I neglected something, or there is still a bug. Or perhaps it's just normal. See [[File:a copy of warning messages from Matlab command windows.pdf]]. When W is singular, this leads to fitness function for the second stage ga being minimized to negative infinity. <br />
*I am trying to put gurobi into a parfor in Matlab. So far, not good. Wanted to figure out how to do CPU-based parallel computing with Gurobi. I cannot find a way to run Gurobi solvers inside a parfor. I believe Matlab's linprog can, but linprog is much slower than Gurobi. There will be some trade off. I need to test this.<br />
<br />
2018-07-06<br />
I really need to understand the code better. Also we probably can run Gurobi inside parfor, but I need to wrap it around inside a function.<br />
<br />
2018-07-09<br />
I have run profiling on the Matlab code several times. It seems that moments.m takes up as much time as calling Gurobi to solve LPs. Probably we should optimize moments.m instead.<br />
<br />
[[File:profiling.png]]<br />
<br />
2018-07-10<br />
Ran profiling again. With big enough R, the parallel code is much faster. Documented in the project [[Matlab, CUDA, and GPU Computing]]<br><br />
<br />
2018-07-11<br />
'''Croatia 2-1 England!!!!!!!!!!!!!!!!!!!!!'''<br />
<br />
2018-07-12<br />
*Helped Minh install Tensorflow on DB Server.<br />
*Learned to use NOTS.<br />
<br />
2018-07-13<br />
*Further parallelize Matlab code (msmf_corr_coeff.m). Now on our 12 cores server, one call to msmf_corr_coeff takes about 35 seconds for R=200, monte_M = 70, mktsize = 30. <br />
*Will try to parallelize moments.m. Currently it takes 10 seconds per call. This in included in the 35 seconds runtime of msmf_corr_coeff.m.<br />
*Reverted back to using Matlab's linprog rather than Gurobi. In a parfor, gurobi takes much longer than the native linprog to solve our LPs. I do not fully understand why this is happening. It might be the way that Gurobi was called inside a function that increases the overhead, or it might be that Gurobi couldn't utilize the full power of our CPU since all 12 cores have been scheduled to work on 12 different LPs (some LP algorithm of Gurobi has parallelism). Note that creating a model for Gurobi takes time. <br />
[[FILE:msmf35seconds.png]]<br />
<br><br />
2018-07-16<br />
*Run monte_data mode with R=200, monte_M = 70, mktsize = 30. The msmf was computed fairly fast.<br />
[[FILE:msmf_monte_data.png]]<br />
*Run data mode with R=200, monte_M=70, mktsize = 30.<br />
[[FILE:msmf_data.png]]<br />
<br><br />
2018-07-18<br />
*Run monte mode with R=200, monte_M = 200, mktsize = 30.<br />
*Helped Maxine with industrial classifier.<br />
*Worked on documentations for NOTS and parallelization <br />
*Worked on running matlab code on NOTS<br />
<br />
2018-08<br />
<br />
Help Marcus with getting familiar with the Matlab code for matching VCs to startups.</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Parallelize_msmf_corr_coeff.m&diff=23907Parallelize msmf corr coeff.m2018-08-03T20:48:31Z<p>WeiWu: </p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallelize msmf corr coeff.m<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code,<br />
|Depends upon it=NOTS Computing for Matching Entrepreneurs to VCs<br />
}}<br />
<br />
This page documents all changes that Wei made to the Matlab code for Matching Entrepreneurs to VC. It also explains the decisions Wei made, and analyzes where further improvement might be achieved in the future.<br />
<br />
=Changes made to the code=<br />
To speed up the valuation of the objective function used by ga, Wei parallelized a couple of lines in msmf_corr_coeff.m.<br />
<br />
==Around line 80 of msmf_corr_coeff== <br />
<br />
Original code:<br />
for m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1: psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12; <br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - reshape(epsim12, [N1(m)*N2(m) R]) <br />
end<br />
<br />
For R=200, monte_M = 70 (hence large M), and mktsize = 30, each call to msmf_corr_coeff takes ~140 seconds. Computing epsimR12 is taking more than 50% time of msmf_corr_coeff.m. Each computation of epsimR12 is taking no more than 2 seconds, but for a large M it will take a long time in total. To improve this, we use a parfor to paralellize this part. <br />
<br><br />
[[FILE:epsimR12_new.png]]<br />
<br><br />
New code:<br />
epsim_cell = cell(M,1); <br />
parfor m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1 : psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12;<br />
epsim_cell{m} = reshape(epsim12, [N1(m)*N2(m) R]); <br />
end <br />
for m = 1 : M<br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - epsim_cell{m};<br />
end<br />
<br />
To avoid data-racing, in each iteration m, we stored epsim12 into epsim_cell, and compute f using another for loop. Using parfor on a 12-cores machine gives a four time speed up for computing msmf_corr_coeff.m. Note that the actual speedup depends how many cores you are using. For the same R, monte_M, and mktsize, it now takes ~ 35 seconds to finish one call to msmf_corr_coeff.<br />
<br />
[[FILE:msmf_corr_coeff_speedup.png]]<br />
<br />
==Around line 120 of msmf_corr_coeff solving LP problems==<br />
For a large R, e.g. R = 200, solving 200 linear programming problems in parallel should be beneficial, especially when the each LP itself is easy to solve (takes less than 0.1 seconds each). Initially we managed to use gurobi inside a parfor:<br />
% Gurobi inisde parfor. For large R (>100). By Wei.<br />
model.rhs = [full(constrB); full(constrM)];<br />
model.A = [sparse(B);sparse(-B)];<br />
model.ub = ones(L,1);<br />
model.lb = zeros(L,1);<br />
model.sense = '<';<br />
for r = 1 : R<br />
% solve_LP is a function wrapper. It calls gurobi to solve a single LP. <br />
asgr(:,r) = solve_LP(r,f,model);<br />
end<br />
<br />
% The function wrapper to use Gurobi<br />
function X = solve_LP(r,f,model)<br />
model.obj = f(:,r); <br />
parameters.outputflag = 0;<br />
result = gurobi(model, parameters);<br />
X = result.x;<br />
end<br />
<br />
However using Gurobi actually did not help with the code performance. This is possibly due to the cost to create models for each Gurobi object. Gurobi is indeed the best commercial solver in the market, but it should be used for solving large LPs (hundreds of thousands of constraints), where as our Lp has only a few thousand constraints. We reversed back to using Matlab's native linprog:<br />
% Solve LPs in parallel with Matlab's native linprog<br />
parfor r = 1 : R<br />
options = optimset('Display', 'off');<br />
[asgr(:, r), ~, eflag(r)] = linprog(f(:, r), [B;-B], [constrB; constrM], [], [], zeros(L, 1), ones(L,1),options);<br />
end<br />
<br />
=Possible Further Improvements=<br />
At this point I believe I have done everything I could for code performance. However if you look at the profiler, you will find that the function moments also takes a good percentage of time. I tried to parallelise moments with parfor, but it did not give any improvements.</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Parallelize_msmf_corr_coeff.m&diff=23906Parallelize msmf corr coeff.m2018-08-03T20:48:08Z<p>WeiWu: </p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallelize msmf corr coeff.m<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code,<br />
|Depends upon it=NOTS Computing for Matching Entrepreneurs to VCs<br />
}}<br />
<br />
This page documents all changes that Wei made to the Matlab code for Matching Entrepreneurs to VC. It also explains the decisions Wei made, and analyzes where further improvement might be achieved in the future.<br />
<br />
=Changes made to the code=<br />
To speed up the valuation of the objective function used by ga, Wei parallelized a couple of lines in msmf_corr_coeff.m.<br />
<br />
==Around line 80 of msmf_corr_coeff== <br />
<br />
Original code:<br />
for m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1: psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12; <br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - reshape(epsim12, [N1(m)*N2(m) R]) <br />
end<br />
<br />
For R=200, monte_M = 70 (hence large M), and mktsize = 30, each call to msmf_corr_coeff takes ~140 seconds. Computing epsimR12 is taking more than 50% time of msmf_corr_coeff.m. Each computation of epsimR12 is taking no more than 2 seconds, but for a large M it will take a long time in total. To improve this, we use a parfor to paralellize this part. <br />
<br><br />
[[FILE:epsimR12_new.png]]<br />
<br><br />
New code:<br />
epsim_cell = cell(M,1); <br />
parfor m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1 : psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12;<br />
epsim_cell{m} = reshape(epsim12, [N1(m)*N2(m) R]); <br />
end <br />
for m = 1 : M<br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - epsim_cell{m};<br />
end<br />
<br />
To avoid data-racing, in each iteration m, we stored epsim12 into epsim_cell, and compute f using another for loop. Using parfor on a 12-cores machine gives a four time speed up for computing msmf_corr_coeff.m. Note that the actual speedup depends how many cores you are using. For the same R, monte_M, and mktsize, it now takes ~ 35 seconds to finish one call to msmf_corr_coeff.<br />
<br />
[[FILE:msmf_corr_coeff_speedup.png]]<br />
<br />
==Around line 120 of msmf_corr_coeff solving LP problems==<br />
For a large R, e.g. R = 200, solving 200 linear programming problems in parallel should be beneficial, especially when the each LP itself is easy to solve (takes less than 0.1 seconds each). Initially we managed to use gurobi inside a parfor:<br />
% Gurobi inisde parfor. For large R (>100). By Wei.<br />
model.rhs = [full(constrB); full(constrM)];<br />
model.A = [sparse(B);sparse(-B)];<br />
model.ub = ones(L,1);<br />
model.lb = zeros(L,1);<br />
model.sense = '<';<br />
for r = 1 : R<br />
% solve_LP is a function wrapper. It calls gurobi to solve a single LP. <br />
asgr(:,r) = solve_LP(r,f,model);<br />
end<br />
<br />
% The function wrapper to use Gurobi<br />
function X = solve_LP(r,f,model)<br />
model.obj = f(:,r); <br />
parameters.outputflag = 0;<br />
result = gurobi(model, parameters);<br />
X = result.x;<br />
end<br />
<br />
However using Gurobi actually did not help with the code performance. This is possibly due to the cost to create models for each Gurobi object. Gurobi is indeed the best commercial solver in the market, but it should be used for solving large LPs (hundreds of thousands of constraints), where as our Lp has only a few thousand constraints. We reversed back to using Matlab's native linprog:<br />
% Solve LPs in parallel with Matlab's native linprog<br />
parfor r = 1 : R<br />
options = optimset('Display', 'off');<br />
[asgr(:, r), ~, eflag(r)] = linprog(f(:, r), [B;-B], [constrB; constrM], [], [], zeros(L, 1), ones(L,1),options);<br />
end<br />
<br />
==Possible Further Improvements==<br />
At this point I believe I have done everything I could for code performance. However if you look at the profiler, you will find that the function moments also takes a good percentage of time. I tried to parallelise moments with parfor, but it did not give any improvements.</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Parallelize_msmf_corr_coeff.m&diff=23784Parallelize msmf corr coeff.m2018-07-25T20:57:34Z<p>WeiWu: /* Around line 120 of msmf_corr_coeff solving LP problems */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallelize msmf corr coeff.m<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code,<br />
|Depends upon it=NOTS Computing for Matching Entrepreneurs to VCs<br />
}}<br />
<br />
This page documents all changes that Wei made to the Matlab code for Matching Entrepreneurs to VC. It also explains the decisions Wei made, and analyzes where further improvement might be achieved in the future.<br />
<br />
=Changes made to the code=<br />
To speed up the valuation of the objective function used by ga, Wei parallelized a couple of lines in msmf_corr_coeff.m.<br />
<br />
==Around line 80 of msmf_corr_coeff== <br />
<br />
Original code:<br />
for m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1: psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12; <br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - reshape(epsim12, [N1(m)*N2(m) R]) <br />
end<br />
<br />
For R=200, monte_M = 70 (hence large M), and mktsize = 30, each call to msmf_corr_coeff takes ~140 seconds. Computing epsimR12 is taking more than 50% time of msmf_corr_coeff.m. Each computation of epsimR12 is taking no more than 2 seconds, but for a large M it will take a long time in total. To improve this, we use a parfor to paralellize this part. <br />
<br><br />
[[FILE:epsimR12_new.png]]<br />
<br><br />
New code:<br />
epsim_cell = cell(M,1); <br />
parfor m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1 : psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12;<br />
epsim_cell{m} = reshape(epsim12, [N1(m)*N2(m) R]); <br />
end <br />
for m = 1 : M<br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - epsim_cell{m};<br />
end<br />
<br />
To avoid data-racing, in each iteration m, we stored epsim12 into epsim_cell, and compute f using another for loop. Using parfor on a 12-cores machine gives a four time speed up for computing msmf_corr_coeff.m. Note that the actual speedup depends how many cores you are using. For the same R, monte_M, and mktsize, it now takes ~ 35 seconds to finish one call to msmf_corr_coeff.<br />
<br />
[[FILE:msmf_corr_coeff_speedup.png]]<br />
<br />
==Around line 120 of msmf_corr_coeff solving LP problems==<br />
For a large R, e.g. R = 200, solving 200 linear programming problems in parallel should be beneficial, especially when the each LP itself is easy to solve (takes less than 0.1 seconds each). Initially we managed to use gurobi inside a parfor:<br />
% Gurobi inisde parfor. For large R (>100). By Wei.<br />
model.rhs = [full(constrB); full(constrM)];<br />
model.A = [sparse(B);sparse(-B)];<br />
model.ub = ones(L,1);<br />
model.lb = zeros(L,1);<br />
model.sense = '<';<br />
for r = 1 : R<br />
% solve_LP is a function wrapper. It calls gurobi to solve a single LP. <br />
asgr(:,r) = solve_LP(r,f,model);<br />
end<br />
<br />
% The function wrapper to use Gurobi<br />
function X = solve_LP(r,f,model)<br />
model.obj = f(:,r); <br />
parameters.outputflag = 0;<br />
result = gurobi(model, parameters);<br />
X = result.x;<br />
end<br />
<br />
However using Gurobi actually did not help with the code performance. This is possibly due to the cost to create models for each Gurobi object. Gurobi is indeed the best commercial solver in the market, but it should be used for solving large LPs (hundreds of thousands of constraints), where as our Lp has only a few thousand constraints. We reversed back to using Matlab's native linprog:<br />
% Solve LPs in parallel with Matlab's native linprog<br />
parfor r = 1 : R<br />
options = optimset('Display', 'off');<br />
[asgr(:, r), ~, eflag(r)] = linprog(f(:, r), [B;-B], [constrB; constrM], [], [], zeros(L, 1), ones(L,1),options);<br />
end</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Matlab,_CUDA,_and_GPU_Computing&diff=23783Matlab, CUDA, and GPU Computing2018-07-25T20:55:11Z<p>WeiWu: /* Further parallelize msmf_corr_coeff */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=Matlab, CUDA, and GPU Computing<br />
|Has owner=Wei Wu<br />
|Has start date=2018-06-22<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Depends upon it=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code<br />
}}<br />
Main Project here: [[Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code]]<br />
<br />
==Synopsis==<br />
On July 2nd of 2018, Chenyu Yang, the original code author of the above main project, requested me to have "an implementation of LP (linear programming) on GPU", and Ed seconded this idea. I henceforth started exploring the possibility of such an endeavor. What is more important, such "an implementation of LP on GPU" should beat our current solver (Gurobi or Matlab's linprog). <br><br />
After some research, my conclusion is that we cannot beat performance of Gurobi on solving a single LP, and we should continue to use Gurobi instead of linprog. However, we might be still able to speed up our code by task-parallelising on CPU cores.<br />
<br />
==Getting Started with our GPU==<br />
<br />
We are running remotely on the Database Server via VNC. The VNC service on DB Server was configured by Wei during Summer 2018. Matlab, CUDA, and the Titan GPU were installed/configured by Office of Information Technology(OIT). <br />
<br />
*To start/configure the VNC service on DB Server and to get connected remotely, see the documentation [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation#Install_and_configure_TightVNC_and_xfce_desktop here].<br />
<br />
*Once you are connected to DB Server through VNC, open a terminal on DB Server and type<br />
matlab<br />
<br />
This will bring up the Matlab GUI.<br />
<br />
*To check if Matlab is working with our Nvidia graphics card, in the Matlab command window, type<br />
gpuDevice. <br />
<br />
[[File:gpuDevice.png]]<br />
<br />
==What Works, and What Doesn't==<br />
<br />
* '''GPU computing does not work well for linear programming.''' The biggest reason is [https://groups.google.com/forum/?fromgroups#!searchin/gurobi/gpu/gurobi/KTP6zDvodII/oPPQT4-mofMJ that GPUs don't work well for sparse linear algebra used for most LP solver]. Another [https://parallellp.wordpress.com/author/parallellp/ case study] published 3 years ago shows that "the overhead of communication between CPU and GPU grows faster than the benefit of parallelizing matrix operations via CUDA." In another [https://www.researchgate.net/publication/308967833_GPU_Computing_Applied_to_Linear_and_Mixed_Integer_Programming technical report], it is confirmed that "all algorithms (for solving LP and MIP problems)... do not fit to the SIMT paradigm".<br />
<br />
* For the above reason, Gurobi '''does not''' support GPU computing.<br />
<br />
* Matlab '''does not''' have GPU-based linprog.<br />
<br />
* Maltba '''does''' have an option to [https://www.mathworks.com/help/gads/how-to-use-parallel-processing.html run ga in parallel].<br />
<br />
* Gurobi '''does''' support [https://www.gurobi.com/documentation/8.0/refman/continuous_models.html CPU parallelism]. In fact, it will use the concurrent solver for LP by default.<br />
<br />
* In general, individual threads of GPU only perform well with simple arithmetic tasks. Task-parallelizing toolboxes/libraries/packages onto threads of GPU is '''NO GO'''.<br />
<br />
==Serial vs Parallel: solving many LPs on CPU==<br />
===Using a function wrapper for Gurobi===<br />
I manage to call Gurobi solvers on many LP problems in a parallel manner. To accomplish this, one needs to wrap the LP solving part into a function, and call this function inside the parfor. See [[File:parallel gurobi.pdf]] for a (lazy) example.<br />
<br />
===Testing performance===<br />
====Testing instance 1====<br />
We used the following parameters for testing:<br />
'''R = 10''', K = 1, S = 2, industry = 4, monte_N = 1, monte_M = 2. quota_per_up = 1, mktsize = 3, n_trail = 2, ini_w_M = 10, n_cores = 12. <br />
<br><br />
Run serial and parallel(using the above method) versions of msmf_corr_coeff.m. With profiling, the parallel version takes much longer time than the serial one. On a closer inspection, a single call to gurobi takes a very small amount of time. Using parfor will probably not give us any benefit. However, I suspect that for large R (~1000) we will see improvements from parallelism. <br> <br />
<br />
Serial: 356s[[File:serial_gurobi_R=10.png]] <br><br />
Parallel: 874s[[File:parallel_call_gurobi_R=10.png]] <br><br />
<br />
====Testing instance 2====<br />
We used the following parameters:<br />
'''R = 1000''', K = 1, S = 2, industry = 4, monte_N = 1, monte_M = 2. quota_per_up = 1, mktsize = 3, n_trail = 2, ini_w_M = 10, n_cores = 12. <br />
<br><br />
<br />
The parallel version now is ~3.5 time faster than the serial version. I believe it might be even faster with more cores. However I do not completely understand how the computation time scales with R (and other parameters). Will run another test with R = 200.<br><br />
Serial: 9000+s[[File:serial_gurobi_R=1000.png]] <br><br />
Parallel: 2873s[[File:parallel_call_gurobi_R=1000.png]] <br><br />
<br />
====Testing instance 3====<br />
We used the following parameters:<br />
'''R = 200''', K = 1, S = 2, industry = 4, monte_N = 1, monte_M = 2. quota_per_up = 1, mktsize = 3, n_trail = 2, ini_w_M = 10, n_cores = 12. <br />
<br><br />
<br />
Result:<br />
Serial: 3231 s [[File:serial_gurobi_R=200.png]] <br><br />
Parallel: 1605 s [[File:parallel_gurobi_R=200.png]] <br><br />
<br />
==Further parallelize msmf_corr_coeff==<br />
At this point I decided to make a new project page to document all the changes I made to the code for parallelism. Please refer to this page: [[Parallelize msmf corr coeff.m]]<br />
<br />
==Deliverable==<br />
===Week July 2 - 6===<br />
<s>In Matlab, use parfor for both CPU and GPU to solve a set of LPs. Profile and compare.</s> <br />
<br>Just try to do CPU instead. Figure out how to do CPU-based parallel computing with Gurobi. I cannot find a way to run Gurobi solvers inside a parfor. I believe Matlab's linprog can, but linprog is much slower than Gurobi. There will be some trade off. I need to test this. <br><br />
It is too hard to test this using the code written by Chenyu. Instead I will write my small piece of code to test this.<br />
<br />
===Week July 9 - 13===<br />
*Continue experimenting with calling gurobi inside parfor.<br />
*<s>Look at ways to speed up moments.m.</s> I was wrong about the computational complexity of moments.m. However it might still benefit us to speed up moments.m, but it is on a lower priority now.<br />
<br />
==Reference==<br />
1. [https://www.mathworks.com/discovery/matlab-gpu.html MATLAB GPU Computing Support for NVIDIA CUDA-Enabled GPUs] <br><br />
2. [https://www.mathworks.com/help/distcomp/getting-started-with-parallel-computing-toolbox.html Getting Started with Parallel Computing Toolbox] <br><br />
3. [https://www.mathworks.com/videos/speeding-up-algorithms-when-parallel-computing-and-gpus-do-and-dont-accelerate-92914.html Speeding Up Algorithms: When Parallel Computing and GPUs Do and Don't Accelerate]</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Matlab,_CUDA,_and_GPU_Computing&diff=23782Matlab, CUDA, and GPU Computing2018-07-25T20:54:58Z<p>WeiWu: /* Further parallelize msmf_corr_coeff */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=Matlab, CUDA, and GPU Computing<br />
|Has owner=Wei Wu<br />
|Has start date=2018-06-22<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Depends upon it=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code<br />
}}<br />
Main Project here: [[Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code]]<br />
<br />
==Synopsis==<br />
On July 2nd of 2018, Chenyu Yang, the original code author of the above main project, requested me to have "an implementation of LP (linear programming) on GPU", and Ed seconded this idea. I henceforth started exploring the possibility of such an endeavor. What is more important, such "an implementation of LP on GPU" should beat our current solver (Gurobi or Matlab's linprog). <br><br />
After some research, my conclusion is that we cannot beat performance of Gurobi on solving a single LP, and we should continue to use Gurobi instead of linprog. However, we might be still able to speed up our code by task-parallelising on CPU cores.<br />
<br />
==Getting Started with our GPU==<br />
<br />
We are running remotely on the Database Server via VNC. The VNC service on DB Server was configured by Wei during Summer 2018. Matlab, CUDA, and the Titan GPU were installed/configured by Office of Information Technology(OIT). <br />
<br />
*To start/configure the VNC service on DB Server and to get connected remotely, see the documentation [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation#Install_and_configure_TightVNC_and_xfce_desktop here].<br />
<br />
*Once you are connected to DB Server through VNC, open a terminal on DB Server and type<br />
matlab<br />
<br />
This will bring up the Matlab GUI.<br />
<br />
*To check if Matlab is working with our Nvidia graphics card, in the Matlab command window, type<br />
gpuDevice. <br />
<br />
[[File:gpuDevice.png]]<br />
<br />
==What Works, and What Doesn't==<br />
<br />
* '''GPU computing does not work well for linear programming.''' The biggest reason is [https://groups.google.com/forum/?fromgroups#!searchin/gurobi/gpu/gurobi/KTP6zDvodII/oPPQT4-mofMJ that GPUs don't work well for sparse linear algebra used for most LP solver]. Another [https://parallellp.wordpress.com/author/parallellp/ case study] published 3 years ago shows that "the overhead of communication between CPU and GPU grows faster than the benefit of parallelizing matrix operations via CUDA." In another [https://www.researchgate.net/publication/308967833_GPU_Computing_Applied_to_Linear_and_Mixed_Integer_Programming technical report], it is confirmed that "all algorithms (for solving LP and MIP problems)... do not fit to the SIMT paradigm".<br />
<br />
* For the above reason, Gurobi '''does not''' support GPU computing.<br />
<br />
* Matlab '''does not''' have GPU-based linprog.<br />
<br />
* Maltba '''does''' have an option to [https://www.mathworks.com/help/gads/how-to-use-parallel-processing.html run ga in parallel].<br />
<br />
* Gurobi '''does''' support [https://www.gurobi.com/documentation/8.0/refman/continuous_models.html CPU parallelism]. In fact, it will use the concurrent solver for LP by default.<br />
<br />
* In general, individual threads of GPU only perform well with simple arithmetic tasks. Task-parallelizing toolboxes/libraries/packages onto threads of GPU is '''NO GO'''.<br />
<br />
==Serial vs Parallel: solving many LPs on CPU==<br />
===Using a function wrapper for Gurobi===<br />
I manage to call Gurobi solvers on many LP problems in a parallel manner. To accomplish this, one needs to wrap the LP solving part into a function, and call this function inside the parfor. See [[File:parallel gurobi.pdf]] for a (lazy) example.<br />
<br />
===Testing performance===<br />
====Testing instance 1====<br />
We used the following parameters for testing:<br />
'''R = 10''', K = 1, S = 2, industry = 4, monte_N = 1, monte_M = 2. quota_per_up = 1, mktsize = 3, n_trail = 2, ini_w_M = 10, n_cores = 12. <br />
<br><br />
Run serial and parallel(using the above method) versions of msmf_corr_coeff.m. With profiling, the parallel version takes much longer time than the serial one. On a closer inspection, a single call to gurobi takes a very small amount of time. Using parfor will probably not give us any benefit. However, I suspect that for large R (~1000) we will see improvements from parallelism. <br> <br />
<br />
Serial: 356s[[File:serial_gurobi_R=10.png]] <br><br />
Parallel: 874s[[File:parallel_call_gurobi_R=10.png]] <br><br />
<br />
====Testing instance 2====<br />
We used the following parameters:<br />
'''R = 1000''', K = 1, S = 2, industry = 4, monte_N = 1, monte_M = 2. quota_per_up = 1, mktsize = 3, n_trail = 2, ini_w_M = 10, n_cores = 12. <br />
<br><br />
<br />
The parallel version now is ~3.5 time faster than the serial version. I believe it might be even faster with more cores. However I do not completely understand how the computation time scales with R (and other parameters). Will run another test with R = 200.<br><br />
Serial: 9000+s[[File:serial_gurobi_R=1000.png]] <br><br />
Parallel: 2873s[[File:parallel_call_gurobi_R=1000.png]] <br><br />
<br />
====Testing instance 3====<br />
We used the following parameters:<br />
'''R = 200''', K = 1, S = 2, industry = 4, monte_N = 1, monte_M = 2. quota_per_up = 1, mktsize = 3, n_trail = 2, ini_w_M = 10, n_cores = 12. <br />
<br><br />
<br />
Result:<br />
Serial: 3231 s [[File:serial_gurobi_R=200.png]] <br><br />
Parallel: 1605 s [[File:parallel_gurobi_R=200.png]] <br><br />
<br />
==Further parallelize msmf_corr_coeff==<br />
At this point I decided to make a new project page to document all the changes I made to the code for parallelism. Please refer to this page:[[Parallelize msmf corr coeff.m]]<br />
<br />
==Deliverable==<br />
===Week July 2 - 6===<br />
<s>In Matlab, use parfor for both CPU and GPU to solve a set of LPs. Profile and compare.</s> <br />
<br>Just try to do CPU instead. Figure out how to do CPU-based parallel computing with Gurobi. I cannot find a way to run Gurobi solvers inside a parfor. I believe Matlab's linprog can, but linprog is much slower than Gurobi. There will be some trade off. I need to test this. <br><br />
It is too hard to test this using the code written by Chenyu. Instead I will write my small piece of code to test this.<br />
<br />
===Week July 9 - 13===<br />
*Continue experimenting with calling gurobi inside parfor.<br />
*<s>Look at ways to speed up moments.m.</s> I was wrong about the computational complexity of moments.m. However it might still benefit us to speed up moments.m, but it is on a lower priority now.<br />
<br />
==Reference==<br />
1. [https://www.mathworks.com/discovery/matlab-gpu.html MATLAB GPU Computing Support for NVIDIA CUDA-Enabled GPUs] <br><br />
2. [https://www.mathworks.com/help/distcomp/getting-started-with-parallel-computing-toolbox.html Getting Started with Parallel Computing Toolbox] <br><br />
3. [https://www.mathworks.com/videos/speeding-up-algorithms-when-parallel-computing-and-gpus-do-and-dont-accelerate-92914.html Speeding Up Algorithms: When Parallel Computing and GPUs Do and Don't Accelerate]</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=NOTS_Computing_for_Matching_Entrepreneurs_to_VCs&diff=23697NOTS Computing for Matching Entrepreneurs to VCs2018-07-20T21:25:58Z<p>WeiWu: /* Wall Time */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=NOTS Computing for Matching Entrepreneurs to VCs<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=NOTS, Matlab<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code, Parallelize msmf corr coeff.m,<br />
|Has image=NOTS.png<br />
}}<br />
When in doubt, consult the full documentation by CRC: https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS<br />
=Synopsis=<br />
Summer 2018. We try to use NOTS ([https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS Night Owls Time-Sharing Service]), a computing cluster of Rice's CRC (Center for Research Computing) to run the Matlab code for Matching Entrepreneurs to VCs. This is a documentation on how to use NOTS. Currently only Wei and Ed have access to NOTS.<br />
<br />
=Getting Started=<br />
SSH to NOTS with your net id. For example, from a Linux/UNIX machine:<br />
ssh -Y (username)@nots.rice.edu<br />
<br />
To check what software is available, type<br />
module spider<br />
<br />
To see what modules are loaded,<br />
module list<br />
<br />
Currently, Matlab 2015a is installed on NOTS. To load a software such as Matlab 2015a,<br />
module load MATLAB/2015a<br />
<br />
To load this module by default at login,<br />
module save<br />
<br />
To unload all the modules,<br />
module purge<br />
<br />
=Files Placement=<br />
Currently all the readjusted code for matching entrepreneurs to VCs are stored under /projects/fox/work<br />
==Transferring files==<br />
We have access to some of the directories on NOTS. In most cases, we will either work within our $HOME directory (4 GB storage quota): <br />
/home/*username*<br />
or within $PROJECT under the group of Prof. Jeremy Fox (100 GB storage quota):<br />
/projects/fox<br />
<br />
To transfer some files from your local Linux/UNIX machine to NOTS, use the following Secure Copy command on your local terminal:<br />
scp some_file.dat *.incl *.txt (your_login_name)@nots.rice.edu:<br />
This will put the files into your $HOME directory on NOTS<br />
<br />
=Set up Matlab Parallel Computing Toolbox (PCT) on NOTS=<br />
Reference: https://docs.rice.edu/confluence/display/CD/Set+up+MATLAB+Parallel+Computing+Toolbox+from+a+cluster+login+node+on+DAVINCI<br />
<br />
==Setting Up Passwordless SSH (SSH Keys) on the Clusters (copied/modified from [https://docs.rice.edu/confluence/display/CD/Setting+Up+Passwordless+SSH+%28SSH+Keys%29+on+the+Clusters this])==<br />
Passwordless SSH is required on the Shared Computing Resources if you need to run MPI jobs using srun, or need to use other specialized software which uses SSH for communication between nodes. The srun command spawns copies of your executable on all of the nodes allocated to you by SLURM. It will communicate with these nodes via SSH so it is necessary that SSH is configured with SSH host keys (passwordless SSH) for your account. This document describes how to enable passwordless SSH on these systems.<br><br />
<br />
*The first step in establishing passwordless SSH is to create your public host keys. Login to the cluster and run the ssh-keygen command. Accept all of the default values and do not enter a passphrase.<br />
ssh-keygen -t rsa<br />
<br />
*After you have created your public host key above, append the contents of ~/.ssh/id_rsa.pub to ~/.ssh/authorized_keys. This will enable mpirun to login from one compute node to another using SSH without a password.<br />
cat .ssh/id_rsa.pub >> .ssh/authorized_keys<br />
<br />
*To avoid ssh prompts when automatically logging into compute nodes allocated by the scheduler, configure ssh to not use strict host key checking. Create the file ~/.ssh/config as shown below.<br />
nano ~/.ssh/config<br />
<br />
Host *<br />
StrictHostKeyChecking no<br />
UserKnownHostsFile /dev/null<br />
LogLevel QUIET<br />
<br />
==Import a cluster configuration on PCT==<br />
Open Matlab:<br />
matlab -nodisplay<br />
<br />
This step only has to be performed once. Once imported, the profile will persist through multiple MATLAB runs. For each cluster the profile includes cluster settings for all available queues. The following commands, run on NOTS, will import the profiles and set Commons as the default:<br />
configCluster<br />
<br />
==Wall Time==<br />
Specifying job time (walltime) is required before submitting a job via matlab and validating a cluster parallel profile. Here is an example, please be as accurate as possible to minimize your wait time.<br />
For example, set the walltime for one day and one hour:<br />
<br />
ClusterInfo.setWallTime('1-01:00:00')<br />
<br />
==Optimal use of the Cluster Configuration==<br />
For performance and file quota reasons, please have your workspace in $SHARED_SCRATCH/your-userid/. In order to do this, you must first create a workspace directory and then modify your cluster configuration to use the new workspace configuration. To change the location for your default profile you can run something like the following in MATLAB:<br />
<br />
workdir = [getenv('SHARED_SCRATCH') filesep getenv('USER') filesep 'MdcsDataLocation'];<br />
mkdir(workdir);<br />
<br />
pc = parcluster;<br />
set(pc, 'JobStorageLocation', workdir);<br />
p.saveProfile;<br />
<br />
=Submitting jobs through Matlab PCT=<br />
https://docs.rice.edu/confluence/display/CD/MATLAB+Parallel+Computing+Toolbox+testing+on+the+Shared+Computing+Clusters</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Parallelize_msmf_corr_coeff.m&diff=23696Parallelize msmf corr coeff.m2018-07-20T21:21:18Z<p>WeiWu: /* Around line 120 of msmf_corr_coeff solving LP problems */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallelize msmf corr coeff.m<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code,<br />
|Depends upon it=NOTS Computing for Matching Entrepreneurs to VCs<br />
}}<br />
<br />
This page documents all changes that Wei made to the Matlab code for Matching Entrepreneurs to VC. It also explains the decisions Wei made, and analyzes where further improvement might be achieved in the future.<br />
<br />
=Changes made to the code=<br />
To speed up the valuation of the objective function used by ga, Wei parallelized a couple of lines in msmf_corr_coeff.m.<br />
<br />
==Around line 80 of msmf_corr_coeff== <br />
<br />
Original code:<br />
for m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1: psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12; <br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - reshape(epsim12, [N1(m)*N2(m) R]) <br />
end<br />
<br />
For R=200, monte_M = 70 (hence large M), and mktsize = 30, each call to msmf_corr_coeff takes ~140 seconds. Computing epsimR12 is taking more than 50% time of msmf_corr_coeff.m. Each computation of epsimR12 is taking no more than 2 seconds, but for a large M it will take a long time in total. To improve this, we use a parfor to paralellize this part. <br />
<br><br />
[[FILE:epsimR12_new.png]]<br />
<br><br />
New code:<br />
epsim_cell = cell(M,1); <br />
parfor m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1 : psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12;<br />
epsim_cell{m} = reshape(epsim12, [N1(m)*N2(m) R]); <br />
end <br />
for m = 1 : M<br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - epsim_cell{m};<br />
end<br />
<br />
To avoid data-racing, in each iteration m, we stored epsim12 into epsim_cell, and compute f using another for loop. Using parfor on a 12-cores machine gives a four time speed up for computing msmf_corr_coeff.m. Note that the actual speedup depends how many cores you are using. For the same R, monte_M, and mktsize, it now takes ~ 35 seconds to finish one call to msmf_corr_coeff.<br />
<br />
[[FILE:msmf_corr_coeff_speedup.png]]<br />
<br />
==Around line 120 of msmf_corr_coeff solving LP problems==<br />
For a large R, e.g. R = 200, solve 200 linear programming problems in parallel should be beneficial, especially when the each LP itself is easy to solve (takes less than 0.1 seconds each). Initially we managed to use gurobi inside a parfor:<br />
% Gurobi inisde parfor. For large R (>100). By Wei.<br />
model.rhs = [full(constrB); full(constrM)];<br />
model.A = [sparse(B);sparse(-B)];<br />
model.ub = ones(L,1);<br />
model.lb = zeros(L,1);<br />
model.sense = '<';<br />
for r = 1 : R<br />
% solve_LP is a function wrapper. It calls gurobi to solve a single LP. <br />
asgr(:,r) = solve_LP(r,f,model);<br />
end<br />
<br />
% The function wrapper to use Gurobi<br />
function X = solve_LP(r,f,model)<br />
model.obj = f(:,r); <br />
parameters.outputflag = 0;<br />
result = gurobi(model, parameters);<br />
X = result.x;<br />
end<br />
<br />
However using Gurobi actually did not help with the code performance. This is possibly due to the cost to create models for each Gurobi object. Gurobi is indeed the best commercial solver in the market, but it should be used for solving large LPs (hundreds of thousands of constraints), where as our Lp has only a few thousand constraints. We reversed back to using Matlab's native linprog:<br />
% Solve LPs in parallel with Matlab's native linprog<br />
parfor r = 1 : R<br />
options = optimset('Display', 'off');<br />
[asgr(:, r), ~, eflag(r)] = linprog(f(:, r), [B;-B], [constrB; constrM], [], [], zeros(L, 1), ones(L,1),options);<br />
end</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Parallelize_msmf_corr_coeff.m&diff=23695Parallelize msmf corr coeff.m2018-07-20T21:19:25Z<p>WeiWu: /* Around line 120 of msmf_corr_coeff solving LP problems */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallelize msmf corr coeff.m<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code,<br />
|Depends upon it=NOTS Computing for Matching Entrepreneurs to VCs<br />
}}<br />
<br />
This page documents all changes that Wei made to the Matlab code for Matching Entrepreneurs to VC. It also explains the decisions Wei made, and analyzes where further improvement might be achieved in the future.<br />
<br />
=Changes made to the code=<br />
To speed up the valuation of the objective function used by ga, Wei parallelized a couple of lines in msmf_corr_coeff.m.<br />
<br />
==Around line 80 of msmf_corr_coeff== <br />
<br />
Original code:<br />
for m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1: psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12; <br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - reshape(epsim12, [N1(m)*N2(m) R]) <br />
end<br />
<br />
For R=200, monte_M = 70 (hence large M), and mktsize = 30, each call to msmf_corr_coeff takes ~140 seconds. Computing epsimR12 is taking more than 50% time of msmf_corr_coeff.m. Each computation of epsimR12 is taking no more than 2 seconds, but for a large M it will take a long time in total. To improve this, we use a parfor to paralellize this part. <br />
<br><br />
[[FILE:epsimR12_new.png]]<br />
<br><br />
New code:<br />
epsim_cell = cell(M,1); <br />
parfor m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1 : psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12;<br />
epsim_cell{m} = reshape(epsim12, [N1(m)*N2(m) R]); <br />
end <br />
for m = 1 : M<br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - epsim_cell{m};<br />
end<br />
<br />
To avoid data-racing, in each iteration m, we stored epsim12 into epsim_cell, and compute f using another for loop. Using parfor on a 12-cores machine gives a four time speed up for computing msmf_corr_coeff.m. Note that the actual speedup depends how many cores you are using. For the same R, monte_M, and mktsize, it now takes ~ 35 seconds to finish one call to msmf_corr_coeff.<br />
<br />
[[FILE:msmf_corr_coeff_speedup.png]]<br />
<br />
==Around line 120 of msmf_corr_coeff solving LP problems==<br />
For a large R, e.g. R = 200, solve 200 linear programming problems in parallel should be beneficial, especially when the each LP itself is easy to solve (takes less than 0.1 seconds each). Initially we managed to use gurobi inside a parfor:<br />
% Gurobi inisde parfor. For large R (>100). By Wei.<br />
model.rhs = [full(constrB); full(constrM)];<br />
model.A = [sparse(B);sparse(-B)];<br />
model.ub = ones(L,1);<br />
model.lb = zeros(L,1);<br />
model.sense = '<';<br />
for r = 1 : R<br />
% solve_LP is a function wrapper. It calls gurobi to solve a single LP. <br />
asgr(:,r) = solve_LP(r,f,model);<br />
end<br />
<br />
% The function wrapper to use Gurobi<br />
function X = solve_LP(r,f,model)<br />
model.obj = f(:,r); <br />
parameters.outputflag = 0;<br />
result = gurobi(model, parameters);<br />
X = result.x;<br />
end<br />
<br />
However using Gurobi actually did not help with the code performance. This is possibly due to the cost to create models for each Gurobi object. Gurobi is indeed the best commercial solver in the market, but it should be used for solving large LPs (hundreds of thousands of constraints), where as our Lp has only a few thousands constraints. We reversed back to using Matlab's native linprog:<br />
% Solve LPs in parallel with Matlab's native linprog<br />
parfor r = 1 : R<br />
options = optimset('Display', 'off');<br />
[asgr(:, r), ~, eflag(r)] = linprog(f(:, r), [B;-B], [constrB; constrM], [], [], zeros(L, 1), ones(L,1),options);<br />
end</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Parallelize_msmf_corr_coeff.m&diff=23694Parallelize msmf corr coeff.m2018-07-20T21:17:48Z<p>WeiWu: /* Around line 120 of msmf_corr_coeff solving LP problems */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallelize msmf corr coeff.m<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code,<br />
|Depends upon it=NOTS Computing for Matching Entrepreneurs to VCs<br />
}}<br />
<br />
This page documents all changes that Wei made to the Matlab code for Matching Entrepreneurs to VC. It also explains the decisions Wei made, and analyzes where further improvement might be achieved in the future.<br />
<br />
=Changes made to the code=<br />
To speed up the valuation of the objective function used by ga, Wei parallelized a couple of lines in msmf_corr_coeff.m.<br />
<br />
==Around line 80 of msmf_corr_coeff== <br />
<br />
Original code:<br />
for m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1: psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12; <br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - reshape(epsim12, [N1(m)*N2(m) R]) <br />
end<br />
<br />
For R=200, monte_M = 70 (hence large M), and mktsize = 30, each call to msmf_corr_coeff takes ~140 seconds. Computing epsimR12 is taking more than 50% time of msmf_corr_coeff.m. Each computation of epsimR12 is taking no more than 2 seconds, but for a large M it will take a long time in total. To improve this, we use a parfor to paralellize this part. <br />
<br><br />
[[FILE:epsimR12_new.png]]<br />
<br><br />
New code:<br />
epsim_cell = cell(M,1); <br />
parfor m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1 : psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12;<br />
epsim_cell{m} = reshape(epsim12, [N1(m)*N2(m) R]); <br />
end <br />
for m = 1 : M<br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - epsim_cell{m};<br />
end<br />
<br />
To avoid data-racing, in each iteration m, we stored epsim12 into epsim_cell, and compute f using another for loop. Using parfor on a 12-cores machine gives a four time speed up for computing msmf_corr_coeff.m. Note that the actual speedup depends how many cores you are using. For the same R, monte_M, and mktsize, it now takes ~ 35 seconds to finish one call to msmf_corr_coeff.<br />
<br />
[[FILE:msmf_corr_coeff_speedup.png]]<br />
<br />
==Around line 120 of msmf_corr_coeff solving LP problems==<br />
For a large R, e.g. R = 200, solve 200 linear programming problems in parallel should be beneficial, especially when the each LP itself is easy to solve (takes less than 0.1 seconds each). Initially we managed to use gurobi inside a parfor:<br />
% Gurobi inisde parfor. For large R (>100). By Wei.<br />
model.rhs = [full(constrB); full(constrM)];<br />
model.A = [sparse(B);sparse(-B)];<br />
model.ub = ones(L,1);<br />
model.lb = zeros(L,1);<br />
model.sense = '<';<br />
for r = 1 : R<br />
% solve_LP is a function wrapper. It calls gurobi to solve a single LP. <br />
asgr(:,r) = solve_LP(r,f,model);<br />
end<br />
<br />
% The function wrapper to use Gurobi<br />
function X = solve_LP(r,f,model)<br />
model.obj = f(:,r); <br />
parameters.outputflag = 0;<br />
result = gurobi(model, parameters);<br />
X = result.x;<br />
end<br />
<br />
However using Gurobi actually did not help with the code performance. This is possibly due to the cost to create models for each Gurobi object. Gurobi is indeed the best commercial solver in the market, but it should be used for solving large LPs (hundreds of thousands of constraints). We reversed back to using Matlab's native linprog:<br />
% Parallelly solve LPs with Matlab's native linprog<br />
parfor r = 1 : R<br />
options = optimset('Display', 'off');<br />
[asgr(:, r), ~, eflag(r)] = linprog(f(:, r), [B;-B], [constrB; constrM], [], [], zeros(L, 1), ones(L,1),options);<br />
end</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Parallelize_msmf_corr_coeff.m&diff=23693Parallelize msmf corr coeff.m2018-07-20T21:12:45Z<p>WeiWu: /* Around line 120 of msmf_corr_coeff solving LP problems */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallelize msmf corr coeff.m<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code,<br />
|Depends upon it=NOTS Computing for Matching Entrepreneurs to VCs<br />
}}<br />
<br />
This page documents all changes that Wei made to the Matlab code for Matching Entrepreneurs to VC. It also explains the decisions Wei made, and analyzes where further improvement might be achieved in the future.<br />
<br />
=Changes made to the code=<br />
To speed up the valuation of the objective function used by ga, Wei parallelized a couple of lines in msmf_corr_coeff.m.<br />
<br />
==Around line 80 of msmf_corr_coeff== <br />
<br />
Original code:<br />
for m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1: psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12; <br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - reshape(epsim12, [N1(m)*N2(m) R]) <br />
end<br />
<br />
For R=200, monte_M = 70 (hence large M), and mktsize = 30, each call to msmf_corr_coeff takes ~140 seconds. Computing epsimR12 is taking more than 50% time of msmf_corr_coeff.m. Each computation of epsimR12 is taking no more than 2 seconds, but for a large M it will take a long time in total. To improve this, we use a parfor to paralellize this part. <br />
<br><br />
[[FILE:epsimR12_new.png]]<br />
<br><br />
New code:<br />
epsim_cell = cell(M,1); <br />
parfor m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1 : psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12;<br />
epsim_cell{m} = reshape(epsim12, [N1(m)*N2(m) R]); <br />
end <br />
for m = 1 : M<br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - epsim_cell{m};<br />
end<br />
<br />
To avoid data-racing, in each iteration m, we stored epsim12 into epsim_cell, and compute f using another for loop. Using parfor on a 12-cores machine gives a four time speed up for computing msmf_corr_coeff.m. Note that the actual speedup depends how many cores you are using. For the same R, monte_M, and mktsize, it now takes ~ 35 seconds to finish one call to msmf_corr_coeff.<br />
<br />
[[FILE:msmf_corr_coeff_speedup.png]]<br />
<br />
==Around line 120 of msmf_corr_coeff solving LP problems==<br />
For a large R, e.g. R = 200, solve 200 linear programming problems in parallel should be beneficial, especially when the each LP itself is easy to solve (takes less than 0.1 seconds each). Initially we managed to use gurobi inside a parfor:<br />
<br />
% Gurobi inisde parfor. For large R (>100). By Wei.<br />
model.rhs = [full(constrB); full(constrM)];<br />
model.A = [sparse(B);sparse(-B)];<br />
model.ub = ones(L,1);<br />
model.lb = zeros(L,1);<br />
model.sense = '<';<br />
for r = 1 : R<br />
% solve_LP is a function wrapper. It calls gurobi to solve a single LP. <br />
asgr(:,r) = solve_LP(r,f,model);<br />
end<br />
<br />
% The function wrapper to use Gurobi<br />
function X = solve_LP(r,f,model)<br />
model.obj = f(:,r); <br />
parameters.outputflag = 0;<br />
result = gurobi(model, parameters);<br />
X = result.x;<br />
end</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Parallelize_msmf_corr_coeff.m&diff=23692Parallelize msmf corr coeff.m2018-07-20T21:10:26Z<p>WeiWu: </p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallelize msmf corr coeff.m<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code,<br />
|Depends upon it=NOTS Computing for Matching Entrepreneurs to VCs<br />
}}<br />
<br />
This page documents all changes that Wei made to the Matlab code for Matching Entrepreneurs to VC. It also explains the decisions Wei made, and analyzes where further improvement might be achieved in the future.<br />
<br />
=Changes made to the code=<br />
To speed up the valuation of the objective function used by ga, Wei parallelized a couple of lines in msmf_corr_coeff.m.<br />
<br />
==Around line 80 of msmf_corr_coeff== <br />
<br />
Original code:<br />
for m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1: psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12; <br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - reshape(epsim12, [N1(m)*N2(m) R]) <br />
end<br />
<br />
For R=200, monte_M = 70 (hence large M), and mktsize = 30, each call to msmf_corr_coeff takes ~140 seconds. Computing epsimR12 is taking more than 50% time of msmf_corr_coeff.m. Each computation of epsimR12 is taking no more than 2 seconds, but for a large M it will take a long time in total. To improve this, we use a parfor to paralellize this part. <br />
<br><br />
[[FILE:epsimR12_new.png]]<br />
<br><br />
New code:<br />
epsim_cell = cell(M,1); <br />
parfor m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1 : psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12;<br />
epsim_cell{m} = reshape(epsim12, [N1(m)*N2(m) R]); <br />
end <br />
for m = 1 : M<br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - epsim_cell{m};<br />
end<br />
<br />
To avoid data-racing, in each iteration m, we stored epsim12 into epsim_cell, and compute f using another for loop. Using parfor on a 12-cores machine gives a four time speed up for computing msmf_corr_coeff.m. Note that the actual speedup depends how many cores you are using. For the same R, monte_M, and mktsize, it now takes ~ 35 seconds to finish one call to msmf_corr_coeff.<br />
<br />
[[FILE:msmf_corr_coeff_speedup.png]]<br />
<br />
==Around line 120 of msmf_corr_coeff solving LP problems==<br />
For a large R, e.g. R = 200, solve 200 linear programming problems in parallel should be beneficial, especially when the each LP itself is easy to solve (takes less than 0.1 seconds each). Initially we managed to use gurobi inside a parfor:</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=NOTS_Computing_for_Matching_Entrepreneurs_to_VCs&diff=23689NOTS Computing for Matching Entrepreneurs to VCs2018-07-20T19:04:42Z<p>WeiWu: /* Set up Matlab Parallel Computing Toolbox (PCT) on NOTS */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=NOTS Computing for Matching Entrepreneurs to VCs<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=NOTS, Matlab<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code, Parallelize msmf corr coeff.m,<br />
|Has image=NOTS.png<br />
}}<br />
When in doubt, consult the full documentation by CRC: https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS<br />
=Synopsis=<br />
Summer 2018. We try to use NOTS ([https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS Night Owls Time-Sharing Service]), a computing cluster of Rice's CRC (Center for Research Computing) to run the Matlab code for Matching Entrepreneurs to VCs. This is a documentation on how to use NOTS. Currently only Wei and Ed have access to NOTS.<br />
<br />
=Getting Started=<br />
SSH to NOTS with your net id. For example, from a Linux/UNIX machine:<br />
ssh -Y (username)@nots.rice.edu<br />
<br />
To check what software is available, type<br />
module spider<br />
<br />
To see what modules are loaded,<br />
module list<br />
<br />
Currently, Matlab 2015a is installed on NOTS. To load a software such as Matlab 2015a,<br />
module load MATLAB/2015a<br />
<br />
To load this module by default at login,<br />
module save<br />
<br />
To unload all the modules,<br />
module purge<br />
<br />
=Files Placement=<br />
Currently all the readjusted code for matching entrepreneurs to VCs are stored under /projects/fox/work<br />
==Transferring files==<br />
We have access to some of the directories on NOTS. In most cases, we will either work within our $HOME directory (4 GB storage quota): <br />
/home/*username*<br />
or within $PROJECT under the group of Prof. Jeremy Fox (100 GB storage quota):<br />
/projects/fox<br />
<br />
To transfer some files from your local Linux/UNIX machine to NOTS, use the following Secure Copy command on your local terminal:<br />
scp some_file.dat *.incl *.txt (your_login_name)@nots.rice.edu:<br />
This will put the files into your $HOME directory on NOTS<br />
<br />
=Set up Matlab Parallel Computing Toolbox (PCT) on NOTS=<br />
Reference: https://docs.rice.edu/confluence/display/CD/Set+up+MATLAB+Parallel+Computing+Toolbox+from+a+cluster+login+node+on+DAVINCI<br />
<br />
==Setting Up Passwordless SSH (SSH Keys) on the Clusters (copied/modified from [https://docs.rice.edu/confluence/display/CD/Setting+Up+Passwordless+SSH+%28SSH+Keys%29+on+the+Clusters this])==<br />
Passwordless SSH is required on the Shared Computing Resources if you need to run MPI jobs using srun, or need to use other specialized software which uses SSH for communication between nodes. The srun command spawns copies of your executable on all of the nodes allocated to you by SLURM. It will communicate with these nodes via SSH so it is necessary that SSH is configured with SSH host keys (passwordless SSH) for your account. This document describes how to enable passwordless SSH on these systems.<br><br />
<br />
*The first step in establishing passwordless SSH is to create your public host keys. Login to the cluster and run the ssh-keygen command. Accept all of the default values and do not enter a passphrase.<br />
ssh-keygen -t rsa<br />
<br />
*After you have created your public host key above, append the contents of ~/.ssh/id_rsa.pub to ~/.ssh/authorized_keys. This will enable mpirun to login from one compute node to another using SSH without a password.<br />
cat .ssh/id_rsa.pub >> .ssh/authorized_keys<br />
<br />
*To avoid ssh prompts when automatically logging into compute nodes allocated by the scheduler, configure ssh to not use strict host key checking. Create the file ~/.ssh/config as shown below.<br />
nano ~/.ssh/config<br />
<br />
Host *<br />
StrictHostKeyChecking no<br />
UserKnownHostsFile /dev/null<br />
LogLevel QUIET<br />
<br />
==Import a cluster configuration on PCT==<br />
Open Matlab:<br />
matlab -nodisplay<br />
<br />
This step only has to be performed once. Once imported, the profile will persist through multiple MATLAB runs. For each cluster the profile includes cluster settings for all available queues. The following commands, run on NOTS, will import the profiles and set Commons as the default:<br />
configCluster<br />
<br />
==Wall Time==<br />
Specifying job time (walltime) is required before submitting a job via matlab and validating a cluster parallel profile. Here are two examples, please be as accurate as possible to minimize your wait time.<br />
For example, set the walltime for one day and one hour:<br />
<br />
ClusterInfo.setWallTime('1-01:00:00')<br />
<br />
==Optimal use of the Cluster Configuration==<br />
For performance and file quota reasons, please have your workspace in $SHARED_SCRATCH/your-userid/. In order to do this, you must first create a workspace directory and then modify your cluster configuration to use the new workspace configuration. To change the location for your default profile you can run something like the following in MATLAB:<br />
<br />
workdir = [getenv('SHARED_SCRATCH') filesep getenv('USER') filesep 'MdcsDataLocation'];<br />
mkdir(workdir);<br />
<br />
pc = parcluster;<br />
set(pc, 'JobStorageLocation', workdir);<br />
p.saveProfile;<br />
<br />
=Submitting jobs through Matlab PCT=<br />
https://docs.rice.edu/confluence/display/CD/MATLAB+Parallel+Computing+Toolbox+testing+on+the+Shared+Computing+Clusters</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=NOTS_Computing_for_Matching_Entrepreneurs_to_VCs&diff=23688NOTS Computing for Matching Entrepreneurs to VCs2018-07-20T19:02:14Z<p>WeiWu: /* Set up Matlab Parallel Computing Toolbox (PCT) on NOTS */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=NOTS Computing for Matching Entrepreneurs to VCs<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=NOTS, Matlab<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code, Parallelize msmf corr coeff.m,<br />
|Has image=NOTS.png<br />
}}<br />
When in doubt, consult the full documentation by CRC: https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS<br />
=Synopsis=<br />
Summer 2018. We try to use NOTS ([https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS Night Owls Time-Sharing Service]), a computing cluster of Rice's CRC (Center for Research Computing) to run the Matlab code for Matching Entrepreneurs to VCs. This is a documentation on how to use NOTS. Currently only Wei and Ed have access to NOTS.<br />
<br />
=Getting Started=<br />
SSH to NOTS with your net id. For example, from a Linux/UNIX machine:<br />
ssh -Y (username)@nots.rice.edu<br />
<br />
To check what software is available, type<br />
module spider<br />
<br />
To see what modules are loaded,<br />
module list<br />
<br />
Currently, Matlab 2015a is installed on NOTS. To load a software such as Matlab 2015a,<br />
module load MATLAB/2015a<br />
<br />
To load this module by default at login,<br />
module save<br />
<br />
To unload all the modules,<br />
module purge<br />
<br />
=Files Placement=<br />
Currently all the readjusted code for matching entrepreneurs to VCs are stored under /projects/fox/work<br />
==Transferring files==<br />
We have access to some of the directories on NOTS. In most cases, we will either work within our $HOME directory (4 GB storage quota): <br />
/home/*username*<br />
or within $PROJECT under the group of Prof. Jeremy Fox (100 GB storage quota):<br />
/projects/fox<br />
<br />
To transfer some files from your local Linux/UNIX machine to NOTS, use the following Secure Copy command on your local terminal:<br />
scp some_file.dat *.incl *.txt (your_login_name)@nots.rice.edu:<br />
This will put the files into your $HOME directory on NOTS<br />
<br />
=Set up Matlab Parallel Computing Toolbox (PCT) on NOTS=<br />
Reference: https://docs.rice.edu/confluence/display/CD/Set+up+MATLAB+Parallel+Computing+Toolbox+from+a+cluster+login+node+on+DAVINCI<br />
<br />
==Setting Up Passwordless SSH (SSH Keys) on the Clusters (copied/modified from [https://docs.rice.edu/confluence/display/CD/Setting+Up+Passwordless+SSH+%28SSH+Keys%29+on+the+Clusters this])==<br />
Passwordless SSH is required on the Shared Computing Resources if you need to run MPI jobs using srun, or need to use other specialized software which uses SSH for communication between nodes. The srun command spawns copies of your executable on all of the nodes allocated to you by SLURM. It will communicate with these nodes via SSH so it is necessary that SSH is configured with SSH host keys (passwordless SSH) for your account. This document describes how to enable passwordless SSH on these systems.<br><br />
<br />
*The first step in establishing passwordless SSH is to create your public host keys. Login to the cluster and run the ssh-keygen command. Accept all of the default values and do not enter a passphrase.<br />
ssh-keygen -t rsa<br />
<br />
*After you have created your public host key above, append the contents of ~/.ssh/id_rsa.pub to ~/.ssh/authorized_keys. This will enable mpirun to login from one compute node to another using SSH without a password.<br />
cat .ssh/id_rsa.pub >> .ssh/authorized_keys<br />
<br />
*To avoid ssh prompts when automatically logging into compute nodes allocated by the scheduler, configure ssh to not use strict host key checking. Create the file ~/.ssh/config as shown below.<br />
nano ~/.ssh/config<br />
<br />
Host *<br />
StrictHostKeyChecking no<br />
UserKnownHostsFile /dev/null<br />
LogLevel QUIET<br />
<br />
==Import a cluster configuration on PCT==<br />
Open Matlab:<br />
matlab -nodisplay<br />
<br />
This step only has to be performed once. Once imported, the profile will persist through multiple MATLAB runs. For each cluster the profile includes cluster settings for all available queues. The following commands, run on NOTS, will import the profiles and set Commons as the default:<br />
configCluster<br />
<br />
=Submitting jobs through Matlab PCT=<br />
https://docs.rice.edu/confluence/display/CD/MATLAB+Parallel+Computing+Toolbox+testing+on+the+Shared+Computing+Clusters</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=NOTS_Computing_for_Matching_Entrepreneurs_to_VCs&diff=23685NOTS Computing for Matching Entrepreneurs to VCs2018-07-20T17:24:14Z<p>WeiWu: </p>
<hr />
<div>{{McNair Projects<br />
|Has title=NOTS Computing for Matching Entrepreneurs to VCs<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=NOTS, Matlab<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code, Parallelize msmf corr coeff.m,<br />
|Has image=NOTS.png<br />
}}<br />
When in doubt, consult the full documentation by CRC: https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS<br />
=Synopsis=<br />
Summer 2018. We try to use NOTS ([https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS Night Owls Time-Sharing Service]), a computing cluster of Rice's CRC (Center for Research Computing) to run the Matlab code for Matching Entrepreneurs to VCs. This is a documentation on how to use NOTS. Currently only Wei and Ed have access to NOTS.<br />
<br />
=Getting Started=<br />
SSH to NOTS with your net id. For example, from a Linux/UNIX machine:<br />
ssh -Y (username)@nots.rice.edu<br />
<br />
To check what software is available, type<br />
module spider<br />
<br />
To see what modules are loaded,<br />
module list<br />
<br />
Currently, Matlab 2015a is installed on NOTS. To load a software such as Matlab 2015a,<br />
module load MATLAB/2015a<br />
<br />
To load this module by default at login,<br />
module save<br />
<br />
To unload all the modules,<br />
module purge<br />
<br />
=Files Placement=<br />
Currently all the readjusted code for matching entrepreneurs to VCs are stored under /projects/fox/work<br />
==Transferring files==<br />
We have access to some of the directories on NOTS. In most cases, we will either work within our $HOME directory (4 GB storage quota): <br />
/home/*username*<br />
or within $PROJECT under the group of Prof. Jeremy Fox (100 GB storage quota):<br />
/projects/fox<br />
<br />
To transfer some files from your local Linux/UNIX machine to NOTS, use the following Secure Copy command on your local terminal:<br />
scp some_file.dat *.incl *.txt (your_login_name)@nots.rice.edu:<br />
This will put the files into your $HOME directory on NOTS<br />
<br />
=Set up Matlab Parallel Computing Toolbox (PCT) on NOTS=<br />
Reference: https://docs.rice.edu/confluence/display/CD/Set+up+MATLAB+Parallel+Computing+Toolbox+from+a+cluster+login+node+on+DAVINCI<br />
<br />
==Setting Up Passwordless SSH (SSH Keys) on the Clusters (copied/modified from [https://docs.rice.edu/confluence/display/CD/Setting+Up+Passwordless+SSH+%28SSH+Keys%29+on+the+Clusters this])==<br />
Passwordless SSH is required on the Shared Computing Resources if you need to run MPI jobs using srun, or need to use other specialized software which uses SSH for communication between nodes. The srun command spawns copies of your executable on all of the nodes allocated to you by SLURM. It will communicate with these nodes via SSH so it is necessary that SSH is configured with SSH host keys (passwordless SSH) for your account. This document describes how to enable passwordless SSH on these systems.<br><br />
<br />
*The first step in establishing passwordless SSH is to create your public host keys. Login to the cluster and run the ssh-keygen command. Accept all of the default values and do not enter a passphrase.<br />
ssh-keygen -t rsa<br />
<br />
*After you have created your public host key above, append the contents of ~/.ssh/id_rsa.pub to ~/.ssh/authorized_keys. This will enable mpirun to login from one compute node to another using SSH without a password.<br />
cat .ssh/id_rsa.pub >> .ssh/authorized_keys<br />
<br />
*To avoid ssh prompts when automatically logging into compute nodes allocated by the scheduler, configure ssh to not use strict host key checking. Create the file ~/.ssh/config as shown below.<br />
nano ~/.ssh/config<br />
<br />
Host *<br />
StrictHostKeyChecking no<br />
UserKnownHostsFile /dev/null<br />
LogLevel QUIET<br />
<br />
=Submitting jobs through Matlab PCT=<br />
https://docs.rice.edu/confluence/display/CD/MATLAB+Parallel+Computing+Toolbox+testing+on+the+Shared+Computing+Clusters</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=NOTS_Computing_for_Matching_Entrepreneurs_to_VCs&diff=23684NOTS Computing for Matching Entrepreneurs to VCs2018-07-20T17:23:51Z<p>WeiWu: </p>
<hr />
<div>{{McNair Projects<br />
|Has title=NOTS Computing for Matching Entrepreneurs to VCs<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=NOTS, Matlab<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code, Parallelize msmf corr coeff.m,<br />
|Has image=NOTS.png<br />
}}<br />
When in doubt, consult the full documentation from CRC: https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS<br />
=Synopsis=<br />
Summer 2018. We try to use NOTS ([https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS Night Owls Time-Sharing Service]), a computing cluster of Rice's CRC (Center for Research Computing) to run the Matlab code for Matching Entrepreneurs to VCs. This is a documentation on how to use NOTS. Currently only Wei and Ed have access to NOTS.<br />
<br />
=Getting Started=<br />
SSH to NOTS with your net id. For example, from a Linux/UNIX machine:<br />
ssh -Y (username)@nots.rice.edu<br />
<br />
To check what software is available, type<br />
module spider<br />
<br />
To see what modules are loaded,<br />
module list<br />
<br />
Currently, Matlab 2015a is installed on NOTS. To load a software such as Matlab 2015a,<br />
module load MATLAB/2015a<br />
<br />
To load this module by default at login,<br />
module save<br />
<br />
To unload all the modules,<br />
module purge<br />
<br />
=Files Placement=<br />
Currently all the readjusted code for matching entrepreneurs to VCs are stored under /projects/fox/work<br />
==Transferring files==<br />
We have access to some of the directories on NOTS. In most cases, we will either work within our $HOME directory (4 GB storage quota): <br />
/home/*username*<br />
or within $PROJECT under the group of Prof. Jeremy Fox (100 GB storage quota):<br />
/projects/fox<br />
<br />
To transfer some files from your local Linux/UNIX machine to NOTS, use the following Secure Copy command on your local terminal:<br />
scp some_file.dat *.incl *.txt (your_login_name)@nots.rice.edu:<br />
This will put the files into your $HOME directory on NOTS<br />
<br />
=Set up Matlab Parallel Computing Toolbox (PCT) on NOTS=<br />
Reference: https://docs.rice.edu/confluence/display/CD/Set+up+MATLAB+Parallel+Computing+Toolbox+from+a+cluster+login+node+on+DAVINCI<br />
<br />
==Setting Up Passwordless SSH (SSH Keys) on the Clusters (copied/modified from [https://docs.rice.edu/confluence/display/CD/Setting+Up+Passwordless+SSH+%28SSH+Keys%29+on+the+Clusters this])==<br />
Passwordless SSH is required on the Shared Computing Resources if you need to run MPI jobs using srun, or need to use other specialized software which uses SSH for communication between nodes. The srun command spawns copies of your executable on all of the nodes allocated to you by SLURM. It will communicate with these nodes via SSH so it is necessary that SSH is configured with SSH host keys (passwordless SSH) for your account. This document describes how to enable passwordless SSH on these systems.<br><br />
<br />
*The first step in establishing passwordless SSH is to create your public host keys. Login to the cluster and run the ssh-keygen command. Accept all of the default values and do not enter a passphrase.<br />
ssh-keygen -t rsa<br />
<br />
*After you have created your public host key above, append the contents of ~/.ssh/id_rsa.pub to ~/.ssh/authorized_keys. This will enable mpirun to login from one compute node to another using SSH without a password.<br />
cat .ssh/id_rsa.pub >> .ssh/authorized_keys<br />
<br />
*To avoid ssh prompts when automatically logging into compute nodes allocated by the scheduler, configure ssh to not use strict host key checking. Create the file ~/.ssh/config as shown below.<br />
nano ~/.ssh/config<br />
<br />
Host *<br />
StrictHostKeyChecking no<br />
UserKnownHostsFile /dev/null<br />
LogLevel QUIET<br />
<br />
=Submitting jobs through Matlab PCT=<br />
https://docs.rice.edu/confluence/display/CD/MATLAB+Parallel+Computing+Toolbox+testing+on+the+Shared+Computing+Clusters</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=NOTS_Computing_for_Matching_Entrepreneurs_to_VCs&diff=23683NOTS Computing for Matching Entrepreneurs to VCs2018-07-20T17:19:05Z<p>WeiWu: </p>
<hr />
<div>{{McNair Projects<br />
|Has title=NOTS Computing for Matching Entrepreneurs to VCs<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=NOTS, Matlab<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code, Parallelize msmf corr coeff.m,<br />
|Has image=NOTS.png<br />
}}<br />
In progress. Building documentation from: https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS<br />
=Synopsis=<br />
Summer 2018. We try to use NOTS ([https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS Night Owls Time-Sharing Service]), a computing cluster of Rice's CRC (Center for Research Computing) to run the Matlab code for Matching Entrepreneurs to VCs. This is a documentation on how to use NOTS. Currently only Wei and Ed have access to NOTS.<br />
<br />
=Getting Started=<br />
SSH to NOTS with your net id. For example, from a Linux/UNIX machine:<br />
ssh -Y (username)@nots.rice.edu<br />
<br />
To check what software is available, type<br />
module spider<br />
<br />
To see what modules are loaded,<br />
module list<br />
<br />
Currently, Matlab 2015a is installed on NOTS. To load a software such as Matlab 2015a,<br />
module load MATLAB/2015a<br />
<br />
To load this module by default at login,<br />
module save<br />
<br />
To unload all the modules,<br />
module purge<br />
<br />
=Files Placement=<br />
Currently all the readjusted code for matching entrepreneurs to VCs are stored under /projects/fox/work<br />
==Transferring files==<br />
We have access to some of the directories on NOTS. In most cases, we will either work within our $HOME directory (4 GB storage quota): <br />
/home/*username*<br />
or within $PROJECT under the group of Prof. Jeremy Fox (100 GB storage quota):<br />
/projects/fox<br />
<br />
To transfer some files from your local Linux/UNIX machine to NOTS, use the following Secure Copy command on your local terminal:<br />
scp some_file.dat *.incl *.txt (your_login_name)@nots.rice.edu:<br />
This will put the files into your $HOME directory on NOTS<br />
<br />
=Set up Matlab Parallel Computing Toolbox (PCT) on NOTS=<br />
Reference: https://docs.rice.edu/confluence/display/CD/Set+up+MATLAB+Parallel+Computing+Toolbox+from+a+cluster+login+node+on+DAVINCI<br />
<br />
==Setting Up Passwordless SSH (SSH Keys) on the Clusters (copied/modified from [https://docs.rice.edu/confluence/display/CD/Setting+Up+Passwordless+SSH+%28SSH+Keys%29+on+the+Clusters this])==<br />
Passwordless SSH is required on the Shared Computing Resources if you need to run MPI jobs using srun, or need to use other specialized software which uses SSH for communication between nodes. The srun command spawns copies of your executable on all of the nodes allocated to you by SLURM. It will communicate with these nodes via SSH so it is necessary that SSH is configured with SSH host keys (passwordless SSH) for your account. This document describes how to enable passwordless SSH on these systems.<br><br />
<br />
*The first step in establishing passwordless SSH is to create your public host keys. Login to the cluster and run the ssh-keygen command. Accept all of the default values and do not enter a passphrase.<br />
ssh-keygen -t rsa<br />
<br />
*After you have created your public host key above, append the contents of ~/.ssh/id_rsa.pub to ~/.ssh/authorized_keys. This will enable mpirun to login from one compute node to another using SSH without a password.<br />
cat .ssh/id_rsa.pub >> .ssh/authorized_keys<br />
<br />
*To avoid ssh prompts when automatically logging into compute nodes allocated by the scheduler, configure ssh to not use strict host key checking. Create the file ~/.ssh/config as shown below.<br />
nano ~/.ssh/config<br />
<br />
Host *<br />
StrictHostKeyChecking no<br />
UserKnownHostsFile /dev/null<br />
LogLevel QUIET<br />
<br />
=Submitting jobs through Matlab PCT=<br />
https://docs.rice.edu/confluence/display/CD/MATLAB+Parallel+Computing+Toolbox+testing+on+the+Shared+Computing+Clusters</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=File:Msmf_corr_coeff_speedup.png&diff=23682File:Msmf corr coeff speedup.png2018-07-20T17:13:57Z<p>WeiWu: </p>
<hr />
<div></div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Parallelize_msmf_corr_coeff.m&diff=23681Parallelize msmf corr coeff.m2018-07-20T17:13:41Z<p>WeiWu: /* Around line 80 of msmf_corr_coeff */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallelize msmf corr coeff.m<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code,<br />
|Depends upon it=NOTS Computing for Matching Entrepreneurs to VCs<br />
}}<br />
<br />
This page documents all changes that Wei made to the Matlab code for Matching Entrepreneurs to VC. It also explains the decisions Wei made, and analyzes where further improvement might be achieved in the future.<br />
<br />
=Changes made to the code=<br />
To speed up the valuation of the objective function used by ga, Wei parallelized a couple of lines in msmf_corr_coeff.m.<br />
<br />
==Around line 80 of msmf_corr_coeff== <br />
<br />
Original code:<br />
for m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1: psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12; <br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - reshape(epsim12, [N1(m)*N2(m) R]) <br />
end<br />
<br />
For R=200, monte_M = 70 (hence large M), and mktsize = 30, each call to msmf_corr_coeff takes ~140 seconds. Computing epsimR12 is taking more than 50% time of msmf_corr_coeff.m. Each computation of epsimR12 is taking no more than 2 seconds, but for a large M it will take a long time in total. To improve this, we use a parfor to paralellize this part. <br />
<br><br />
[[FILE:epsimR12_new.png]]<br />
<br><br />
New code:<br />
epsim_cell = cell(M,1); <br />
parfor m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1 : psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12;<br />
epsim_cell{m} = reshape(epsim12, [N1(m)*N2(m) R]); <br />
end <br />
for m = 1 : M<br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - epsim_cell{m};<br />
end<br />
<br />
To avoid data-racing, in each iteration m, we stored epsim12 into epsim_cell, and compute f using another for loop. Using parfor on a 12-cores machine gives a four time speed up for computing msmf_corr_coeff.m. Note that the actual speedup depends how many cores you are using. For the same R, monte_M, and mktsize, it now takes ~ 35 seconds to finish one call to msmf_corr_coeff.<br />
<br />
[[FILE:msmf_corr_coeff_speedup.png]]</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Parallelize_msmf_corr_coeff.m&diff=23680Parallelize msmf corr coeff.m2018-07-20T17:13:23Z<p>WeiWu: /* Around line 80 of msmf_corr_coeff */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallelize msmf corr coeff.m<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code,<br />
|Depends upon it=NOTS Computing for Matching Entrepreneurs to VCs<br />
}}<br />
<br />
This page documents all changes that Wei made to the Matlab code for Matching Entrepreneurs to VC. It also explains the decisions Wei made, and analyzes where further improvement might be achieved in the future.<br />
<br />
=Changes made to the code=<br />
To speed up the valuation of the objective function used by ga, Wei parallelized a couple of lines in msmf_corr_coeff.m.<br />
<br />
==Around line 80 of msmf_corr_coeff== <br />
<br />
Original code:<br />
for m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1: psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12; <br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - reshape(epsim12, [N1(m)*N2(m) R]) <br />
end<br />
<br />
For R=200, monte_M = 70 (hence large M), and mktsize = 30, each call to msmf_corr_coeff takes ~140 seconds. Computing epsimR12 is taking more than 50% time of msmf_corr_coeff.m. Each computation of epsimR12 is taking no more than 2 seconds, but for a large M it will take a long time in total. To improve this, we use a parfor to paralellize this part. <br />
<br><br />
[[FILE:epsimR12_new.png]]<br />
<br><br />
New code:<br />
epsim_cell = cell(M,1); <br />
parfor m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1 : psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12;<br />
epsim_cell{m} = reshape(epsim12, [N1(m)*N2(m) R]); <br />
end <br />
for m = 1 : M<br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - epsim_cell{m};<br />
end<br />
<br />
To avoid data-racing, in each iteration m, we stored epsim12 into epsim_cell, and compute f using another for loop. Using parfor on a 12-cores machine gives a four time speed up for computing msmf_corr_coeff.m. Note that the actual speedup depends how many cores you are using. For the same R, monte_M, and mktsize, it now takes ~ 35 seconds to finish one call to msmf_corr_coeff.<br />
<br />
[[msmf_corr_coeff_speedup.png]]</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Parallelize_msmf_corr_coeff.m&diff=23679Parallelize msmf corr coeff.m2018-07-20T17:10:32Z<p>WeiWu: /* Around line 80 of msmf_corr_coeff */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallelize msmf corr coeff.m<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code,<br />
|Depends upon it=NOTS Computing for Matching Entrepreneurs to VCs<br />
}}<br />
<br />
This page documents all changes that Wei made to the Matlab code for Matching Entrepreneurs to VC. It also explains the decisions Wei made, and analyzes where further improvement might be achieved in the future.<br />
<br />
=Changes made to the code=<br />
To speed up the valuation of the objective function used by ga, Wei parallelized a couple of lines in msmf_corr_coeff.m.<br />
<br />
==Around line 80 of msmf_corr_coeff== <br />
<br />
Original code:<br />
for m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1: psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12; <br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - reshape(epsim12, [N1(m)*N2(m) R]) <br />
end<br />
<br />
For R=200, monte_M = 70 (hence large M), and mktsize = 30, each call to msmf_corr_coeff takes ~140 seconds. Computing epsimR12 is taking more than 50% time of msmf_corr_coeff.m. Each computation of epsimR12 is taking no more than 2 seconds, but for a large M it will take a long time in total. To improve this, we use a parfor to paralellize this part. <br />
<br><br />
[[FILE:epsimR12_new.png]]<br />
<br><br />
New code:<br />
epsim_cell = cell(M,1); <br />
parfor m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1 : psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12;<br />
epsim_cell{m} = reshape(epsim12, [N1(m)*N2(m) R]); <br />
end <br />
for m = 1 : M<br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - epsim_cell{m};<br />
end<br />
<br />
To avoid data-racing, in each iteration m, we stored epsim12 into epsim_cell, and compute f using another for loop. Using parfor on a 12-cores machine gives a four time speed up for computing msmf_corr_coeff.m (apology for not having a profiling screenshot here). Note that the actual speedup depends how many cores you are using.</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=File:EpsimR12_new.png&diff=23678File:EpsimR12 new.png2018-07-20T17:08:31Z<p>WeiWu: </p>
<hr />
<div></div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Parallelize_msmf_corr_coeff.m&diff=23676Parallelize msmf corr coeff.m2018-07-20T17:07:18Z<p>WeiWu: /* Around line 80 of msmf_corr_coeff */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallelize msmf corr coeff.m<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code,<br />
|Depends upon it=NOTS Computing for Matching Entrepreneurs to VCs<br />
}}<br />
<br />
This page documents all changes that Wei made to the Matlab code for Matching Entrepreneurs to VC. It also explains the decisions Wei made, and analyzes where further improvement might be achieved in the future.<br />
<br />
=Changes made to the code=<br />
To speed up the valuation of the objective function used by ga, Wei parallelized a couple of lines in msmf_corr_coeff.m.<br />
<br />
==Around line 80 of msmf_corr_coeff== <br />
<br />
Original code:<br />
for m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1: psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12; <br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - reshape(epsim12, [N1(m)*N2(m) R]) <br />
end<br />
<br />
For large R, monte_M (hence large M), and mktsize, computing epsimR12 is taking more than 50% time of msmf_corr_coeff.m. Each call is taking no more than 2 seconds, but for a large M it will take a long time in total. To improve this, we use a parfor to paralellize this part. <br />
<br><br />
[[FILE:epsimR12_new.png]]<br />
<br><br />
New code:<br />
epsim_cell = cell(M,1); <br />
parfor m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1 : psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12;<br />
epsim_cell{m} = reshape(epsim12, [N1(m)*N2(m) R]); <br />
end <br />
for m = 1 : M<br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - epsim_cell{m};<br />
end<br />
<br />
To avoid data-racing, in each iteration m, we stored epsim12 into epsim_cell, and compute f using another for loop. Using parfor on a 12-cores machine gives a four time speed up for computing msmf_corr_coeff.m (apology for not having a profiling screenshot here). Note that the actual speedup depends how many cores you are using.</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=File:EpsimR12.png&diff=23675File:EpsimR12.png2018-07-20T17:06:46Z<p>WeiWu: </p>
<hr />
<div></div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Parallelize_msmf_corr_coeff.m&diff=23673Parallelize msmf corr coeff.m2018-07-20T16:29:44Z<p>WeiWu: /* Around line 80 of msmf_corr_coeff */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallelize msmf corr coeff.m<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code,<br />
|Depends upon it=NOTS Computing for Matching Entrepreneurs to VCs<br />
}}<br />
<br />
This page documents all changes that Wei made to the Matlab code for Matching Entrepreneurs to VC. It also explains the decisions Wei made, and analyzes where further improvement might be achieved in the future.<br />
<br />
=Changes made to the code=<br />
To speed up the valuation of the objective function used by ga, Wei parallelized a couple of lines in msmf_corr_coeff.m.<br />
<br />
==Around line 80 of msmf_corr_coeff== <br />
<br />
Original code:<br />
for m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1: psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12; <br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - reshape(epsim12, [N1(m)*N2(m) R]) <br />
end<br />
<br />
For large R, monte_M (hence large M), and mktsize, computing epsimR12 is taking more than 50% time of msmf_corr_coeff.m. Each call is taking no more than 2 seconds, but for a large M it will take a long time in total. To improve this, we use a parfor to paralellize this part. <br />
<br><br />
[[FILE:epsimR12.png]]<br />
<br><br />
New code:<br />
epsim_cell = cell(M,1); <br />
parfor m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1 : psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12;<br />
epsim_cell{m} = reshape(epsim12, [N1(m)*N2(m) R]); <br />
end <br />
for m = 1 : M<br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - epsim_cell{m};<br />
end<br />
<br />
To avoid data-racing, in each iteration m, we stored epsim12 into epsim_cell, and compute f using another for loop. Using parfor on a 12-cores machine gives a four time speed up for computing msmf_corr_coeff.m (apology for not having a profiling screenshot here). Note that the actual speedup depends how many cores you are using.</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Parallelize_msmf_corr_coeff.m&diff=23671Parallelize msmf corr coeff.m2018-07-20T16:28:18Z<p>WeiWu: </p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallelize msmf corr coeff.m<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code,<br />
|Depends upon it=NOTS Computing for Matching Entrepreneurs to VCs<br />
}}<br />
<br />
This page documents all changes that Wei made to the Matlab code for Matching Entrepreneurs to VC. It also explains the decisions Wei made, and analyzes where further improvement might be achieved in the future.<br />
<br />
=Changes made to the code=<br />
To speed up the valuation of the objective function used by ga, Wei parallelized a couple of lines in msmf_corr_coeff.m.<br />
<br />
==Around line 80 of msmf_corr_coeff== <br />
<br />
Original code:<br />
for m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1: psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12; <br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - reshape(epsim12, [N1(m)*N2(m) R]) <br />
end<br />
<br />
For large R, monte_M (hence large M), and mktsize, computing epsimR12 is taking more than 50% time of msmf_corr_coeff.m. Each call is taking no more than 2 seconds, but for a large M it will take a long time in total. To improve this, we use a parfor to paralellize this part. <br />
<br><br />
[[FILE:epsimR12.png]]<br />
<br><br />
New code:<br />
epsim_cell = cell(M,1); <br />
parfor m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1 : psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12;<br />
epsim_cell{m} = reshape(epsim12, [N1(m)*N2(m) R]); <br />
end <br />
for m = 1 : M<br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - epsim_cell{m};<br />
end<br />
<br />
Using parfor on a 12-cores machine gives a four time speed up for computing msmf_corr_coeff.m (apology for not having a profiling screenshot here). Note that to avoid data-racing, in each iteration m, we stored epsim12 into epsim_cell, and compute f using another for loop.</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Parallelize_msmf_corr_coeff.m&diff=23670Parallelize msmf corr coeff.m2018-07-20T16:26:12Z<p>WeiWu: /* Around line 80 of msmf_corr_coeff */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallelize msmf corr coeff.m<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code,<br />
|Depends upon it=NOTS Computing for Matching Entrepreneurs to VCs<br />
}}<br />
<br />
This page documents all changes that Wei made to the Matlab code for Matching Entrepreneurs to VC. It also explains the decisions Wei made, and where further improvement might be achieved in the future.<br />
<br />
=Changes made to the code=<br />
To speed up the valuation of the objective function used by ga, Wei parallelized a couple of lines in msmf_corr_coeff.m.<br />
<br />
==Around line 80 of msmf_corr_coeff== <br />
<br />
Original code:<br />
for m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1: psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12; <br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - reshape(epsim12, [N1(m)*N2(m) R]) <br />
end<br />
<br />
For large R, monte_M (hence large M), and mktsize, computing epsimR12 is taking more than 50% time of msmf_corr_coeff.m. Each call is taking no more than 2 seconds, but for a large M it will take a long time in total. To improve this, we use a parfor to paralellize this part. <br />
<br><br />
[[FILE:epsimR12.png]]<br />
<br><br />
New code:<br />
epsim_cell = cell(M,1); <br />
parfor m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1 : psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12;<br />
epsim_cell{m} = reshape(epsim12, [N1(m)*N2(m) R]); <br />
end <br />
for m = 1 : M<br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - epsim_cell{m};<br />
end<br />
<br />
Using parfor on a 12-cores machine gives a four time speed up for computing msmf_corr_coeff.m (apology for not having a profiling screenshot here). Note that to avoid data-racing, in each iteration m, we stored epsim12 into epsim_cell, and compute f using another for loop.</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Parallelize_msmf_corr_coeff.m&diff=23668Parallelize msmf corr coeff.m2018-07-20T16:02:50Z<p>WeiWu: /* Changes made to the code */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallelize msmf corr coeff.m<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code,<br />
|Depends upon it=NOTS Computing for Matching Entrepreneurs to VCs<br />
}}<br />
<br />
This page documents all changes that Wei made to the Matlab code for Matching Entrepreneurs to VC. It also explains the decisions Wei made, and where further improvement might be achieved in the future.<br />
<br />
=Changes made to the code=<br />
To speed up the valuation of the objective function used by ga, Wei parallelized a couple of lines in msmf_corr_coeff.m.<br />
<br />
==Around line 80 of msmf_corr_coeff== <br />
<br />
Original code:<br />
for m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1: psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12; <br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - reshape(epsim12, [N1(m)*N2(m) R]) <br />
end<br />
<br />
For large R, monte_M (hence large M), and mktsize, computing epsimR12 is taking more than 50% time of msmf_corr_coeff.m. <br />
[[FILE:epsimR12.png]]</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Parallelize_msmf_corr_coeff.m&diff=23666Parallelize msmf corr coeff.m2018-07-20T16:01:39Z<p>WeiWu: /* Changes made to the code */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallelize msmf corr coeff.m<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code,<br />
|Depends upon it=NOTS Computing for Matching Entrepreneurs to VCs<br />
}}<br />
<br />
This page documents all changes that Wei made to the Matlab code for Matching Entrepreneurs to VC. It also explains the decisions Wei made, and where further improvement might be achieved in the future.<br />
<br />
=Changes made to the code=<br />
To speed up the valuation of the objective function used by ga, Wei parallelized a couple of lines in msmf_corr_coeff.m.<br />
<br />
*Around line 80 of msmf_corr_coeff: <br />
<br />
Original code:<br />
for m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1: psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12; <br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - reshape(epsim12, [N1(m)*N2(m) R]) <br />
end<br />
<br />
For large R, monte_M (hence large M), and mktsize, computing epsimR12 is taking more than 50% time of msmf_corr_coeff.m. <br />
[[FILE:epsimR12.png]]</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Parallelize_msmf_corr_coeff.m&diff=23665Parallelize msmf corr coeff.m2018-07-20T15:57:18Z<p>WeiWu: </p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallelize msmf corr coeff.m<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code,<br />
|Depends upon it=NOTS Computing for Matching Entrepreneurs to VCs<br />
}}<br />
<br />
This page documents all changes that Wei made to the Matlab code for Matching Entrepreneurs to VC. It also explains the decisions Wei made, and where further improvement might be achieved in the future.<br />
<br />
=Changes made to the code=<br />
To speed up the valuation of the objective function used by ga, Wei parallelized a couple of lines in msmf_corr_coeff.m.<br />
<br />
*Around line 80 of msmf_corr_coeff: <br />
<br />
Original code:<br />
for m = 1 : M<br />
cholA = chol(exchsig(N1(m)-1, N2(m)-1, th));<br />
epsim12 = zeros(N1(m), N2(m), R);<br />
epsimR12 = reshape(mtimesx(repmat(cholA', [1 1 R]), reshape(epsimR(psa(m)+1: psa(m)+(N1(m)-1)*(N2(m)-1), :, :), [(N1(m)-1)*(N2(m)-1) 1 R])), [N1(m)-1 N2(m)-1 R]);<br />
epsim12(2:end, 2:end, :) = epsimR12;<br />
epsim_cell{m} = reshape(epsim12, [N1(m)*N2(m) R]); % Added by Wei <br />
f(psa(m)+1 : psa(m+1), :) = f(psa(m)+1 : psa(m+1), :) - reshape(epsim12, [N1(m)*N2(m) R]) <br />
end</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Parallelize_msmf_corr_coeff.m&diff=23663Parallelize msmf corr coeff.m2018-07-20T15:51:10Z<p>WeiWu: </p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallelize msmf corr coeff.m<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code,<br />
|Depends upon it=NOTS Computing for Matching Entrepreneurs to VCs<br />
}}<br />
<br />
This page documents all changes that Wei made to the Matlab code for Matching Entrepreneurs to VC. It also explains the decisions Wei made, and where further improvement might be achieved in the future.<br />
<br />
=Changes made to the code=<br />
To speed up the valuation of the objective function used by ga, Wei parallelized a couple of lines in msmf_corr_coeff.m.<br />
<br />
*Around line 80 of msmf_corr_coeff: <br />
<br />
Original code:</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Wei_Wu(Work_log)&diff=23643Wei Wu(Work log)2018-07-18T21:59:28Z<p>WeiWu: /* Summer 2018 */</p>
<hr />
<div>==Notes from Ed==<br />
<br />
Detail about the install/config/basic use of software on the db server is on the [[Database Server Documentation]] documentation page.<br />
<br />
Please build and link to project pages that describe what you have done to date!<br />
<br />
==Summer 2018==<br />
<br />
<onlyinclude><br />
[[Wei Wu]] [[Work Logs]] [[Wei Wu(Work log)|(log page)]] <br><br />
<br />
2018-06-11 <br><br />
*Set up wiki page and RDP for work. Installed CUDA on dbserver. Waiting for Matlab and Gurobi to be <br />
installed on the dbserver (or I will do it myself later this week). <br />
*Started looking at the paper in progress [http://mcnair.bakerinstitute.org/wiki/Estimating_Unobserved_Complementarities_between_Entrepreneurs_and_Venture_Capitalists ''Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists''] and its related Matlab code. <br />
*Began looking for ways to incorporate CUDA parallel computation with Matlab and Julia. <br />
<br />
2018-06-12 <br><br />
*Installed tightVNC on dbserver by following steps 1-3 from this [https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-vnc-on-ubuntu-14-04 tightVNC tutorial]<br />
<br />
*Tested connection via localhost port. <br />
<br />
*Matlab matching code with Ed, James, and Chenyu via Skype.<br />
<br />
2018-06-13 <br><br />
*Continued searching for a method to set up vnc for dbserver without ssh. <br />
*Started moving the selenium box, monitors, keyboards, etc, from Room 310. <br />
*Matlab matching code with Ed, James, and Chenyu via Skype. <br />
<br />
2018-06-18 <br><br />
Trained on using PostgreSQL for DBServer. <br />
<br />
2018-06-19 <br><br />
Further training with SQL and SDC Platinum. Job assignment among team members. <br />
<br />
2018-06-20 <br> <br />
*Started reading a short tutorial on GMM and its implementation[https://www.kevinsheppard.com/images/5/55/Chapter6.pdf][https://ocw.mit.edu/courses/economics/14-385-nonlinear-econometric-analysis-fall-2007/lecture-notes/lec13_gmm.pdf]. Should have a good grasp before the end of the week. <br />
*[http://www.gurobi.com/documentation/8.0/quickstart_windows.pdf Gurobi interface guide]<br />
*It seems that Gurobi does not support GPGPU computation here in [http://www.gurobi.com/pdfs/webinar-parallel-and-distributed-optimization-english.pdf page 36], and here is [https://groups.google.com/forum/?fromgroups#!searchin/gurobi/gpu/gurobi/KTP6zDvodII/oPPQT4-mofMJ a slightly more elaborate communication] between the engineering director of Gurobi and the community regarding GPGPU computation support. Need to figure out how to do parallel computation in Matlab[https://www.mathworks.com/discovery/matlab-gpu.html][https://www.mathworks.com/help/distcomp/getting-started-with-parallel-computing-toolbox.html], and where we need it in the Startup-VC Code. <br />
</onlyinclude><br />
<br />
2018-06-21 <br><br />
*Huge problem with code '''gmm_2stage_estimated.m'''. In line 80, we compute ''W'' by taking the inverse of matrix ''Om''. We kept getting ''W'' as an ill-conditioned matrix, whose entries are infinitely large. There might be a bug in the readjusted code. I will try to catch it by comparing with the original code. If I can't, will try to set up another skype phone call with Chenyu.<br />
'''Update''': This might be related to the bug reported in the [http://mcnair.bakerinstitute.org/wiki/Estimating_Unobserved_Complementarities_between_Entrepreneurs_and_Venture_Capitalists_Matlab_Code#Bugs Matlab Code page]. I also don't think the fix was correct. I will look into that.<br />
<br />
2018-06-22 <br><br />
*Want to test Matlab and its parallel computing toolbox on DBServer. Cannot use the Matlab GUI remotely. This is possible due to the environment variable setting for remote access. '''Update''': now we have Matlab GUI. Nvidia CUDA is configured correctly as well. Today is a good day for a Linux user. <br />
*Probably it's the right time to further configure the VNC server on DBServer. [https://www.tightvnc.com/vncserver.1.php Documentation for TightVNC configuration]. '''Done'''. Documented in the [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation Database Server Documentation] <br />
<br />
2018-06-25/26 <br><br />
Looking at quick tutorials for C/C++ and CUDA, in case that I will need to read CUDA code in the future. <br />
[[Category:Work Log]]<br />
<br />
2018-06-27 <br><br />
Sick. Working from home.<br />
<br />
2018-06-28 <br><br />
Emailed Chenyu about the bug in '''gmm_2stage_estimated.m'''.<br />
<br />
2018-06-29 <br><br />
Set up a meeting with Jeremy to talk about the Matlab code and paper.<br />
<br />
2018-07-02 <br><br />
*Meeting with Jeremy. <br />
*Here is a [http://hpc.fs.uni-lj.si/sites/default/files/HPC_for_dummies.pdf gentle introduction to HPC] for myself (and whoever might benefit from it), even though this was published in 2009 and by AMD (hence it is less likely to have recent information on GPU computing).<br />
*This recent paper [https://www.researchgate.net/publication/308967833_GPU_Computing_Applied_to_Linear_and_Mixed_Integer_Programming GPU Computing Applied to Linear and Mixed Integer Programming] should be helpful. It summarizes recent advancement in GPU computing in the OR community.<br />
<br />
2018-07-03 <br><br />
Finally learned from Chenyu that the "bug" reported on June 21 was not a bug at all. We are getting singular matrices because we are using too small R and monte_M.<br />
<br />
2018-07-05 <br><br />
*There is still problems with W being singular. I have changed R and monte_M to be as big as in the original code. Either I neglected something, or there is still a bug. Or perhaps it's just normal. See [[File:a copy of warning messages from Matlab command windows.pdf]]. When W is singular, this leads to fitness function for the second stage ga being minimized to negative infinity. <br />
*I am trying to put gurobi into a parfor in Matlab. So far, not good. Wanted to figure out how to do CPU-based parallel computing with Gurobi. I cannot find a way to run Gurobi solvers inside a parfor. I believe Matlab's linprog can, but linprog is much slower than Gurobi. There will be some trade off. I need to test this.<br />
<br />
2018-07-06<br />
I really need to understand the code better. Also we probably can run Gurobi inside parfor, but I need to wrap it around inside a function.<br />
<br />
2018-07-09<br />
I have run profiling on the Matlab code several times. It seems that moments.m takes up as much time as calling Gurobi to solve LPs. Probably we should optimize moments.m instead.<br />
<br />
[[File:profiling.png]]<br />
<br />
2018-07-10<br />
Ran profiling again. With big enough R, the parallel code is much faster. Documented in the project [[Matlab, CUDA, and GPU Computing]]<br><br />
<br />
2018-07-11<br />
'''Croatia 2-1 England!!!!!!!!!!!!!!!!!!!!!'''<br />
<br />
2018-07-12<br />
*Helped Minh install Tensorflow on DB Server.<br />
*Learned to use NOTS.<br />
<br />
2018-07-13<br />
*Further parallelize Matlab code (msmf_corr_coeff.m). Now on our 12 cores server, one call to msmf_corr_coeff takes about 35 seconds for R=200, monte_M = 70, mktsize = 30. <br />
*Will try to parallelize moments.m. Currently it takes 10 seconds per call. This in included in the 35 seconds runtime of msmf_corr_coeff.m.<br />
*Reverted back to using Matlab's linprog rather than Gurobi. In a parfor, gurobi takes much longer than the native linprog to solve our LPs. I do not fully understand why this is happening. It might be the way that Gurobi was called inside a function that increases the overhead, or it might be that Gurobi couldn't utilize the full power of our CPU since all 12 cores have been scheduled to work on 12 different LPs (some LP algorithm of Gurobi has parallelism). Note that creating a model for Gurobi takes time. <br />
[[FILE:msmf35seconds.png]]<br />
<br><br />
2018-07-16<br />
*Run monte_data mode with R=200, monte_M = 70, mktsize = 30. The msmf was computed fairly fast.<br />
[[FILE:msmf_monte_data.png]]<br />
*Run data mode with R=200, monte_M=70, mktsize = 30.<br />
[[FILE:msmf_data.png]]<br />
<br><br />
2018-07-18<br />
*Run monte mode with R=200, monte_M = 200, mktsize = 30.<br />
*Helped Maxine with industrial classifier.<br />
*Worked on documentations for NOTS and parallelization <br />
*Worked on running matlab code on NOTS</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Parallelize_msmf_corr_coeff.m&diff=23642Parallelize msmf corr coeff.m2018-07-18T21:56:33Z<p>WeiWu: </p>
<hr />
<div>{{McNair Projects<br />
|Has title=Parallelize msmf corr coeff.m<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=Matlab, parallel computing, CPU<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code,<br />
|Depends upon it=NOTS Computing for Matching Entrepreneurs to VCs<br />
}}<br />
<br />
This page documents all changes that Wei made to the Matlab code for Matching Entrepreneurs to VC. It also explains the decisions Wei made, and where further improvement might be achieved in the future.</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=NOTS_Computing_for_Matching_Entrepreneurs_to_VCs&diff=23639NOTS Computing for Matching Entrepreneurs to VCs2018-07-18T21:32:50Z<p>WeiWu: /* Transferring files */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=NOTS Computing for Matching Entrepreneurs to VCs<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=NOTS, Matlab<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code, Parallelize msmf corr coeff.m,<br />
|Has image=NOTS.png<br />
}}<br />
In progress. Building documentation from: https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS<br />
=Synopsis=<br />
Summer 2018. We try to use NOTS ([https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS Night Owls Time-Sharing Service]), a computing cluster of Rice's CRC (Center for Research Computing) to run the Matlab code for Matching Entrepreneurs to VCs. This is a documentation on how to use NOTS. Currently only Wei and Ed have access to NOTS.<br />
<br />
=Getting Started=<br />
SSH to NOTS with your net id. For example, from a Linux/UNIX machine:<br />
ssh -Y (username)@nots.rice.edu<br />
<br />
To check what software is available, type<br />
module spider<br />
<br />
To see what modules are loaded,<br />
module list<br />
<br />
Currently, Matlab 2015a is installed on NOTS. To load a software such as Matlab 2015a,<br />
module load MATLAB/2015a<br />
<br />
To load this module by default at login,<br />
module save<br />
<br />
To unload all the modules,<br />
module purge<br />
<br />
=Files Placement=<br />
Currently all the readjusted code for matching entrepreneurs to VCs are stored under /projects/fox/work<br />
==Transferring files==<br />
We have access to some of the directories on NOTS. In most cases, we will either work within our $HOME directory (4 GB storage quota): <br />
/home/*username*<br />
or within $PROJECT under the group of Prof. Jeremy Fox (100 GB storage quota):<br />
/projects/fox<br />
<br />
To transfer some files from your local Linux/UNIX machine to NOTS, use the following Secure Copy command on your local terminal:<br />
scp some_file.dat *.incl *.txt (your_login_name)@nots.rice.edu:<br />
This will put the files into your $HOME directory on NOTS<br />
<br />
=Set up Matlab Parallel Computing Toolbox on NOTS=<br />
Reference: https://docs.rice.edu/confluence/display/CD/Set+up+MATLAB+Parallel+Computing+Toolbox+from+a+cluster+login+node+on+DAVINCI<br />
<br />
==Setting Up Passwordless SSH (SSH Keys) on the Clusters (copied/modified from [https://docs.rice.edu/confluence/display/CD/Setting+Up+Passwordless+SSH+%28SSH+Keys%29+on+the+Clusters this])==<br />
Passwordless SSH is required on the Shared Computing Resources if you need to run MPI jobs using srun, or need to use other specialized software which uses SSH for communication between nodes. The srun command spawns copies of your executable on all of the nodes allocated to you by SLURM. It will communicate with these nodes via SSH so it is necessary that SSH is configured with SSH host keys (passwordless SSH) for your account. This document describes how to enable passwordless SSH on these systems.<br><br />
<br />
*The first step in establishing passwordless SSH is to create your public host keys. Login to the cluster and run the ssh-keygen command. Accept all of the default values and do not enter a passphrase.<br />
ssh-keygen -t rsa<br />
<br />
*After you have created your public host key above, append the contents of ~/.ssh/id_rsa.pub to ~/.ssh/authorized_keys. This will enable mpirun to login from one compute node to another using SSH without a password.<br />
cat .ssh/id_rsa.pub >> .ssh/authorized_keys<br />
<br />
*To avoid ssh prompts when automatically logging into compute nodes allocated by the scheduler, configure ssh to not use strict host key checking. Create the file ~/.ssh/config as shown below.<br />
nano ~/.ssh/config<br />
<br />
Host *<br />
StrictHostKeyChecking no<br />
UserKnownHostsFile /dev/null<br />
LogLevel QUIET</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=NOTS_Computing_for_Matching_Entrepreneurs_to_VCs&diff=23638NOTS Computing for Matching Entrepreneurs to VCs2018-07-18T21:28:20Z<p>WeiWu: /* To transfer files */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=NOTS Computing for Matching Entrepreneurs to VCs<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=NOTS, Matlab<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code, Parallelize msmf corr coeff.m,<br />
|Has image=NOTS.png<br />
}}<br />
In progress. Building documentation from: https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS<br />
=Synopsis=<br />
Summer 2018. We try to use NOTS ([https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS Night Owls Time-Sharing Service]), a computing cluster of Rice's CRC (Center for Research Computing) to run the Matlab code for Matching Entrepreneurs to VCs. This is a documentation on how to use NOTS. Currently only Wei and Ed have access to NOTS.<br />
<br />
=Getting Started=<br />
SSH to NOTS with your net id. For example, from a Linux/UNIX machine:<br />
ssh -Y (username)@nots.rice.edu<br />
<br />
To check what software is available, type<br />
module spider<br />
<br />
To see what modules are loaded,<br />
module list<br />
<br />
Currently, Matlab 2015a is installed on NOTS. To load a software such as Matlab 2015a,<br />
module load MATLAB/2015a<br />
<br />
To load this module by default at login,<br />
module save<br />
<br />
To unload all the modules,<br />
module purge<br />
<br />
=Files Placement=<br />
Currently all the readjusted code for matching entrepreneurs to VCs are stored under /projects/fox/work<br />
==Transferring files==<br />
We have access to some of the directory on NOTS. In most cases, we will either work within our $HOME directory (4 GB storage quota): <br />
/home/*username*<br />
or within $PROJECT under the group of Prof. Jeremy Fox (100 GB storage quota):<br />
/projects/fox<br />
<br />
To transfer some files from your local Linux/UNIX machine to NOTS, use the following Secure Copy command on your local terminal:<br />
scp some_file.dat *.incl *.txt (your_login_name)@nots.rice.edu:<br />
This will put the files into your $HOME directory on NOTS<br />
<br />
=Set up Matlab Parallel Computing Toolbox on NOTS=<br />
Reference: https://docs.rice.edu/confluence/display/CD/Set+up+MATLAB+Parallel+Computing+Toolbox+from+a+cluster+login+node+on+DAVINCI<br />
<br />
==Setting Up Passwordless SSH (SSH Keys) on the Clusters (copied/modified from [https://docs.rice.edu/confluence/display/CD/Setting+Up+Passwordless+SSH+%28SSH+Keys%29+on+the+Clusters this])==<br />
Passwordless SSH is required on the Shared Computing Resources if you need to run MPI jobs using srun, or need to use other specialized software which uses SSH for communication between nodes. The srun command spawns copies of your executable on all of the nodes allocated to you by SLURM. It will communicate with these nodes via SSH so it is necessary that SSH is configured with SSH host keys (passwordless SSH) for your account. This document describes how to enable passwordless SSH on these systems.<br><br />
<br />
*The first step in establishing passwordless SSH is to create your public host keys. Login to the cluster and run the ssh-keygen command. Accept all of the default values and do not enter a passphrase.<br />
ssh-keygen -t rsa<br />
<br />
*After you have created your public host key above, append the contents of ~/.ssh/id_rsa.pub to ~/.ssh/authorized_keys. This will enable mpirun to login from one compute node to another using SSH without a password.<br />
cat .ssh/id_rsa.pub >> .ssh/authorized_keys<br />
<br />
*To avoid ssh prompts when automatically logging into compute nodes allocated by the scheduler, configure ssh to not use strict host key checking. Create the file ~/.ssh/config as shown below.<br />
nano ~/.ssh/config<br />
<br />
Host *<br />
StrictHostKeyChecking no<br />
UserKnownHostsFile /dev/null<br />
LogLevel QUIET</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=NOTS_Computing_for_Matching_Entrepreneurs_to_VCs&diff=23637NOTS Computing for Matching Entrepreneurs to VCs2018-07-18T21:27:25Z<p>WeiWu: /* Set up Matlab Parallel Computing Toolbox on NOTS */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=NOTS Computing for Matching Entrepreneurs to VCs<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=NOTS, Matlab<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code, Parallelize msmf corr coeff.m,<br />
|Has image=NOTS.png<br />
}}<br />
In progress. Building documentation from: https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS<br />
=Synopsis=<br />
Summer 2018. We try to use NOTS ([https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS Night Owls Time-Sharing Service]), a computing cluster of Rice's CRC (Center for Research Computing) to run the Matlab code for Matching Entrepreneurs to VCs. This is a documentation on how to use NOTS. Currently only Wei and Ed have access to NOTS.<br />
<br />
=Getting Started=<br />
SSH to NOTS with your net id. For example, from a Linux/UNIX machine:<br />
ssh -Y (username)@nots.rice.edu<br />
<br />
To check what software is available, type<br />
module spider<br />
<br />
To see what modules are loaded,<br />
module list<br />
<br />
Currently, Matlab 2015a is installed on NOTS. To load a software such as Matlab 2015a,<br />
module load MATLAB/2015a<br />
<br />
To load this module by default at login,<br />
module save<br />
<br />
To unload all the modules,<br />
module purge<br />
<br />
=Files Placement=<br />
Currently all the readjusted code for matching entrepreneurs to VCs are stored under /projects/fox/work<br />
==To transfer files==<br />
We have access to some of the directory on NOTS. In most cases, we will either work within our $HOME directory (4 GB storage quota): <br />
/home/*username*<br />
or within $PROJECT under the group of Prof. Jeremy Fox (100 GB storage quota):<br />
/projects/fox<br />
<br />
To transfer some files from your local Linux/UNIX machine to NOTS, use the following Secure Copy command on your local terminal:<br />
scp some_file.dat *.incl *.txt (your_login_name)@nots.rice.edu:<br />
This will put the files into your $HOME directory on NOTS<br />
<br />
=Set up Matlab Parallel Computing Toolbox on NOTS=<br />
Reference: https://docs.rice.edu/confluence/display/CD/Set+up+MATLAB+Parallel+Computing+Toolbox+from+a+cluster+login+node+on+DAVINCI<br />
<br />
==Setting Up Passwordless SSH (SSH Keys) on the Clusters (copied/modified from [https://docs.rice.edu/confluence/display/CD/Setting+Up+Passwordless+SSH+%28SSH+Keys%29+on+the+Clusters this])==<br />
Passwordless SSH is required on the Shared Computing Resources if you need to run MPI jobs using srun, or need to use other specialized software which uses SSH for communication between nodes. The srun command spawns copies of your executable on all of the nodes allocated to you by SLURM. It will communicate with these nodes via SSH so it is necessary that SSH is configured with SSH host keys (passwordless SSH) for your account. This document describes how to enable passwordless SSH on these systems.<br><br />
<br />
*The first step in establishing passwordless SSH is to create your public host keys. Login to the cluster and run the ssh-keygen command. Accept all of the default values and do not enter a passphrase.<br />
ssh-keygen -t rsa<br />
<br />
*After you have created your public host key above, append the contents of ~/.ssh/id_rsa.pub to ~/.ssh/authorized_keys. This will enable mpirun to login from one compute node to another using SSH without a password.<br />
cat .ssh/id_rsa.pub >> .ssh/authorized_keys<br />
<br />
*To avoid ssh prompts when automatically logging into compute nodes allocated by the scheduler, configure ssh to not use strict host key checking. Create the file ~/.ssh/config as shown below.<br />
nano ~/.ssh/config<br />
<br />
Host *<br />
StrictHostKeyChecking no<br />
UserKnownHostsFile /dev/null<br />
LogLevel QUIET</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=NOTS_Computing_for_Matching_Entrepreneurs_to_VCs&diff=23635NOTS Computing for Matching Entrepreneurs to VCs2018-07-18T20:07:16Z<p>WeiWu: </p>
<hr />
<div>{{McNair Projects<br />
|Has title=NOTS Computing for Matching Entrepreneurs to VCs<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=NOTS, Matlab<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code, Parallelize msmf corr coeff.m,<br />
|Has image=NOTS.png<br />
}}<br />
In progress. Building documentation from: https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS<br />
=Synopsis=<br />
Summer 2018. We try to use NOTS ([https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS Night Owls Time-Sharing Service]), a computing cluster of Rice's CRC (Center for Research Computing) to run the Matlab code for Matching Entrepreneurs to VCs. This is a documentation on how to use NOTS. Currently only Wei and Ed have access to NOTS.<br />
<br />
=Getting Started=<br />
SSH to NOTS with your net id. For example, from a Linux/UNIX machine:<br />
ssh -Y (username)@nots.rice.edu<br />
<br />
To check what software is available, type<br />
module spider<br />
<br />
To see what modules are loaded,<br />
module list<br />
<br />
Currently, Matlab 2015a is installed on NOTS. To load a software such as Matlab 2015a,<br />
module load MATLAB/2015a<br />
<br />
To load this module by default at login,<br />
module save<br />
<br />
To unload all the modules,<br />
module purge<br />
<br />
=Files Placement=<br />
Currently all the readjusted code for matching entrepreneurs to VCs are stored under /projects/fox/work<br />
==To transfer files==<br />
We have access to some of the directory on NOTS. In most cases, we will either work within our $HOME directory (4 GB storage quota): <br />
/home/*username*<br />
or within $PROJECT under the group of Prof. Jeremy Fox (100 GB storage quota):<br />
/projects/fox<br />
<br />
To transfer some files from your local Linux/UNIX machine to NOTS, use the following Secure Copy command on your local terminal:<br />
scp some_file.dat *.incl *.txt (your_login_name)@nots.rice.edu:<br />
This will put the files into your $HOME directory on NOTS<br />
<br />
=Set up Matlab Parallel Computing Toolbox on NOTS=<br />
https://docs.rice.edu/confluence/display/CD/Set+up+MATLAB+Parallel+Computing+Toolbox+from+a+cluster+login+node+on+DAVINCI</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=NOTS_Computing_for_Matching_Entrepreneurs_to_VCs&diff=23630NOTS Computing for Matching Entrepreneurs to VCs2018-07-18T17:28:02Z<p>WeiWu: /* Synopsis */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=NOTS Computing for Matching Entrepreneurs to VCs<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=NOTS, Matlab<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code, Parallelize msmf corr coeff.m,<br />
|Has image=NOTS.png<br />
}}<br />
In progress. Building documentation from: https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS<br />
=Synopsis=<br />
Summer 2018. We try to use NOTS ([https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS Night Owls Time-Sharing Service]), a computing cluster of Rice's CRC (Center for Research Computing) to run the Matlab code for Matching Entrepreneurs to VCs. This is a documentation on how to use NOTS. Currently only Wei and Ed have access to NOTS.<br />
<br />
=Getting Started=<br />
SSH to NOTS with your net id. For example, from a Linux/UNIX machine:<br />
ssh -Y (username)@nots.rice.edu<br />
<br />
To check what software is available, type<br />
module spider<br />
<br />
To see what modules are loaded,<br />
module list<br />
<br />
Currently, Matlab 2015a is installed on NOTS. To load a software such as Matlab 2015a,<br />
module load MATLAB/2015a<br />
<br />
To load this module by default at login,<br />
module save<br />
<br />
To unload all the modules,<br />
module purge<br />
<br />
=Files Placement=<br />
Currently all the readjusted code for matching entrepreneurs to VCs are stored under /projects/fox/work<br />
==To transfer files==<br />
We have access to some of the directory on NOTS. In most cases, we will either work within our $HOME directory (4 GB storage quota): <br />
/home/*username*<br />
or within $PROJECT under the group of Prof. Jeremy Fox (100 GB storage quota):<br />
/projects/fox<br />
<br />
To transfer some files from your local Linux/UNIX machine to NOTS, use the following Secure Copy command on your local terminal:<br />
scp some_file.dat *.incl *.txt (your_login_name)@nots.rice.edu:<br />
This will put the files into your $HOME directory on NOTS</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=NOTS_Computing_for_Matching_Entrepreneurs_to_VCs&diff=23629NOTS Computing for Matching Entrepreneurs to VCs2018-07-18T17:27:30Z<p>WeiWu: /* To transfer files */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=NOTS Computing for Matching Entrepreneurs to VCs<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=NOTS, Matlab<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code, Parallelize msmf corr coeff.m,<br />
|Has image=NOTS.png<br />
}}<br />
In progress. Building documentation from: https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS<br />
=Synopsis=<br />
Summer 2018. We try to use NOTS ([https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS Night Owls Time-Sharing Service]), a computing cluster of Rice's CRC (Center for Research Computing) to run the Matlab code for Matching Entrepreneurs to VCs. This is a documentation on how to log onto and use NOTS. Currently only Wei and Ed have access to NOTS.<br />
<br />
=Getting Started=<br />
SSH to NOTS with your net id. For example, from a Linux/UNIX machine:<br />
ssh -Y (username)@nots.rice.edu<br />
<br />
To check what software is available, type<br />
module spider<br />
<br />
To see what modules are loaded,<br />
module list<br />
<br />
Currently, Matlab 2015a is installed on NOTS. To load a software such as Matlab 2015a,<br />
module load MATLAB/2015a<br />
<br />
To load this module by default at login,<br />
module save<br />
<br />
To unload all the modules,<br />
module purge<br />
<br />
=Files Placement=<br />
Currently all the readjusted code for matching entrepreneurs to VCs are stored under /projects/fox/work<br />
==To transfer files==<br />
We have access to some of the directory on NOTS. In most cases, we will either work within our $HOME directory (4 GB storage quota): <br />
/home/*username*<br />
or within $PROJECT under the group of Prof. Jeremy Fox (100 GB storage quota):<br />
/projects/fox<br />
<br />
To transfer some files from your local Linux/UNIX machine to NOTS, use the following Secure Copy command on your local terminal:<br />
scp some_file.dat *.incl *.txt (your_login_name)@nots.rice.edu:<br />
This will put the files into your $HOME directory on NOTS</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=NOTS_Computing_for_Matching_Entrepreneurs_to_VCs&diff=23628NOTS Computing for Matching Entrepreneurs to VCs2018-07-18T16:28:09Z<p>WeiWu: /* To transfer files */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=NOTS Computing for Matching Entrepreneurs to VCs<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=NOTS, Matlab<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code, Parallelize msmf corr coeff.m,<br />
|Has image=NOTS.png<br />
}}<br />
In progress. Building documentation from: https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS<br />
=Synopsis=<br />
Summer 2018. We try to use NOTS ([https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS Night Owls Time-Sharing Service]), a computing cluster of Rice's CRC (Center for Research Computing) to run the Matlab code for Matching Entrepreneurs to VCs. This is a documentation on how to log onto and use NOTS. Currently only Wei and Ed have access to NOTS.<br />
<br />
=Getting Started=<br />
SSH to NOTS with your net id. For example, from a Linux/UNIX machine:<br />
ssh -Y (username)@nots.rice.edu<br />
<br />
To check what software is available, type<br />
module spider<br />
<br />
To see what modules are loaded,<br />
module list<br />
<br />
Currently, Matlab 2015a is installed on NOTS. To load a software such as Matlab 2015a,<br />
module load MATLAB/2015a<br />
<br />
To load this module by default at login,<br />
module save<br />
<br />
To unload all the modules,<br />
module purge<br />
<br />
=Files Placement=<br />
Currently all the readjusted code for matching entrepreneurs to VCs are stored under /projects/fox/work<br />
==To transfer files==<br />
We have access to some of the directory on NOTS. In most cases, we will either work within our $HOME directory (4 GB storage quota): <br />
/home/*username*<br />
or within $PROJECT under Jeremy Fox (100 GB storage quota):<br />
/projects/fox<br />
<br />
To transfer some files from your local Linux/UNIX machine to NOTS, use the following Secure Copy command on your local terminal:<br />
scp some_file.dat *.incl *.txt (your_login_name)@nots.rice.edu:<br />
This will put the files into your $HOME directory on NOTS</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=NOTS_Computing_for_Matching_Entrepreneurs_to_VCs&diff=23627NOTS Computing for Matching Entrepreneurs to VCs2018-07-18T16:26:23Z<p>WeiWu: /* Synopsis */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=NOTS Computing for Matching Entrepreneurs to VCs<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=NOTS, Matlab<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code, Parallelize msmf corr coeff.m,<br />
|Has image=NOTS.png<br />
}}<br />
In progress. Building documentation from: https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS<br />
=Synopsis=<br />
Summer 2018. We try to use NOTS ([https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS Night Owls Time-Sharing Service]), a computing cluster of Rice's CRC (Center for Research Computing) to run the Matlab code for Matching Entrepreneurs to VCs. This is a documentation on how to log onto and use NOTS. Currently only Wei and Ed have access to NOTS.<br />
<br />
=Getting Started=<br />
SSH to NOTS with your net id. For example, from a Linux/UNIX machine:<br />
ssh -Y (username)@nots.rice.edu<br />
<br />
To check what software is available, type<br />
module spider<br />
<br />
To see what modules are loaded,<br />
module list<br />
<br />
Currently, Matlab 2015a is installed on NOTS. To load a software such as Matlab 2015a,<br />
module load MATLAB/2015a<br />
<br />
To load this module by default at login,<br />
module save<br />
<br />
To unload all the modules,<br />
module purge<br />
<br />
=Files Placement=<br />
Currently all the readjusted code for matching entrepreneurs to VCs are stored under /projects/fox/work<br />
==To transfer files==<br />
We have access to some of the directory on NOTS. In most cases, we will either work within our $HOME directory: <br />
/home/*username*<br />
or within $PROJECT under Jeremy Fox:<br />
/projects/fox<br />
<br />
To transfer some files from your local Linux/UNIX machine to NOTS, use the following Secure Copy command on your local terminal:<br />
scp some_file.dat *.incl *.txt (your_login_name)@nots.rice.edu:<br />
This will put the files into your $HOME directory on NOTS</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=NOTS_Computing_for_Matching_Entrepreneurs_to_VCs&diff=23626NOTS Computing for Matching Entrepreneurs to VCs2018-07-18T16:25:20Z<p>WeiWu: /* To transfer files */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=NOTS Computing for Matching Entrepreneurs to VCs<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=NOTS, Matlab<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code, Parallelize msmf corr coeff.m,<br />
|Has image=NOTS.png<br />
}}<br />
In progress. Building documentation from: https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS<br />
=Synopsis=<br />
Summer 2018. We try to use NOTS (Night Owls Time-Sharing Service), a computing cluster of Rice's CRC (Center for Research Computing) to run the Matlab code for Matching Entrepreneurs to VCs. This is a documentation on how to log onto and use NOTS. Currently only Wei and Ed have access to NOTS.<br />
<br />
=Getting Started=<br />
SSH to NOTS with your net id. For example, from a Linux/UNIX machine:<br />
ssh -Y (username)@nots.rice.edu<br />
<br />
To check what software is available, type<br />
module spider<br />
<br />
To see what modules are loaded,<br />
module list<br />
<br />
Currently, Matlab 2015a is installed on NOTS. To load a software such as Matlab 2015a,<br />
module load MATLAB/2015a<br />
<br />
To load this module by default at login,<br />
module save<br />
<br />
To unload all the modules,<br />
module purge<br />
<br />
=Files Placement=<br />
Currently all the readjusted code for matching entrepreneurs to VCs are stored under /projects/fox/work<br />
==To transfer files==<br />
We have access to some of the directory on NOTS. In most cases, we will either work within our $HOME directory: <br />
/home/*username*<br />
or within $PROJECT under Jeremy Fox:<br />
/projects/fox<br />
<br />
To transfer some files from your local Linux/UNIX machine to NOTS, use the following Secure Copy command on your local terminal:<br />
scp some_file.dat *.incl *.txt (your_login_name)@nots.rice.edu:<br />
This will put the files into your $HOME directory on NOTS</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=NOTS_Computing_for_Matching_Entrepreneurs_to_VCs&diff=23625NOTS Computing for Matching Entrepreneurs to VCs2018-07-18T16:24:41Z<p>WeiWu: /* Transferring files from local to NOTS */</p>
<hr />
<div>{{McNair Projects<br />
|Has title=NOTS Computing for Matching Entrepreneurs to VCs<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=NOTS, Matlab<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code, Parallelize msmf corr coeff.m,<br />
|Has image=NOTS.png<br />
}}<br />
In progress. Building documentation from: https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS<br />
=Synopsis=<br />
Summer 2018. We try to use NOTS (Night Owls Time-Sharing Service), a computing cluster of Rice's CRC (Center for Research Computing) to run the Matlab code for Matching Entrepreneurs to VCs. This is a documentation on how to log onto and use NOTS. Currently only Wei and Ed have access to NOTS.<br />
<br />
=Getting Started=<br />
SSH to NOTS with your net id. For example, from a Linux/UNIX machine:<br />
ssh -Y (username)@nots.rice.edu<br />
<br />
To check what software is available, type<br />
module spider<br />
<br />
To see what modules are loaded,<br />
module list<br />
<br />
Currently, Matlab 2015a is installed on NOTS. To load a software such as Matlab 2015a,<br />
module load MATLAB/2015a<br />
<br />
To load this module by default at login,<br />
module save<br />
<br />
To unload all the modules,<br />
module purge<br />
<br />
=Files Placement=<br />
Currently all the readjusted code for matching entrepreneurs to VCs are stored under /projects/fox/work<br />
==To transfer files==<br />
We have access to some of the directory on NOTS. In most cases, we will either work within our $HOME directory: <br />
/home/*username*<br />
or within $PROJECT under Jeremy Fox:<br />
/projects/fox<br />
<br />
To transfer some files from your local Linux/UNIX machine to NOTS, use the following Secure Copy command:<br />
scp some_file.dat *.incl *.txt (your_login_name)@nots.rice.edu:<br />
This will put the files into your $HOME directory on NOTS</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=NOTS_Computing_for_Matching_Entrepreneurs_to_VCs&diff=23624NOTS Computing for Matching Entrepreneurs to VCs2018-07-18T16:22:44Z<p>WeiWu: </p>
<hr />
<div>{{McNair Projects<br />
|Has title=NOTS Computing for Matching Entrepreneurs to VCs<br />
|Has owner=Wei Wu,<br />
|Has start date=2018-07-09<br />
|Has keywords=NOTS, Matlab<br />
|Has project status=Active<br />
|Is dependent on=Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists Matlab Code, Parallelize msmf corr coeff.m,<br />
|Has image=NOTS.png<br />
}}<br />
In progress. Building documentation from: https://docs.rice.edu/confluence/display/CD/Getting+Started+on+NOTS<br />
=Synopsis=<br />
Summer 2018. We try to use NOTS (Night Owls Time-Sharing Service), a computing cluster of Rice's CRC (Center for Research Computing) to run the Matlab code for Matching Entrepreneurs to VCs. This is a documentation on how to log onto and use NOTS. Currently only Wei and Ed have access to NOTS.<br />
<br />
=Getting Started=<br />
SSH to NOTS with your net id. For example, from a Linux/UNIX machine:<br />
ssh -Y (username)@nots.rice.edu<br />
<br />
To check what software is available, type<br />
module spider<br />
<br />
To see what modules are loaded,<br />
module list<br />
<br />
Currently, Matlab 2015a is installed on NOTS. To load a software such as Matlab 2015a,<br />
module load MATLAB/2015a<br />
<br />
To load this module by default at login,<br />
module save<br />
<br />
To unload all the modules,<br />
module purge<br />
<br />
=Transferring files from local to NOTS=<br />
We have access to some of the directory on NOTS. In most cases, we will either work within our $HOME directory: <br />
/home/*username*<br />
or within $PROJECT under Jeremy Fox:<br />
/projects/fox<br />
<br />
To transfer some files from your local Linux/UNIX machine to NOTS, use the following Secure Copy command:<br />
scp some_file.dat *.incl *.txt (your_login_name)@nots.rice.edu:<br />
This will put the files into your $HOME directory on NOTS</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=File:Msmf_data.png&diff=23594File:Msmf data.png2018-07-16T20:53:48Z<p>WeiWu: </p>
<hr />
<div></div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Wei_Wu(Work_log)&diff=23593Wei Wu(Work log)2018-07-16T20:53:30Z<p>WeiWu: /* Summer 2018 */</p>
<hr />
<div>==Notes from Ed==<br />
<br />
Detail about the install/config/basic use of software on the db server is on the [[Database Server Documentation]] documentation page.<br />
<br />
Please build and link to project pages that describe what you have done to date!<br />
<br />
==Summer 2018==<br />
<br />
<onlyinclude><br />
[[Wei Wu]] [[Work Logs]] [[Wei Wu(Work log)|(log page)]] <br><br />
<br />
2018-06-11 <br><br />
*Set up wiki page and RDP for work. Installed CUDA on dbserver. Waiting for Matlab and Gurobi to be <br />
installed on the dbserver (or I will do it myself later this week). <br />
*Started looking at the paper in progress [http://mcnair.bakerinstitute.org/wiki/Estimating_Unobserved_Complementarities_between_Entrepreneurs_and_Venture_Capitalists ''Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists''] and its related Matlab code. <br />
*Began looking for ways to incorporate CUDA parallel computation with Matlab and Julia. <br />
<br />
2018-06-12 <br><br />
*Installed tightVNC on dbserver by following steps 1-3 from this [https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-vnc-on-ubuntu-14-04 tightVNC tutorial]<br />
<br />
*Tested connection via localhost port. <br />
<br />
*Matlab matching code with Ed, James, and Chenyu via Skype.<br />
<br />
2018-06-13 <br><br />
*Continued searching for a method to set up vnc for dbserver without ssh. <br />
*Started moving the selenium box, monitors, keyboards, etc, from Room 310. <br />
*Matlab matching code with Ed, James, and Chenyu via Skype. <br />
<br />
2018-06-18 <br><br />
Trained on using PostgreSQL for DBServer. <br />
<br />
2018-06-19 <br><br />
Further training with SQL and SDC Platinum. Job assignment among team members. <br />
<br />
2018-06-20 <br> <br />
*Started reading a short tutorial on GMM and its implementation[https://www.kevinsheppard.com/images/5/55/Chapter6.pdf][https://ocw.mit.edu/courses/economics/14-385-nonlinear-econometric-analysis-fall-2007/lecture-notes/lec13_gmm.pdf]. Should have a good grasp before the end of the week. <br />
*[http://www.gurobi.com/documentation/8.0/quickstart_windows.pdf Gurobi interface guide]<br />
*It seems that Gurobi does not support GPGPU computation here in [http://www.gurobi.com/pdfs/webinar-parallel-and-distributed-optimization-english.pdf page 36], and here is [https://groups.google.com/forum/?fromgroups#!searchin/gurobi/gpu/gurobi/KTP6zDvodII/oPPQT4-mofMJ a slightly more elaborate communication] between the engineering director of Gurobi and the community regarding GPGPU computation support. Need to figure out how to do parallel computation in Matlab[https://www.mathworks.com/discovery/matlab-gpu.html][https://www.mathworks.com/help/distcomp/getting-started-with-parallel-computing-toolbox.html], and where we need it in the Startup-VC Code. <br />
</onlyinclude><br />
<br />
2018-06-21 <br><br />
*Huge problem with code '''gmm_2stage_estimated.m'''. In line 80, we compute ''W'' by taking the inverse of matrix ''Om''. We kept getting ''W'' as an ill-conditioned matrix, whose entries are infinitely large. There might be a bug in the readjusted code. I will try to catch it by comparing with the original code. If I can't, will try to set up another skype phone call with Chenyu.<br />
'''Update''': This might be related to the bug reported in the [http://mcnair.bakerinstitute.org/wiki/Estimating_Unobserved_Complementarities_between_Entrepreneurs_and_Venture_Capitalists_Matlab_Code#Bugs Matlab Code page]. I also don't think the fix was correct. I will look into that.<br />
<br />
2018-06-22 <br><br />
*Want to test Matlab and its parallel computing toolbox on DBServer. Cannot use the Matlab GUI remotely. This is possible due to the environment variable setting for remote access. '''Update''': now we have Matlab GUI. Nvidia CUDA is configured correctly as well. Today is a good day for a Linux user. <br />
*Probably it's the right time to further configure the VNC server on DBServer. [https://www.tightvnc.com/vncserver.1.php Documentation for TightVNC configuration]. '''Done'''. Documented in the [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation Database Server Documentation] <br />
<br />
2018-06-25/26 <br><br />
Looking at quick tutorials for C/C++ and CUDA, in case that I will need to read CUDA code in the future. <br />
[[Category:Work Log]]<br />
<br />
2018-06-27 <br><br />
Sick. Working from home.<br />
<br />
2018-06-28 <br><br />
Emailed Chenyu about the bug in '''gmm_2stage_estimated.m'''.<br />
<br />
2018-06-29 <br><br />
Set up a meeting with Jeremy to talk about the Matlab code and paper.<br />
<br />
2018-07-02 <br><br />
*Meeting with Jeremy. <br />
*Here is a [http://hpc.fs.uni-lj.si/sites/default/files/HPC_for_dummies.pdf gentle introduction to HPC] for myself (and whoever might benefit from it), even though this was published in 2009 and by AMD (hence it is less likely to have recent information on GPU computing).<br />
*This recent paper [https://www.researchgate.net/publication/308967833_GPU_Computing_Applied_to_Linear_and_Mixed_Integer_Programming GPU Computing Applied to Linear and Mixed Integer Programming] should be helpful. It summarizes recent advancement in GPU computing in the OR community.<br />
<br />
2018-07-03 <br><br />
Finally learned from Chenyu that the "bug" reported on June 21 was not a bug at all. We are getting singular matrices because we are using too small R and monte_M.<br />
<br />
2018-07-05 <br><br />
*There is still problems with W being singular. I have changed R and monte_M to be as big as in the original code. Either I neglected something, or there is still a bug. Or perhaps it's just normal. See [[File:a copy of warning messages from Matlab command windows.pdf]]. When W is singular, this leads to fitness function for the second stage ga being minimized to negative infinity. <br />
*I am trying to put gurobi into a parfor in Matlab. So far, not good. Wanted to figure out how to do CPU-based parallel computing with Gurobi. I cannot find a way to run Gurobi solvers inside a parfor. I believe Matlab's linprog can, but linprog is much slower than Gurobi. There will be some trade off. I need to test this.<br />
<br />
2018-07-06<br />
I really need to understand the code better. Also we probably can run Gurobi inside parfor, but I need to wrap it around inside a function.<br />
<br />
2018-07-09<br />
I have run profiling on the Matlab code several times. It seems that moments.m takes up as much time as calling Gurobi to solve LPs. Probably we should optimize moments.m instead.<br />
<br />
[[File:profiling.png]]<br />
<br />
2018-07-10<br />
Ran profiling again. With big enough R, the parallel code is much faster. Documented in the project [[Matlab, CUDA, and GPU Computing]]<br><br />
<br />
2018-07-11<br />
'''Croatia 2-1 England!!!!!!!!!!!!!!!!!!!!!'''<br />
<br />
2018-07-12<br />
*Helped Minh install Tensorflow on DB Server.<br />
*Learned to use NOTS.<br />
<br />
2018-07-13<br />
*Further parallelize Matlab code (msmf_corr_coeff.m). Now on our 12 cores server, one call to msmf_corr_coeff takes about 35 seconds for R=200, monte_M = 70, mktsize = 30. <br />
*Will try to parallelize moments.m. Currently it takes 10 seconds per call. This in included in the 35 seconds runtime of msmf_corr_coeff.m.<br />
*Reverted back to using Matlab's linprog rather than Gurobi. In a parfor, gurobi takes much longer than the native linprog to solve our LPs. I do not fully understand why this is happening. It might be the way that Gurobi was called inside a function that increases the overhead, or it might be that Gurobi couldn't utilize the full power of our CPU since all 12 cores have been scheduled to work on 12 different LPs (some LP algorithm of Gurobi has parallelism). Note that creating a model for Gurobi takes time. <br />
[[FILE:msmf35seconds.png]]<br />
<br />
2018-07-16<br />
*Run monte_data mode with R=200, monte_M = 70, mktsize = 30. The msmf was computed fairly fast.<br />
[[FILE:msmf_monte_data.png]]<br />
*Run data mode with R=200, monte_M=70, mktsize = 30.<br />
[[FILE:msmf_data.png]]</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=Wei_Wu(Work_log)&diff=23592Wei Wu(Work log)2018-07-16T20:53:09Z<p>WeiWu: /* Summer 2018 */</p>
<hr />
<div>==Notes from Ed==<br />
<br />
Detail about the install/config/basic use of software on the db server is on the [[Database Server Documentation]] documentation page.<br />
<br />
Please build and link to project pages that describe what you have done to date!<br />
<br />
==Summer 2018==<br />
<br />
<onlyinclude><br />
[[Wei Wu]] [[Work Logs]] [[Wei Wu(Work log)|(log page)]] <br><br />
<br />
2018-06-11 <br><br />
*Set up wiki page and RDP for work. Installed CUDA on dbserver. Waiting for Matlab and Gurobi to be <br />
installed on the dbserver (or I will do it myself later this week). <br />
*Started looking at the paper in progress [http://mcnair.bakerinstitute.org/wiki/Estimating_Unobserved_Complementarities_between_Entrepreneurs_and_Venture_Capitalists ''Estimating Unobserved Complementarities between Entrepreneurs and Venture Capitalists''] and its related Matlab code. <br />
*Began looking for ways to incorporate CUDA parallel computation with Matlab and Julia. <br />
<br />
2018-06-12 <br><br />
*Installed tightVNC on dbserver by following steps 1-3 from this [https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-vnc-on-ubuntu-14-04 tightVNC tutorial]<br />
<br />
*Tested connection via localhost port. <br />
<br />
*Matlab matching code with Ed, James, and Chenyu via Skype.<br />
<br />
2018-06-13 <br><br />
*Continued searching for a method to set up vnc for dbserver without ssh. <br />
*Started moving the selenium box, monitors, keyboards, etc, from Room 310. <br />
*Matlab matching code with Ed, James, and Chenyu via Skype. <br />
<br />
2018-06-18 <br><br />
Trained on using PostgreSQL for DBServer. <br />
<br />
2018-06-19 <br><br />
Further training with SQL and SDC Platinum. Job assignment among team members. <br />
<br />
2018-06-20 <br> <br />
*Started reading a short tutorial on GMM and its implementation[https://www.kevinsheppard.com/images/5/55/Chapter6.pdf][https://ocw.mit.edu/courses/economics/14-385-nonlinear-econometric-analysis-fall-2007/lecture-notes/lec13_gmm.pdf]. Should have a good grasp before the end of the week. <br />
*[http://www.gurobi.com/documentation/8.0/quickstart_windows.pdf Gurobi interface guide]<br />
*It seems that Gurobi does not support GPGPU computation here in [http://www.gurobi.com/pdfs/webinar-parallel-and-distributed-optimization-english.pdf page 36], and here is [https://groups.google.com/forum/?fromgroups#!searchin/gurobi/gpu/gurobi/KTP6zDvodII/oPPQT4-mofMJ a slightly more elaborate communication] between the engineering director of Gurobi and the community regarding GPGPU computation support. Need to figure out how to do parallel computation in Matlab[https://www.mathworks.com/discovery/matlab-gpu.html][https://www.mathworks.com/help/distcomp/getting-started-with-parallel-computing-toolbox.html], and where we need it in the Startup-VC Code. <br />
</onlyinclude><br />
<br />
2018-06-21 <br><br />
*Huge problem with code '''gmm_2stage_estimated.m'''. In line 80, we compute ''W'' by taking the inverse of matrix ''Om''. We kept getting ''W'' as an ill-conditioned matrix, whose entries are infinitely large. There might be a bug in the readjusted code. I will try to catch it by comparing with the original code. If I can't, will try to set up another skype phone call with Chenyu.<br />
'''Update''': This might be related to the bug reported in the [http://mcnair.bakerinstitute.org/wiki/Estimating_Unobserved_Complementarities_between_Entrepreneurs_and_Venture_Capitalists_Matlab_Code#Bugs Matlab Code page]. I also don't think the fix was correct. I will look into that.<br />
<br />
2018-06-22 <br><br />
*Want to test Matlab and its parallel computing toolbox on DBServer. Cannot use the Matlab GUI remotely. This is possible due to the environment variable setting for remote access. '''Update''': now we have Matlab GUI. Nvidia CUDA is configured correctly as well. Today is a good day for a Linux user. <br />
*Probably it's the right time to further configure the VNC server on DBServer. [https://www.tightvnc.com/vncserver.1.php Documentation for TightVNC configuration]. '''Done'''. Documented in the [http://mcnair.bakerinstitute.org/wiki/Database_Server_Documentation Database Server Documentation] <br />
<br />
2018-06-25/26 <br><br />
Looking at quick tutorials for C/C++ and CUDA, in case that I will need to read CUDA code in the future. <br />
[[Category:Work Log]]<br />
<br />
2018-06-27 <br><br />
Sick. Working from home.<br />
<br />
2018-06-28 <br><br />
Emailed Chenyu about the bug in '''gmm_2stage_estimated.m'''.<br />
<br />
2018-06-29 <br><br />
Set up a meeting with Jeremy to talk about the Matlab code and paper.<br />
<br />
2018-07-02 <br><br />
*Meeting with Jeremy. <br />
*Here is a [http://hpc.fs.uni-lj.si/sites/default/files/HPC_for_dummies.pdf gentle introduction to HPC] for myself (and whoever might benefit from it), even though this was published in 2009 and by AMD (hence it is less likely to have recent information on GPU computing).<br />
*This recent paper [https://www.researchgate.net/publication/308967833_GPU_Computing_Applied_to_Linear_and_Mixed_Integer_Programming GPU Computing Applied to Linear and Mixed Integer Programming] should be helpful. It summarizes recent advancement in GPU computing in the OR community.<br />
<br />
2018-07-03 <br><br />
Finally learned from Chenyu that the "bug" reported on June 21 was not a bug at all. We are getting singular matrices because we are using too small R and monte_M.<br />
<br />
2018-07-05 <br><br />
*There is still problems with W being singular. I have changed R and monte_M to be as big as in the original code. Either I neglected something, or there is still a bug. Or perhaps it's just normal. See [[File:a copy of warning messages from Matlab command windows.pdf]]. When W is singular, this leads to fitness function for the second stage ga being minimized to negative infinity. <br />
*I am trying to put gurobi into a parfor in Matlab. So far, not good. Wanted to figure out how to do CPU-based parallel computing with Gurobi. I cannot find a way to run Gurobi solvers inside a parfor. I believe Matlab's linprog can, but linprog is much slower than Gurobi. There will be some trade off. I need to test this.<br />
<br />
2018-07-06<br />
I really need to understand the code better. Also we probably can run Gurobi inside parfor, but I need to wrap it around inside a function.<br />
<br />
2018-07-09<br />
I have run profiling on the Matlab code several times. It seems that moments.m takes up as much time as calling Gurobi to solve LPs. Probably we should optimize moments.m instead.<br />
<br />
[[File:profiling.png]]<br />
<br />
2018-07-10<br />
Ran profiling again. With big enough R, the parallel code is much faster. Documented in the project [[Matlab, CUDA, and GPU Computing]]<br><br />
<br />
2018-07-11<br />
'''Croatia 2-1 England!!!!!!!!!!!!!!!!!!!!!'''<br />
<br />
2018-07-12<br />
*Helped Minh install Tensorflow on DB Server.<br />
*Learned to use NOTS.<br />
<br />
2018-07-13<br />
*Further parallelize Matlab code (msmf_corr_coeff.m). Now on our 12 cores server, one call to msmf_corr_coeff takes about 35 seconds for R=200, monte_M = 70, mktsize = 30. <br />
*Will try to parallelize moments.m. Currently it takes 10 seconds per call. This in included in the 35 seconds runtime of msmf_corr_coeff.m.<br />
*Reverted back to using Matlab's linprog rather than Gurobi. In a parfor, gurobi takes much longer than the native linprog to solve our LPs. I do not fully understand why this is happening. It might be the way that Gurobi was called inside a function that increases the overhead, or it might be that Gurobi couldn't utilize the full power of our CPU since all 12 cores have been scheduled to work on 12 different LPs (some LP algorithm of Gurobi has parallelism). Note that creating a model for Gurobi takes time. <br />
[[FILE:msmf35seconds.png]]<br />
<br />
2018-07-16<br />
*Run monte_data mode with R=200, monte_M = 70, mktsize = 30. The msmf was computed fairly fast.<br />
[[FILE:msmf_monte_data.png]]<br />
*Run data mode with R=200, monte_M=70, mktsize = 30.<br />
[[msmf_data.png]]</div>WeiWuhttp://www.edegan.com/mediawiki/index.php?title=File:Msmf_monte_data.png&diff=23591File:Msmf monte data.png2018-07-16T20:50:08Z<p>WeiWu: </p>
<hr />
<div></div>WeiWu