parallel computation - run iterations on multiple computers?

Hi all,

I want to take advantage of all the unused desktops that my company has sitting around at night. I'm searching for a way to create something like a render farm or server computing through GH.

When using GH as a translator to run multiple iterations through 3rd party programs like Radiance or E+ (see honeybee modules), the calc doesn't live in GH/rhino. GH simply calls these programs. Unfortunately, every iteration is called in series. Once one ends, the next begins. I'd like to send these iterations to multiple computers on a server so that I can run in parallel and reduce my wall-clock time.

I'd like to generate all 500 batch files from one machine, then somehow call those routines on other computers on my network, say 100 runs on 5 machines. Or 5 runs on 100 machines (even better!). I hope this is possible because we are not splitting any single computation between multiple machines. We only want to call unique runs across a network.

Has anyone explored this? If we figure this out, companies or students could take advantage of all the unused desktops in offices or computer labs to calculate large sets overnight. Which would be amazing.

Thanks for your help.

-Leland

Replies to This Discussion

Permalink Reply by peter fotiadis on October 17, 2015 at 12:13am

Hmm ... assuming (by the question) that you are familiar with code matters get this very entry level simple test case and we can further "elaborate" ... or not.

Note: // computing is a giant pile of worms. Actually it's a colossal pile of worms (without the proper scheduler. BTW: are your iterations asynchronous?).

Attachments:

Parallel_12A_V1.gh, 124 KB

Permalink Reply by Leland Curtis on October 19, 2015 at 11:48am

Peter,

Thanks for responding. I am not an expert with code matters but I know what I need. My iterations are asynchronous, which I hope will simplify things. Once the iterations have been properly parceled using your code or something similar, how can the resulting batch files be sent and executed by a set of clustered computers?

Permalink Reply by James Ramsden on October 19, 2015 at 4:19pm

Very interesting stuff! How sophisticated do you want to be? If you don't mind setting each machine up manually (e.g. your five machines and 100 runs example) you could use the RunItAll component with the sliders on each machine's GH file set up to run their own unique slider combinations. Then, once all machines are finished, you can collate results files manually. This has worked well for me in the past.

A more automated workflow (necessary for 100 machines and 5 runs!) would be much more challenging. For one, I'm not sure if there's even a way to open Grasshopper and a specified file from the command line (correct me if I'm wrong!).

Presumably Rhino licensing is not an issue if you were able to get this working?

Permalink Reply by Andrew Heumann on October 19, 2015 at 4:58pm

There is in fact a way to launch rhino + GH with a particular file. This is the approach we used on pollination, back at the 2014 AEC hackathon in NY. All that code is available (though it's not especially well documented, so use at your own risk). https://github.com/HoneyPatch/pollination

The general approach with launching Rhino + GH around a file is to use a .bat file with the following command:

@ECHO OFF
cd **PATH TO DIRECTORY CONTAINING GH FILE**

"**PATH TO RHINO.EXE**" /nosplash /runscript="-grasshopper editor load document open **GRASSHOPPER FILE NAME** _enter" "**PATH TO ASSOCIATED RHINO FILE**"

Permalink Reply by Andrew Heumann on October 19, 2015 at 5:00pm

on pollination, if memory serves, we were actually launching all identical instances of Rhino w/ grasshopper opening the SAME file - but the GH file looked at a .json file containing all the parameters, and we auto-generated different json files for each parameter set we wanted to run.

Permalink Reply by Yijiang Huang on September 25, 2016 at 9:52pm