Cyberhelp

for Researchers & Teams

Run Code in Parallel on a Windows Virtual Machine

If you have a program that runs only on Windows systems, and you would like to run the program in parallel, it is not possible to use SESYNC’s Slurm cluster. However, it is possible to take advantage of SESYNC’s Winanalytics virtual machine, which has multiple cores and much more available RAM than the typical laptop. You will need to write a little bit of code in PowerShell, which is Windows’ shell scripting languge and is fairly similar to Bash scripting. Here is a quick walkthrough of how to do this.

Writing a PowerShell script to run multiple processes in parallel

PowerShell scripting language provides a construction called a “workflow” in which you can write a loop that is executed in parallel. This FAQ does not go into great detail about how to write PowerShell scripts, but the following is a basic template that you can modify to create a simple script that will run multiple instances of the same program in parallel, each time taking a different file as input. (Borrowed from this StackOverflow answer by user js2010)

workflow run_in_parallel {
  $file_names = "foo.txt","bar.txt","baz.txt"
  foreach -parallel ($name in $file_names) { 
    <insert full path to your program here> "Z:\files\$($name)" 
    "$($name) done" 
  }
}

run_in_parallel

Note: If the file path to your program has spaces in it, as is sometimes the case with Windows (for example, C:\Program Files (x86)\...), you will need to wrap the command on line 4 in the above script with an “inline script,” and quote the path name, as follows: InlineScript { & "<insert full path to your program here>" "Z:\files\$($name)" }. See this Microsoft documentation page for more information.

Save this script to somewhere on your research data directory. In this example it is saved in a subdirectory called scripts.

Running the script on the Winanalytics VM

net use Z: \\storage.research.sesync.org\<your sesync username or group name>-data /persistent:no 
Set-ExecutionPolicy Unrestricted -Scope CurrentUser

You will be asked to enter “Y” to confirm.

cd Z:\scripts
.\<name of the script>.ps1

You will need to enter “R” to confirm that you want to run the script once. If your PowerShell script does not contain errors, it will run your program in parallel!

Related