Dispy parallelization issue

[hecmay]

When launching multiple independent DATuner workflows sequentially on different machines, only the first works and all other flows will just hang as following, without any further response.

2018-08-19 20:35:18 dispynode - dispynode version: 4.9.1, PID: 25844
2018-08-19 20:35:18 pycos - version 4.8.0 with epoll I/O notifier
2018-08-19 20:35:18 dispynode - "en-ec-zhang08.coecis.cornell.edu" serving 8 cpus
2018-08-19 20:35:18 dispy - Transfer of computation "tune_function" to 128.84.48.155 failed
2018-08-19 20:35:18 dispy - Failed to setup 128.84.48.155 for compute "tune_function": -1
2018-08-19 20:35:18 dispynode - Computation "1534725316722" is not valid

[jg925]

In order to be able to run multiple independent DATuner instances on the same server regardless of user, dispyscheduler needs to be implemented. Currently, DATuner only allows for parallelization among one instance through the '-p' argument (letting the maximum be 8 -- the max number of cpus total) and through the configuration file indicating how many machines will be used. (If 2 machines are used, the maximum for '-p' increases to 16.)

[hecmay]

Sorry for the missing words. I was trying to launch multiple independent DATuner instances on multiple machines. And only the first instance will work.

[hecmay]

The error is most likely caused by communication conflicts in the local area network with multiple dispy job clusters. A possible solution is to add an unique identifier to each cluster with --secret parameter like:

# assign a uuid for each cluster
import uuid
secret = uuid.uuid4()
cluster = dispy.JobCluster(tune_function,
                           depends = ['package.zip'],
                           secret  = str(secret),
                           cleanup = False)

and assign the same uuid with servers (running dispynode.py) in datuner.py

  for i in range(len(machines)):
    machine_addr = machines[i % len(machines)]

    subprocess.call(['scp', DATUNER_HOME + '/releases/Linux_x86_64/install/bin/dispynode.py', machine_addr + ':' +workspace]);
    sshProcess = subprocess.Popen(['ssh',
                                   machine_addr],
                                   stdin=subprocess.PIPE,
                                   stdout=subprocess.PIPE,
                                   universal_newlines=True,
                                   bufsize=0)
    sshProcess.stdin.write("cd " + workspace + "\n")
    sshProcess.stdin.write("python dispynode.py --serve 1 --clean --secret " + str(secret) + " --dest_path_prefix dispytmp_" + str(i) + "\n")
    sshProcess.stdin.close()

[zhangzhiru]

@hecmay Can you elaborate? What does communication conflict mean?
Also, is "--secret" a standard option defined for dispy? If so, what is the typical usage of this option according to the dispy document?

[zhangzhiru]

I also notice that we are hardcoding "/releases/Linux_x86_64/install/bin/" in the program. This piece of code will break our tool when it's compiled on a 32b machine.

[hecmay]

1. Communication conflict means that the server (worker) nodes cannot match with and receive computation tasks correctly from the client (master) node they belong to, since there are more than one clusters existing on the local areas network, but without valid information to determine their identities.

2. secret is a pre-defined standard option in dispy/dispynode. In the document of dispy (master):

secret is a string that is (hashed and) used for handshaking of communication with nodes; i.e., this cluster will only work with nodes that use same secret.

and the secret of the dispynode (worker) nodes will be assigned by parameter --secret like
python dispynode.py --serve 1 --clean --secret 1111

[zhangzhiru]

Can we add a unit test to make sure we don't run into the same issue again?

[hecmay]

Sure, the unit test for this case can be implemented by setting up the CircleCI configuration.