Share model via thunderbolt instead of re-download

[dardevelin]

Use the fast thunderbolt link already established to copy the model over to the machine that doesn't have it opposed to triggering a re-download.

[JDelgadoCMG]

That's an efficient bandwidth utilization and process optimization when a model already exists in a node. Also, I have seen that one node advances faster than another (50-60% percent difference) when downloading a new model. When the first node finishes, then waiting for the others to complete is a waste; better would be to set one node to download the model and then copy the model to the other node(s).

[andrewseago]

I was going to make this same feature request.

[AndreiCravtov]

This feature is most likely gated behind an overlay routing feature we may have planned at some point.

On this branch is an experimental version of this routing that currently achieves upto ~26gbps transfers from any connected node in the network to any other connected node in the network.