q: best way to sweep over optimizers

[SauravMaheshkar]

Related: #4

The intention behind #4 was to enable easily experimenting with various optimizer's for a single strategy. At the moment, to do this one would have to instantiate a new strategy for each new optimizer and then call the .fit() method on a trainer. Considering the example from the README.

trainer = LocalTrainer(model, train_dataset, val_dataset)

for optim in ["adam", "sgd"]:
	strategy = DiLoCoStrategy(
	  inner_optim=optim,
	  H=100
	)
	
	trainer.fit(
	  strategy=strategy,
	  num_nodes=4,
	  device='mps'
	)

Opening this issue to start a conversation about potential alternatives (if at all) for this. If the only thing that's changed are optimizer's (or inner steps or any other hyper-parameter) does one need to instantiate a new strategy for each?

Possible Solutions

• a single configuration shared between Trainer and Strategy.
• allow changing Strategy attributes from within .fit() (as proposed in feat: support passing optimizers as strings #4)
• This is the way?