Training restarts from epoch 1 on rerun even though previous epoch checkpoints exist

[mukherjeesougata-eros]

My training stopped midway due to an issue. Before stopping, checkpoints for epoch 1, epoch 2, and epoch 3 were successfully saved. However, when I rerun the training script using the same configuration and checkpoint directory, the training starts again from epoch 1 instead of resuming from the latest saved checkpoint. New checkpoints are created in the same path, but the previously saved checkpoints do not seem to be picked up for resuming.

Can you kindly please fix this so that the training automatically resumes from the latest checkpoint?

[mukherjeesougata-eros]

@zhu-han Can you kindly please get this fix done?

[zhu-han]

You should pass --start-epoch argument to the training script to resume from middle epochs.

See https://github.com/k2-fsa/ZipVoice/blob/master/zipvoice/bin/train_zipvoice.py#L139

[mukherjeesougata-eros]

You should pass --start-epoch argument to the training script to resume from middle epochs.

See https://github.com/k2-fsa/ZipVoice/blob/master/zipvoice/bin/train_zipvoice.py#L139

May I know how can I resume my training from the last saved checkpoint instead of resuming it from a particular epoch?

[mukherjeesougata-eros]

You should pass --start-epoch argument to the training script to resume from middle epochs.
See https://github.com/k2-fsa/ZipVoice/blob/master/zipvoice/bin/train_zipvoice.py#L139

May I know how can I resume my training from the last saved checkpoint instead of resuming it from a particular epoch?

Can I get this done after passing --checkpoint as exp-dir/checkpoint-{params.batch_idx_train}.pt in the training script @zhu-han ?

[mukherjeesougata-eros]

You should pass --start-epoch argument to the training script to resume from middle epochs.
See https://github.com/k2-fsa/ZipVoice/blob/master/zipvoice/bin/train_zipvoice.py#L139

May I know how can I resume my training from the last saved checkpoint instead of resuming it from a particular epoch?

Can I get this done after passing --checkpoint as exp-dir/checkpoint-{params.batch_idx_train}.pt in the training script where checkpoint-{params.batch_idx_train}.pt is the last saved checkpoint? @zhu-han

[zhu-han]

You should pass --start-epoch argument to the training script to resume from middle epochs.
See https://github.com/k2-fsa/ZipVoice/blob/master/zipvoice/bin/train_zipvoice.py#L139

May I know how can I resume my training from the last saved checkpoint instead of resuming it from a particular epoch?

Can I get this done after passing --checkpoint as exp-dir/checkpoint-{params.batch_idx_train}.pt in the training script @zhu-han ?

Passing --checkpoint will fine-tune the corresponding checkpoint with from-scratch optimizer states and learning rate. Passing --start-epoch will resume the learning rate and optimizer states from the specified epoch. If you don't want to change the training code and want to resume from latest checkpoint. You can remane your latest checkpoint to epoch-{*}.pt and then use --start-epoch to resume from that checkpoint.