Evaluating runs with overrides

[Aceticia]

When there are overrides for an experiment and they affect the model architecture, simply doing things like python src/eval.py ckpt=path/to/ckpt/this.ckpt doesn't cut it since the configs are not restored. What would be one way to get this addressed?

The way I'm currently doing this is via python src/eval.py ckpt=path/to/ckpt/this.ckpt and then manually appending all of the old overrides. Since my checkpoint folders are named by my overrides.

My 2 cents:

1. For eval task, we add some lines of code to optionally load the .hydra folder.
2. Before calling the eval task, we override the default configs/eval.yaml with the specified .hydra folder for your past run.

This should be nice since things like checkpoint path would also be contained in the logs in .hydra folder. What do you think?