[NMT] Add support for multiple validation and test dataloaders #2113
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: ericharper <[email protected]>
Signed-off-by: ericharper <[email protected]>
Signed-off-by: ericharper <[email protected]>
Signed-off-by: ericharper <[email protected]>
Signed-off-by: ericharper <[email protected]>
Signed-off-by: ericharper <[email protected]>
Signed-off-by: ericharper <[email protected]>
Signed-off-by: ericharper <[email protected]>
Signed-off-by: ericharper <[email protected]>
Signed-off-by: ericharper <[email protected]>
Signed-off-by: ericharper <[email protected]>
Signed-off-by: ericharper <[email protected]>
Signed-off-by: ericharper <[email protected]>
okuchaiev
reviewed
Apr 26, 2021
nemo/collections/nlp/data/machine_translation/machine_translation_dataset.py
Show resolved
Hide resolved
Signed-off-by: ericharper <[email protected]>
okuchaiev
approved these changes
Apr 26, 2021
Signed-off-by: ericharper <[email protected]>
Signed-off-by: ericharper <[email protected]>
Signed-off-by: ericharper <[email protected]>
Signed-off-by: ericharper <[email protected]>
karpnv
pushed a commit
to karpnv/NeMo
that referenced
this pull request
May 21, 2021
…A#2113) * add multiple validation dataloaders Signed-off-by: ericharper <[email protected]> * loop over outputs in eval_epoch_end Signed-off-by: ericharper <[email protected]> * looping over val dataloaders in eval epoch end Signed-off-by: ericharper <[email protected]> * style Signed-off-by: ericharper <[email protected]> * add sync_dist Signed-off-by: ericharper <[email protected]> * rearrange name Signed-off-by: ericharper <[email protected]> * compute eval loss for each data loader Signed-off-by: ericharper <[email protected]> * set loss attr for each val dl Signed-off-by: ericharper <[email protected]> * set loss attr for each val dl Signed-off-by: ericharper <[email protected]> * default to dl index 0 Signed-off-by: ericharper <[email protected]> * default to dl index 0 Signed-off-by: ericharper <[email protected]> * clean up Signed-off-by: ericharper <[email protected]> * normalize one val dataset outputs Signed-off-by: ericharper <[email protected]> * add jenkins test for multi-val Signed-off-by: ericharper <[email protected]> * add multi-test Signed-off-by: ericharper <[email protected]> * add multi-test Signed-off-by: ericharper <[email protected]> * add multi-test Signed-off-by: ericharper <[email protected]> * add testing to jenkins tests Signed-off-by: ericharper <[email protected]> Signed-off-by: Карпов Николай Вячеславович <[email protected]>
michalivne
pushed a commit
to michalivne/NeMo
that referenced
this pull request
Jun 23, 2021
…A#2113) * add multiple validation dataloaders Signed-off-by: ericharper <[email protected]> * loop over outputs in eval_epoch_end Signed-off-by: ericharper <[email protected]> * looping over val dataloaders in eval epoch end Signed-off-by: ericharper <[email protected]> * style Signed-off-by: ericharper <[email protected]> * add sync_dist Signed-off-by: ericharper <[email protected]> * rearrange name Signed-off-by: ericharper <[email protected]> * compute eval loss for each data loader Signed-off-by: ericharper <[email protected]> * set loss attr for each val dl Signed-off-by: ericharper <[email protected]> * set loss attr for each val dl Signed-off-by: ericharper <[email protected]> * default to dl index 0 Signed-off-by: ericharper <[email protected]> * default to dl index 0 Signed-off-by: ericharper <[email protected]> * clean up Signed-off-by: ericharper <[email protected]> * normalize one val dataset outputs Signed-off-by: ericharper <[email protected]> * add jenkins test for multi-val Signed-off-by: ericharper <[email protected]> * add multi-test Signed-off-by: ericharper <[email protected]> * add multi-test Signed-off-by: ericharper <[email protected]> * add multi-test Signed-off-by: ericharper <[email protected]> * add testing to jenkins tests Signed-off-by: ericharper <[email protected]> Signed-off-by: Micha Livne <[email protected]>
mousebaiker
pushed a commit
to mousebaiker/NeMo
that referenced
this pull request
Jul 8, 2021
…A#2113) * add multiple validation dataloaders Signed-off-by: ericharper <[email protected]> * loop over outputs in eval_epoch_end Signed-off-by: ericharper <[email protected]> * looping over val dataloaders in eval epoch end Signed-off-by: ericharper <[email protected]> * style Signed-off-by: ericharper <[email protected]> * add sync_dist Signed-off-by: ericharper <[email protected]> * rearrange name Signed-off-by: ericharper <[email protected]> * compute eval loss for each data loader Signed-off-by: ericharper <[email protected]> * set loss attr for each val dl Signed-off-by: ericharper <[email protected]> * set loss attr for each val dl Signed-off-by: ericharper <[email protected]> * default to dl index 0 Signed-off-by: ericharper <[email protected]> * default to dl index 0 Signed-off-by: ericharper <[email protected]> * clean up Signed-off-by: ericharper <[email protected]> * normalize one val dataset outputs Signed-off-by: ericharper <[email protected]> * add jenkins test for multi-val Signed-off-by: ericharper <[email protected]> * add multi-test Signed-off-by: ericharper <[email protected]> * add multi-test Signed-off-by: ericharper <[email protected]> * add multi-test Signed-off-by: ericharper <[email protected]> * add testing to jenkins tests Signed-off-by: ericharper <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Specify multiple validation (or test) dataloaders by inputting a list of src (tgt) files.
Usage
Replace
src_file_nameandtgt_file_namewith a list of file paths.Becomes:
When using
val_lossorval_sacreBLEUfor theexp_manager.checkpoint_callback_params.monitorthen the 0th indexed dataloader will be used as the monitor. This ensures backwards compatibility.Additionally, any validation dataloader can now be used as a monitor by appending the index as below:
Multiple test datasets works exactly the same way as validation datasets, simply replace
validation_dsbytest_dsin the above examples.