Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add Type Validation parameter for Pipeline Connections #8875

Open
wants to merge 31 commits into
base: main
Choose a base branch
from

Conversation

sjrl
Copy link
Contributor

@sjrl sjrl commented Feb 19, 2025

Related Issues

Proposed Changes

This PR introduces a new connection_type_validation parameter for type compatibility checks and adds _types_are_compatible, and _strict_types_are_compatible functions to support it. The connection_type_validation parameter allows users to configure whether type validation is performed or not. If True then type validation is performed if False then all connections are allowed.

Additionally, I allow this option to be set at the pipeline level or at the individual connection level. I've updated the to_dict and from_dict methods of the pipeline to reflect this.

How did you test it?

  • Added unit tests
  • Updated existing tests

Notes for the reviewer

@mpangrazzi I'd appreciate your thoughts on the name of the variable and whether you think it makes sense to have the option available at both the pipeline level and the connection level or if it would make sense to only have it present in one spot.

Checklist

@github-actions github-actions bot added the type:documentation Improvements on the docs label Feb 19, 2025
@coveralls
Copy link
Collaborator

coveralls commented Feb 19, 2025

Pull Request Test Coverage Report for Build 13455277332

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 23 unchanged lines in 2 files lost coverage.
  • Overall coverage increased (+0.008%) to 90.163%

Files with Coverage Reduction New Missed Lines %
core/type_utils.py 2 95.45%
core/pipeline/base.py 21 94.67%
Totals Coverage Status
Change from base Build 13455055613: 0.008%
Covered Lines: 9505
Relevant Lines: 10542

💛 - Coveralls

@anakin87
Copy link
Member

I have marked the failing HF API test -depending on HF API instability- as xfail, so if you merge main, the errors should disappear.

@sjrl sjrl changed the title feat: feat: New Type Validation Mode for Pipeline Connections Feb 19, 2025
@sjrl sjrl changed the title feat: New Type Validation Mode for Pipeline Connections feat: Add Type Validation parameter for Pipeline Connections Feb 20, 2025
@sjrl sjrl marked this pull request as ready for review February 20, 2025 09:30
@sjrl sjrl requested review from a team as code owners February 20, 2025 09:30
@sjrl sjrl requested review from dfokina and mpangrazzi and removed request for a team February 20, 2025 09:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
3 participants