-
Notifications
You must be signed in to change notification settings - Fork 309
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ARCTIC-1025][FLINK] Fix data duplication even if there is a primary key table with upsert enabled #1180
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@zstraw Thanks for your contribution, I left some comments.
flink/v1.12/flink/src/test/java/com/netease/arctic/flink/table/TestKeyed.java
Outdated
Show resolved
Hide resolved
flink/v1.12/flink/src/main/java/com/netease/arctic/flink/write/ArcticFileWriter.java
Show resolved
Hide resolved
flink/v1.12/flink/src/main/java/com/netease/arctic/flink/write/ArcticFileWriter.java
Outdated
Show resolved
Hide resolved
Codecov ReportPatch coverage has no change and project coverage change:
Additional details and impacted files@@ Coverage Diff @@
## master #1180 +/- ##
=============================================
+ Coverage 27.86% 55.95% +28.09%
+ Complexity 4928 1884 -3044
=============================================
Files 657 287 -370
Lines 69172 12824 -56348
Branches 7978 1222 -6756
=============================================
- Hits 19274 7176 -12098
+ Misses 47986 5023 -42963
+ Partials 1912 625 -1287
Flags with carried forward coverage won't be shown. Click here to find out more. Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
FIX #1025
Why are the changes needed?
To make sure only one primary key in key table with upsert enabled, it's necessary to delete data firstly when update_after not followed by update_before.
It may lead to some unnecessary delete data if there are some data interspersed with other primary key data. e.g. K1-UB, K2-UB, K1-UA, K2-UB
Brief change log
How was this patch tested?
Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before making a pull request
Documentation