Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature][Spark]: Support duplicate-key-check when insert into and insert overwrite #484

Closed
2 tasks done
baiyangtx opened this issue Oct 18, 2022 · 0 comments · Fixed by #522
Closed
2 tasks done

[Feature][Spark]: Support duplicate-key-check when insert into and insert overwrite #484

baiyangtx opened this issue Oct 18, 2022 · 0 comments · Fixed by #522
Assignees
@hellojinsilei
Labels
module:mixed-spark Spark module for Mixed Format type:feature Feature Requests
Milestone
Release 0.4.0

Comments

@baiyangtx
Copy link
Contributor

Description

for current, our insert overwrite && insert into dosen't check for duplicate key of input data.

Use case/motivation

insert into xxx as select * from source.
insert overwrite xxx as select * from source.

the source table may has duplicate row with same key. arctic spark writer shoud check input data.

Describe the solution

No response

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hellojinsilei hellojinsilei

Labels
module:mixed-spark Spark module for Mixed Format type:feature Feature Requests
Projects
None yet
Milestone
Release 0.4.0
Development

Successfully merging a pull request may close this issue.

[SPARK]Feature support duplicate-key-check when insert into hellojinsilei/arctic
2 participants
@baiyangtx @hellojinsilei