Add update source API #5636
Open
Add update source API #5636
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rdettai
commented
Jan 16, 2025
rdettai
commented
Jan 16, 2025
guilload
reviewed
Jan 16, 2025
quickwit/quickwit-metastore/src/metastore/index_metadata/mod.rs
Outdated
Show resolved
Hide resolved
quickwit/quickwit-metastore/src/metastore/postgres/metastore.rs
Outdated
Show resolved
Hide resolved
rdettai
force-pushed
the
update-source
branch
from
6f7bcd2 to
39a8506
Compare
rdettai
commented
Jan 21, 2025
rdettai
force-pushed
the
update-source
branch
from
39a8506 to
abf7723
Compare
rdettai
commented
Jan 21, 2025
rdettai
force-pushed
the
update-source
branch
2 times, most recently
from
cc21afd to
d7c318d
Compare
Currently the pipeline fingerpring only contains index settings. The source setting changes should also trigger a pipeline restart. Note that this change will change all the existing pipeline fingerprints. This means that once the control plane runs this for the first time, it will restart all indexing pipelines.
Also include some comment and naming improvments.
guilload
reviewed
Jan 21, 2025
rdettai
force-pushed
the
update-source
branch
from
b630b98 to
ce61a97
Compare
guilload
approved these changes
Jan 23, 2025
| let mut hasher = SipHasher::new(); | ||
| self.doc_mapping.doc_mapping_uid.hash(&mut hasher); | ||
| self.indexing_settings.hash(&mut hasher); | ||
| hasher.finish() | ||
| } | ||
|
|
||
| /// Compare IndexConfig level fingerprints |
There was a problem hiding this comment.
Suggested change
| /// Compare IndexConfig level fingerprints | |
| /// Compares IndexConfig level fingerprints |
| fn validate_update(&self, _other: &Self) -> anyhow::Result<()> { | ||
| // In the Pulsar source, we use use combinations of the topic+partition | ||
| // (generated by the Pulsar client library) as metastore checkpoint | ||
| // PartitionId, and those are guarantied to be unique across topics. |
There was a problem hiding this comment.
Suggested change
| // PartitionId, and those are guarantied to be unique across topics. | |
| // PartitionId, and those are "guaranteed" to be unique across topics. |
| @@ -47,6 +47,7 @@ tracing = { workspace = true } | |||
| ulid = { workspace = true } | |||
| utoipa = { workspace = true } | |||
| vrl = { workspace = true, optional = true } | |||
| warp = { workspace = true, optional = true } | |||
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Closes #5634
How was this PR tested?
Describe how you tested this PR.