Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Add ES Auto Index Cleaner #6425

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

akstron
Copy link
Contributor

@akstron akstron commented Dec 27, 2024

Which problem is this PR solving?

Description of the changes

  • Added an index cleaner

How was this change tested?

  • Not tested yet

Checklist

Signed-off-by: Alok Kumar Singh <[email protected]>
Signed-off-by: Alok Kumar Singh <[email protected]>
Signed-off-by: Alok Kumar Singh <[email protected]>
Copy link

codecov bot commented Dec 27, 2024

Codecov Report

Attention: Patch coverage is 0% with 66 lines in your changes missing coverage. Please review.

Project coverage is 34.64%. Comparing base (39e3bfb) to head (6370ca0).
Report is 93 commits behind head on main.

Files with missing lines Patch % Lines
plugin/storage/es/cleaner/cleaner.go 0.00% 35 Missing ⚠️
pkg/es/client/index_client.go 0.00% 22 Missing ⚠️
plugin/storage/es/factory.go 0.00% 9 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (39e3bfb) and HEAD (6370ca0). Click for more details.

HEAD has 8 uploads less than BASE
Flag BASE (39e3bfb) HEAD (6370ca0)
kafka-v1 1 0
elasticsearch-6.x-v1 1 0
opensearch-1.x-v1 1 0
opensearch-2.x-v1 1 0
elasticsearch-7.x-v1 1 0
elasticsearch-8.x-v1 1 0
unittests 1 0
kafka-v2 1 0
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #6425       +/-   ##
===========================================
- Coverage   96.20%   34.64%   -61.57%     
===========================================
  Files         356      192      -164     
  Lines       20416    11643     -8773     
===========================================
- Hits        19642     4034    -15608     
- Misses        585     7301     +6716     
- Partials      189      308      +119     
Flag Coverage Δ
badger_v1 10.17% <0.00%> (+1.27%) ⬆️
badger_v2 2.92% <0.00%> (+1.29%) ⬆️
cassandra-4.x-v1-manual 15.84% <0.00%> (+1.00%) ⬆️
cassandra-4.x-v2-auto 2.86% <0.00%> (+1.28%) ⬆️
cassandra-4.x-v2-manual 2.86% <0.00%> (+1.28%) ⬆️
cassandra-5.x-v1-manual 15.84% <0.00%> (+1.00%) ⬆️
cassandra-5.x-v2-auto 2.86% <0.00%> (+1.28%) ⬆️
cassandra-5.x-v2-manual 2.86% <0.00%> (+1.28%) ⬆️
elasticsearch-6.x-v1 ?
elasticsearch-7.x-v1 ?
elasticsearch-8.x-v1 ?
elasticsearch-8.x-v2 2.95% <0.00%> (+1.31%) ⬆️
grpc_v1 11.77% <0.00%> (+1.40%) ⬆️
grpc_v2 8.97% <0.00%> (+1.07%) ⬆️
kafka-2.x-v1 10.02% <0.00%> (?)
kafka-2.x-v2 2.95% <0.00%> (?)
kafka-3.x-v1 10.02% <0.00%> (?)
kafka-3.x-v2 2.95% <0.00%> (?)
kafka-v1 ?
kafka-v2 ?
memory_v2 2.95% <0.00%> (+1.31%) ⬆️
opensearch-1.x-v1 ?
opensearch-2.x-v1 ?
opensearch-2.x-v2 2.95% <0.00%> (+1.31%) ⬆️
tailsampling-processor 0.54% <0.00%> (+0.08%) ⬆️
unittests ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

i.logger.Info("Indices before this date will be deleted", zap.String("date", deleteIndicesBefore.Format(time.RFC3339)))

indices = filter.ByDate(indices, deleteIndicesBefore)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will add check to not delete indices with write alias.

@@ -41,6 +44,32 @@ type IndicesClient struct {
MasterTimeoutSeconds int
}

// Create the indices only client using config.Configuration
func CreateIndicesClient(c *config.Configuration, logger *zap.Logger) (*IndicesClient, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't follow why we need this function. The pattern in es.Factory is to use config.NewClient, which takes configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants