Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

enhance: Optimize the task scheduling logic in DataCoord #39084

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

enhance: Optimize the task scheduling logic in DataCoord #39084

wants to merge 1 commit into from
+1,377 −719

Conversation

xiaocai2333
Copy link
Contributor

@xiaocai2333 xiaocai2333 commented Jan 8, 2025 •
edited
Loading

issue: #39101

@sre-ci-robot sre-ci-robot added the size/L Denotes a PR that changes 100-499 lines. label Jan 8, 2025
@xiaocai2333 xiaocai2333 force-pushed the refine_index_scheduler branch from 5cc499d to 4fb4229 Compare January 8, 2025 09:14
@mergify mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Jan 8, 2025
@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 8, 2025

@xiaocai2333 Please associate the related issue to the body of your Pull Request. (eg. “issue: #”)

@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 8, 2025

@xiaocai2333 go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 8, 2025

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@codecov Codecov
Copy link

codecov bot commented Jan 8, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 79.59559% with 111 lines in your changes missing coverage. Please review.

Project coverage is 79.75%. Comparing base (316534e) to head (04c81d5).
Report is 10 commits behind head on master.

Files with missing lines Patch % Lines
internal/datacoord/task_queue.go 48.10% 40 Missing and 1 partial ⚠️
internal/datacoord/task_scheduler.go 85.15% 36 Missing and 2 partials ⚠️
internal/datacoord/session/indexnode_manager.go 74.28% 7 Missing and 2 partials ⚠️
internal/datacoord/task_index.go 64.00% 6 Missing and 3 partials ⚠️
pkg/util/merr/utils.go 0.00% 6 Missing ⚠️
internal/datacoord/task_analyze.go 55.55% 4 Missing ⚠️
internal/datacoord/task_stats.go 84.21% 1 Missing and 2 partials ⚠️
internal/datacoord/job_manager.go 0.00% 1 Missing ⚠️

❌ Your patch check has failed because the patch coverage (79.59%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #39084      +/-   ##
==========================================
- Coverage   79.79%   79.75%   -0.05%     
==========================================
  Files        1496     1501       +5     
  Lines      206993   207708     +715     
==========================================
+ Hits       165175   165649     +474     
- Misses      35821    36021     +200     
- Partials     5997     6038      +41
Components Coverage Δ
Client 79.24% <ø> (ø)
Core 69.62% <ø> (+0.04%) ⬆️
Go 81.40% <79.59%> (-0.07%) ⬇️
Files with missing lines Coverage Δ
internal/datacoord/index_meta.go 95.31% <100.00%> (-0.16%) ⬇️
internal/datacoord/server.go 73.87% <100.00%> (+0.54%) ⬆️
internal/datacoord/stats_task_meta.go 77.64% <100.00%> (+0.13%) ⬆️
internal/indexnode/indexnode.go 80.86% <100.00%> (+0.08%) ⬆️
internal/indexnode/indexnode_service.go 83.08% <100.00%> (+0.12%) ⬆️
internal/indexnode/task_analyze.go 80.14% <100.00%> (+0.28%) ⬆️
internal/indexnode/task_index.go 72.06% <100.00%> (+0.19%) ⬆️
internal/indexnode/task_scheduler.go 88.66% <100.00%> (+1.48%) ⬆️
internal/indexnode/task_stats.go 70.46% <100.00%> (+0.91%) ⬆️
internal/indexnode/util.go 61.29% <100.00%> (+18.43%) ⬆️
... and 10 more

... and 44 files with indirect coverage changes

@mergify mergify bot added the ci-passed label Jan 8, 2025
@xiaocai2333
Copy link
Contributor Author

rerun go-sdk

@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 9, 2025

@xiaocai2333 go-sdk check failed, comment rerun go-sdk can trigger the job again.

@xiaocai2333 xiaocai2333 force-pushed the refine_index_scheduler branch from a69d6ee to 242bff0 Compare January 9, 2025 03:22
@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 9, 2025

@xiaocai2333 cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 9, 2025

@xiaocai2333 go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 9, 2025

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@xiaocai2333
Copy link
Contributor Author

/run-cpu-e2e

@mergify mergify bot added the ci-passed label Jan 9, 2025
@xiaocai2333 xiaocai2333 force-pushed the refine_index_scheduler branch from 9d0bd08 to 30fbbf6 Compare January 13, 2025 11:49
@sre-ci-robot sre-ci-robot added size/XL Denotes a PR that changes 500-999 lines. and removed size/L Denotes a PR that changes 100-499 lines. labels Jan 13, 2025
@xiaocai2333 xiaocai2333 force-pushed the refine_index_scheduler branch from 30fbbf6 to 4b4d61f Compare January 13, 2025 11:49
@mergify mergify bot removed the ci-passed label Jan 13, 2025
@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 13, 2025

@xiaocai2333 go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify mergify bot added the ci-passed label Jan 20, 2025
@xiaocai2333 xiaocai2333 force-pushed the refine_index_scheduler branch from 2fd0e7d to ad9c3a8 Compare February 8, 2025 03:25
@mergify mergify bot removed the ci-passed label Feb 8, 2025
@xiaocai2333
Copy link
Contributor Author

/unhold

@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 8, 2025

@xiaocai2333 go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 8, 2025

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@xiaocai2333 xiaocai2333 force-pushed the refine_index_scheduler branch from 32db11e to a8e8126 Compare February 10, 2025 08:15
@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 10, 2025

@xiaocai2333 go-sdk check failed, comment rerun go-sdk can trigger the job again.

@xiaocai2333
Copy link
Contributor Author

rerun go-sdk

@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 10, 2025

@xiaocai2333 go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 10, 2025

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@xiaocai2333 xiaocai2333 force-pushed the refine_index_scheduler branch from a8e8126 to e3fe504 Compare February 12, 2025 11:00
@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 12, 2025

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 12, 2025

@xiaocai2333 cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 13, 2025

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 13, 2025

@xiaocai2333 go-sdk check failed, comment rerun go-sdk can trigger the job again.

@xiaocai2333 xiaocai2333 force-pushed the refine_index_scheduler branch from bf3752a to 86a0300 Compare February 17, 2025 01:57
@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 17, 2025

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@xiaocai2333 xiaocai2333 force-pushed the refine_index_scheduler branch from 86a0300 to a0b57fe Compare February 18, 2025 04:02
@xiaocai2333 xiaocai2333 force-pushed the refine_index_scheduler branch from a0b57fe to 04c81d5 Compare February 18, 2025 06:38
@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 18, 2025

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@xiaocai2333
Copy link
Contributor Author

/run-cpu-e2e

@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 19, 2025

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@xiaocai2333 xiaocai2333 mentioned this pull request Feb 21, 2025
Merged
sre-ci-robot pushed a commit that referenced this pull request Feb 22, 2025
@xiaocai2333
enhance: Refine index task scheduler policy (#40104)
be30f53 
master pr: #39084 

issue: #39101

---------

Signed-off-by: Cai Zhang <[email protected]>
@xiaocai2333 xiaocai2333 mentioned this pull request Feb 23, 2025
Merged
sre-ci-robot pushed a commit that referenced this pull request Feb 23, 2025
@xiaocai2333
fix: [2.5] Fix task scheduler dead lock (#40121)
6c81463 
issue: #39101 

master pr: #39084

Signed-off-by: cai.zhang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@czs007 czs007 czs007 left review comments

@bigsheeper bigsheeper Awaiting requested review from bigsheeper

@congqixia congqixia Awaiting requested review from congqixia

Assignees
No one assigned
Labels
area/test dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement sig/testing size/XXL Denotes a PR that changes 1000+ lines.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@xiaocai2333 @sre-ci-robot @czs007