changefeed get stuck ...in test case "cdc_scale_sync"

[Tammyxia]

What did you do?

• test infra test case: cdc_scale_sync, main branch
• the changefeed get stuck already for 6 hours: https://tcms.pingcap.net/dashboard/executions/plan/630680
• owner is not stuck because if creating new changefeed, the new one works fine.

What did you expect to see?

No response

What did you see instead?

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

main

TiCDC version (execute cdc version):

[release-version=v5.5.0-nightly] [git-hash=7a227b421dbfcdafee02148e787138798edadf31] [git-branch=heads/refs/tags/v5.5.0-nightly] [utc-build-time="2022-01-24 18:10:05"] [go-version="go version go1.16.4 linux/amd64"] [failpoint-build=false]

[Tammyxia]

Reproduced again in main version.
After cdc scale finished for about 2 hours, while waiting for sync task completing, changefeed stuck.
It seems tikv is the main reason, because tikv has no grpc message output any more from metrics

And tikv log ERROR:
[2022/02/06 20:36:48.101 +00:00] [ERROR] [transport.rs:86] ["failed to send significant msg"] [msg="CaptureChange { cmd: ChangeObserver { cdc_id: Some(ObserveHandle { id: ObserveID(1174), observing: true }), rts_id: None, region_id: 1228 }, region_epoch: conf_ver: 5 version: 99, callback: Callback::Read(..) }"]
[2022/02/06 20:36:48.101 +00:00] [ERROR] [endpoint.rs:637] ["cdc initialize fail: Request error message: "region 1228 not found" region_not_found { region_id: 1228 }"] [region_id=1228]
[2022/02/06 20:36:48.109 +00:00] [ERROR] [transport.rs:86] ["failed to send significant msg"] [msg="CaptureChange { cmd: ChangeObserver { cdc_id: Some(ObserveHandle { id: ObserveID(1177), observing: true }), rts_id: None, region_id: 2 }, region_epoch: conf_ver: 5 version: 101, callback: Callback::Read(..) }"]
[2022/02/06 20:36:48.109 +00:00] [ERROR] [endpoint.rs:637] ["cdc initialize fail: Request error message: "region 2 not found" region_not_found { region_id: 2 }"] [region_id=2]

[Tammyxia]

Reproduce again in v5.1.4.

[Tammyxia]

Reproduced this issue in v6.0.