Closed
Description
Enhancement
Now tidb will save an ExecDetails for each request, and will not release the memory until the end of the SQL. In generally, the number of requests that need to be sent by a SQL is not very large, so this part of the memory will not be very large. But in some extreme scenarios, when the number of requests reaches 10 million, the memory usage of this part will reach more than GB. This may cause frequent GC, CPU usage and memory usage. Therefore, this part of memory also needs to be tracked in order to intervene in extreme scenarios.
For example,
Reproduce Steps
use test;
create table base(t int);
insert into base values(1);
insert into base select * from base;
insert into base select * from base;
insert into base select * from base;
insert into base select * from base;
insert into base select * from base;
insert into base select * from base;
insert into base select * from base;
insert into base select * from base;
insert into base select * from base;
insert into base select * from base;
create table base2(a int);
insert into base2 select row_number() over() from base;
insert into base2 select a from base2;
insert into base2 select a from base2;
insert into base2 select a from base2;
insert into base2 select a from base2;
insert into base2 select a from base2;
insert into base2 select a from base2;
insert into base2 select a from base2;
insert into base2 select a from base2;
insert into base2 select a from base2;
insert into base2 select a from base2;
create table t(pk int primary key auto_increment, a int, b int, index idx(a));
insert into t(a) select a from base2;
insert into t(a) select a from base2;
insert into t(a) select a from base2;
insert into t(a) select a from base2;
insert into t(a) select a from base2;
insert into t(a) select a from base2;
insert into t(a) select a from base2;
insert into t(a) select a from base2;
insert into t(a) select a from base2;
insert into t(a) select a from base2;
insert into t(a) select a from base2;
insert into t(a) select a from base2;
set tidb_wait_split_region_finish=on;
split table t between (0) and (1048576) regions 1000;
split table t between (1048576) and (1048576*2) regions 1000;
split table t between (1048576*2) and (1048576*3) regions 1000;
split table t between (1048576*3) and (1048576*4) regions 1000;
split table t between (1048576*4) and (1048576*5) regions 1000;
split table t between (1048576*5) and (1048576*6) regions 1000;
split table t between (1048576*6) and (1048576*7) regions 1000;
split table t between (1048576*7) and (1048576*8) regions 1000;
split table t between (1048576*8) and (1048576*9) regions 1000;
split table t between (1048576*9) and (1048576*10) regions 1000;
split table t between (1048576*10) and (1048576*11) regions 1000;
split table t between (1048576*11) and (1048576*12) regions 1000;
set tidb_index_lookup_size = 10000;
desc analyze select /*+ use_index(t,idx) */ count(b) from t;
One SQL will send 12 million request and the runtime info will occury 1GB memory.
Activity