http connection leak

[tiancaiamao]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

In the master branch we have the lint check for http close #28755

In 5.1 5.2 it's not added...
and some user meet the connection leak issues like https://asktug.com/t/topic/183219/16

2. What did you expect to see? (Required)

golangci-lint run -c ./.golangci.yml --disable-all -E bodyclose

3. What did you see instead (Required)

Some error detected, for example in 5.1.1

4. What is your TiDB version? (Required)

[tiancaiamao]

A hotfix for 5.1.1 is #30570

We need to check the 5.1 5.2 5.3 branch and fix all of them

[tiancaiamao]

https://github.com/pingcap/tidb/pull/30570/files#diff-d19169611c4613e5d3f09dc49da1c5d99bc6c7adb90a4b2ebbd75ac869806f50R697

This is the most important change, it visits "/pd/api/v1/config"

One possible case call stack could be from telemetry, and the rate is 1min/op

goroutine 643 [running]:
runtime/debug.Stack(0x0, 0x26, 0x0)
        /home/genius/project/go/src/runtime/debug/stack.go:24 +0x9f
runtime/debug.PrintStack()
        /home/genius/project/go/src/runtime/debug/stack.go:16 +0x25
github.com/pingcap/tidb/domain/infosync.(*InfoSyncer).getPrometheusAddr(0xc0005a6880, 0x13c33cad3a0, 0x55b5ce0, 0x551e70799, 0x7f4c0c2d9e68)
        /home/genius/project/src/github.com/pingcap/tidb/domain/infosync/info.go:700 +0x1b0
github.com/pingcap/tidb/domain/infosync.GetPrometheusAddr(0x203000, 0x8, 0x55eeb48, 0x203000)
        /home/genius/project/src/github.com/pingcap/tidb/domain/infosync/info.go:661 +0xb2
github.com/pingcap/tidb/telemetry.querySQLMetric(0x3c14568, 0xc000054090, 0xc06492c52bbc21b2, 0x14185b1d689, 0x55b5ce0, 0x37af363, 0x2e, 0x0, 0x0, 0x0, ...)
        /home/genius/project/src/github.com/pingcap/tidb/telemetry/data_window.go:123 +0xa5
github.com/pingcap/tidb/telemetry.readSQLMetric(0xc06492c52bbc21b2, 0x14185b1d689, 0x55b5ce0, 0xc00113a068, 0x2, 0xc000dbdf8c)
        /home/genius/project/src/github.com/pingcap/tidb/telemetry/data_window.go:109 +0x70
github.com/pingcap/tidb/telemetry.RotateSubWindow()
        /home/genius/project/src/github.com/pingcap/tidb/telemetry/data_window.go:191 +0x293
github.com/pingcap/tidb/domain.(*Domain).TelemetryRotateSubWindowLoop.func1(0xc00014e870)
        /home/genius/project/src/github.com/pingcap/tidb/domain/domain.go:1100 +0x65
created by github.com/pingcap/tidb/domain.(*Domain).TelemetryRotateSubWindowLoop
        /home/genius/project/src/github.com/pingcap/tidb/domain/domain.go:1089 +0x7f

Other possible way to trigger it is visit the METRICS_SCHEMA tables, for example,

use METRICS_SCHEMA;
select * from up;

The call stack is

goroutine 643 [running]:
runtime/debug.Stack(0x0, 0x26, 0x0)
        /home/genius/project/go/src/runtime/debug/stack.go:24 +0x9f
runtime/debug.PrintStack()
        /home/genius/project/go/src/runtime/debug/stack.go:16 +0x25
github.com/pingcap/tidb/domain/infosync.(*InfoSyncer).getPrometheusAddr(0xc0005a6880, 0x1796773891a, 0x55b5ce0, 0xdf85e98bc, 0xc0018ff680)
        /home/genius/project/src/github.com/pingcap/tidb/domain/infosync/info.go:700 +0x1b0
github.com/pingcap/tidb/domain/infosync.GetPrometheusAddr(0x203000, 0x8, 0x0, 0x203000)
        /home/genius/project/src/github.com/pingcap/tidb/domain/infosync/info.go:661 +0xb2
github.com/pingcap/tidb/telemetry.querySQLMetric(0x3c14568, 0xc000054090, 0xc06493102c77b059, 0x1875fd21d30, 0x55b5ce0, 0x37af363, 0x2e, 0x0, 0x0, 0x0, ...)
        /home/genius/project/src/github.com/pingcap/tidb/telemetry/data_window.go:123 +0xa5
github.com/pingcap/tidb/telemetry.readSQLMetric(0xc06493102c77b059, 0x1875fd21d30, 0x55b5ce0, 0xc000653f68, 0x2, 0xc0009a4f8c)
        /home/genius/project/src/github.com/pingcap/tidb/telemetry/data_window.go:109 +0x70
github.com/pingcap/tidb/telemetry.RotateSubWindow()
        /home/genius/project/src/github.com/pingcap/tidb/telemetry/data_window.go:191 +0x293
github.com/pingcap/tidb/domain.(*Domain).TelemetryRotateSubWindowLoop.func1(0xc00014e870)
        /home/genius/project/src/github.com/pingcap/tidb/domain/domain.go:1100 +0x65
created by github.com/pingcap/tidb/domain.(*Domain).TelemetryRotateSubWindowLoop
        /home/genius/project/src/github.com/pingcap/tidb/domain/domain.go:1089 +0x7f
goroutine 12831 [running]:
runtime/debug.Stack(0x0, 0x26, 0x0)
        /home/genius/project/go/src/runtime/debug/stack.go:24 +0x9f
runtime/debug.PrintStack()
        /home/genius/project/go/src/runtime/debug/stack.go:16 +0x25
github.com/pingcap/tidb/domain/infosync.(*InfoSyncer).getPrometheusAddr(0xc0005a6880, 0x1875fe2bf7d, 0x55b5ce0, 0x2df12fdcb, 0x38)
        /home/genius/project/src/github.com/pingcap/tidb/domain/infosync/info.go:700 +0x1b0
github.com/pingcap/tidb/domain/infosync.GetPrometheusAddr(0x2dee1ec6c1ea2561, 0x2c91b58, 0x2d0000c000092c00, 0x0)
        /home/genius/project/src/github.com/pingcap/tidb/domain/infosync/info.go:661 +0xb2
github.com/pingcap/tidb/executor.(*MetricRetriever).queryMetric(0xc001a52d60, 0x3c145d8, 0xc001130300, 0x3c53fd0, 0xc00134c000, 0xc064927d44bc2a21, 0xfe8c2968f8, 0x55b5ce0, 0xc064931344bc2a21, 0x18a3ef2d8f8, ...)
        /home/genius/project/src/github.com/pingcap/tidb/executor/metrics_reader.go:99 +0xa5
github.com/pingcap/tidb/executor.(*MetricRetriever).retrieve(0xc001a52d60, 0x3c145d8, 0xc001130300, 0x3c53fd0, 0xc00134c000, 0x0, 0x0, 0xc000a44c28, 0x116d6dd, 0xc00076cfc0)
        /home/genius/project/src/github.com/pingcap/tidb/executor/metrics_reader.go:77 +0x2ef
github.com/pingcap/tidb/executor.(*MemTableReaderExec).Next(0xc000879710, 0x3c145d8, 0xc001130300, 0xc00189b5e0, 0xc0003f7c70, 0xc00118a000)
        /home/genius/project/src/github.com/pingcap/tidb/executor/memtable_reader.go:113 +0x497
github.com/pingcap/tidb/executor.Next(0x3c145d8, 0xc001130300, 0x3c18838, 0xc000879710, 0xc00189b5e0, 0x0, 0x0)
        /home/genius/project/src/github.com/pingcap/tidb/executor/executor.go:286 +0x2de
github.com/pingcap/tidb/executor.(*recordSet).Next(0xc00189b540, 0x3c145d8, 0xc001130300, 0xc00189b5e0, 0x0, 0x0)
        /home/genius/project/src/github.com/pingcap/tidb/executor/adapter.go:145 +0xb5
github.com/pingcap/tidb/server.(*tidbResultSet).Next(0xc00189b590, 0x3c145d8, 0xc001130300, 0xc00189b5e0, 0xc00137e360, 0x400)
        /home/genius/project/src/github.com/pingcap/tidb/server/driver_tidb.go:304 +0x51
github.com/pingcap/tidb/server.(*clientConn).writeChunks(0xc000de0100, 0x3c145d8, 0xc001130300, 0x3c2ac60, 0xc00189b590, 0x20000, 0x55e8de0, 0x0, 0x0)
        /home/genius/project/src/github.com/pingcap/tidb/server/conn.go:1875 +0x4de
github.com/pingcap/tidb/server.(*clientConn).writeResultset(0xc000de0100, 0x3c145d8, 0xc001130300, 0x3c2ac60, 0xc00189b590, 0x2ac00, 0x0, 0x0, 0x0, 0x0)
        /home/genius/project/src/github.com/pingcap/tidb/server/conn.go:1823 +0x20e
github.com/pingcap/tidb/server.(*clientConn).handleStmt(0xc000de0100, 0x3c14530, 0xc001130300, 0x3c2eca8, 0xc001976780, 0x55e8de0, 0x0, 0x0, 0x1, 0x0, ...)
        /home/genius/project/src/github.com/pingcap/tidb/server/conn.go:1717 +0x3e5
github.com/pingcap/tidb/server.(*clientConn).handleQuery(0xc000de0100, 0x3c14530, 0xc00076cfc0, 0xc0010a2379, 0x10, 0x0, 0x0)
        /home/genius/project/src/github.com/pingcap/tidb/server/conn.go:1563 +0x492
github.com/pingcap/tidb/server.(*clientConn).dispatch(0xc000de0100, 0x3c14530, 0xc00076cfc0, 0xc0010a2378, 0x11, 0x10, 0x0, 0x0)
        /home/genius/project/src/github.com/pingcap/tidb/server/conn.go:1097 +0xafd
github.com/pingcap/tidb/server.(*clientConn).Run(0xc000de0100, 0x3c145d8, 0xc001a28180)
        /home/genius/project/src/github.com/pingcap/tidb/server/conn.go:860 +0x296
github.com/pingcap/tidb/server.(*Server).onConn(0xc000f7e340, 0xc000de0100)
        /home/genius/project/src/github.com/pingcap/tidb/server/server.go:485 +0xa7e
created by github.com/pingcap/tidb/server.(*Server).Run

[bb7133]

@tiancaiamao Can we close this issue because of #30462 and #28756? Usually when an issue is fixed in master branch, we can close it and treat it as backlog for backporting for release branch.

[tiancaiamao]

OK, now I've verified the following actions:

• Hotfix for 5.1.1
• Make sure we have the lint check on master branch so we can prevent this kind of error in the future
• Check and cherry pick to 5.1
• Check and cherry pick to 5.2
• Check and cherry pick to 5.3, just a tiny fix, 5.3 is not really affected