Skip to content

Avoid unnecessary processing when namespace is terminating #12858

Open
0 of 1 issue completed
Open
@bartsmykla

Description

Kuma Version

0.0.0-preview.v7bf53c40b

Describe the bug

When deleting a namespace with an injected mesh, the system continues to process resources even though Kubernetes services and other objects may already be deleted. This results in unnecessary errors and processing. In one case, deleting a namespace with 800 services (2 replicas each) generated 125,000 log lines.

Two types of repeated errors occur:

  1. Pod-to-dataplane conversion errors due to missing services:

    ERROR discovery.k8s.pod-to-dataplane-converter could not get K8S Service for service tag {"error": "failed to get Service \"kuma-test/fake-service-799\": Service \"fake-service-799\" not found"}
    
  2. DataplaneInsight errors due to namespace termination:

    ERROR xds.sink failed to flush DataplaneInsight {"dataplaneid": {"Mesh":"default","Name":"fake-service-736-6b479fb987-zjsxr.kuma-test"}, "error": "failed to create k8s resource: dataplaneinsights.kuma.io \"fake-service-736-6b479fb987-zjsxr\" is forbidden: unable to create new content in namespace kuma-test because it is being terminated"}
    

To Reproduce

  1. Create a namespace with an injected mesh.
  2. Deploy ~1000 services with 2 replicas each.
  3. Delete the namespace.
  4. Observe logs attempting to convert pods to dataplanes and flush DataplaneInsights while the namespace is terminating.

Expected behavior

The system should detect namespace termination and avoid processing resources that may already be deleted.

Additional context (optional)

The system should recognize when a namespace is terminating and avoid unnecessary operations like pod-to-dataplane conversion and DataplaneInsight updates.

Sub-issues

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    kind/bugA bugtriage/acceptedThe issue was reviewed and is complete enough to start working on it

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions