If Safeguard probe is invalid, CTK just ignores it and continues running.

[WixoLeo]

Here is an example of a simple experiment:

{
  "version": "1.0.0",
  "title": "test",
  "description": "test",
  "configuration": {
    "myarg": "exp.json",
    "another_arg": "another.json"
  },
  "tags": [
    "network",
    "lala"
  ],
  "controls": [
    {
      "name": "Safeguards",
      "provider": {
        "type": "python",
        "module": "chaosaddons.controls.safeguards",
        "arguments": {
          "probes": [
            {
              "name": "Non existing probe",
              "description": "Probe that doesn't exist",
              "type": "probe",
              "provider": {
                "type": "python",
                "module": "blablabla",
                "func": "bla",
                "arguments": {}
              },
              "tolerance": 2
            }
          ]
        }
      }
    }
  ],
  "steady-state-hypothesis": {
    "title": "Check something",
    "probes": [
      {
        "name": "My probe",
        "description": "My best probe",
        "tolerance": {
          "type": "regex",
          "pattern": "exp.json",
          "target": "stdout"
        },
        "type": "probe",
        "provider": {
          "type": "process",
          "path": "echo",
          "arguments": "${myarg}"
        }
      }
    ]
  },
  "method": [
    {
      "name": "My probe",
      "description": "My best probe",
      "tolerance": {
        "type": "regex",
        "pattern": "exp.json",
        "target": "stdout"
      },
      "type": "probe",
      "provider": {
        "type": "process",
        "path": "echo",
        "arguments": "${myarg}"
      }
    }
  ],
  "rollbacks": [
    {
      "name": "My probe",
      "description": "My best probe",
      "tolerance": {
        "type": "regex",
        "pattern": "exp.json",
        "target": "stdout"
      },
      "type": "probe",
      "provider": {
        "type": "process",
        "path": "echo",
        "arguments": "${myarg}"
      }
    }
  ]
}```

Notice the probe with module and function as "blabla"
Obviously it doesn't exists. But if you run the experiment, it just runs normally by ignoring that probe.
If I put the real probe there, it works fine.

I think the expected result would be some kind of an exception and abort everything.
We've been running experiments for a long time without knowing that we've had a non working probe.

[Lawouach]

Oh right. The ctk validator is unaware that this particular control uses probes to achieve its goal. So they are not validated.

I can't make the validator support that but I could bring the validation into the control itself, when we configure it.

[WixoLeo]

Oh right. The ctk validator is unaware that this particular control uses probes to achieve its goal. So they are not validated.

I can't make the validator support that but I could bring the validation into the control itself, when we configure it.

Sure, as long as it stops the experiment and throws an error, would be fantastic :)

[Lawouach]

Yeap. I can't work on this today but I'll try tomorrow, ping me otherwise :)

[WixoLeo]

Awesome thank you! @Lawouach

[Lawouach]

Hey @WixoLeo could you try the latest commit on master and see if that works?

[WixoLeo]

Hey @WixoLeo could you try the latest commit on master and see if that works?

Hi, sorry for late response, I tested it, and still the same problem for some reason

[WixoLeo]

Python:

__all__ = [
    "some_probe"
]

def some_probe():
    print("Hello mate")
    return True

exp.json

{
  "version": "1.0.0",
  "title": "test",
  "description": "test",
  "configuration": {
    "myarg": "exp.json",
    "another_arg": "another.json"
  },
  "variables": {
    "selected_instance": null
  },
  "tags": [
    "network",
    "lala"
  ],
  "controls": [
    {
      "name": "Safeguards",
      "provider": {
        "type": "python",
        "module": "chaosaddons.controls.safeguards",
        "arguments": {
          "probes": [
            {
              "name": "Non existing probe",
              "description": "Probe that doesn't exist",
              "type": "probe",
              "provider": {
                "type": "python",
                "module": "testoplugin.testo.probes1",
                "func": "some_probe",
                "arguments": {}
              },
              "tolerance": true
            }
          ]
        }
      }
    }
  ],
  "steady-state-hypothesis": {
    "title": "Check something",
    "probes": [
      {
        "name": "Non existing probe",
        "description": "Probe that doesn't exist",
        "type": "probe",
        "provider": {
          "type": "python",
          "module": "testoplugin.testo.probes",
          "func": "some_probe",
          "arguments": {}
        },
        "tolerance": true
      }
    ]
  },
  "method": [
    {
      "name": "My probe",
      "description": "My best probe",
      "tolerance": {
        "type": "regex",
        "pattern": "exp.json",
        "target": "stdout"
      },
      "type": "probe",
      "provider": {
        "type": "process",
        "path": "echo",
        "arguments": "${myarg}"
      }
    }
  ],
  "rollbacks": [
    {
      "name": "My probe",
      "description": "My best probe",
      "tolerance": {
        "type": "regex",
        "pattern": "exp.json",
        "target": "stdout"
      },
      "type": "probe",
      "provider": {
        "type": "process",
        "path": "echo",
        "arguments": "${myarg}"
      }
    }
  ]
}```

[Lawouach]

Found the reason why my fix wasn't enough. I'll have to update the ctk core lib and release it for this. On it.

[Lawouach]

Once this one is merged and released, i'll update this repo accordingly chaostoolkit/chaostoolkit-lib#225

[WixoLeo]

Once this one is merged and released, i'll update this repo accordingly chaostoolkit/chaostoolkit-lib#225

Awesome, thank you @Lawouach, absolute pro!

[Lawouach]

Hey @WixoLeo I have updated the master branch with the changes from ctklib 1.20.0, feel free to give it a spin :)

[WixoLeo]

Hey @WixoLeo I have updated the master branch with the changes from ctklib 1.20.0, feel free to give it a spin :)

So I've tested it, and it seems like it's still not working..
I used ctk, ctk-lib and addons from master branch

[Lawouach]

Could you upload a ctk log file here and your experiment again?

[WixoLeo]

chaostoolkit.log