Detect unexpected errors via logs in envtest and e2e tests

[sbueringer]

It would be great if we find a way to surface unexpected conditions via logs and then check in envtest and e2e tests that they never occur.

Of course this requires us to have an easy way to identify such logs (e.g. they could contain a specific keyword).

Recently we implemented something similar for panics. After envtest and e2e tests are completed we check controller-runtime metrics to ensure no panics occured

[sbueringer]

/triage accepted

But next step is

Of course this requires us to have an easy way to identify such logs (e.g. they could contain a specific keyword).

[sbueringer]

/help

[k8s-ci-robot]

@sbueringer:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[faermanj]

@sbueringer I'm interested in helping with this one. Do we want to catch known patterns (rules based) or anything out of the ordinary (ML-based)?

[sbueringer]

Nothing fancy, ideally just looking for a substring. We have to figure out this one first though:

Of course this requires us to have an easy way to identify such logs (e.g. they could contain a specific keyword).