ETCD filling the ETCD_QUOTA_BACKEND_BYTES frequently and causing DB spaces exceeded

[IamSatyaonline]

Hi
We are suing ETCD 3.5.3 in our deployment. We are facing the DB space exceeded issue frequently. We are using lease to store key/values to ETCD but after expiry of lease Data is not getting removed from the ETCD .
Could you please help us to know is it the expected ETCD behavior ?

Thanks,
Satya

[vivekpatani]

@IamSatyaonline thank you for reaching out.

Do you have a test to consistently reproduce the problem? It might be difficult to advise, if the complete situation isn't available. The expected behavior as you mentioned is the keys should be removed when the Lease revoke is executed, which happens once the lease expires.

[IamSatyaonline]

It's not possible to reproduce the issue every time. We are not sure about the reproduction steps.

[IamSatyaonline]

Hi @vivekpatani
The restart happened on a worker node and due to it. we have seen several keys were not deleted and were consuming the volumes and resulted into DB space exceeded issue.

Thanks,
Satya

[vivekpatani]

Do you have any logs to kind of look into the problem?

[IamSatyaonline]

We have checked the logs but there is no information being printed of keys/values which belongs to expired leases

Thanks,
Satya

[jmhbnz]

Hey @IamSatyaonline - What are your settings for etcd database compaction and defragmentation?

Please refer to: https://etcd.io/docs/v3.5/op-guide/maintenance

[IamSatyaonline]

Hi @jmhbnz
Below are the configuration for compaction and we are running defragmentation every hour.
ETCD_AUTO_COMPACTION_MODE: "revision"
ETCD_AUTO_COMPACTION_RETENTION: 100

Thanks,
Satya