Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add a timeout(e.g. 1min) of get disk in CSI driver during disk attach, to avoid get stuck in 12min #2716

Open
andyzhangx opened this issue Dec 12, 2024 · 0 comments

Comments

@andyzhangx
Copy link
Member

andyzhangx commented Dec 12, 2024

What happened:
add a timeout(e.g. 1min) of get disk in CSI driver during disk attach, to avoid get stuck in 12min

I1205 08:04:32.288626       1 utils.go:77] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1205 08:04:32.288705       1 utils.go:78] GRPC request: {"node_id":"aks-esse103az3-17761304-vmss000004","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"DiskIOPSReadWrite":"3000","DiskMBpsReadWrite":"125","cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-deb52d4d-113b-4d89-aeca-c80693a72331","csi.storage.k8s.io/pvc/name":"es-es-search-774d4bbd89-pfm82-elasticsearch-data","csi.storage.k8s.io/pvc/namespace":"project-da2e5a1d4e1140fd95a14735475025b9","requestedsizegib":"65","skuName":"PremiumV2_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1731592903910-8495-disk.csi.azure.com"},"volume_id":"/subscriptions/xxx/resourceGroups/aks-esp-stg-azucac-cp-internal-app-1/providers/Microsoft.Compute/disks/pvc-deb52d4d-113b-4d89-aeca-c80693a72331"}
time="2024-12-05T08:04:32Z" level=info msg="Proxy method: GET uri: /subscriptions/xxx/resourceGroups/aks-esp-stg-azucac-cp-internal-app-1/providers/Microsoft.Compute/disks/pvc-deb52d4d-113b-4d89-aeca-c80693a72331?api-version=2022-07-02, localid: 8f983235-5059-40f6-ac0b-d16f667c2b32"
time="2024-12-05T08:04:32Z" level=info msg="Original request uri: https://management.azure.com/subscriptions/xxx/resourceGroups/aks-esp-stg-azucac-cp-internal-app-1/providers/Microsoft.Compute/disks/pvc-deb52d4d-113b-4d89-aeca-c80693a72331?api-version=2022-07-02, target type: "
time="2024-12-05T08:20:09Z" level=error msg="Failed to proxy request, localid: 8f983235-5059-40f6-ac0b-d16f667c2b32, err: Get \"https://management.azure.com/subscriptions/xxx/resourceGroups/aks-esp-stg-azucac-cp-internal-app-1/providers/Microsoft.Compute/disks/pvc-deb52d4d-113b-4d89-aeca-c80693a72331?api-version=2022-07-02\": read tcp 172.31.55.164:33838->4.150.240.10:443: read: connection timed out"
I1205 08:20:09.818823       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/xxx/resourceGroups/aks-esp-stg-azucac-cp-internal-app-1/providers/Microsoft.Compute/disks/pvc-deb52d4d-113b-4d89-aeca-c80693a72331, error: Retriable: true, RetryAfter: 0s, HTTPStatusCode: -1, RawError: Get "http://localhost:7788/subscriptions/xxx/resourceGroups/aks-esp-stg-azucac-cp-internal-app-1/providers/Microsoft.Compute/disks/pvc-deb52d4d-113b-4d89-aeca-c80693a72331?api-version=2022-07-02": EOF
E1205 08:20:09.818869       1 utils.go:82] GRPC error: rpc error: code = NotFound desc = Volume not found, failed with error: Retriable: true, RetryAfter: 0s, HTTPStatusCode: -1, RawError: Get "http://localhost:7788/subscriptions/xxx/resourceGroups/aks-esp-stg-azucac-cp-internal-app-1/providers/Microsoft.Compute/disks/pvc-deb52d4d-113b-4d89-aeca-c80693a72331?api-version=2022-07-02": EOF

What you expected to happen:

How to reproduce it:

Anything else we need to know?:

Environment:

  • CSI Driver version:
  • Kubernetes version (use kubectl version):
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant