BUG: Race in PyArray_UpdateFlags under free threading

[hawkinsp]

Describe the issue:

thread-sanitizer reports a race in PyArray_UpdateFlags under free threading.

It may take a few runs to reproduce this race.

Reproduce the code example:

import concurrent.futures
import functools
import threading

import numpy as np

num_threads = 8


def closure(b, x):
  b.wait()
  for _ in range(100):
    list(x.flat)

with concurrent.futures.ThreadPoolExecutor(max_workers=num_threads) as executor:
  for _ in range(100):
    b = threading.Barrier(num_threads)
    x = np.arange(20).reshape(5, 4).T
    for _ in range(num_threads):
      executor.submit(functools.partial(closure, b, x))

Error message:

WARNING: ThreadSanitizer: data race (pid=409824)
  Write of size 4 at 0x7fadb72e6db0 by thread T135:
    #0 PyArray_UpdateFlags <null> (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x296e4b) (BuildId: a138a702a237ca030803af4168ee423ada9702f7)
    #1 PyArray_RawIterBaseInit <null> (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x2ac9cd) (BuildId: a138a702a237ca030803af4168ee423ada9702f7)
    #2 PyArray_IterNew <null> (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x2ad200) (BuildId: a138a702a237ca030803af4168ee423ada9702f7)
    #3 array_flat_get getset.c (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x29a045) (BuildId: a138a702a237ca030803af4168ee423ada9702f7)
    #4 getset_get /usr/local/google/home/phawkins/p/cpython/Objects/descrobject.c:193:16 (python3.13+0x1ffa68) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #5 _PyObject_GenericGetAttrWithDict /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1635:19 (python3.13+0x295de1) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #6 PyObject_GenericGetAttr /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1717:12 (python3.13+0x295c32) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #7 PyObject_GetAttr /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1231:18 (python3.13+0x294597) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #8 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:3766:28 (python3.13+0x3eddf0) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #9 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de62a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #10 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1811:12 (python3.13+0x3de62a)
    #11 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb61f) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #12 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x571bb2) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #13 partial_vectorcall /usr/local/google/home/phawkins/p/cpython/./Modules/_functoolsmodule.c:252:16 (python3.13+0x571bb2)
    #14 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb293) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #15 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb293)
    #16 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb315) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #17 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e46e2) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #18 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de62a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #19 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1811:12 (python3.13+0x3de62a)
    #20 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb61f) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #21 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef5ef) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #22 method_vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef5ef)
    #23 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb293) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #24 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb293)
    #25 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb315) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #26 thread_run /usr/local/google/home/phawkins/p/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x564292) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #27 pythread_wrapper /usr/local/google/home/phawkins/p/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bd637) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)

  Previous write of size 4 at 0x7fadb72e6db0 by thread T132:
    #0 PyArray_UpdateFlags <null> (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x296e4b) (BuildId: a138a702a237ca030803af4168ee423ada9702f7)
    #1 PyArray_RawIterBaseInit <null> (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x2ac9cd) (BuildId: a138a702a237ca030803af4168ee423ada9702f7)
    #2 PyArray_IterNew <null> (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x2ad200) (BuildId: a138a702a237ca030803af4168ee423ada9702f7)
    #3 array_flat_get getset.c (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x29a045) (BuildId: a138a702a237ca030803af4168ee423ada9702f7)
    #4 getset_get /usr/local/google/home/phawkins/p/cpython/Objects/descrobject.c:193:16 (python3.13+0x1ffa68) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #5 _PyObject_GenericGetAttrWithDict /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1635:19 (python3.13+0x295de1) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #6 PyObject_GenericGetAttr /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1717:12 (python3.13+0x295c32) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #7 PyObject_GetAttr /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1231:18 (python3.13+0x294597) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #8 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:3766:28 (python3.13+0x3eddf0) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #9 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de62a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #10 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1811:12 (python3.13+0x3de62a)
    #11 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb61f) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #12 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x571bb2) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #13 partial_vectorcall /usr/local/google/home/phawkins/p/cpython/./Modules/_functoolsmodule.c:252:16 (python3.13+0x571bb2)
    #14 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb293) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #15 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb293)
    #16 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb315) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #17 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e46e2) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #18 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de62a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #19 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1811:12 (python3.13+0x3de62a)
    #20 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb61f) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #21 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef5ef) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #22 method_vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef5ef)
    #23 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb293) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #24 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb293)
    #25 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb315) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #26 thread_run /usr/local/google/home/phawkins/p/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x564292) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #27 pythread_wrapper /usr/local/google/home/phawkins/p/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bd637) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)

Python and NumPy Versions:

2.3.0.dev0+git20241219.35b2c4a
3.13.1 experimental free-threading build (tags/v3.13.1:06714517797, Dec 15 2024, 15:38:01) [Clang 18.1.8 (11)]

Runtime Environment:

[{'numpy_version': '2.3.0.dev0+git20241219.35b2c4a',
'python': '3.13.1 experimental free-threading build '
'(tags/v3.13.1:06714517797, Dec 15 2024, 15:38:01) [Clang 18.1.8 '
'(11)]',
'uname': uname_result(system='Linux', node='', release='', version='https://github.com//pull/1 SMP PREEMPT_DYNAMIC Debian 6.redacted (2024-10-16)', machine='x86_64')},
{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
'found': ['SSSE3',
'SSE41',
'POPCNT',
'SSE42',
'AVX',
'F16C',
'FMA3',
'AVX2'],
'not_found': ['AVX512F',
'AVX512CD',
'AVX512_KNL',
'AVX512_SKX',
'AVX512_CLX',
'AVX512_CNL',
'AVX512_ICL']}},
{'architecture': 'Zen',
'filepath': '/usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.27.so',
'internal_api': 'openblas',
'num_threads': 128,
'prefix': 'libopenblas',
'threading_layer': 'pthreads',
'user_api': 'blas',
'version': '0.3.27'}]

Context for the issue:

Found when working on free-threading support in JAX.

[seberg]

This call to updateflags is just an unnecessary remnant of a more defensive past: Flags must always be up to date.