TypeError 'NoneType' object is not subscriptable

[mthompson313]

I am seeing an issue where asgiref/local.py is throwing a TypeError 'NoneType' object is not subscriptable

this is being thrown from:

def __getattr__(self, key):
        storage_object = self._data.get({})
        try:
            return storage_object[key]
        except KeyError:
            raise AttributeError(f"{self!r} object has no attribute {key!r}")

[andrewgodwin]

Could I get a full traceback please plus the code that's calling it?

[mthompson313]

Full Stack Trace:

TypeError: 'NoneType' object does not support item assignment
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "ddtrace/contrib/trace_utils.py", line 334, in wrapper
    return func(mod, pin, wrapped, instance, args, kwargs)
  File "ddtrace/contrib/django/patch.py", line 302, in wrapped
    return func(*args, **kwargs)
  File "axes/middleware.py", line 37, in __call__
    response = self.get_response(request)
  File "asgiref/local.py", line 124, in __setattr__
    setattr(storage, key, value)
  File "asgiref/local.py", line 28, in __setattr__
    storage_object[key] = value
TypeError: 'NoneType' object is not subscriptable
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "ddtrace/contrib/trace_utils.py", line 334, in wrapper
    return func(mod, pin, wrapped, instance, args, kwargs)
  File "ddtrace/contrib/django/patch.py", line 302, in wrapped
    return func(*args, **kwargs)
  File "discovery_service/middleware.py", line 119, in __call__
    response = self.get_response(request)
  File "django/core/handlers/exception.py", line 57, in inner
    response = response_for_exception(request, exc)
  File "django/core/handlers/exception.py", line 141, in response_for_exception
    request, get_resolver(get_urlconf()), sys.exc_info()
  File "django/urls/base.py", line 145, in get_urlconf
    return getattr(_urlconfs, "value", default)
  File "asgiref/local.py", line 118, in __getattr__
    return getattr(storage, key)
  File "asgiref/local.py", line 19, in __getattr__
    return storage_object[key]
TypeError: 'NoneType' object is not subscriptable
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "ddtrace/contrib/trace_utils.py", line 334, in wrapper
    return func(mod, pin, wrapped, instance, args, kwargs)
  File "ddtrace/contrib/django/patch.py", line 302, in wrapped
    return func(*args, **kwargs)
  File "discovery_service/middleware.py", line 16, in __call__
    response = self.get_response(request)
  File "django/core/handlers/exception.py", line 57, in inner
    response = response_for_exception(request, exc)
  File "django/core/handlers/exception.py", line 141, in response_for_exception
    request, get_resolver(get_urlconf()), sys.exc_info()
  File "django/urls/base.py", line 145, in get_urlconf
    return getattr(_urlconfs, "value", default)
  File "asgiref/local.py", line 118, in __getattr__
    return getattr(storage, key)
  File "asgiref/local.py", line 19, in __getattr__
    return storage_object[key]
TypeError: 'NoneType' object is not subscriptable
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "ddtrace/contrib/trace_utils.py", line 334, in wrapper
    return func(mod, pin, wrapped, instance, args, kwargs)
  File "ddtrace/contrib/django/patch.py", line 302, in wrapped
    return func(*args, **kwargs)
  File "django/utils/deprecation.py", line 134, in __call__
    response = response or self.get_response(request)
  File "django/core/handlers/exception.py", line 57, in inner
    response = response_for_exception(request, exc)
  File "django/core/handlers/exception.py", line 141, in response_for_exception
    request, get_resolver(get_urlconf()), sys.exc_info()
  File "django/urls/base.py", line 145, in get_urlconf
    return getattr(_urlconfs, "value", default)
  File "asgiref/local.py", line 118, in __getattr__
    return getattr(storage, key)
  File "asgiref/local.py", line 19, in __getattr__
    return storage_object[key]
TypeError: 'NoneType' object is not subscriptable
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "ddtrace/contrib/trace_utils.py", line 334, in wrapper
    return func(mod, pin, wrapped, instance, args, kwargs)
  File "ddtrace/contrib/django/patch.py", line 302, in wrapped
    return func(*args, **kwargs)
  File "django/utils/deprecation.py", line 134, in __call__
    response = response or self.get_response(request)
  File "django/core/handlers/exception.py", line 57, in inner
    response = response_for_exception(request, exc)
  File "django/core/handlers/exception.py", line 141, in response_for_exception
    request, get_resolver(get_urlconf()), sys.exc_info()
  File "django/urls/base.py", line 145, in get_urlconf
    return getattr(_urlconfs, "value", default)
  File "asgiref/local.py", line 118, in __getattr__
    return getattr(storage, key)
  File "asgiref/local.py", line 19, in __getattr__
    return storage_object[key]

[mthompson313]

@andrewgodwin the code calling it I am having hard time finding as this came from a sentry alert on a production box. but the farthest down line given is:

 else:
        @wraps(get_response)
        def inner(request):
            try:
                response = get_response(request)
            except Exception as exc:
                response = response_for_exception(request, exc)
            return response
        return inner

from django/core/handlers/exception.py

[andrewgodwin]

Yeah, we're not going to be able to confirm if this is a true bug or not unless it's possible to get a reproduceable test case that doesn't have ddtrace in there mucking things up.

[mthompson313]

I believe the most recent commit to local.py should fix this issue

[natdempk]

Hey folks, I have also seen this in production seemingly in random methods where I'd have no other reason to suspect anything is weird, and across a wide-enough spread of endpoints / application code that I would probably rule out the application/endpoint code itself. We do also run ddtrace.

The stack traces look like this:

TypeError: 'NoneType' object does not support item assignment
  File "django/core/handlers/base.py", line 253, in _get_response_async
    response = await wrapped_callback(
  File "ddtrace/contrib/trace_utils.py", line 335, in wrapper
    return func(mod, pin, wrapped, instance, args, kwargs)
  File "ddtrace/contrib/django/patch.py", line 303, in wrapped
    return func(*args, **kwargs)
  File "ninja/operation.py", line 400, in _sync_view
    return operation.run(request, *a, **kw)
  File "ninja/operation.py", line 114, in run
    return self.api.on_exception(request, e)
  File "ninja/main.py", line 490, in on_exception
    return handler(request, exc)
  File "ninja/errors.py", line 108, in _default_exception
    raise exc  # let django deal with it
  File "ninja/operation.py", line 107, in run
    result = self.view_func(request, **values)
  File "our_api/foo/bar/api.py", line 214, in get_foo_info
    foo_quota = async_to_sync(usage.get_one_time_foo_usage)(request.user)
  File "asgiref/local.py", line 124, in __setattr__
    setattr(storage, key, value)
  File "asgiref/local.py", line 28, in __setattr__
    storage_object[key] = value
TypeError: 'NoneType' object is not subscriptable
  File "django/core/handlers/exception.py", line 42, in inner
    response = await get_response(request)
  File "django/core/handlers/base.py", line 257, in _get_response_async
    response = await sync_to_async(
  File "asgiref/local.py", line 118, in __getattr__
    return getattr(storage, key)
  File "asgiref/local.py", line 19, in __getattr__
    return storage_object[key]

From reading #473 it sounds like the changes there would probably fix our issue. Symptoms there sound right to cause the type of issue this stack trace is referring to from reading the code.

I can't reproduce this locally, but if we had a release with those changes, I could upgrade to it easily and see if the errors stop within a few days. We see this relatively consistently, but I don't know that our production setup has an easy way for me to put the latest main into production via source.

@andrewgodwin / @carltongibson would it be possible to cut a release with the changes from local.py so we can try out the fix? Is there any reason that fix hasn't had a release yet?

[carltongibson]

@natdempk Thanks for the extra info! This could then be a duplicate of #473. (Still difficult to say 100% without a reproduce...)

You can install directly from git using Pip's VCS support: https://pip.pypa.io/en/stable/topics/vcs-support/ — If you could confirm it that fixes the issue for you that would be a great help.

Likely we can roll a release in the new year.

Happy holidays! 🎄

[natdempk]

With the latest main running, I infrequently see a new Sentry/stack trace rarely:

AttributeError: <asgiref.local._CVar object at 0x7fcfdd027880> object has no attribute 'value'
  File "django/core/handlers/asgi.py", line 160, in __call__
    await self.handle(scope, receive, send)
  File "django/core/handlers/asgi.py", line 190, in handle
    await self.send_response(response, send)
  File "django/core/handlers/asgi.py", line 296, in send_response
    await sync_to_async(response.close, thread_sensitive=True)()
  File "concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "ddtrace/contrib/futures/threading.py", line 36, in _wrap_execution
    return fn(*args, **kwargs)
  File "django/http/response.py", line 335, in close
    signals.request_finished.send(sender=self._handler_class)
  File "django/dispatch/dispatcher.py", line 176, in send
    return [
  File "django/dispatch/dispatcher.py", line 177, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "django/core/handlers/base.py", line 370, in reset_urlconf
    set_urlconf(None)
  File "django/urls/base.py", line 137, in set_urlconf
    del _urlconfs.value
  File "asgiref/local.py", line 129, in __delattr__
    delattr(storage, key)
  File "asgiref/local.py", line 38, in __delattr__
    raise AttributeError(f"{self!r} object has no attribute {key!r}")

Not really sure why __delattr__ in _CVar needs to throw if we try to delete an attribute that doesn't exist. Does this indicate a programming/synchronization bug somewhere orr is this just a bit over-restricting?

[natdempk]

We saw the error above arguably more frequently than we saw the original error, similarly across a wide and random spread of requests/endpoints for a small subset of requests, so I backed out the latest main, but the stack traces were basically always the same. Just putting one more in this comment in case its helpful, but I think its identical to the above.

I created #485 for this issue as it is probably blocking a release here given frequency we observe it at.

AttributeError: <asgiref.local._CVar object at 0x7f05030333a0> object has no attribute 'value'
  File "django/core/handlers/asgi.py", line 160, in __call__
    await self.handle(scope, receive, send)
  File "django/core/handlers/asgi.py", line 190, in handle
    await self.send_response(response, send)
  File "django/core/handlers/asgi.py", line 296, in send_response
    await sync_to_async(response.close, thread_sensitive=True)()
  File "concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "ddtrace/contrib/futures/threading.py", line 36, in _wrap_execution
    return fn(*args, **kwargs)
  File "django/http/response.py", line 335, in close
    signals.request_finished.send(sender=self._handler_class)
  File "django/dispatch/dispatcher.py", line 176, in send
    return [
  File "django/dispatch/dispatcher.py", line 177, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "django/core/handlers/base.py", line 370, in reset_urlconf
    set_urlconf(None)
  File "django/urls/base.py", line 137, in set_urlconf
    del _urlconfs.value
  File "asgiref/local.py", line 129, in __delattr__
    delattr(storage, key)
  File "asgiref/local.py", line 38, in __delattr__
    raise AttributeError(f"{self!r} object has no attribute {key!r}")