Python: Passing data without passing it

In this post I will show you how to pass data to places where you cannot pass it as arguments.

As an example, let’s say we have a function that we cannot change which calls our code (MyAction.action). Because we cannot change function_i_cannot_change, we cannot make it pass dynamic_value through to our code. Also note that we cannot pass something else than a number as the arg parameter because of its implementation. (Yes, in this simple example there would be alternative solutions, see note at the end.)

from typing import Callable

def function_i_cannot_change(callback : Callable[[int], int], arg : int) -> int:
    arg += 1
    # lots of code
    result = callback(arg)
    # more code
    return result


def action(arg):
    # hmm, how do I get `dynamic_value` into this method?
    return arg + dynamic_value


def usage():
    dynamic_value = 100
    function_i_cannot_change(action, 1)

We can solve this by passing the data through a global variable.

DYNAMIC_GLOBAL = 1

def action(arg):
    return arg + DYNAMIC_GLOBAL

def usage():
    global DYNAMIC_GLOBAL 
    DYNAMIC_GLOBAL = 100
    function_i_cannot_change(action, 1)  # returns 102

Yes, you’re right… Using globals is bad. It makes your code hard to follow and breaks thread-safety.

To address the threading issue, we can use a threading.local instead:

import threading

DYNAMIC_TL = threading.local()
DYNAMIC_TL.value = 1

def action(arg):
    return arg + DYNAMIC_TL.value

def usage():
    DYNAMIC_TL.value = 100
    function_i_cannot_change(action, 1)  # returns 102

This solves threading issues, but we still have to know the thread local in the consuming code and we have to trust all consumers to be well-behaved concerning what they set on DYNAMIC_TL.value and that they reset it to the original value.

Let’s use a context manager to make the usage more safe and clear for the consumers.

import threading
import contextlib

DYNAMIC_TL = threading.local()
DYNAMIC_TL.value = 1

@contextlib.contextmanager
def override_dynamic_value(value):
    # we can do assertions on the value here
    old_value = DYNAMIC_TL.value
    DYNAMIC_TL.value = value
    yield
    DYNAMIC_TL.value = old_value

def get_dynamic_value():
    return DYNAMIC_TL.value


def action(arg):
    return arg + get_dynamic_value()


def usage():
    with override_dynamic_value(100):
        function_i_cannot_change(action, 1)  # returns 102

I also added get_dynamic_value to hide the usage of the thread local from the implementation of action, too.

This sounds a bit crazy, does anybody really do this?

Yes. Have you ever wondered how flask.request works?

Can I really use this?

Well, I would say: Yes, if there is no simpler alternative. The pattern comes with the general problem of global variables:

a) They are available everywhere and thus people will starting using them in more and more places. This leads to unintended coupling.

b) The dataflow is not obvious. You will get surprising and hard to debug errors.

For me, this means that I only use the pattern in a very explicit way that makes sure it will be used responsibly. I would move the thread local and it’s accessors to their own module and make the thread local itself private.

I often find myself using this technique in tests, when I need to get 3rd party code to cooperate. In production code I would try very hard to find an alternative.

Note: Alternative solutions for the simple example

The simple example has a couple pretty simple, preferable solutions,

E.g. there is partial application of functions:

import functools

def action(dynamic_value, arg):
    return arg + dynamic_value

def usage():
    function_i_cannot_change(functools.partial(action, 100), 1)  # returns 102

Or a function that returns a function:

def action_factory(dynamic_value):
    def action(arg):
        return arg + dynamic_value

def usage():
    function_i_cannot_change(action_factory(100), 1)  # returns 102

Or a class:

class MyAction:
    def __init__(self, dynamic_value):
        self.dynamic_value = dynamic_value

    def action(self, arg):
        return arg + self.dynamic_value

def usage():
    function_i_cannot_change(MyAction(100), 1)  # returns 102

Moserei