Skip to content

retry-toolkit

(Yet Another) Retry implementation for python.

Do you have code that may be subjected to intermittent failures? Then you should have a retry wapper for it. This module includes a simple retry decorator (factory) you can use. Or peek inside and copy the implementation into your own project where you can make your own tweaks.

No dependencies outside of standard python libraries

Installation

pip install retry-toolkit

Examples

Defaults to 3 tries, no delays between, retry all exceptions:

from retry.simple import retry

@retry()
def foo():
    some_networking_stuff()

Customize the basic behaviors like so:

from retry.simple import retry

@retry(tries=4, backoff=1, exceptions=SomeConnectionError)
def foo():
    some_other_networking_stuff()

The arguments can take callables for more customization.

API Reference

retry_toolkit.simple.retry

Decorator factory, enables retries.

This is a decorator factory with some arguments to customize the retry behavior. Either specify constants or callables that will return the appropriate constants.

The parameters can also be set to callables returning the type indicated below. For backoff, the callable must be able to take a single argument which will be the retry number (1 for first retry, 2 for second, etc).

Parameters:

Name Type Description Default
tries int | Callable[[], int]

Number of times to try an operation including the first attempt which is not technically a RE-try.

If set to a callable, it must have signature:

() -> int

If not present in arguments, Defaults.TRIES is used.

 None
backoff int | Callable[[int], int]

Value in seconds of an amount to wait before next attempt. Can also be set to a callable taking the number of retries that must return the time to wait.

If set to a callable, it must have signature:

(int) -> float

If not present in arguments, Defaults.BACKOFF is used.

 None
exceptions type(Exception) | ExceptionTuple | ExceptionFunc

Defines the exceptions to to catch for retrying. Exceptions thrown that are not caught will bypass further retries, be raised normally, and not result in a GiveUp being thrown.

if set to a callable, it must have signature:

() -> tuple[Exception,...] | Exception

If not present in arguments, Defaults.EXC is used.

 None

Returns:

Type Description
*

This decorator factory returns a decorator used to wrap a function. The wrapped function will have retry behavior and when called it will return whatever it normally would.

Raises:

Type Description
GiveUp

Thrown when retries are exhausted.
Source code in src/retry_toolkit/simple.py 
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
def retry(
    tries      : int | Callable[[],int] = None,
    backoff    : int | Callable[[int],int] = None,
    exceptions : type(Exception) | ExceptionTuple | ExceptionFunc = None
    ):
    '''Decorator factory, enables retries.

    This is a decorator factory with some arguments to customize the retry
    behavior. Either specify constants or callables that will return the
    appropriate constants.

    The parameters can also be set to callables returning the type indicated
    below. For backoff, the callable must be able to take a single argument
    which will be the retry number (1 for first retry, 2 for second, etc).

    Parameters
    ----------
    tries:
        Number of times to try an operation including the first attempt which
        is not technically a RE-try.

        If set to a callable, it must have signature:

            () -> int

        If not present in arguments, Defaults.TRIES is used.

    backoff:
        Value in seconds of an amount to wait before next attempt. Can also
        be set to a callable taking the number of retries that must return
        the time to wait.

        If set to a callable, it must have signature:

            (int) -> float

        If not present in arguments, Defaults.BACKOFF is used.

    exceptions:
        Defines the exceptions to to catch for retrying. Exceptions thrown that
        are not caught will bypass further retries, be raised normally, and
        not result in a GiveUp being thrown.

        if set to a callable, it must have signature:

            () -> tuple[Exception,...] | Exception

        If not present in arguments, Defaults.EXC is used.

    Returns
    -------
    : *
        This decorator factory returns a decorator used to wrap a function. The
        wrapped function will have retry behavior and when called it will return
        whatever it normally would.

    Raises
    ------
    GiveUp
        Thrown when retries are exhausted.
    '''

    def _retry_decorator(func):
        @wraps(func)
        def _retry_wrapper(*args, **kwargs):
            # configure at call-time to allow any changes to defaults
            # to properly take effect each time func is used
            n_tries_f = _ensure_callable(tries      , Defaults.TRIES  )
            backoff_f = _ensure_callable(backoff    , Defaults.BACKOFF)
            exc_f     = _ensure_callable(exceptions , Defaults.EXC    )
            sleep_f   = Defaults.SLEEP_FUNC

            n_tries = n_tries_f()
            exc     = exc_f()

            # context/state
            total_sleep    = 0.0
            exception_list = []

            for try_num in range(n_tries):

                if try_num > 0:
                    sleep_time = backoff_f(try_num-1)
                    total_sleep += sleep_time
                    sleep_f(sleep_time)

                try:
                    return func(*args, **kwargs)
                except exc as e:
                    exception_list.append(e)

            raise GiveUp(try_num+1, total_sleep, func, exception_list)

        return _retry_wrapper
    return _retry_decorator

retry_toolkit.simple.GiveUp

Bases: Exception

Exception class thrown when retries are exhausted.

Parameters:

Name Type Description Default
n_tries int

number of tries attempted

 required
total_wait float

total seconds of wait used

 required
target_func callable

the target function that was attempted (the wrapped function)

 required
exceptions list

list of exceptions for each failed try

 required
Source code in src/retry_toolkit/simple.py 
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
class GiveUp(Exception):
    '''Exception class thrown when retries are exhausted.

    Parameters
    ----------
    n_tries    : int
        number of tries attempted

    total_wait : float
        total seconds of wait used

    target_func: callable
        the target function that was attempted (the wrapped function)

    exceptions : list
        list of exceptions for each failed try
    '''

    def __init__(self, n_tries: int, total_wait: float, target_func: callable,
                 exceptions: list):
        self.n_tries     = n_tries
        self.total_wait  = total_wait
        self.target_func = target_func
        self.exceptions  = exceptions

retry_toolkit.simple.Defaults

Defaults for retry behavior.

These values are used if not specified during retry decorator generation or if not overriden here (sleep function). For these defaults, it is also acceptable to set them to a callable returning the required type using the same convention as if it were used as an argument to the retry decorator generator.

Source code in src/retry_toolkit/simple.py 
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
class Defaults:
    '''Defaults for retry behavior.

    These values are used if not specified during retry decorator generation
    or if not overriden here (sleep function). For these defaults, it is
    also acceptable to set them to a callable returning the required type
    using the same convention as if it were used as an argument to the
    retry decorator generator.
    '''

    TRIES = 3
    '''integer: How many times to try an operation.'''

    BACKOFF = 0
    '''float: is or returns how long to wait before next retry.'''

    EXC = Exception
    '''
    Defines what exceptions are used for retrying. If any
    exceptions are thrown that do not match this specification then a retry
    will not occur and exception will be raised.
    '''

    SLEEP_FUNC = time.sleep
    '''callable: used as the sleep waiter'''
TRIES = 3 class-attribute instance-attribute

integer: How many times to try an operation.

BACKOFF = 0 class-attribute instance-attribute

float: is or returns how long to wait before next retry.

EXC = Exception class-attribute instance-attribute

Defines what exceptions are used for retrying. If any exceptions are thrown that do not match this specification then a retry will not occur and exception will be raised.

SLEEP_FUNC = time.sleep class-attribute instance-attribute

callable: used as the sleep waiter

Backoff Functions

retry_toolkit.simple.constant

Simple constant backoff. Suitable for demonstration.

Source code in src/retry_toolkit/simple.py 
33
34
35
36
37
def constant(c: float):
    '''Simple constant backoff. Suitable for demonstration.'''
    def _constant(x: float) -> float:
        return c
    return _constant

retry_toolkit.simple.linear

Linear backoff. Suitable for demonstration.

Source code in src/retry_toolkit/simple.py 
40
41
42
43
44
def linear(m: float, b: float =0):
    '''Linear backoff. Suitable for demonstration.'''
    def _linear(x: float) -> float:
        return m*x + b
    return _linear

retry_toolkit.simple.exponential

Exponential backoff. Slightly more realistic.

Source code in src/retry_toolkit/simple.py 
47
48
49
50
51
def exponential(n: float, b: float = 0):
    '''Exponential backoff. Slightly more realistic.'''
    def _exponential(x: float) -> float:
        return n*(2**x) + b
    return _exponential