Fluent Python

Table of Contents

1 Philosophy

  • EAFP(python way)

Easier to ask for forgiveness than permission. This common Python coding style assumes the existence of valid keys or attributes and catches exceptions if the assumption proves false. This clean and fast style is characterized by the presence of many try and except statements. The technique contrasts with the LBYL style common to many other languages such as C.

  • LBYL(contrast with EAFP)

Look before you leap. This coding style explicitly tests for pre-conditions before making calls or lookups. This style contrasts with the EAFP approach and is characterized by the presence of many if statements. In a multi-threaded environment, the LBYL approach can risk introducing a race condition between “the looking” and “the leaping”. For example, the code, if key in mapping: return mapping[key] can fail if another thread removes key from mapping after the test, but before the lookup. This issue can be solved with locks or by using the EAFP approach.

2 Data Model

2.1 Special Methods

Category Method names
String/bytes representation __repr__ , __str__ , __format__ , __bytes__
Conversion to number __abs__, __bool__, __complex__, __int__, __float__, __hash__, __index__
Emulating collections __len__, __getitem__, __setitem__, __delitem__, __contains__
Iteration __iter__, __reversed__, __next__
Emulating callables __call__
Context management __enter__, __exit__
Instance creation and destruction __new__, __init__, __del__
Attribute management __getattr__, __getattribute__, __setattr__, __delattr__, __dir__
Attribute descriptors __get__, __set__, __delete__
Class services __prepare__, __instancecheck__, __subclasscheck__
  • operators
Category Method names
Unary numeric operators __neg__ -, __pos__ +, __abs__ abs()
Rich comparison operators __lt__ >, __le__ <=, __eq__ ==, __ne__ !=, __gt__ >, __ge__ >=
Arithmetic operators __add__ +, __sub__ -, __mul__ =, =__truediv__ /, __floordiv__ //, __mod__\n%, __divmod__ divmod(), __pow__ ** or pow(), __round__ round()
Reversed arithmetic operators __radd__, __rsub__, __rmul__, __rtruediv__, __rfloordiv__, __rmod__, __rdivmod__, __rpow__
Augmented assignment arithmetic operators __iadd__, __isub__, __imul__, __itruediv__, __ifloordiv__, __imod__, __ipow__
Bitwise operators __invert__ ~, __lshift__ <<, __rshift__ >>, __and__ &, __or__, __xor__ ^
Reversed bitwise operators __rlshift__, __rrshift__, __rand__, __rxor__, __ror__
Augmented assignment bitwise operators __ilshift__, __irshift__, __iand__, __ixor__, __ior__

2.2 numeric types

  • decimal.Decimal
  • fractions.Fraction

2.3 Sequence categories

2.3.1 Flat(hold one type)

str, bytes, bytearray, memoryview, array.array

  • If the list will only contain numbers, array.array is more efficient than a list

2.3.2 Mutable

list, bytearray, array.array, collections.deque, memoryview

2.3.3 Immutable

tuple, str, bytes

2.4 Collection Hierarchy

interface Container {
{method} __contains__
}

interface Iterable {
{method} __iter__
}

interface Sized {
{method} __len__
}

interface Sequence {
{method} __getitem__
{method} __reversed__
{method} index
{method} count
}
Container <|-- Sequence
Iterable <|-- Sequence
Sized <|-- Sequence

3 Metaprogramming

3.1 Monkey Patching

Changing a class or module at runtime, without touching the source code. 

some_obj.__getitem__ = some_get_function(some_class, position)

4 Iterator

4.1 Compare with Iterable

  • Iterator is a Iterable, they both have a __iter__ method
  • In addition, Iterator has a __next__ method
  • Iterator __iter__ often return itself
  • To support multiple traversals of aggregate objects

5 Functional Programming

5.1 Filtering

  • built-in filter
  • filterfalse(predicate, it)
  • islice(it, stop) or islice(it, start, stop, step=1)
  • compress(it, selector_it): Consumes two iterables in parallel; yields items from it whenever the corresponding item in selector_it is truthy
  • dropwhile(predicate, it): Consumes it skipping items while predicate computes truthy, then yields every remaining item
  • takewhile(predicate, it): Yields items while predicate computes truthy, then stops

5.2 Mapping

  • built-in enumerate(iterable, start=0)
  • built-in map(func, it1, [it2, …, itN])
  • accumulate(it, [func]): Yields accumulated sums(default); if func is provided, yields the result of applying it to the first pair of items, then to the first result and next item
  • starmap(func, it): Applies func to each item of it, yielding the result; the input iterable should yield iterable items iit, and func is applied as func(*iit)

5.3 Merging Multiple Inputs

  • chain(it1, …, itN)
  • chain.from_iterable(it): Yield all items from each iterable produced by it, one after the other, seamlessly
  • product(it1, …, itN, repeat=1): Cartesian product; yields N-tuples made by combining items from each input iterable like nested for loops could produce; repeat allows the input iterables to be consumed more than once.
  • (built-in) zip(it1, …, itN)~: Silently stopping when the first iterable is exhausted
  • zip_longest(it1, …, itN, fillvalue=None): Stopping only when the last iterable is exhausted, filling the blanks with the fillvalue

5.4 Rearranging

  • groupby(it, key=None): Yields 2-tuples of the form (key, group)
  • reversed(seq)
  • tee(it, n=2): Yields multiple generators from a single input iterable

5.5 Reducing

  • built-in all(it)
  • built-in any(it)
  • built-in max(it, [key=,] [default=])
  • built-in min(it, [key=,] [default=])
  • functools reduce(func, it, [initial])
  • built-in sum(it, start=0)

5.6 Others

  • count(start=0, step=1): indefinitely 

    def aritprog_gen(begin, step, end=None):
    first = type(begin + step)(begin)
    ap_gen = itertools.count(first, step)
    if end is not None:
	ap_gen = itertools.takewhile(lambda n: n < end, ap_gen)
    return ap_gen
  • cycle(it): indefinitely
  • repeat(item, [times]): Yield the given item repeadedly, indefinetly unless a number of times is given
  • combinations(it, out_len): Yield combinations of out_len items from the items yielded by it
  • combinations_with_replacement(it, out_len)
  • permutations(it, out_len=None)

5.7 iter Tricks

iter(callable, sentinel) 

def d6():
    return randint(1, 6)
d6_iter = iter(d6, 1)  # the second argument is a sentinel

# another useful example
with open('mydata.txt') as fp:
    for line in iter(fp.readline, ''):
	process_line(line)
  • when a sentinel returned by the callable, causes the iterator to raise StopIteration instead of yielding the sentinel.

6 else keyword

6.1 with for/while

for item in my_list:
    if item.flavor == 'banana':
	break
else:
    raise ValueError('No banana flavor found!')

6.2 with try

try:
    dangerous_call()
except OSError:
    log('OSError...')
else:
    after_call()

7 Context Manager

  • The with statement was designed to simplify the try/finally pattern
  • __enter__()~
  • __exit__(self, exc_type, exc_value, traceback): arguments are None, None, None if all went well
    • return True to tell the interpreter that the exception was handled.
    • If returns None or anything but True, any exception raised in the with block will be propagated.

7.1 Novel Usages

  • Managing transactions in the sqlite3 module
  • Holding locks, conditions, and semaphores in threading code(RAII)
  • Setting up environments for arithmetic operations with Decimal objects
  • Applying temporary patches to objects for testing

7.2 contextlib

7.2.1 @contextmanager & @asynccontextmanager

A decorator that lets you build a context manager from a simple generator function, instead of creating a class and implementing the protocol.

  • Use yield to produce whatever you want the __enter__ method to return
  • Essentially the contextlib.contextmanager decorator wraps the function in a class that implements the __enter__ and __exit__ methods.
  • __enter__
    1. Invokes the generator function and holds on to the generator object—let's call it gen.
    2. Calls next(gen) to make it run to the yield keyword.
    3. Returns the value yielded by next(gen), so it can be bound to a target variable in the with/as form.
  • __exit__
    1. Checks an exception was passed as exc_type; if so, gen.throw(exception) is invoked, causing the exception to be raised in the yield line inside the generator function body.
    2. Otherwise, next(gen) is called, resuming the execution of the generator function body after the yield.
  • Tips
    • The __exit__ method provided by the decorator assumes any exception sent into the generator is handled and should be suppressed.

    You must explicitly re-raise an exception in the decorated function if you don't want @contextmanager to suppress it.

    • Having a try/finally (or a with block) around the yield is an unavoidable price of using @contextmanager,

    because you never know what the users of your context manager are going to do inside their with block.

  • Example

    A context manager for rewriting files in place 

    import csv

with inplace(csvfilename, 'r', newline='') as (infh, outfh):
    reader = csv.reader(infh)
    writer = csv.writer(outfh)

    for row in reader:
	row += ['new', 'columns']
	writer.writerow(row)

7.2.2 closing

A function to build context managers out of objects that provide a close() method but don’t implement the __enter__/__exit__ protocol. 

from contextlib import closing
from urllib.request import urlopen

with closing(urlopen('http://www.python.org')) as page:
    for line in page:
	print(line)

7.2.3 suppress

A context manager to temporarily ignore specified exceptions. 

from contextlib import suppress

with suppress(FileNotFoundError):
    os.remove('somefile.tmp')

7.2.4 ContextDecorator

A base class for defining class-based context managers that can also be used as function decorators, running the entire function within a managed context. 

from contextlib import ContextDecorator
class mycontext(ContextDecorator):
    def __enter__(self):
	print('Starting')
	return self

    def __exit__(self, *exc):
	print('Finishing')
	return False

@mycontext()
def function():
    print('The bit in the middle')

7.2.5 ExitStack & AsyncExitStack

A context manager that lets you enter a variable number of context managers. When the with block ends, ExitStack calls the stacked context managers' __exit__ methods in LIFO order 

with ExitStack() as stack:
    files = [stack.enter_context(open(fname)) for fname in filenames]
    # All opened files will automatically be closed at the end of
    # the with statement, even if attempts to open files later
    # in the list raise an exception

7.2.6 others

  • redirect_stdout, redirect_stderr

8 Coroutine

  • Generators produce data for iteration
  • Coroutines are consumers of data
  • To keep your brain from exploding, you don't mix the two concepts together
  • Coroutines are not related to iteration
  • .send() allows two-way data exchange between the client code and the generator in contrast with .__next__()
def simple_coroutine():
    print('-> coroutine started')
    x = yield # received by send(x)
    print('-> coroutine received:', x)

coro = simple_coroutine() # GEN_CREATED
next(coro) # equivalent to coro.send(None). GEN_CREATED->GEN_RUNNING->GEN_SUSPENDED
#-> coroutine started
coro.send(42) # GEN_SUSPENDED->GEN_RUNNING->GEN_CLOSED
#-> coroutine received: 42
  • next(coro) is often describe as "priming" the coroutine(advancing it to the first yield, the EXPR after first yield will be executed)

8.1 states

use inspect.getgeneratorstate() to get state

  • GEN_CREATED: Waiting to start execution
  • GEN_RUNNING: Currently being executed by the interpreter
  • GEN_SUSPENDED: Currently suspended at a yield expression
  • GEN_CLOSED

8.2 Termination & Exception

The coroutine will only terminate when the caller calls .close() on it, or when it's garbage collected

8.2.1 Return Value

The value attribute of the StopIteration carries the value returned

8.2.2 Exception From Generator

An exception within a coroutine propagates to the caller that triggered it.

8.2.3 Terminate Generator

  • throw(exc_type[, exc_value[, traceback]]): Causes the yield expression where the generator was paused to raise the exception given.
  • close: Causes the yield expression where the generator was paused to raise a GeneratorExit exception.

8.2.4 Sentinel Values

None, ...

8.3 yield from

The main feature of yield from is to open a bidirectional channel from the outermost caller to the innermost subgenerator.

  • Any values that the subgenerator yields are passed directly to the caller of the delegating generator
  • Any values sent to the delegating generator using send() are passed directly to the subgenerator.
  • return expr in a generator (or subgenerator) causes StopIteration(expr) to be raised upon exit from the generator.
  • The value of the yield from expression is the first argument to the StopIteration exception raised by the subgenerator when it terminates.
  • Exception Handling: Exceptions other than GeneratorExit thrown into the delegating generator are passed to the throw() method of

the subgenerator. If the call raises StopIteration, the delegating generator is resumed. Any other exception is propagated to the delegating generator.

  • If a GeneratorExit exception is thrown into the delegating generator, or the close() method of the delegating generator is called,

then the close() method of the subgenerator is called if it has one. If this call results in an exception, it is propagated to the delegating generator. Otherwise, GeneratorExit is raised in the delegating generator.

8.3.1 Pseudocode

  • Simple Version
    _i = iter(EXPR)
try:
    _y = next(_i)
except StopIteration as _e:
    _r = _e.value # the RESULT in the simplest case
else: # channel between the caller and the subgenerator
    while 1:
	_s = yield _y
	try:
	    _y = _i.send(_s)
	except StopIteration as _e:
	    _r = _e.value
	    break
RESULT = _r
    • _i(iterator): The subgenerator
    • _y(yielded): A value yielded from the subgenerator
    • _r(result): The eventual result
    • _s(sent): A value sent by the caller to the delegating generator, which is forwarded to the subgenerator
    • _e(exception): An exception
  • Complete Version
    _i = iter(EXPR)
try:
    _y = next(_i)
except StopIteration as _e:
    _r = _e.value
else:
    while 1:
	try:
	    _s = yield _y
	except GeneratorExit as _e:
	    try:
		_m = _i.close
	    except AttributeError:
		pass
	    else:
		_m()
	    raise _e
	except BaseException as _e:
	    _x = sys.exc_info()
	    try:
		_m = _i.throw
	    except AttributeError:
		raise _e
	    else:
		try:
		    _y = _m(*_x)
		except StopIteration as _e:
		    _r = _e.value
		    break
	else:
	    try:
		if _s is None:
		    _y = next(_i)
		else:
		    _y = _i.send(_s)
	    except StopIteration as _e:
		_r = _e.value
		break

RESULT = _r

8.4 Discrete Event Simulations(DES)

SimPy

9 Currency

9.1 Threads & Processes

9.1.1 Executor

from concurrent import futures
with futures.ThreadPoolExecutor(4) as executor:
    results = executor.map(process_one, sorted(args))
    # map: 1. create and schedule futures 2. retrive the future.result() one by one
    print(results)

# equivalent to
with futures.ThreadPoolExecutor(4) as executor:
    fs = [executor.submit(process_one, arg) for arg in args]
    for completed_future in futures.as_completed(fs):
	print(completed_future.result())
    return len(list(results))
  • The combination of executor.submit and futures.as_completed is more flexible than executor.map

because you can submit different callables and arguments, while executor.map is designed to run the same callable on the different arguments.

  • Future is hashable

9.1.2 GIL

Restricts only one thread at a time to execute Python bytecodes. However, all standard library functions that perform blocking I/O release the GIL when waiting for a result from the OS.

  • All standard library functions that perform blocking I/O release the GIL when waiting for a result from the OS. I/O bound program can benefit from using threads.

9.1.3 Spin Example

class Signal:
    go = True

def spin(msg, signal):
    write, flush = sys.stdout.write, sys.stdout.flush
    for char in itertools.cycle('|/-\\'):
	status = char + ' ' + msg
	write(status)
	flush()
	write('\x08' * len(status))
	time.sleep(.1)
	if not signal.go:
	    break
    write(' ' * len(status) + '\x08' * len(status))

def supervisor():
    signal = Signal()
    spinner = threading.Thread(target=spin,
			       args=('thinking!', signal))
    print('spinner object:', spinner)
    spinner.start()
    result = slow_function()
    signal.go = False
    spinner.join()
    return result

9.2 Future

Interface: cancel, cancelled, running, done, result(timeout=None), exception(timeout=None), add_done_callback(fn)

9.3 asyncio + executor

import asyncio
import time
from concurrent.futures import ThreadPoolExecutor as Executor


def blocking_task():
    time.sleep(2)
    return 42


async def run_tasks(executor):
    loop = asyncio.get_event_loop()
    blocking_tasks = []
    for _ in range(40):
	task = loop.run_in_executor(executor, blocking_task)
	blocking_tasks.append(task)
    completed, pending = await asyncio.wait(blocking_tasks)
    for t in completed:
	print(t.result())


if __name__ == '__main__':
    event_loop = asyncio.get_event_loop()
    executor = Executor(20)
    event_loop.run_until_complete(run_tasks(executor))

9.4 asyncio + aioXXX

import asyncio


async def aiotask():
    await asyncio.sleep(2)
    return 42

event_loop = asyncio.get_event_loop()
tasks = [aiotask() for _ in range(1000)]
results = event_loop.run_until_complete(asyncio.gather(*tasks))
for result in results:
    print(result)

10 asyncio

10.1 loop.create_future

This is a preferred way to create futures in asyncio, as event loop implementations can provide alternative implementations of the Future class

10.2 loop.create_task vs. asyncio.ensure_future

Always use loop.create_task to schedule coroutines, however use asyncio.ensure_future for other awaitable objects(like Futures, Tasks)

11 Doctest

12 Utilities

  • math.hypot: Return the Euclidean distance, sqrt(x*x + y*y)
  • b, a = a, b: swap values without using a temporary variable
  • Vaurien: chaos monkey, the Chaos TCP Proxy
  • requests resp: resp.raise_for_status()
  • httpbin
  • SimPy: DES simulation
  • lelo
  • python-parallelize
  • Celery