Why?¶

The temptation¶

You are building a tool that needs a function's code, because its values are not enough: a query DSL that wants the expression the user wrote, an instrumenter that must add checks inside the body, a runner that ships the function to another machine. Python appears to make this easy:

import ast, inspect, textwrap

source = textwrap.dedent(inspect.getsource(fn))
tree = ast.parse(source)
# ... transform ...
exec(compile(tree, "<rewritten>", "exec"), namespace)

Four lines, and they work in the demo. Every deployed implementation of these four lines that we audited in the 500 most-downloaded PyPI packages works in the demo. The question this library answers is what happens outside the demo, because we measured that too. Across eighteen execution contexts and five CPython versions, the four lines fail in eight contexts, fail silently in the worst ones, and need seven distinct repairs to be correct where they work.

The notebook story¶

The worst failure deserves to be told concretely, because it is the one nothing deployed catches.

A user of your tool works in a notebook. They write a cell defining process(), run it, and your decorator captures the source and rewrites it. They edit the cell and run it again. Between those two runs, Python's source cache (linecache) can hold the old text while inspect happily serves it for the new function. Your four lines now parse valid source, find a plausible function definition, rewrite it, and return something that works. It is the wrong code. No exception or warning is raised, and the user is now debugging behaviour that corresponds to text they already deleted.

We demonstrated this end to end on a real DSL prototype: a re-edited cell was silently projected from its stale text, and in a variant, the tool's error message quoted an expression the user had never written. redef catches both: the recovered definition's name is verified, and the recovered source is recompiled and compared against the live code object, so a stale body raises at decoration time with the reason named. That second check exists nowhere else. Verification describes it: we measured its false-positive rate at zero across 330 cells before turning it on by default.

The quieter failures¶

The notebook story is the sharpest of a family. The same audit and measurements produced the rest of the list, each one silent or misleading in the naive pipeline:

redef is those seven repairs on by default, the refusals typed, and the two verification layers. The whole pipeline costs tens to hundreds of microseconds per decorated function, once, at definition time.

Why not something simpler?¶

A wrapper (functools.wraps) cannot see inside. Wrappers observe arguments and returns. They cannot capture the expression a user wrote, add a check to an inner statement, or ship the body elsewhere. If a wrapper serves your case, use a wrapper; this library is for the cases where it cannot.

Symbolic tracing runs the operands. Query builders that overload operators (Product.price > 500) never see and or in: Python coerces them through __bool__ and __contains__ before any object can record them, and and does not fail, it silently drops a conjunct. That is why every tracing DSL makes users write &. Recovering the source captures the expression as written.

An import hook is a different tool with different costs. Hooks rewrite whole modules, silently, for every user of the process. They need a deployment story (installation order, cache tags), and static checkers pass vacuously on everything they load. We measured the most complete hook-based macro expander taxing every import in the process 4× to 5× cold, on modules with no macros in them. A decorator announces itself in the file, touches one function, and leaves the file meaning what it says.

Bytecode is pinned to the interpreter. The one production system that decompiles bytecode to recover expressions must track opcode churn per CPython release; we measured 27 distinct opcodes across five releases for one generator expression, 8 of them stable. Source is the stable representation.

What you should still not expect¶

No library makes source exist where it does not: frozen apps, stdin, and the bare REPL are out for the whole approach, and the support matrix says exactly where the edges are and which edges have remedies. That page is as much a part of the answer to "why?" as this one: the reason to use a kit is that somebody already found the edges the hard way.

Next: Use cases, each with working code.