In the spirit of toolz, we provide missing features for Python, mainly from the list processing tradition, but with some Haskellisms mixed in. We extend the language with a set of syntactic macros. We also provide an in-process, background REPL server for live inspection and hot-patching. The emphasis is on clear, pythonic syntax, making features work together, and obsessive correctness.
Some hypertext features of this README, such as local links to detailed documentation, and expandable example highlights, are not supported when viewed on PyPI; view on GitHub to have those work properly.
None required.
- MacroPy optional, to enable the syntactic macro layer.
- imacropy optional, to enable the improved interactive macro REPL.
The officially supported language version is Python 3.6, on both CPython and PyPy3.
The 0.14.x series should run on CPythons 3.4 through 3.7, and on PyPy3; the CI process verifies the tests pass on those platforms. Support for 3.8 is planned in one of the next few releases. Pure-Python features should already work; but macro code still needs changes to account for AST representation changes in 3.8, see issue #16.
Pure-Python feature set
Syntactic macro feature set
REPL server: interactively hot-patch your running Python program.
Design notes: for more insight into the design choices of unpythonic.
The features of unpythonic are built out of, in increasing order of magic:
- Pure Python (e.g. batteries for
itertools), - Macros driving a pure-Python core (
do,let), - Pure macros (e.g.
continuations,lazify,dbg).
This depends on the purpose of each feature, as well as ease-of-use considerations. See the design notes for more information.
Small, limited-space overview of the overall flavor. There's a lot more that doesn't fit here, especially in the pure-Python feature set. See the full documentation and unit tests for more examples.
Loop functionally, with tail call optimization.
[docs]
from unpythonic import looped, looped_over
@looped
def result(loop, acc=0, i=0):
if i == 10:
return acc
else:
return loop(acc + i, i + 1) # tail call optimized, no call stack blowup.
assert result == 45
@looped_over(range(3), acc=[])
def result(loop, i, acc):
acc.append(lambda x: i * x) # fresh "i" each time, no mutation of loop counter.
return loop()
assert [f(10) for f in result] == [0, 10, 20]Introduce dynamic variables.
[docs]
from unpythonic import dyn, make_dynvar
make_dynvar(x=42) # set a default value
def f():
assert dyn.x == 17
with dyn.let(x=23):
assert dyn.x == 23
g()
assert dyn.x == 17
def g():
assert dyn.x == 23
assert dyn.x == 42
with dyn.let(x=17):
assert dyn.x == 17
f()
assert dyn.x == 42Interactively hot-patch your running Python program.
[docs]
To opt in, add just two lines of code to your main program:
from unpythonic.net import server
server.start(locals={}) # automatically daemonic
import time
def main():
while True:
time.sleep(1)
if __name__ == '__main__':
main()Or if you just want to take this for a test run, start the built-in demo app:
python3 -m unpythonic.net.serverOnce a server is running, to connect:
python3 -m unpythonic.net.client 127.0.0.1This gives you a REPL, inside your live process, with all the power of Python. You can importlib.reload any module, and through sys.modules, inspect or overwrite any name at the top level of any module. You can pickle.dump your data. Or do anything you want with/to the live state of your app.
You can have multiple REPL sessions connected simultaneously. When your app exits (for any reason), the server automatically shuts down, closing all connections if any remain. But exiting the client leaves the server running, so you can connect again later - that's the whole point.
Optionally, if you have MacroPy, the REPL sessions support importing and invoking macros. If you additionally have imacropy, the improved interactive macro REPL is used automatically.
Industrial-strength scan and fold.
[docs]
from operator import add
from unpythonic import scanl, foldl, unfold, take
assert tuple(scanl(add, 0, range(1, 5))) == (0, 1, 3, 6, 10)
def op(e1, e2, acc):
return acc + e1 * e2
assert foldl(op, 0, (1, 2), (3, 4)) == 11 # we accept multiple input sequences, like Racket
def nextfibo(a, b): # *oldstates
return (a, b, a + b) # value, *newstates
assert tuple(take(10, unfold(nextfibo, 1, 1))) == (1, 1, 2, 3, 5, 8, 13, 21, 34, 55)Allow a lambda to call itself. Name a lambda.
[docs for withself] [docs for namelambda]
from unpythonic import withself, namelambda
fact = withself(lambda self, n: n * self(n - 1) if n > 1 else 1) # see @trampolined to do this with TCO
assert fact(5) == 120
square = namelambda("square")(lambda x: x**2)
assert square.__name__ == "square"
assert square.__qualname__ == "square" # or e.g. "somefunc.<locals>.square" if inside a function
assert square.__code__.co_name == "square" # used by stack tracesBreak infinite recursion cycles.
[docs]
from typing import NoReturn
from unpythonic import fix
@fix()
def a(k):
return b((k + 1) % 3)
@fix()
def b(k):
return a((k + 1) % 3)
assert a(0) is NoReturnBuild number sequences by example. Slice general iterables.
[docs for s] [docs for islice]
from unpythonic import s, islice
seq = s(1, 2, 4, ...)
assert tuple(islice(seq)[:10]) == (1, 2, 4, 8, 16, 32, 64, 128, 256, 512)Memoize functions and generators.
[docs for memoize] [docs for gmemoize]
from itertools import count, takewhile
from unpythonic import memoize, gmemoize, islice
ncalls = 0
@memoize # <-- important part
def square(x):
global ncalls
ncalls += 1
return x**2
assert square(2) == 4
assert ncalls == 1
assert square(3) == 9
assert ncalls == 2
assert square(3) == 9
assert ncalls == 2 # called only once for each unique set of arguments
# "memoize lambda": classic evaluate-at-most-once thunk
thunk = memoize(lambda: print("hi from thunk"))
thunk() # the message is printed only the first time
thunk()
@gmemoize # <-- important part
def primes(): # FP sieve of Eratosthenes
yield 2
for n in count(start=3, step=2):
if not any(n % p == 0 for p in takewhile(lambda x: x*x <= n, primes())):
yield n
assert tuple(islice(primes())[:10]) == (2, 3, 5, 7, 11, 13, 17, 19, 23, 29)Functional updates.
[docs]
from itertools import repeat
from unpythonic import fup
t = (1, 2, 3, 4, 5)
s = fup(t)[0::2] << tuple(repeat(10, 3))
assert s == (10, 2, 10, 4, 10)
assert t == (1, 2, 3, 4, 5)Lispy data structures.
[docs for box] [docs for cons] [docs for frozendict]
from unpythonic import box, unbox # mutable single-item container
cat = object()
cardboardbox = box(cat)
assert cardboardbox is not cat # the box is not the cat
assert unbox(cardboardbox) is cat # but the cat is inside the box
assert cat in cardboardbox # ...also syntactically
dog = object()
cardboardbox << dog # hey, it's my box! (replace contents)
assert unbox(cardboardbox) is dog
from unpythonic import cons, nil, ll, llist # lispy linked lists
lst = cons(1, cons(2, cons(3, nil)))
assert ll(1, 2, 3) == lst # make linked list out of elements
assert llist([1, 2, 3]) == lst # convert iterable to linked list
from unpythonic import frozendict # immutable dictionary
d1 = frozendict({'a': 1, 'b': 2})
d2 = frozendict(d1, c=3, a=4)
assert d1 == frozendict({'a': 1, 'b': 2})
assert d2 == frozendict({'a': 4, 'b': 2, 'c': 3})Live list slices.
[docs]
from unpythonic import view
lst = list(range(10))
v = view(lst)[::2] # [0, 2, 4, 6, 8]
v[2:4] = (10, 20) # re-slicable, still live.
assert lst == [0, 1, 2, 3, 10, 5, 20, 7, 8, 9]
lst[2] = 42
assert v == [0, 42, 10, 20, 8]Pipes: focus on data flow in function composition.
[docs]
from unpythonic import piped, exitpipe
double = lambda x: 2 * x
inc = lambda x: x + 1
x = piped(42) | double | inc | exitpipe
assert x == 85Conditions: resumable, modular error handling, like in Common Lisp.
[docs]
Contrived example:
from unpythonic import error, restarts, handlers, invoke, use_value, unbox
class MyError(ValueError):
def __init__(self, value): # We want to act on the value, so save it.
self.value = value
def lowlevel(lst):
_drop = object() # gensym/nonce
out = []
for k in lst:
# Provide several different error recovery strategies.
with restarts(use_value=(lambda x: x),
halve=(lambda x: x // 2),
drop=(lambda: _drop)) as result:
if k > 9000:
error(MyError(k))
# This is reached when no error occurs.
# `result` is a box, send k into it.
result << k
# Now the result box contains either k,
# or the return value of one of the restarts.
r = unbox(result) # get the value from the box
if r is not _drop:
out.append(r)
return out
def highlevel():
# Choose which error recovery strategy to use...
with handlers((MyError, lambda c: use_value(c.value))):
assert lowlevel([17, 10000, 23, 42]) == [17, 10000, 23, 42]
# ...on a per-use-site basis...
with handlers((MyError, lambda c: invoke("halve", c.value))):
assert lowlevel([17, 10000, 23, 42]) == [17, 5000, 23, 42]
# ...without changing the low-level code.
with handlers((MyError, lambda: invoke("drop"))):
assert lowlevel([17, 10000, 23, 42]) == [17, 23, 42]
highlevel()Conditions only shine in larger systems, with restarts set up at multiple levels of the call stack; this example is too small to demonstrate that. The single-level case here could be implemented as a error-handling mode parameter for the example's only low-level function.
With multiple levels, it becomes apparent that this mode parameter must be threaded through the API at each level, unless it is stored as a dynamic variable (see unpythonic.dyn). But then, there can be several types of errors, and the error-handling mode parameters - one for each error type - have to be shepherded in an intricate manner. A stack is needed, so that an inner level may temporarily override the handler for a particular error type...
The condition system is the clean, general solution to this problem. It automatically scopes handlers to their dynamic extent, and manages the handler stack automatically. In other words, it dynamically binds error-handling modes (for several types of errors, if desired) in a controlled, easily understood manner. The local programmability (i.e. the fact that a handler is not just a restart name, but an arbitrary function) is a bonus for additional flexibility.
If this sounds a lot like an exception system, that's because conditions are the supercharged sister of exceptions. The condition model cleanly separates mechanism from policy, while otherwise remaining similar to the exception model.
unpythonic.test.fixtures: a minimalistic test framework for macro-enabled Python code.
[docs TODO]; for now, look at the docstrings of the various constructs in the example below, the module unpythonic.test.fixtures (which provides much of this), and the automated tests of unpythonic itself. (Particularly, how to test code using conditions and restarts can be found in unpythonic.test.test_conditions.)
from unpythonic.syntax import macros, test, test_raises, fail, error, the
from unpythonic.test.fixtures import session, testset, terminate, returns_normally
def f():
raise RuntimeError("argh!")
def g(a, b):
return a * b
fail["this line should be unreachable"]
count = 0
def counter():
global count
count += 1
return count
with session("simple framework demo"):
with testset():
test[2 + 2 == 4]
test_raises[RuntimeError, f()]
test[returns_normally(g(2, 3))]
test[g(2, 3) == 6]
# Use `the[]` in a `test[]` to declare what you want to inspect if the test fails.
test[counter() < the[counter()]]
with testset("outer"):
with testset("inner 1"):
test[g(6, 7) == 42]
with testset("inner 2"):
test[None is None]
with testset("inner 3"): # an empty testset is considered 100% passed.
pass
with testset("integration"):
try:
import blargly
except ImportError:
error["blargly not installed, cannot test integration with it."]
else:
... # blargly integration tests go here
with testset(postproc=terminate):
test[2 * 2 == 5] # fails, terminating the nearest dynamically enclosing `with session`
test[2 * 2 == 4] # not reachedWe provide the low-level syntactic constructs test[], test_raises[] and test_signals[], with the usual meanings. The last one is for testing code that uses conditions and restarts; see unpythonic.conditions.
Inside a test[] expression, the[] can be used to declare a subexpression as the interesting part, for displaying its value as "result" in the test failure message if the test fails. By default (if no the[] is present), test[] captures the leftmost term if the top-level expression is a comparison (common use case), and otherwise the whole expression.
There can be at most one the[] in each test. In case of nested test[], each the[...] is understood as belonging to the lexically innermost surrounding one.
The test macros also come in block variants, with test, with test_raises, with test_signals.
We provide the helper macros fail[message], error[message] and warn[message] for producing unconditional failures, errors or warnings. Examples of the intended meanings:
fail[...]if reaching that point means that the test failed, e.g. on a line that should be unreachable.error[...]if the test cannot run, e.g. if an optional dependency for an integration test is not installed.warn[...]if the test is temporarily disabled and needs future attention, e.g. for syntactic compatibility to make the code run for now on an old Python version.
Warnings produced by warn[] are currently (v0.14.3) not counted in the total number of tests run.
As usual in test frameworks, the testing constructs behave somewhat like assert, with the difference that a failure or error will not abort the whole unit (unless explicitly asked to do so).
The with session() is optional. The human-readable session name is also optional, used for display purposes only. The session serves two roles: it provides an exit point for terminate, and defines an implicit top-level testset.
Tests can optionally be grouped into testsets. Each testset tallies passed, failed and errored tests within it, and displays the totals when it exits. Testsets can be named and nested.
Testsets also provide the option to locally install a postproc handler that gets a copy of each failure or error in that testset (and by default, any of its inner testsets), after the failure or error has been printed. In nested testsets, the dynamically innermost postproc wins. A failure is an instance of unpythonic.test.fixtures.TestFailure, and an error is an instance of unpythonic.test.fixtures.TestError. Both inherit from unpythonic.test.fixtures.TestingException.
If you want to set a default global postproc, which is used when no local postproc is in effect, see the TestConfig bunch of constants in unpythonic.test.fixtures. It also contains some other configuration options.
The with testset construct comes with one other important feature. The nearest dynamically enclosing with testset catches any stray exceptions or signals that occur within its dynamic extent, but outside a test. In case of an uncaught signal, the error is reported, and the testset resumes. In case of an uncaught exception, the error is reported, and the testset terminates (because the exception model does not support resuming).
Catching of uncaught signals, in both the low-level test constructs and the high-level testset, can be disabled using with catch_signals(False). This is useful in testing code that uses conditions and restarts; sometimes allowing a signal (e.g. from unpythonic.conditions.warn) to remain uncaught is the right thing to do.
We provide a custom testing framework, because unpythonic is effectively a language extension. Inspired by Julia's standard-library Test package.
let: expression-local variables.
[docs]
from unpythonic.syntax import macros, let, letseq, letrec
x = let[((a, 1), (b, 2)) in a + b]
y = letseq[((c, 1), # LET SEQuential, like Scheme's let*
(c, 2 * c),
(c, 2 * c)) in
c]
z = letrec[((evenp, lambda x: (x == 0) or oddp(x - 1)), # LET mutually RECursive, like in Scheme
(oddp, lambda x: (x != 0) and evenp(x - 1)))
in evenp(42)]let-over-lambda: stateful functions.
[docs]
from unpythonic.syntax import macros, dlet
@dlet((x, 0)) # let-over-lambda for Python
def count():
return x << x + 1 # `name << value` rebinds in the let env
assert count() == 1
assert count() == 2do: code imperatively in any expression position.
[docs]
from unpythonic.syntax import macros, do, local, delete
x = do[local[a << 21],
local[b << 2 * a],
print(b),
delete[b], # do[] local variables can be deleted, too
4 * a]
assert x == 84Automatically apply tail call optimization (TCO), à la Scheme/Racket.
[docs]
from unpythonic.syntax import macros, tco
with tco:
# expressions are automatically analyzed to detect tail position.
evenp = lambda x: (x == 0) or oddp(x - 1)
oddp = lambda x: (x != 0) and evenp(x - 1)
assert evenp(10000) is TrueCurry automatically, à la Haskell.
[docs]
from unpythonic.syntax import macros, curry
from unpythonic import foldr, composerc as compose, cons, nil, ll
with curry:
def add3(a, b, c):
return a + b + c
assert add3(1)(2)(3) == 6
mymap = lambda f: foldr(compose(cons, f), nil)
double = lambda x: 2 * x
assert mymap(double, (1, 2, 3)) == ll(2, 4, 6)Lazy functions, a.k.a. call-by-need.
[docs]
from unpythonic.syntax import macros, lazify
with lazify:
def my_if(p, a, b):
if p:
return a # b never evaluated in this code path
else:
return b # a never evaluated in this code path
assert my_if(True, 23, 1/0) == 23
assert my_if(False, 1/0, 42) == 42Genuine multi-shot continuations (call/cc).
[docs]
from unpythonic.syntax import macros, continuations, call_cc
with continuations: # enables also TCO automatically
# McCarthy's amb() operator
stack = []
def amb(lst, cc):
if not lst:
return fail()
first, *rest = tuple(lst)
if rest:
remaining_part_of_computation = cc
stack.append(lambda: amb(rest, cc=remaining_part_of_computation))
return first
def fail():
if stack:
f = stack.pop()
return f()
# Pythagorean triples using amb()
def pt():
z = call_cc[amb(range(1, 21))] # capture continuation, auto-populate cc arg
y = call_cc[amb(range(1, z+1))]
x = call_cc[amb(range(1, y+1))]
if x*x + y*y != z*z:
return fail()
return x, y, z
t = pt()
while t:
print(t)
t = fail() # note pt() has already returned when we call this.PyPI
pip3 install unpythonic --user
or
sudo pip3 install unpythonic
GitHub
Clone (or pull) from GitHub. Then,
python3 setup.py install --user
or
sudo python3 setup.py install
Uninstall
Uninstallation must be invoked in a folder which has no subfolder called unpythonic, so that pip recognizes it as a package name (instead of a filename). Then,
pip3 uninstall unpythonic
or
sudo pip3 uninstall unpythonic
Not working as advertised? Missing a feature? Documentation needs improvement?
Issue reports and pull requests are welcome. Contribution guidelines.
While unpythonic is intended as a serious tool for improving productivity as well as for teaching, right now my work priorities mean that it's developed and maintained on whatever time I can spare for it. Thus getting a response may take a while, depending on which project I happen to be working on.
All original code is released under the 2-clause BSD license.
For sources and licenses of fragments originally seen on the internet, see AUTHORS.
Thanks to TUT for letting me teach RAK-19006 in spring term 2018; early versions of parts of this library were originally developed as teaching examples for that course. Thanks to @AgenttiX for feedback.
The trampoline implementation of unpythonic.tco takes its remarkably clean and simple approach from recur.tco in fn.py. Our main improvements are a cleaner syntax for the client code, and the addition of the FP looping constructs.
Another important source of inspiration was tco by Thomas Baruchel, for thinking about the possibilities of TCO in Python.
Links to blog posts, online articles and papers on topics relevant in the context of unpythonic have been collected to a separate document.