Blocks In Python

ReFactored from PythonVsRuby. The below code-snippets were meant to show off Python's ability to handle LexicalClosures, matching the power of BlocksInRuby (or not). This has been refactored by a PythonPerson?, but it could still use some work and is missing some ruby examples.

---

Blocks are AnonymousFunctions. HigherOrderFunctions are those which take any function as a parameter or return them, they are separate and unrelated concepts, though often used together.

Python's anonymous function declaration, the lambda, is somewhat neutered in that it can only take expressions, not statements. But python allows functions to be defined in any scope (i.e. nested functions).

''It doesn't appear to me that BlocksInPython are closures, I hope someone will please explain what I'm missing.''

Note that as of Python 2.5 lambda still don't allow you to assign stuff to variables and are not closures in the scheme/lisp/ruby sense (see samples at http://www.paulgraham.com/accgen.html). But Python3000 is gaining a nonlocal keyword which will allow functions to alter variables in the enclosing scope.

lambdas do allow you to assign stuff to variables but you wouldn't do things that way. See the AsciiArt on the ObfuscatedPython page for proof of concept.

---

 f =lambda x: x**2 + 5

is equivalent to

 def f(x): return x**2 + 5

While Python has nested scopes and allows you to write HigherOrderFunctions, you can't call them as conveniently as you can call the built-in control structures. Compare:

if foo:
bar()

with

def _foo():
bar()
my_if(foo, _foo)

As a result, in practice, people rarely use HOFs in Python.

Yet all @decorators in python are HigherOrderFunctions, and many methods of builtin types and standard library, such as sort, take a optional argument that is a function.

In more functional languages even very complicated things like list comprehensions and parsers are just instances of using HigherOrderFunctions in MonadicProgramming style. In contrast, these are special constructs in Python.

"The Python Way" is to define support for language constructs that already exist, like "for...in";

In Python, all "iterator constructs" (for...in, if...in..., list comprehensions, indexing) can be hooked by adding a few methods. Writing "for elem in myList" isn't very different from writing (as in RubyLanguage) "myList collect {|elem|}"

This class shows that defining collect() is quite the same in Ruby and Python (except that in Python you pass f explicitly, while in Ruby you have special syntax for the stuff: the block syntax and yield() instead of f().)

 class myList(list):
def collect(self, f):
return myList([f(e) for e in self])

 l = myList()
 l.append('2')
 l.append('4')
 l2 = l.collect(lambda x: 2*x) #similar to ruby myList.collect {|x| 2*x}

---

"I wouldn't call blocks new ControlStructures?".

Blocks themselves aren't new control structures; methods that use blocks are new ControlStructures?.

Ruby lets a language grow, Python restricts you to its existing syntax, the difference is huge.

Python also does, it just takes care to prevent things like flow control macros and obfuscated code.

 for node in tree.depthFirst(followForiegn = False, recursive = True): 

is a lot more readable then the block equivalent, which will really require a comment to explain what the parameters are explaining here. And there is nothing stopping the user from writing their own stuff to tack on too; you aren't limited to "for x in obj", "for x in collect(obj)" works too:

 def collect(obj, f):
for x in obj:
 yield f(x)

 l = [1, 2, 3]
 for element in collect(l, lambda x: 2 * x):
print element

The syntax isn't "restrictive" because the syntax doesn't require you to iterate only directly on objects, to the extent that even makes sense as a restriction. Note that this "collect" is lazy, too.

There isn't a great deal of expressive difference between the "for" and "each" approaches, when all you're comparing is ways of iterating over collections of items. But blocks and HOFs let you model different behaviors, such as:

• "execute this block repeatedly until it completes without throwing an exception";

In python

 @auto_repeat
 def f():
     ...code here...
 f()

• "execute this block under the protection of all of these mutexes";

In python, using contextmanager class and with statement:

  with nested(mutex1, mutex2, mutex3):
       ...execute code under mutexes...

• or even "execute this block for each element of this collection, but do them all concurrently, and stop at the first result returned".

Insert ruby example, please

These are easy and natural things to model in Ruby. The great thing about "each" is that it does not hold a privileged position among these possibilities (other than the fact that it's famous).

try/catch/finally is a control structure...If it didn't exist, blocks would allow you to invent it, hooking "for in" wouldn't. Blocks allow you to invent "new" things, hooking existing structures allows you to extend "existing" things. Control structures can do anything, control structures let you hide complexity behind a simple language tailored to the problem. All truly powerful languages let you extend them by adding new elements to the language. Ruby, as a language, is better than Python because it doesn't tie your hands to existing language elements.

What Ruby is doing is also "hooking existing structures" - but those structures seem to me somewhat richer, simpler and more generic than Python's. Isn't it fun that such different tasks as spinning off another thread, iterating over a collection, attaching a callback, managing a filehandle resource and synchronizing a code block via a mutex - all those tasks use the same "block syntax", and it fits all of them perfectly? (See BlocksInRuby) Isn't it even more fun that the programmer can take advantage of this idiom for other tasks? And for some really inventive uses of "block syntax", take a look at Smalltalk. For example, it doesn't have an "if" statement - it uses block syntax in a method of the "boolean" class instead. Ruby is Perl dragged halfway in the Smalltalk direction.

The new with statement in python is designed exactly for this usecase, see http://www.python.org/dev/peps/pep-0343/ Actually hooking "for in" does allow it, but it's not pretty: you make every routine return an iterator of zero or one element, then do [result for x in op1() for y in op2(x) for ...] but this is of theoretic interest only.

I would actually like to see you implement try/catch with blocks, because this seems to concern two of the ways in which the comparability of blocks to higher order functions can be doubted: the ability to pass more than one block to a routine, and the ability of making blocks evaluation environments, like the monadic style. TryCatchWithRubyBlocks, anyone?

I thought (and I'd liked to be proved wrong) that try/catch requires continuations to work. The exception handler has to return to the code following the try block, not to the code that called it. Thus it can't be a normal HigherOrderFunction.

Ruby does have continuations. But if Python had them too, a problem would still remain. In Ruby you can easily set up a nice-looking syntax for your try-catch construct, something like:

 my_try {
...some code...
 }.my_catch(MyExceptionClass) {|e|
...error handling...
 }

Which looks hardly worse than the built-in construct. (Classes are objects; the handler gets the exception as a parameter if it matches the class; "my_try" works by capturing the closure in an object with a single "my_catch" method.) Ruby blocks let you do things that have nothing to do with iteration. It doesn't matter if python can somehow fake it, or emulate it by hooking iteration structures, it matters than Ruby doesn't make you. Can you do something like this in Python? What would a usage example look like?

I don't think that would work...how do you catch an exception so that method can use it, without first having a method to catch all exceptions? It's chicken-and-egg.

You could probably do something like:

 @my_try
 def block():
    ...some code...
 @block.my_catch(MyExceptionClass)
 def handler(e):
    ...error handling...

But python tries to have all necessary 'complete' control structures built in. Wasn't it proved that you only really need a branch and loop structure to cover all possible cases [Djikstra's GotoConsideredHarmful etc.]

My problem (and it may well be an instance of the BlubParadox) is that I've yet to see on any of the PythonVsRuby style pages where blocks really appear to provide a superior effect to Python's "hooking existing stuctures". How about providing a ruby program of reasonable complexity using blocks. Not too long - is a max of 80-100 lines reasonable to demonstrate the overwhelming superiority of blocks? Please describe expected behavior/output/etc. Then we can have a crack at translating it to a Pythonic form. Then this page should really be refactored...

---

From BlocksInRuby: "Match the terseness of this stuff in Python." Sure thing!

 ##Unit testing: 
 ##
 ## assert_raises(SomeExceptionClass) do
 ##foo(arg1, arg2)
 ##bar(arg3)
 ## end

(we have two statements in the block) And oh, don't talk about moving the calls to "foo" and "bar" into a separate function: the whole point of blocks is to keep related logic together. Blocks are just syntactic sugar to make passing a _single_ lexical closure into a function really easy - and pretend to be a "custom control structure" (do..end or {}) on top of that. Turns out that this covers a lot of useful cases. )

If you using something like nose, you can just wrap the test in a raises decorator, like so:

 class TestException
@raises(MyException)
def test_exception(self):
 foo(arg1)
 bar(arg2, arg3)

see http://python-nose.googlecode.com/svn/trunk/nose/tools.py for the implementation of @raises you could probably do something very similar to the ruby example by using the with statement:

 with assert_raises(MyException):
foo(arg1)
bar(arg2, arg3)

assuming assert_raises is defined like so:

 @contextmanager
 def assert_raises(MyException):
    try:
        yield
    except MyException:
        pass
    else:
        raise TestFailure

 ##Custom sorting (closures can take 2 parameters, as can lambdas): 
 ## my_array.sort {|x,y| y <=> x }  # sorts an array in reverse order
 my_array.sort(reversed=True) # or exactly translating from ruby:
 my_array.sort(key=lambda x: -x)

 ##made easier in 1.8: silently applying a SchwartzianTransform: 
 ## my_array.sort_by { |x| x.someproperty }
 my_array.sort(key=lambda x: x.someproperty) # inplace
 my_array = sorted(my_array, key=lambda x: x.someproperty)

 ## dbh.select_all("select name, phone from phonebook") do |name, phone|
 ##...this code will be executed for each row selected...
 ## end
 ##
 for name, phone in dbh.select_all("select name, phone from phonebook"):
...this code will be executed for each row selected...

 ## "hello".gsub(/./) {|s| s[0].to_s + ' '}  # converts each char to its code, from Pickaxe book
 "".join(str(ord(c)) for c in "hello")  # Python's strings are immutable

The Ruby version doesn't simply iterate over all chars - it iterates over all matches of a pattern, substituting in the string as it goes. The pattern just happened to be a dot, but it's often something more complex. But this was overkill (and besides I try to avoid regexes ;-)...

 import re
 re.sub(".",
lambda s: "%s " % ord(s.group(0)),
"hello")

Or the more readable alternative:

 "".join(str(ord(c)) for c in re.findall('.', "hello"))

---

The idiom I frequently use in Ruby that I don't know of a way to do in python is having 'with' methods. e.g.,

  with_current_directory "/tmp" do ... end.
  with_database_transaction do ... end.
  obj = with_object_that_is_complicated_to_construct do |builder| builder.a; builder.b; end.

Any takers on a Python equivalent?

python 2.5 can do this:

 # no longer needed in 2.6
 # http://docs.python.org/dev/whatsnew/pep-343.html
 from __future__ import with_statement  

 with open('/etc/passwd') as file:
for line in file:
print line,

The protocol works with exceptions, i.e., finally will still __exit__ cleanly.

---

I conjecture that all blocks can be transformed into an iterable (ie. a generator) and a for loop. This is illustrated below:

  def withOpenFile(fn) :
f = file(fn)
yield f
f.close()

  for f in withOpenFile("data") :
print f.read()

Generators yielding tuples can be used to bind multiple values at once if needed.

If the body of the for loop throws an exception, the generator will never reach the end and so f.close() will never be called.

This is similar to how the generator 'templating' for the with statement works, but they handle exceptions

 @contextmanager
 def openFile(fn):
     f = file(fn)
     try:
        yield f
     finally:
        f.close()

 with openFile("data") as f:
      print f.read()

Given the above openFile, what is different between with .. as and this:

 for f in openFile("data"):
      print f.read()

The only difference (on the surface) seems to be that the with .. as syntax is much more clear for the purpose. Actually the file object f is guaranteed to be closed after the with block. No such guarantee is true with the for loop. See the with statement PEP for details.

Authors

 * LuisBruno
 * VladimirSlepnev?
 * JonathanTang
 * RyanFreckleton?
 * TimN
 * JasonFelice

---

See PythonVsRuby, BlocksInManyLanguages, SmalltalkInsteadOfPython

CategoryPython CategoryClosure