Python Functions & Closure: Understanding First-Class Functions & Closure Conversion - Pro, Assignments of Electrical and Electronics Engineering

Download Python Functions & Closure: Understanding First-Class Functions & Closure Conversion - Pro and more Assignments Electrical and Electronics Engineering in PDF only on Docsity! Assignment 7: functions and closure conversion (part 1) ECEN 4553 & 5013, CSCI 4555 & 5525 Prof. Jeremy G. Siek November 12, 2008 The main ideas for this week are: • first-class functions • lexical scoping of variables • closures and closure conversion 1 Syntax of P4 For this assignment add functions to the language. We introduce two con- structs for creating functions, the def statement and the lambda expression, and a new expression for calling a function with some arguments. For sim- plicity, we leave out function calls with keyword arguments. The concrete syntax of the P4 subset of Python is shown in Figure 1. expression ::= expression "(" [expression ("," expression)*] ")" | "lambda" [identifier ("," identifier)*] ":" expression statement ::= "return" expression | "def" identifier "(" [identifier ("," identifier)*] ")" ":" statement Figure 1: Concrete syntax for the P4 subset of Python. (In addition to that of P3.) Figure 2 shows the additional Python classes for the P4 AST. 1 class CallFunc(Node): def __init__(self, node, args): self.node = node self.args = args class Function(Node): def __init__(self, decorators, name, argnames, defaults, \ flags, doc, code): self.decorators = decorators # ignore self.name = name self.argnames = argnames self.defaults = defaults # ignore self.flags = flags # ignore self.doc = doc # ignore self.code = code class Lambda(Node): def __init__(self, argnames, defaults, flags, code): self.argnames = argnames self.defaults = defaults # ignore self.flags = flags # ignore self.code = code class Return(Node): def __init__(self, value): self.value = value Figure 2: The Python classes for P4 ASTs. 2 >>> def h(): ... b = a + 2 ... a = 1 ... return b + a >>> h() Traceback (most recent call last): File "<stdin>", line 1, in ? File "<stdin>", line 2, in h UnboundLocalError: local variable ’a’ referenced before assignment Exercise 2.1. Write five programs in the P4 subset of Python that help you understand the language. Look for corner cases or unusual aspects of the language to test in your programs. Run the programs using the standard python interpreter. Add your test programs to the test directory. 3 Closure conversion The major challenge in compiling Python functions to C functions is that functions may not be nested in C. (There is a GNU extension to enable nested functions, but we won’t be using that extension.) When compiling to C, we must unravel the nesting and define each function in the global scope. Moving the function definitions is straightforward, but it takes a bit more work to make sure that the function behaves the same way it use to, after all, many of the variables that were in scope at the point where the function was originally defined are not in the global scope. When we move a function, we have to worry about it’s free variables. A variable is free with respect to a given expression or statement, let’s call it P , if the variable is referenced anywhere inside P but not bound in P . A variable is bound with respect to a given expression or statement if there is a function or lambda inside P that has that variable as a parameter or local variable. In the following, the variables y and z are bound in function f, but the variable x is free in function f. x = 3 def f(y): z = 3 return x + y + z The definition of free variables applies in the same way to nested functions. In the following, the variables x, y, and z are free in the lambda expression whereas w is bound in the lambda expression. The variable x is free in func- tion f, whereas the variables w, y, and z are bound in f. 5 x = 3 def f(y): z = 3 return lambda w: x + y + z + w Figure 4 gives part of the definition of a function that computes the free variables of an expression. Finishing this function and defining a similar function for statements is left to you. def free_vars(n): if isinstance(n, Const): return Set([]) elif isinstance(n, Name): if n.name == ’True’ or n.name == ’False’: return Set([]) else: return Set([n.name]) elif isinstance(n, Add): return free_vars(n.left) | free_vars(n.right) elif isinstance(n, CallFunc): fv_args = [free_vars(e) for e in n.args] return free_vars(n.node) | reduce(lambda a, b: a | b, fv_args, Set([])) elif isinstance(n, Lambda): return free_vars(n.code) - Set(n.argnames) ... Figure 4: Computing the free variables of an expression. The process of closure conversion turns a function with free variables into an behaviorally equivalent function without any free variables. A function without any free variables is called “closed”, hence the term “closure con- version”. The main trick in closure conversion is to turn each function into an ob- ject that contains a pointer to the function and a list that stores the values of the free variables. This object is called a closure and you’ll need to extend your definition of pyobj to include a closure inside the union. In the expla- nation below, we’ll use two primitive operators, get func and get free to access the two parts of a closure object. When a closure is called, the free variables list is passed as an extra argument to the function so that it can obtain the values of what use to be free variables from the list. Figure 5 shows the result of applying closure conversion to the deriva- tive example. The lambda expression has been removed and the associated code placed in the anonymous1 function. For each of the free variable of the lambda (f and epsilon) we add assignments inside the body of anonymous1 to initialize those variables by subscripting into the fvs list. The lambda ex- pression inside derivative has been replaced by a new kind of primitive 6 operation, creating a closure, that takes two arguments. The first argument is the anonymous function and the second is a list containing the values of the free variables. Now when we call the derivative function we get back a closure. To invoke a closure, we call the closures’s function, passing the closure’s free variable array as the first argument. The rest of the argu- ments are the normal arguments from the call site. Note that in the global functions generated for derivative and square, we added a statement that creates a closure out of the function itself. The reason for doing this is to allow functions to call themselves recursively. The way we’re doing it here is not exactly right with respect to Python, but we’ll fix this in part 2 of the assignment. def anonymous1(fvs, x): f = fvs[0] epsilon = fvs[1] return (get_func(f)(get_free(f), x+epsilon) \ - get_func(f)(get_free(f), x)) / epsilon def derivative1(fvs, f): derivative = make_closure(derivative1, fvs) epsilon = 0.0001 return make_closure(anonymous1, [f,epsilon]) def square1(fvs, x): square = make_closure(square1, fvs) return x * x derivative = make_closure(derivative1,[]) square = make_closre(square1, []) ds = get_func(derivative)(get_free(derivative), square) get_func(ds)(get_free(ds), 10) Figure 5: Closure conversion applied to the derivative example. To implement closure conversion, write a recursive function convert_closures that takes one parameter, the current AST node, and returns new version of the current AST node and a list of function definitions that need to be added to the global scope. Let us look at the two interesting the cases in the closure conversion functions. 7 can be eliminated in the simplification pass by introducing Let expressions just as we do for built-in operations. ef(e1, . . ., en) =⇒ get_func(ef)(get_free(ef), e1, . . ., en) )} 3.4 Heapifying variables Closure conversion as described so far, which copies the values of the free variables into the closure’s array, works as long as the variables are not updated by a latter assignment. Consider the following program and the output of the python interpreter. >>> f = lambda: x + 1 >>> x = 7 >>> f() 8 The read from variable x should be performed when the function is called, and at that time the value of the variable is 7. Unfortunately, if we sim- ply copy the value of x into the closure’s array, then the program would incorrectly access an undefined variable. We can solve this problem by storing the values of variables on the heap and storing just a pointer to the variable’s value in the closure’s array. We’ll call this process heapification. As this assignment is already rather compli- cated we’ll save heapification for part 2 of this assignment. 4 Overview of the implementation Figure 6 shows the suggested organization for the compiler. Exercise 4.1. Update the pyobj union to include a member that is a function pointer. Exercise 4.2. Update the function you wrote that figures out where variable declarations should be inserted to take into account function definitions and lambdas. Exercise 4.3. Update the SSA conversion and type analysis to take func- tions into account. When analyzing function, assume the parameters have type pyobj. 10 Generate C Lex & Parse AST Python text C text IR0Simplify Type AnalysisType Specialization IR3 IR4 IR3 Insert variable declarations Closure Conversion convert to SSA form convert out of SSA form IR3 IR1 IR2 Figure 6: Organization of the compiler passes. Exercise 4.4. Finish the function that determines the free variables of ex- pressions and write a function that computes the free variables of a state- ment. Hint: for function statements (defs), make sure to subtract local vari- ables from the set of free variables. Exercise 4.5. Extend your Python to C compiler to handle P4. Implement closure conversion pass that converts an AST to another AST. After closure conversion, the AST should not have any nested functions. 11