Skip to main content

Interacting with Python

ScalaPy offers a variety of ways to interact with the Python interpreter, enabling you to calculate any Python expression from Scala code.

Global Scope​

The primary entrypoint into the Python interpreter from ScalaPy is py.global, which acts similarly to Scala.js's js.Dynamic.global to provide a dynamically-typed interface for the interpreter's global scope. With py.global, you can call any global method and access any global value.

For example, we can create a Python range with the range() method, and calculate the sum of its elements with sum().

import me.shadaj.scalapy.py
import me.shadaj.scalapy.py.SeqConverters

// Python ranges are exclusive
val list = py.Dynamic.global.range(1, 3 + 1)
// list: py.Dynamic = range(1, 4)

// 1 + 2 + 3 == 6
val listSum = py.Dynamic.global.sum(list)
// listSum: py.Dynamic = 6

Importing Modules​

If you're working with a Python library, you'll likely need to import some modules. You can do this in ScalaPy with the py.module method. This method returns an object representing the imported module, which can be used just like py.global but with the contents referencing the module instead of the global scope.

For example we can import NumPy, a popular package for scientific computing with Python.

val np = py.module("numpy")
// np: py.Module = <module 'numpy' from '/opt/buildhome/.pyenv/versions/3.9.9/lib/python3.9/site-packages/numpy/__init__.py'>

val a = np.array(Seq(
  Seq(1, 0),
  Seq(0, 12)
).toPythonProxy)
// a: py.Dynamic = [[ 1  0]
//  [ 0 12]]

val aSquared = np.matmul(a, a)
// aSquared: py.Dynamic = [[  1   0]
//  [  0 144]]

Scala-Python Conversions​

In the previous example, you'll notice that we passed in a Scala Seq[Seq[Int]] into np.array, which usually takes a Python list. When using Python APIs, ScalaPy will automatically convert scalar Scala values into their Python equivalents (through the Writer type). This handles the integer values, but not sequences, which have multiple options for conversion in ScalaPy.

If you'd like to create a copy of the sequence, which can be accessed by Python code with high performance but miss any mutations you later make in Scala code, you can use toPythonCopy:

val mySeqToCopy = Seq(Seq(1, 2), Seq(3))
// mySeqToCopy: Seq[Seq[Int]] = List(List(1, 2), List(3))
mySeqToCopy.toPythonCopy
// res0: py.Any = [[1, 2], [3]]

If you'd like to create a proxy of the sequence instead, which uses less memory and can observe mutations but comes with a larger overhead for repeated access from Python, you can use toPythonProxy:

val mySeqToProxy = Array(1, 2, 3)
// mySeqToProxy: Array[Int] = Array(1, 2, 100)
val myProxy = mySeqToProxy.toPythonProxy
// myProxy: py.Any = <SequenceProxy object at 0x7f4ee6eadee0>
println(py.Dynamic.global.list(myProxy))
// [1, 2, 3]
mySeqToProxy(2) = 100
println(py.Dynamic.global.list(myProxy))
// [1, 2, 100]

To convert Python values back into their Scala equivalents, ScalaPy comes with the .as API to automatically perform conversions for supported types (those that have a Reader implementation). Unlike writing, where there were multiple options for converting sequence types, there is a single .as[] API for converting back. If you load a collection into an immutable Scala sequence type, it will be loaded as a copy. If you load it as a mutable.Seq, however, it will be loaded as a proxy and can observe underlying changes

import scala.collection.mutable
val myPythonList = py.Dynamic.global.list(py.Dynamic.global.range(10))
// myPythonList: py.Dynamic = [200, 1, 2, 3, 4, 5, 6, 7, 8, 9]
val copyLoad = myPythonList.as[Vector[Int]]
// copyLoad: Vector[Int] = Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
val proxyLoad = myPythonList.as[mutable.Seq[Int]]
// proxyLoad: mutable.Seq[Int] = Seq(200, 1, 2, 3, 4, 5, 6, 7, 8, 9)

println(copyLoad)
// Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
println(proxyLoad)
// Seq(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)

myPythonList.bracketUpdate(0, 100)

println(copyLoad)
// Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
println(proxyLoad)
// Seq(100, 1, 2, 3, 4, 5, 6, 7, 8, 9)

proxyLoad(0) = 200

println(myPythonList)
// [200, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Custom Python Snippets​

Sometimes, you might run into a situation where you need to express a Python construct that can't be done through an existing ScalaPy API. For this situation and to make converting Python code easier, ScalaPy provides an escape hatch via the py"" string interpolator. This lets you run arbitrary strings as Python code with the additional power of being able to interpolate in Scala values.

For example, we might want to use Python map which takes a lambda. Instead, we can use the py"" interpolator to write the expression as a piece of Python code.

import py.PyQuote

val mappedList = py.Dynamic.global.list(
  py"map(lambda elem: elem + 1, ${Seq(1, 2, 3).toPythonProxy})"
)
// mappedList: py.Dynamic = [2, 3, 4]

If you need to run arbitrary strings of Python that are dynamically generated, you can use py.eval:

py.eval("1 + 2")
// res11: py.Dynamic = 3

Special Python Syntax​

ScalaPy includes APIs to make it possible to use Python features that require special syntax from Scala.

py.with​

Python includes a "try-with-resources" feature in the form of the with keyword. In ScalaPy, you can use this feature by calling py.with with the value you want to open and a curried function using that value. For example, we can open a file with the following code:

val myFile = py.Dynamic.global.open("../README.md")
// myFile: py.Dynamic = <_io.TextIOWrapper name='../README.md' mode='r' encoding='UTF-8'>
py.`with`(myFile) { file =>
  println(file.encoding.as[String])
}
// UTF-8

bracketAccess, bracketUpdate, and bracketDelete​

To index into a sequence-like Python value, py.Dynamic offers the bracketAccess, bracketUpdate, and bracketDelete APIs to load, set, or delete a value through an indexing operation. For example, we could update values of a Python list:

val pythonList = py.Dynamic.global.list(Seq(1, 2, 3).toPythonProxy)
// pythonList: py.Dynamic = [1, 100, 3]
println(pythonList)
// [1, 2, 3]
pythonList.bracketAccess(0)
// res14: py.Dynamic = 1
pythonList.bracketUpdate(1, 100)
println(pythonList)
// [1, 100, 3]

We can also delete elements of a Python dictionary:

val myDict = py.Dynamic.global.dict()
// myDict: py.Dynamic = {}
myDict.bracketUpdate("hello", "world")
println(myDict)
// {'hello': 'world'}
myDict.bracketDelete("hello")
println(myDict)
// {}

attrDelete​

On supported objects, you can also delete an attribute with the attrDelete APIs:

import me.shadaj.scalapy.interpreter.CPythonInterpreter
CPythonInterpreter.execManyLines(
  """class MyClass:
    |  myAttribute = 0""".stripMargin
)
py.Dynamic.global.MyClass.attrDelete("myAttribute")

.del()​

Some Python APIs require you to explicitly delete a reference to a value with the del keyword. In ScalaPy, you can perform the equivalent operation by calling del on a Python value.

val myValue = py.Dynamic.global.MyClass()
myValue.del()
println(myValue)
// java.lang.IllegalAccessException: The Python value you are try to access has already been released by a call to py.Any.del()
//  at me.shadaj.scalapy.py.Any.__scalapy_value(Any.scala:15)
//  at me.shadaj.scalapy.py.Any.__scalapy_value$(Any.scala:13)
//  at me.shadaj.scalapy.py.FacadeValueProvider.__scalapy_value(Facades.scala:7)
//  at me.shadaj.scalapy.py.Any.toString(Any.scala:21)
//  at me.shadaj.scalapy.py.Any.toString$(Any.scala:21)
//  at me.shadaj.scalapy.py.FacadeValueProvider.toString(Facades.scala:7)
//  at java.lang.String.valueOf(String.java:2994)
//  at java.io.PrintStream.println(PrintStream.java:821)

There are two key points to note when using this API. First, although the Python value is still available in Scala, any attempts to access it will result in an exception since the value has been released. Second, if there are multiple references to a single Python value from your Scala code, del will only delete a single reference and the underlying value will not be freed since other Scala code still holds a reference to it.

Zoned Memory Management​

By default, ScalaPy uses the JVM's garbage collector to determine when to free memory on the Python side. However, if your code allocates many Python values and immediately releases them, your application may use more memory than needed since the JVM lazily performs garbage collection. In addition, Scala Native currently does not support the necessary APIs to automatically release memory on the Python side. In these situations, you can use the py.local API to mark a chunk of your application where all new Python values will be freed at the end of the block.

For example, we can use py.local to allocate a large number of Python values and then immediately release them:

py.local {
  (1 to 100).foreach { _ =>
    py.Dynamic.global.len(Seq(1, 2, 3).toPythonCopy)
  }
}

In this code, we can manipulate the values allocated inside the py.local block as long as we are still in its scope. If we try to access the values after the block ends, we will get an exception.

Note: using py.local requires careful reasoning about where different Python values will be initialized, since ScalaPy only uses the stack to compute which values should be cleaned. For example, if an object is lazily initialized inside a py.local block, any Python values stored there will be cleaned because they will be seen as part of the local block.