Skip to main content

Python computation

The Python computation reads from /input and writes results to /output.

It can execute standard Python 3.11.6 code with the following supported libraries.

Python version: 3.11.6

Python computation example

Library versions
  • zipp 3.16.2
  • zict 3.0.0
  • yarl 1.9.2
  • xgboost 2.0.1
  • wrapt 1.14.1
  • watchdog 3.0.0
  • validators 0.22.0
  • urllib3 2.0.7
  • tzlocal 5.0.1
  • tzdata 2023.3
  • typing_extensions 4.7.1
  • tqdm 4.66.1
  • tornado 6.3.3
  • toolz 0.12.0
  • tomlkit 0.12.1
  • toml 0.10.2
  • tifffile 2023.8.30
  • threadpoolctl 3.1.0
  • tenacity 8.2.3
  • tblib 2.0.0
  • tabulate 0.9.0
  • sympy 1.12
  • streamlit 1.28.1
  • statsmodels 0.14.0
  • sortedcontainers 2.4.0
  • smmap 5.0.0
  • smbprotocol 1.11.0
  • smac 1.3.3
  • six 1.16.0
  • setuptools 68.2.2.post0
  • seaborn 0.13.0
  • scipy 1.11.3
  • scikit-survival 0.22.1
  • scikit-learn 1.3.0
  • scikit-image 0.21.0
  • ruamel.yaml.clib 0.2.7
  • ruamel.yaml 0.17.32
  • ruamel.base 1.0.0
  • rpds-py 0.10.3
  • rich 13.5.2
  • requests 2.31.0
  • referencing 0.30.2
  • qdldl 0.1.7.post0
  • pyzmq 25.1.1
  • pytz-deprecation-shim 0.1.0.post0
  • pytz 2023.3.post1
  • python-gnupg 0.5.1
  • python-dateutil 2.8.2
  • pyspnego 0.9.2
  • pyspark 3.5.6
  • pyrfr 0.8.2
  • pyreadstat 1.2.4
  • pyparsing 3.0.9
  • pynisher 0.6.4
  • pydeck 0.8.0b4
  • pycparser 2.21
  • pycares 4.3.0
  • pyasn1 0.5.0
  • pyarrow 14.0.1
  • py4j 0.10.9.7
  • py 1.11.0
  • psutil 5.9.6
  • protobuf 4.24.4
  • poetry-dynamic-versioning 1.0.1
  • patsy 0.5.3
  • partd 1.4.0
  • paramiko 3.3.1
  • pandas 2.1.1
  • packaging 23.1
  • osqp 0.6.3
  • openpyxl 3.1.2
  • olefile 0.46
  • numpy 1.26.1
  • numexpr 2.8.6
  • networkx 3.1
  • multidict 6.0.4
  • msgpack 1.0.5
  • mpmath 1.3.0
  • mdurl 0.1.2
  • matplotlib 3.8.0
  • markdown-it-py 3.0.0
  • lz4 4.3.2
  • lxml 4.9.3
  • locket 1.0.0
  • lime 0.2.0.1
  • lifelines 0.27.8
  • liac-arff 2.5.0
  • lazy_loader 0.3
  • kiwisolver 1.4.5
  • jsonschema-specifications 2023.7.1
  • jsonschema 4.19.0
  • joblib 1.3.2
  • jdcal 1.4.1
  • interface_meta 1.3.0
  • importlib-metadata 6.8.0
  • imbalanced-learn 0.11.0
  • imageio 2.33.0
  • idna 3.4
  • gssapi 1.8.3
  • gitdb 4.0.10
  • future 0.18.3
  • fsspec 2023.10.0
  • frozenlist 1.4.0
  • formulaic 0.6.6
  • fonttools 4.42.1
  • faust-cchardet 2.1.19
  • et-xmlfile 1.1.0
  • emcee 3.1.4
  • ecos 2.0.11
  • dunamai 1.18.0
  • dq_sql_worker 0.1.0
  • distro 1.8.0
  • distributed 2023.10.0
  • defusedxml 0.7.1
  • decorator 5.1.1
  • decentriq_util 0.1.0
  • ddt 1.6.0
  • dask 2023.10.1
  • cycler 0.11.0
  • cryptography 41.0.3
  • contourpy 1.1.0
  • cloudpickle 2.2.1
  • click 8.1.7
  • charset-normalizer 3.2.0
  • cffi 1.16.0
  • certifi 2023.7.22
  • faust-cchardet 2.1.19
  • cachetools 5.3.0
  • brotlicffi 1.1.0.0
  • bottle 0.12.25
  • blinker 1.6.2
  • bcrypt 4.0.1
  • autograd-gamma 0.4.3
  • autograd 1.6.2
  • auto-sklearn 0.15.0
  • attrs 23.1.0
  • async-timeout 4.0.3
  • astor 0.8.1
  • altair 5.1.2
  • aiosignal 1.3.1
  • aiohttp 3.8.6
  • aiodns 3.0.0
  • Pympler 1.0.1
  • Pygments 2.16.1
  • PyYAML 6.0.1
  • PyWavelets 1.4.1
  • PyNaCl 1.5.0
  • Pillow 10.1.0
  • MarkupSafe 2.1.3
  • Jinja2 3.1.2
  • GitPython 3.1.37
  • Cython 0.29.36
  • ConfigSpace 0.5.0
  • Brotli 1.1.0
  • Babel 2.12.1

Note that by default the Python computations cannot access the network or internet. Also note that GPU-computations are currently not supported.

Input

The Python computation can access data from datasets and computations which have been listed as its dependencies in the Available data section from the /input directory.

The paths differ based on the type of the dependency:

  • A Table dataset Table 1 can be loaded as /input/Table 1/dataset.csv
  • A File dataset File 1 can be loaded as /input/File 1
  • The results of a depending Python or R computation can be loaded under /input/<computation name>/<filename> the same way those results have been written to /output/<filename> in the depending computation
  • The results of a SQL computation can be loaded under /input/<computation name>/dataset.csv

Note that /input is slow remote-backed storage. Computations that read from it repeatedly or do a lot of random access on it will be slow — copy the data to /scratch first in that case (see the Scratch section below), and read /input directly only for a single pass.

Output

The computation should write results to the /output directory. The results as well as any text streamed to stdout are accessible by all users with data analyst permission on the computation.

Scratch

Computations can write temporary data to the /scratch directory, a fast encrypted SSD with around 800 GB of usable space. Unlike the slow remote-backed /input and /output directories, /scratch is fast local storage. Data written there is always encrypted and is deleted after the computation has run, so it's safe to use for sensitive intermediate data.