Parallel Computing with R and Python
Objectives
– R-Julia-Python: Detecting the slow parts of a script by using profiling and benchmarking tools. Students will be able to detect the parts of a script where the code should be improved and where the memory allocations should be reduced.rn- Julia: Understanding why a just-in-time compiled language can specialize code on types, and how this reduces the per-operation overhead, with Python as a familiar point of comparison.rn- Julia: Understanding multiple dispatch as a language mechanism, contrasted with the object-oriented style of Python.rn- R-Julia-Python: Knowing the various ways of implementing parallel computations.rn- Julia & Python: Implementing multithreaded computations on the CPU and recognizing the classic pitfalls, such as race conditions and the limits that some language runtimes place on threading.rn- Julia & Python: Distinguishing concurrency from parallelism and using asynchronous tasks to overlap waiting time on a single core.rn- Julia: Gaining an awareness of the memory hierarchy and of data locality and, time permitting, an overview of how a computation can be offloaded to a GPU.rn- R: Learning and using the futureverse ecosystem of packages, a unifying parallelization framework in R with which you can parallelize locally or on clusters.rn- R: Learning and using the mirai and mori packages which offer the same capabilities with different trade- offs and support shared memory access.rn- R: Using parallel computing for exploratory data analysis within the tidyverse.rn- R: Using parallel computing for tuning ML models within the tidymodels.rn- Improving code performance through parallel computation on the CPU and, time permitting, on the GPU.
Course outline
In the R section, we will learn how to profile the code to look for slow parts or memory-heavy parts. We will then learn a few tricks to make sure the basic R code is optimized before thinking about parallelization. Next, we will introduce various ways of implementing parallel computations in R with their pros and cons. Finally, we will learn about the futureverse framework which is a unifying framework for parallel computing in R.rnWe will dive into the core concepts of futures and showcase the recent futurize and progressify packages that make turning sequential code into parallel code frictionless. We will also learn about the mirai and mori packages for parallelization which use different tradeoffs and support shared memory access. The course will then be oriented towards showing how simple it is to use parallelization within the tidyverse ecosystem for exploratory data analysis and within the tidymodels ecosystem for machine learning thanks to the futureverse and the mirai package.rnIn the Julia & Python section, the guiding idea is that speed does not come from a "magic" language but from understanding what the machine is actually doing. Python serves as the familiar baseline and Julia as the test bench. Through a series of small examples, we look at how code specialized on types differs from interpreted code, how dispatch on types compares with the object-oriented style, and how concurrency differs from parallelism: first overlapping waiting time with asynchronous tasks on a single core, then multithreading and its classic pitfalls, such as race conditions. Timings are compared along the way, so that students can see where the speed-ups actually come from. Depending on the time available, the section may also offer an overview of the usual Python tools for numerical and parallel computing, and of offloading a computation to a GPU. The precise selection of topics and libraries may vary from one year to the next.
Prerequisites
Basic knowledge of R and Python. No prior knowledge of Julia is required — it is introduced from scratch, with Python as the point of comparison.