Skip to Content
šŸ‘† We offer 1-on-1 classes as well check now
PythonAdvanced TopicsConcurrency and Parallelism in Python

Concurrency and Parallelism in Python

Concurrency and parallelism are two related but distinct concepts in programming that can help improve the performance and responsiveness of applications. In this section, we will delve into the world of concurrency and parallelism in Python, exploring the different techniques, tools, and best practices available.

Introduction to Concurrency and Parallelism

Concurrency refers to the ability of a program to execute multiple tasks simultaneously, sharing the same resources such as memory and I/O devices. Parallelism, on the other hand, refers to the simultaneous execution of multiple tasks on multiple processing units, such as CPUs or cores.

Python provides several modules and libraries to support concurrency and parallelism, including threading, multiprocessing, asyncio, and concurrent.futures. In this section, we will focus on the threading and multiprocessing modules, as well as the asyncio library.

Threading in Python

Threading in Python is achieved using the threading module. Threads are lightweight processes that share the same memory space as the parent process. Here is an example of a simple threading program:

import threading
import time
 
def print_numbers():
    for i in range(10):
        time.sleep(1)
        print(i)
 
def print_letters():
    for letter in 'abcdefghij':
        time.sleep(1)
        print(letter)
 
# Create threads
thread1 = threading.Thread(target=print_numbers)
thread2 = threading.Thread(target=print_letters)
 
# Start threads
thread1.start()
thread2.start()
 
# Wait for threads to finish
thread1.join()
thread2.join()

In this example, we define two functions print_numbers and print_letters that print numbers and letters respectively. We then create two threads thread1 and thread2 that execute these functions concurrently.

Best Practices for Threading

  • Use threads for I/O-bound tasks, such as reading or writing to files, network sockets, or databases.
  • Avoid using threads for CPU-bound tasks, as the Global Interpreter Lock (GIL) can limit performance.
  • Use synchronization primitives such as locks, semaphores, and condition variables to coordinate access to shared resources.

Multiprocessing in Python

Multiprocessing in Python is achieved using the multiprocessing module. Processes are heavyweight processes that have their own memory space and do not share resources with the parent process. Here is an example of a simple multiprocessing program:

import multiprocessing
import time
 
def square_numbers(numbers):
    for number in numbers:
        time.sleep(1)
        print(f'Square of {number} is {number ** 2}')
 
def cube_numbers(numbers):
    for number in numbers:
        time.sleep(1)
        print(f'Cube of {number} is {number ** 3}')
 
# Create processes
process1 = multiprocessing.Process(target=square_numbers, args=(range(10),))
process2 = multiprocessing.Process(target=cube_numbers, args=(range(10),))
 
# Start processes
process1.start()
process2.start()
 
# Wait for processes to finish
process1.join()
process2.join()

In this example, we define two functions square_numbers and cube_numbers that calculate the square and cube of numbers respectively. We then create two processes process1 and process2 that execute these functions concurrently.

Best Practices for Multiprocessing

  • Use processes for CPU-bound tasks, such as scientific computing, data compression, or encryption.
  • Avoid using processes for I/O-bound tasks, as the overhead of creating and managing processes can be high.
  • Use synchronization primitives such as queues, pipes, and shared memory to coordinate access to shared resources.

Asynchronous Programming with Asyncio

Asyncio is a library for writing single-threaded concurrent code using coroutines, multiplexing I/O access over sockets and other resources, and implementing network clients and servers. Here is an example of a simple asyncio program:

import asyncio
 
async def print_numbers():
    for i in range(10):
        await asyncio.sleep(1)
        print(i)
 
async def print_letters():
    for letter in 'abcdefghij':
        await asyncio.sleep(1)
        print(letter)
 
async def main():
    # Create tasks
    task1 = asyncio.create_task(print_numbers())
    task2 = asyncio.create_task(print_letters())
 
    # Wait for tasks to finish
    await task1
    await task2
 
# Run the main function
asyncio.run(main())

In this example, we define two coroutines print_numbers and print_letters that print numbers and letters respectively. We then create two tasks task1 and task2 that execute these coroutines concurrently.

Best Practices for Asyncio

  • Use asyncio for I/O-bound tasks, such as reading or writing to files, network sockets, or databases.
  • Avoid using asyncio for CPU-bound tasks, as it can limit performance.
  • Use synchronization primitives such as locks, semaphores, and condition variables to coordinate access to shared resources.

Real-World Examples

Concurrency and parallelism have many real-world applications, such as:

  • Web servers: handling multiple requests concurrently to improve responsiveness and throughput.
  • Scientific computing: using multiple processes or threads to perform complex calculations and simulations.
  • Data processing: using multiple processes or threads to process large datasets and improve performance.
  • Machine learning: using multiple processes or threads to train models and improve performance.

In conclusion, concurrency and parallelism are powerful techniques for improving the performance and responsiveness of applications. By using the right tools and techniques, such as threading, multiprocessing, and asyncio, developers can write efficient and scalable code that takes advantage of multiple processing units and improves overall system performance.

Last updated on
Asynchronous Programming with asyncio