Multiprocessing in Python

Multiprocessing is a powerful feature in which we can do multiple processes at the same time simultaneously. We can use the advantage of multiple CPU cores to perform parallel processing. With the help of multiprocessing we can complete our tasks in a small amount of time and enhancing the improvement in our allocated tasks. Multiprocessing module in python provides an interface to create and manage process similar to the threading method but it done with a separate memory. We can also avoids and resolve some issues regarding to locks while allocating resources to the processes.

Basic concepts of Multiprocessing

• Process:

In multiprocessing, processes are an independent entity which is executed in its own memory space by the operating system.

• pool:

With the help of pool we can perform data parallelism by distributing the input data across processes in-order to parallelize the execution. We can also use this method for task distribution.

• Queue:

The term queue is mainly used to communicate between processes. It can offer the possibility to run different analysis in parallel processes which in result increasing the speed of calculating different risks simultaneously.

• pipe:

pipe() functions usually represents the two ends of the pipe. It is basically similar to the queue in performing multiprocessing. Each end point contains send() and receive() methods which are used to send and receive the data respectively from other processes.

Usage of the basic concepts in multiprocessing

• Creating processes

import multiprocessing
def task(num):
"""Thread Processing Functon"""
print(f'task: {number}')
if __name__ == '__main__':
process=[]
for i in range(3):
p= multiprocessing.process(targets=task, temp=(i,))
process.append(p)
p.start()
for p in process:
p.join()

Output:
task:0
task:1
task:2

Using pool in multiprocessing:

from multiprocessing import pool
def squarevalue(i):
return i*i
if __name__='__main__':
with pool(5) as x:
result=x.map(squarevalue,[1,2,3,4,5,6])
print(result)

Output:
[1,4,9,16,25,36]

Using queue in multiprocessing:

from multiprocessing import process,Queue
def task(q):
q.put('Hello')
if __name__=='__main__':
q=Queue()
p=process(target=task,temp=(q,))
p.start()
print(q.get())
p.join()

Output:
Hello

Using pipe in multiprocessing:

from multiprocessing import process,pipe
def task(connect):
connect.send('Hello')
connect.close()
if __name__=='__main__':
p_connect,c_connect=pipe()
p=process(target=task,temp=(c_connect,))
p.start()
print(p_connect.recv())
p.join()

Output:
Hello

Synchronization in multiprocessing:

Synchronization is very crucial while ensuring that multiple processes can access the resources or data without any conflcits while accessing the resources. Multiprocessing provides several synchronization primitives for the accessing of shared resources.

Essential Synchronization primitives:

• Lock
• RLock
• Semaphore
• Bounded semaphore
• Event
• Condition
• Barrier

from multiprocessing import process,lock
def task(locker,number):
locker.acquire()
print(f'Locker was acquired by {number}')
locker.release()
if __name__=='__main__':
locker=lock()
for number in range(4):
process(target=task,temp=(locker,number)).start()

Output:
Locker was acquired by 0
Locker was acquired by 1
Locker was acquired by 2
Locker was acquired by 3

Here each line indicates that the respective process has acquired the lock successfully and then it prints the message that it has acquired the lock and then releases the lock. Lock ensures that messages are printing in correct order without any overlapping of messages or interleaving. The main important aspect is that no two messages are printed simultaneously i.e, after a message printed only next message will be printed like one after another which will be possible through the synchronization and lock concept.

 

Inorder to summarize on this topic, With the help of multiprocessing we can complete various tasks like parallel execution, Process management, Synchonization, communication and process pools.
We can also do other tasks like increasing the performance overhead, Cleaning up of resources, Testing thoroughly the current  code which we write and execute the code in parallel with another code which helps in enhancing the multiprocessing function.