解决Python多进程长时间运行导致内存爆炸的问题
Python的子进程不能被垃圾回收机制很好的回收全部的资源,所以当某个程序运行的时间足够长的时候,没有回收的资源逐渐累积,就导致了内存飙升;
为解决这个问题,主要还得靠手动处理;
方案就是每次检查尚在工作的子进程,发现后将其主动杀死;
核心方法:
def kill_child_processes(self, main_pid): for process in psutil.process_iter(): try: # psutil会缓存已经关闭的子进程 if process.ppid() == main_pid and process.status() == psutil.STATUS_RUNNING: # if process.ppid() == main_pid: process.terminate() except (psutil.NoSuchProcess, psutil.AccessDenied, psutil.ZombieProcess): pass
完整代码如下
```
import os import time import psutil class Test: def req(self): time.sleep(100) def kill_child_processes(self, main_pid): for process in psutil.process_iter(): try: # psutil会缓存已经关闭的子进程 if process.ppid() == main_pid and process.status() == psutil.STATUS_RUNNING: # if process.ppid() == main_pid: process.terminate() except (psutil.NoSuchProcess, psutil.AccessDenied, psutil.ZombieProcess): pass def run(self): main_pid = os.getpid() num = 0 # 30s钟跑一次 while True: num += 1 self.req() time.sleep(30) # 杀死30s前创建的所有子进程 self.kill_child_processes(main_pid)
```