python内存泄漏

最新推荐文章于 2025-06-04 15:49:15 发布

原创最新推荐文章于 2025-06-04 15:49:15 发布 · 3.2k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#python #merge #date #class #command #system

一直没有管python的内存泄漏，等产品已经达到7×365运行了，运行了2天后，占用了6G，而且还在增长。看来不得不解决python的这个问题了。

占用内存：31*19.3%=5.983G=6126M
运行时间：(2days,22:51:21.637736)=255081s
完成任务数：1396projects
泄漏速度：24.59k/s 4.388m/project

python的这个问题是循环引用，导致gc不知道如何释放内存。

以下语句就会造成a和b都不会释放：
class MyClass:
def __init__(self):
self.next=None;
a=MyClass();
b=MyClass();
a.next=b;
b.next=a;

解决方法很简单，写dispose函数，至上而下的打开引用就可以。

class MyClass:
def dispose():
self.next = None;

需要主动打断循环引用：
a.dispose();
b.dispose():

修改后，系统占用内存稳定在2.1%约666M，没有继续上升。

=================================================================================================================================================================================
Mon Aug 29 14:35:31 CST 2011
root 16855 11.6 0.5 342976 176832 ? Sl 13:07 10:09 /usr/local/tvie/bin/transcode_system/tvie_scheduler -c /usr/local/tvie/config/transcode_system/scheduler.conf
running time: 1:28:20.319298.
projects summary: all:861, finish:120, fail:1, abort:0, active:740(init:4, slice:549, trans:44, merge:143).
31*1024*0.5/100=158.72M
1:28:20.319298=5300s
泄漏速度：30.66k/s 188.76k/project 1.31k/piece
=================================================================================================================================================================================
Mon Aug 29 15:05:47 CST 2011
root 16855 9.7 0.7 406844 242692 ? Sl 13:07 11:27 /usr/local/tvie/bin/transcode_system/tvie_scheduler -c /usr/local/tvie/config/transcode_system/scheduler.conf
running time: 1:58:24.360514.
projects summary: all:901, finish:164, fail:1, abort:0, active:736(init:0, slice:487, trans:50, merge:199).
31*1024*0.7/100=222.208M
1:58:24.360514=7104
泄漏速度：32.02k/s 252.5k/project 1.75k/piece 泄漏加速度：(222-158)/(7104-5300)=36.32k/s
=================================================================================================================================================================================
Mon Aug 29 15:52:39 CST 2011
root 16855 8.1 0.9 479948 317512 ? Sl 13:07 13:27 /usr/local/tvie/bin/transcode_system/tvie_scheduler -c /usr/local/tvie/config/transcode_system/scheduler.conf
running time: 2:45:15.494224.
projects summary: all:965, finish:224, fail:1, abort:0, active:740(init:4, slice:402, trans:44, merge:290).
31*1024*0.9/100=285.696M
2:45:15.494224=9915s
泄漏速度：29.5k/s 303k/project 2.1k/piece 泄漏加速度：(285-222)/(9915-7104)=22.95k/s
=================================================================================================================================================================================
2011-08-29 17:12:35
root 16855 6.8 1.2 560408 413756 ? Sl 13:07 16:50 /usr/local/tvie/bin/transcode_system/tvie_scheduler -c /usr/local/tvie/config/transcode_system/scheduler.conf
running time: 4:04:42.085104.
projects summary: all:1077, finish:335, fail:3, abort:0, active:739(init:2, slice:251, trans:49, merge:437).
31*1024*1.2/100=380.928M
4:04:42.085104=14682s
泄漏速度：26.57k/s 362.2k/project 2.52k/piece 泄漏加速度：(380-285)/(14682-9915)=20.41k/s
=================================================================================================================================================================================
Tue Aug 30 10:19:44 CST 2011
root 16855 3.7 2.1 895880 714088 ? Sl Aug29 47:32 /usr/local/tvie/bin/transcode_system/tvie_scheduler -c /usr/local/tvie/config/transcode_system/scheduler.conf
running time: 21:12:21.625271.
projects summary: all:3081, finish:2297, fail:47, abort:0, active:737(init:4, slice:9, trans:1, merge:723).
31*1024*2.1/100=666.624M
21:12:21.625271=76341s
泄漏速度：8.94k/s 221.56k/project 1.539k/piece 泄漏加速度：(666-380)/(76341-14682)=4.75k/s
=================================================================================================================================================================================
Tue Aug 30 12:13:35 CST 2011
root 16855 3.6 2.1 895880 714088 ? Sl Aug29 50:34 /usr/local/tvie/bin/transcode_system/tvie_scheduler -c /usr/local/tvie/config/transcode_system/scheduler.conf
running time: 23:08:22.715334.
projects summary: all:3303, finish:2518, fail:48, abort:0, active:737(init:3, slice:5, trans:4, merge:725).
31*1024*2.1/100=666.624M
23:08:22.715334=83302s
泄漏速度：8.18k/s 206.4k/project 1.4k/piece 泄漏加速度：(666-666)/(83302-76341)=0k/s
=================================================================================================================================================================================
Tue Aug 30 14:12:11 CST 2011
root 16855 3.5 2.1 896544 714804 ? Sl Aug29 53:50 /usr/local/tvie/bin/transcode_system/tvie_scheduler -c /usr/local/tvie/config/transcode_system/scheduler.conf
running time: 1 day, 1:04:47.502380.
projects summary: all:3532, finish:2745, fail:51, abort:0, active:736(init:4, slice:3, trans:5, merge:724).
31*1024*2.1/100=666.624M
(1 day, 1:04:47.502380)=90287s
泄漏速度：7.55k/s 193.1k/project 1.34k/piece 泄漏加速度：(666-666)/(90287-83302)=0k/s
=================================================================================================================================================================================

config: /usr/local/tvie/config/transcode_system/cli.conf. start to execute command for 1 times: status.
transcode system report. status: no_project_idling. start at: 2011-11-21 16:01:44 running time: 25 days, 2:46:10.603138.
build:1.0.10812 build date:2011-11-21 15:56:47 release:1.0(alpha) sm_ver:1.0 description:project-code fate.
lisence expire date: 2012-05-17. now: 2011-12-16 18:47:54
total worker count:16. the detail is "[id][Local|Remote]ip:work_count" :
total slicer worker 2, details:
[2][L&R]192.168.2.39,120.72.48.39:21[3][L&R]192.168.2.39,120.72.48.39:22
total transcoder worker 6, details:
[4][R]192.168.2.39,120.72.48.39:23[5][R]192.168.2.39,120.72.48.39:20[6][R]192.168.2.39,120.72.48.39:23[7][R]192.168.2.39,120.72.48.39:19
[8][R]192.168.2.39,120.72.48.39:16[9][R]192.168.2.39,120.72.48.39:20
total merger worker 2, details:
[0][L]192.168.2.39,120.72.48.39:21[1][L]192.168.2.39,120.72.48.39:23
total integrater worker 6, details:
[10][R]192.168.2.39,120.72.48.39:1[11][R]192.168.2.39,120.72.48.39:2[12][R]192.168.2.39,120.72.48.39:1[13][R]192.168.2.39,120.72.48.39:1
[14][R]192.168.2.39,120.72.48.39:2[15][R]192.168.2.39,120.72.48.39:41
projects summary: all:63, finish:50, fail:8, abort:5, active:0(init:0, slice:0, trans:0, merge:0).
time costs summary: total time(s): slice=5036.613, transcode=22858.25(total:down=1.51,trans=57673.211,up=0.898), merge=825.533