首页 » 编程语言 » PHP » php进程导致服务器cpu 100问题追查过程

php进程导致服务器cpu 100问题追查过程

 

前段时间,出现了一次服务器cpu 占用100的问题。以下为追查原因的过程。仅当抛砖引玉,欢迎拍砖。
查看占用cpu高的进程
想找出占用cpu高的进程,用top命令就可以搞定。

$top
.....此处省略n多行...
10434 root     20   0  509m 174m 1528 R 99.7  0.5   8:42.43 php                                                                                                        
 5638 root     20   0  509m 174m 1528 R 99.1  0.5   9:12.35 php                                                                                                        
16390 root     20   0  541m 182m 5244 R 98.4  0.6   8:40.92 php        

此时,轻轻按下C键。就会看到如下信息。

10434  root     20   0  540m 182m 5244 R 101.0  0.6   9:57.39 php /home/gearman_manager/pecl-manager.php -P /home/gearman_manager/manager.pid -u root -d -v
5638 root     20   0  509m 174m 1528 R 101.0  0.5  10:15.28 php /home/gearman_manager/pecl-manager.php -P /home/gearman_manager/manager.pid -u root -d -v
16390 root     20   0  509m 174m 1528 R 99.5  0.5   9:41.18 php /home/gearman_manager/pecl-manager.php -P /home/gearman_manager/manager.pid -u root -d -vv

找出进程占用cpu高的原因
进程占用cpu高,一般是由于进程长时间占用cpu,又没有主动释放占用。如果想主动释放cpu,可以调用sleep。在写程序的时候,尤其要注意while 等循环的地方。

找出php进程在执行那段代码

$sudo gdb -p 10434
(gdb) print (char *)executor_globals.active_op_array->filename
$13 = 0x2924118 "/home/gearman_manager/pecl-manager.php"
(gdb) print executor_globals->current_execute_data->opline->lineno
$14 = 55
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x00000031d32306d0 in sigprocmask () from /lib64/libc.so.6
(gdb) print executor_globals->current_execute_data->opline->lineno
$15 = 71
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x00000000006250e1 in zend_hash_find ()
(gdb) print executor_globals->current_execute_data->opline->lineno
$16 = 53

如果对上面的命令有疑问,可以查看 [当cpu飙升时,找出php中可能有问题的代码行]
根据上面的信息,我们可以知道,cpu高时,正在执行/home/gearman_manager/pecl-manager.php文件。并且正在执行53和71行附近的代码。

分析相关代码
php的相关代码如下:

while(!$this->stop_work){ //此处是第51行
    if(@$thisWorker->work() ||
        $thisWorker->returnCode() == GEARMAN_IO_WAIT ||
        $thisWorker->returnCode() == GEARMAN_NOT_CONNECTED ||
        $thisWorker->returnCode() == GEARMAN_NO_JOBS) {
         if ($thisWorker->returnCode() == GEARMAN_SUCCESS) continue;
         if (!@$thisWorker->wait()){
             if ($thisWorker->returnCode() == GEARMAN_NO_ACTIVE_FDS){
                 sleep(5);
             }
         }
    }

    /**
     * Check the running time of the current child. If it has
     * been too long, stop working.
     */
     //此处是第71行
     if($this->max_run_time > 0 && time() - $start > $this->max_run_time) {
         $this->log("Been running too long, exiting", GearmanManager::LOG_LEVEL_WORKER_INFO);
         $this->stop_work = true;
      }

      if(!empty($this->config["max_runs_per_worker"]) && $this->job_execution_count >= $this->config["max_runs_per_worker"]) {
          $this->log("Ran $this->job_execution_count jobs which is over the maximum({$this->config['max_runs_per_worker']}), exiting", GearmanManager::LOG_LEVEL_WORKER_INFO);
           $this->stop_work = true;
       }
}

看来作者已经考虑到某些情况可能导致cpu 100,因此在代码中使用了sleep方法。现在php进程占用cpu 100,看来是没执行sleep(5)。如果$thisWorker->work() 和 $thisWorker->returnCode() 方法中没有IO等操作,能释放cpu的占用,又没有调用sleep的情况下,很容易导致进程占用cpu 100。
根据代码我们可以得出如下几个结论:
- 肯定没执行slepp(5)
- $thisWorker->work() 和 $thisWorker->returnCode() 方法中没有IO等操作,能释放cpu的占用。

进一步跟踪代码中可疑部分
分析 $thisWorker->returnCode() 的返回值。如果很容易复现cpu 100的话,你可以在程序中echo $thisWorker->returnCode() 来获得返回值。但是现在cpu 100的问题,复现比较麻烦。所以,还是使用gdb 来搞吧。

$gdb -p 10434
(gdb)b zif_gearman_worker_return_code 
Breakpoint 1, zif_gearman_worker_return_code (ht=0, return_value=0x2a1b648, return_value_ptr=0x0, this_ptr=0x2a14090, return_value_used=1)
    at /home/php_src/pecl/gearman-1.1.1/php_gearman.c:3186
3186    in /home/php_src/pecl/gearman-1.1.1/php_gearman.c
(gdb) p return_value
$3 = (zval *) 0x2a1b648
(gdb) p *$3
$4 = {value = {lval = 0, dval = 0, str = {val = 0x0, len = 0}, ht = 0x0, obj = {handle = 0, handlers = 0x0}}, refcount__gc = 1, type = 0 '\0', is_ref__gc = 0 '\0'}
(gdb) s
3189    in /home/php_src/pecl/gearman-1.1.1/php_gearman.c
(gdb) p *$3
$5 = {value = {lval = 0, dval = 0, str = {val = 0x0, len = 0}, ht = 0x0, obj = {handle = 0, handlers = 0x0}}, refcount__gc = 1, type = 0 '\0', is_ref__gc = 0 '\0'}
(gdb) s
3186    in /home/php_src/pecl/gearman-1.1.1/php_gearman.c
(gdb) p *$3
$6 = {value = {lval = 0, dval = 0, str = {val = 0x0, len = 0}, ht = 0x0, obj = {handle = 0, handlers = 0x0}}, refcount__gc = 1, type = 0 '\0', is_ref__gc = 0 '\0'}
(gdb) s
3189    in /home/php_src/pecl/gearman-1.1.1/php_gearman.c
(gdb) p *$3
$7 = {value = {lval = 0, dval = 0, str = {val = 0x0, len = 0}, ht = 0x0, obj = {handle = 0, handlers = 0x0}}, refcount__gc = 1, type = 0 '\0', is_ref__gc = 0 '\0'}
(gdb) s
3190    in /home/php_src/pecl/gearman-1.1.1/php_gearman.c
(gdb) p *$3
$8 = {value = {lval = 0, dval = 0, str = {val = 0x0, len = 0}, ht = 0x0, obj = {handle = 0, handlers = 0x0}}, refcount__gc = 1, type = 0 '\0', is_ref__gc = 0 '\0'}
(gdb) s
3191    in /home/php_src/pecl/gearman-1.1.1/php_gearman.c
(gdb) p *$3
$9 = {value = {lval = 25, dval = 1.2351641146031164e-322, str = {val = 0x19 <Address 0x19 out of bounds>, len = 0}, ht = 0x19, obj = {handle = 25, handlers = 0x0}}, 
  refcount__gc = 1, type = 1 '\001', is_ref__gc = 0 '\0'}

从最后的lval = 25 和 type =1 我们可以看出 returnCode()方法的最后返回值是 25 。
根据文档,返回值25 对应的是 GEARMAN_NOT_CONNECTED。

解决问题

既然出问题时, returnCode()方法的最后返回值是 25,而现在程序中对这种情况又没进行出来,导致了cpu 100。那我们只要在出现这种情况时,sleep几秒就ok了。

if(@$thisWorker->work() ||
               $thisWorker->returnCode() == GEARMAN_IO_WAIT ||
               $thisWorker->returnCode() == GEARMAN_NOT_CONNECTED ||
               $thisWorker->returnCode() == GEARMAN_NO_JOBS) {
                //下面一行是新增的
                if ($thisWorker->returnCode() == GEARMAN_NOT_CONNECTED) sleep(5);
                if ($thisWorker->returnCode() == GEARMAN_SUCCESS) continue;

                if (!@$thisWorker->wait()){
                    if ($thisWorker->returnCode() == GEARMAN_NO_ACTIVE_FDS){
                        sleep(5);
                    }
                }

            }

技术交流

原文链接:php进程导致服务器cpu 100问题追查过程,转载请注明来源!

原文链接:php进程导致服务器cpu 100问题追查过程,转载请注明来源!

10