python - Celery connection broker lost makes CPU usage go to 100% -


we've been getting strange issues celery when mainprocess loses connection broker. celery generates error below , starts using 100% cpu. workers still continue work fine tho. can see in rabbitmq thinks connection timing out. we've been getting these error after updating celery 3.

i have feeling has non-blocking message consuming i'm not progressing in understanding code.

is there way detect these earlier or prevent celery using 100% cpu?

  • celery: 3.0.4
  • amqp: 1.0.10
  • rabbitmq: 2.8.4

the timestamp 2 hours apart because rabbitmq reports gmt , celery localtime.

celery error

[2013-05-09 18:20:20,204: error/mainprocess] consumer: connection broker lost. trying re-establish connection... traceback (most recent call last):   file "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 369, in start     self.consume_messages()   file "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 450, in consume_messages     readers[fileno](fileno, event)   file "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 290, in drain_nowait     self.drain_events(timeout=0)   file "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 279, in drain_events     return self.transport.drain_events(self.connection, **kwargs)   file "/usr/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 91, in drain_events     return connection.drain_events(**kwargs)   file "/usr/local/lib/python2.7/site-packages/amqp/connection.py", line 266, in drain_events     chanmap, none, timeout=timeout,   file "/usr/local/lib/python2.7/site-packages/amqp/connection.py", line 328, in _wait_multiple     channel, method_sig, args, content = read_timeout(timeout)   file "/usr/local/lib/python2.7/site-packages/amqp/connection.py", line 299, in read_timeout     return self.method_reader.read_method()   file "/usr/local/lib/python2.7/site-packages/amqp/method_framing.py", line 187, in read_method     raise m ioerror: socket closed

rabbitmq error

=error report==== 9-may-2013::16:20:20 === closing amqp connection <0.1813.0> (192.168.201.104:12809 -> 192.168.201.104:5672): {timeout,running}

i turned broker hearbeats off seems have fixed problem. however, i'm not quite don't have way of reproducing bug.


Comments

Post a Comment

Popular posts from this blog

Java sticky instances of class com.mysql.jdbc.Field aggregating -

i noticed (via jmap+jhat) app leaking storing many instances com.mysql.jdbc.field  class - double checked code make sure closing preparedstatements, find few missing didn't change outcome. - preparedstatement created , closed right away - several others created @ start , reused many times - using org.apache.tomcat.jdbc.pool.datasource connection pool what can cause type of behavior? are closing of preparedstatements / result sets using: preparedstatement stmt (possible assignment here); try { // work prepared statement resultset rs; try { while (rs.next()) ... // process results } { rs.close(); } } { stmt.close(); } so guarantee result sets closed when finished (regardless of exceptions) , statements closed (regardless of exceptions)? there other ways write that, idea same. if you're doing of properly, unless have many references it's taking inordinate amount of memory, it's ok , nothing needs done it.
Read more