python - Celery connection broker lost makes CPU usage go to 100% -


we've been getting strange issues celery when mainprocess loses connection broker. celery generates error below , starts using 100% cpu. workers still continue work fine tho. can see in rabbitmq thinks connection timing out. we've been getting these error after updating celery 3.

i have feeling has non-blocking message consuming i'm not progressing in understanding code.

is there way detect these earlier or prevent celery using 100% cpu?

  • celery: 3.0.4
  • amqp: 1.0.10
  • rabbitmq: 2.8.4

the timestamp 2 hours apart because rabbitmq reports gmt , celery localtime.

celery error

[2013-05-09 18:20:20,204: error/mainprocess] consumer: connection broker lost. trying re-establish connection... traceback (most recent call last):   file "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 369, in start     self.consume_messages()   file "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 450, in consume_messages     readers[fileno](fileno, event)   file "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 290, in drain_nowait     self.drain_events(timeout=0)   file "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 279, in drain_events     return self.transport.drain_events(self.connection, **kwargs)   file "/usr/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 91, in drain_events     return connection.drain_events(**kwargs)   file "/usr/local/lib/python2.7/site-packages/amqp/connection.py", line 266, in drain_events     chanmap, none, timeout=timeout,   file "/usr/local/lib/python2.7/site-packages/amqp/connection.py", line 328, in _wait_multiple     channel, method_sig, args, content = read_timeout(timeout)   file "/usr/local/lib/python2.7/site-packages/amqp/connection.py", line 299, in read_timeout     return self.method_reader.read_method()   file "/usr/local/lib/python2.7/site-packages/amqp/method_framing.py", line 187, in read_method     raise m ioerror: socket closed

rabbitmq error

=error report==== 9-may-2013::16:20:20 === closing amqp connection <0.1813.0> (192.168.201.104:12809 -> 192.168.201.104:5672): {timeout,running}

i turned broker hearbeats off seems have fixed problem. however, i'm not quite don't have way of reproducing bug.


Comments

Post a Comment

Popular posts from this blog

linux - xterm copying to CLIPBOARD using copy-selection causes automatic updating of CLIPBOARD upon mouse selection -

i've tried various ways coerce xterm (versions 285 , 292) copy selection clipboard clipboard whenever press ctrl-shift-c. promising way far has been, in ~/.xresources, put this: xterm*vt100.translations: #override \ ctrl shift <keypress> c: copy-selection(clipboard) \n\ ctrl shift <keypress> v: insert-selection(clipboard) ctrl-shift-v works perfectly, copying has nuance... if restart xterm, highlighting text puts things in primary clipboard; expected, proper, default behavior. if hit ctrl-shift-c, copies current selection clipboard clipboard. the bug, however, if highlight text after first time pressed ctrl-shift-c, you'll see highlighting copies both primary and clipboard. cannot figure out how stop xterm updating clipboard upon selection... , doesn't make sense me in first place. i said, @ specific point in time, copy selection clipboard... yet starts updating automatically upon selection after doing once... anyone have workaround...
Read more

qt - Errors in generated MOC files for QT5 from cmake -

i generated moc files qt5 using set (cmake_automoc on) set(cmake_include_current_dir on) then add moc files src using set(src src/main.cpp src/video_widget_surface.cpp src/video_widget.cpp src/video_player.cpp #moc files moc/moc_video_player.cpp moc/moc_video_widget.cpp moc/moc_video_widget_surface.cpp finally add executable using add_executable(somegui ${src}) but errors in moc files saying : /other/workspace/perception/somestuff/moc/moc_video_widget.cpp:54:6: error: 'videowidget' has not been declared /other/workspace/perception/somestuff/moc/moc_video_widget.cpp:62:19: error: 'videowidget' has not been declared /other/workspace/perception/somestuff/moc/moc_video_widget.cpp:68:20: error: 'videowidget' has not been declared /other/workspace/perception/somestuff/moc/moc_video_widget.cpp:68:46: error: non-member function 'const qmetaobject* metaobject()' cannot have cv-qualifier /other/qt5.0.1/5.0.1/gcc_64/include/qtcore/qo...
Read more