ruby - Hanging on rake db:migrate ts:conf -
i have jenkins running deploy script on of our app machines. lately, half builds don't finish , keep hanging while trying run same thing. last of output looks this:
** [app@app1 :: stdout] generating configuration /var/www/app/releases/20130509192657/config/production.sphinx.conf ** [app@app2 :: stdout] generating configuration /var/www/app/releases/20130509192657/config/production.sphinx.conf ** [app@app3 :: stdout] generating configuration /var/www/app/releases/20130509192657/config/production.sphinx.conf ** [app@app4 :: stdout] generating configuration /var/www/app/releases/20130509192657/config/production.sphinx.conf ** [app@app6 :: stdout] generating configuration /var/www/app/releases/20130509192657/config/production.sphinx.conf ** [app@app7 :: stdout] generating configuration /var/www/app/releases/20130509192657/config/production.sphinx.conf
app5 machine seems have problem, , occurs when tries run:
/usr/local/bin/ruby /usr/local/bin/bundle exec rake db:migrate ts:conf
production running ruby 1.9.3p194, , due legacy reasons we're still running thinkingsphinx v. 0.9.8. we're running rails 3.2.13 , thinkingsphinx 2.0.7.
running strace on hanging process shows me this:
... 29802 select(4, [3], null, null, null <unfinished ...> 29790 restart_syscall(<... resuming interrupted call ...>) = -1 etimedout (connection timed out) 29790 futex(0x64a88e8, futex_wake_private, 1) = 0 29790 write(4, "!", 1 <unfinished ...> 29802 <... select resumed> ) = 1 (in [3]) 29790 <... write resumed> ) = 1 29802 read(3, <unfinished ...> 29790 futex(0x1d47f64, futex_wait_private, 3, null <unfinished ...> 29802 <... read resumed> "!", 1024) = 1 29802 select(4, [3], null, null, {0, 100000}) = 0 (timeout) 29802 select(4, [3], null, null, {0, 100000}) = 0 (timeout) 29802 select(4, [3], null, null, {0, 100000}) = 0 (timeout) 29802 select(4, [3], null, null, {0, 100000}) = 0 (timeout) 29802 select(4, [3], null, null, {0, 100000}) = 0 (timeout) 29802 select(4, [3], null, null, {0, 100000}) = 0 (timeout) 29802 select(4, [3], null, null, {0, 100000}) = 0 (timeout) 29802 select(4, [3], null, null, {0, 100000}) = 0 (timeout) 29802 select(4, [3], null, null, {0, 100000}) = 0 (timeout) 29802 select(4, [3], null, null, {0, 100000}) = 0 (timeout) 29802 select(4, [3], null, null, {0, 100000}) = 0 (timeout) 29802 select(4, [3], null, null, {0, 100000}) = 0 (timeout) 29802 select(4, [3], null, null, {0, 100000}) = 0 (timeout) 29802 select(4, [3], null, null, {0, 100000}) = 0 (timeout) 29802 select(4, [3], null, null, {0, 100000}) = 0 (timeout) 29802 select(4, [3], null, null, {0, 100000}) = 0 (timeout) 29802 select(4, [3], null, null, {0, 100000}) = 0 (timeout) 29802 select(4, [3], null, null, {0, 100000}) = 0 (timeout) 29802 select(4, [3], null, null, {0, 100000}) = 0 (timeout) 29802 select(4, [3], null, null, {0, 100000}) = 0 (timeout) ...
has ever seen before? not having of background in sysops, there specific approach should taking in trying tackle problem?
if db:migrate
locking, may have active, or hung - perhaps zombie - process locking database table (or other resource) being referred in migration. have experienced data fix-up script ran engineer had crashed (over week before deployment attempting), not exited - holding open transaction prevented alteration table. fix terminate stuck process, , migrate worked normal.
without knowing system architecture, it's difficult precise conflicting resource be. rdbms toolkit may allow take @ db hosted on server , see open connections to.
Comments
Post a Comment