ruby on rails - Experience with web crawlers on heroku -


does have experience coding web crawlers gems such anemone , deploying them heroku own person use? such continuously running programs violate of heroku's toa/tos?

i don't have experience using web crawlers in heroku (i interested in reading that!). here points:

  1. this prohibited content. illegal activity prohibited (duh) , since sites "prohibit" web crawlers , screen scrapers (such imdb), considered illegal. let's ignore now.

  2. these prohibited actions. following prohibited:

    data mining web property (including heroku) find email addresses or other user account information;

  3. these usage limits:

    • network bandwidth: 2tb/month - soft
    • shared db processing: max 200msec per second cpu time - soft
    • dyno ram usage: 512mb - hard
    • slug size: 200mb - hard
    • request length: 30 seconds - hard

  4. in tos, point 2.5., it's explained:

    repeated exceeding of hard or soft usage limits may lead termination of account.

emphasis mine. heroku gives each app 750 dyno hours. long don't abuse heroku's services , don't use gather personal info, believe you're in clear. suggest:

  1. somehow cap web crawler. should limit rate of api requests, should have common courtesy of limiting speed of crawler.

  2. keep eye on dyno hours. can here.


Comments

Post a Comment

Popular posts from this blog

linux - xterm copying to CLIPBOARD using copy-selection causes automatic updating of CLIPBOARD upon mouse selection -

i've tried various ways coerce xterm (versions 285 , 292) copy selection clipboard clipboard whenever press ctrl-shift-c. promising way far has been, in ~/.xresources, put this: xterm*vt100.translations: #override \ ctrl shift <keypress> c: copy-selection(clipboard) \n\ ctrl shift <keypress> v: insert-selection(clipboard) ctrl-shift-v works perfectly, copying has nuance... if restart xterm, highlighting text puts things in primary clipboard; expected, proper, default behavior. if hit ctrl-shift-c, copies current selection clipboard clipboard. the bug, however, if highlight text after first time pressed ctrl-shift-c, you'll see highlighting copies both primary and clipboard. cannot figure out how stop xterm updating clipboard upon selection... , doesn't make sense me in first place. i said, @ specific point in time, copy selection clipboard... yet starts updating automatically upon selection after doing once... anyone have workaround...
Read more

c++ - qgraphicsview horizontal scrolling always has a vertical delta -

Image
i've subclassed qgraphicsview custom canvas used in cad application. i've reimplemented qgraphicsview::wheelevent check keyboard modifiers control key and, if control key pressed, zoom. i'm trying implement horizontal scroll when user holds shift , uses wheel. the problem i'm having horizontal scrolling always scrolls 0.279. not huge problem, hugely annoying , points else being wrong. so, here questions: is right way implement horizontal scrolling? if not, is? how eliminate delta of 0.279? thanks in advance. code , sample output below void myview::zoom(int delta) { double factor = pow(1.2, delta/abs(delta)); this->scale(factor, factor); } void myview::scrollhorizontal(int level) { qpointf center = maptoscene(viewport()->rect().center()); qdebug() << "center: " << center.x() << ", " << center.y(); centeron(qpointf(center.x() - level, center.y())); } void myview::wheelevent(qwheeleve...
Read more