web applications - Does Google spider read robots.txt before accessing a resource? -


i have been wondering while. know if, instance, link here site example.com , domain hosts robots.txt file disallow everything, google won't index it.

however, happens if i, instance, insert here in stackoverflow js (script src ...) loads resource example.com? google read robots.txt first , decide load resource, or going load anyway, not index it? (this question)

basically worries me since have tracker , not spiders increase number of visits. of course there possibility of blocking several spiders in code putting user-agent, not 100% valid measure since many other spiders (not google, or not search spiders) count.

in theory, google says not read resources excluded robots.txt. in practice, google has not been consistent on this. several months year (and 2012), google not obeying robots.txt sure received complaints - provided mine! however, of mid-may 2013, googlebot seems reading , respecting robots.txt. means reads robots.txt, , reads resources not excluded.

why there time when did not respect robots.txt? possible answers: bad programming. maybe because google has grown much, has been more difficult enforce quality standards. maybe wanted more content. google if ever admits errors, unlikely admit not respecting robots.txt not them forthright on topic.


Comments

Post a Comment

Popular posts from this blog

linux - xterm copying to CLIPBOARD using copy-selection causes automatic updating of CLIPBOARD upon mouse selection -

i've tried various ways coerce xterm (versions 285 , 292) copy selection clipboard clipboard whenever press ctrl-shift-c. promising way far has been, in ~/.xresources, put this: xterm*vt100.translations: #override \ ctrl shift <keypress> c: copy-selection(clipboard) \n\ ctrl shift <keypress> v: insert-selection(clipboard) ctrl-shift-v works perfectly, copying has nuance... if restart xterm, highlighting text puts things in primary clipboard; expected, proper, default behavior. if hit ctrl-shift-c, copies current selection clipboard clipboard. the bug, however, if highlight text after first time pressed ctrl-shift-c, you'll see highlighting copies both primary and clipboard. cannot figure out how stop xterm updating clipboard upon selection... , doesn't make sense me in first place. i said, @ specific point in time, copy selection clipboard... yet starts updating automatically upon selection after doing once... anyone have workaround...
Read more

c++ - qgraphicsview horizontal scrolling always has a vertical delta -

Image
i've subclassed qgraphicsview custom canvas used in cad application. i've reimplemented qgraphicsview::wheelevent check keyboard modifiers control key and, if control key pressed, zoom. i'm trying implement horizontal scroll when user holds shift , uses wheel. the problem i'm having horizontal scrolling always scrolls 0.279. not huge problem, hugely annoying , points else being wrong. so, here questions: is right way implement horizontal scrolling? if not, is? how eliminate delta of 0.279? thanks in advance. code , sample output below void myview::zoom(int delta) { double factor = pow(1.2, delta/abs(delta)); this->scale(factor, factor); } void myview::scrollhorizontal(int level) { qpointf center = maptoscene(viewport()->rect().center()); qdebug() << "center: " << center.x() << ", " << center.y(); centeron(qpointf(center.x() - level, center.y())); } void myview::wheelevent(qwheeleve...
Read more