java - crawler4j always returns fatal transport error -


this seed add crawler4j.

error [crawler 1] fatal transport error: connection http://example.com refused while fetching http://example.com/page.html (link found in doc #0)

this weird me. don't know causes it.

try increase politeness setting of crawler. happens me when server want crawler not good.


Comments

Post a Comment

Popular posts from this blog

Java sticky instances of class com.mysql.jdbc.Field aggregating -

i noticed (via jmap+jhat) app leaking storing many instances com.mysql.jdbc.field  class - double checked code make sure closing preparedstatements, find few missing didn't change outcome. - preparedstatement created , closed right away - several others created @ start , reused many times - using org.apache.tomcat.jdbc.pool.datasource connection pool what can cause type of behavior? are closing of preparedstatements / result sets using: preparedstatement stmt (possible assignment here); try { // work prepared statement resultset rs; try { while (rs.next()) ... // process results } { rs.close(); } } { stmt.close(); } so guarantee result sets closed when finished (regardless of exceptions) , statements closed (regardless of exceptions)? there other ways write that, idea same. if you're doing of properly, unless have many references it's taking inordinate amount of memory, it's ok , nothing needs done it.
Read more