Web tarayıcısı için tipik nezaket faktörü?

Question

Web tarayıcısı için tipik bir nezaket faktörü nedir? Apart hep robot.txt uymaktan

Hem "Disallow:" ve standart olmayan "Tarama gecikmesi:"

Ama bir site açık bir tarama-gecikme girmezse varsayılan değer ne olsun

kurmak?

Answer 1

Kullandığımız algoritmadır:

// If we are blocked by robots.txt 
// Make sure it is obeyed. 
// Our bots user-agent string contains a link to a html page explaining this. 
// Also an email address to be added to so that we never even consider their domain in the future 

// If we receive more that 5 consecutive responses with HTTP response code of 500+ (or timeouts) 
// Then we assume the domain is either under heavy load and does not need us adding to it. 
// Or the URL we are crawling are completely wrong and causing problems 
// Wither way we suspend crawling from this domain for 4 hours. 

// There is a non-standard parameter in robots.txt that defines a min crawl delay 
// If it exists then obey it. 
// 
// see: http://www.searchtools.com/robots/robots-txt-elements.html 
double PolitenssFromRobotsTxt = getRobotPolitness(); 


// Work Size politeness 
// Large popular domains are designed to handle load so we can use a 
// smaller delay on these sites then for smaller domains (thus smaller domains hosted by 
// mom and pops by the family PC under the desk in the office are crawled slowly). 
// 
// But the max delay here is 5 seconds: 
// 
// domainSize => Range 0 -> 10 
// 
double workSizeTime = std::min(exp(2.52166863221 + -0.530185027289 * log(domainSize)), 5); 
// 
// You can find out how important we think your site is here: 
//  http://www.opensiteexplorer.org 
// Look at the Domain Authority and diveide by 10. 
// Note: This is not exactly the number we use but the two numbers are highly corelated 
//  Thus it will usually give you a fair indication. 



// Take into account the response time of the last request. 
// If the server is under heavy load and taking a long time to respond 
// then we slow down the requests. Note time-outs are handled above 
double responseTime = pow(0.203137637588 + 0.724386103344 * lastResponseTime, 2); 

// Use the slower of the calculated times 
double result = std::max(workSizeTime, responseTime); 

//Never faster than the crawl-delay directive 
result = std::max(result, PolitenssFromRobotsTxt); 


// Set a minimum delays 
// So never hit a site more than every 10th of a second 
result = std::max(result, 0.1); 

// The maximum delay we have is every 2 minutes. 
result = std::min(result, 120.0)