crawl-delay: # (advantage and disadvantages)

[jutplus]

Hi friends,

i'm thinking using crawl-delay in my robots, but i want to know further the strong points of having high and lower values, and what's the most appropriate values for websites.

thanks.

[h0ster]

It's quite a large subject but I will say that the values that you set are a guideline and not a rule for the engines to follow.

Maybe give me a few more details, ie. context of site, page structure, and i'll see if I can help.