Friday, October 5, 2018
I see bots crawling devise pages in my log:
-- 141.8.143.179 - - [06/Oct/2018:00:35:45 -0400] "GET /users/sign_in HTTP/1.1" 200 4022 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
Should I disallow those paths in robots.txt ?
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-talk+unsubscribe@googlegroups.com.
To post to this group, send email to rubyonrails-talk@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/5091e20f-ec00-4764-9bb3-9a29a9bcbbf9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment