Re: [Rails] disallow devise paths in robots.txt ?

Ruby on Rails Sunday, October 7, 2018

On Fri, Oct 5, 2018 at 11:04 PM, fugee ohu <fugee279@gmail.com> wrote:
> I see bots crawling devise pages in my log:

> Should I disallow those paths in robots.txt ?

1) why?

2) do your logs show these bots *reading* robots.txt?

--
Hassan Schroeder ------------------------ hassan.schroeder@gmail.com
twitter: @hassan
Consulting Availability : Silicon Valley or remote

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-talk+unsubscribe@googlegroups.com.
To post to this group, send email to rubyonrails-talk@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/CACmC4yBoWsK3rHNEHckW5vTenKG4HWh-xUTKrtgepK274VBo3w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Ruby on Rails