Ruby on Rails

On Fri, Sep 1, 2017 at 12:02 PM, Ben Edwards <lo...@funkytwig.com> wrote:

> Now a bit of context is probably needed.  I will have quite a lot of scripts
> (possibly 40, each one is a web scraper).  They do very gentle scraping
> (i.e. a http request every 15 seconds) so currently I run them concurrently
> using cron.

Out of curiousity -- what OS are you using that gives you sub-minute
granularity for cron?

> Doing this all within rails seems a great idea but I am worried
> about the (resource) cost of doing this (initializing 40 rails environments)
> will be quite high.  Did some googeling and could not find out what the cost
> is.  Don't want to put undue stress on the server so although I like the
> ideal it may be better to keep them as separate stand alone scripts outside
> rails.  Like I said I would rather run them inside rails.  What so you
> think?

Depends on the server, but yes, starting 40 full rails environments
every 15 seconds seems like a bad idea. At this point, you should
probably be thinking in terms of background jobs. Or leaving them
as standalone scripts.

A lot depends on the details -- how close to exact that 15 seconds
needs to be, how to handle timeouts, what you're doing with the data
they're collecting, etc.

But set it up, try it, see how it goes :-)

--
Hassan Schroeder ------------------------ hassan.s...@gmail.com
twitter: @hassan
Consulting Availability : Silicon Valley or remote