Saturday, September 2, 2017
Yes, I'me building a website that lists what's on at local venues. Others have suggested I move the scripts into rails and using ActiveRecords certainly appals to me. Ime very open to suggestions/advice and if I can use ActiveRecord and the models from my ruby app, and keep the scripts stand alone that may be a way to go.
I am using terms like outside rails and stand alone but this may be the wrong terminology. It just seems logical to have access to rails infrastructure and put stuff there so there is not duplication, or I could use the rals infrastructure and have the batch processing classes separate, wit the scripts using both.
Maybe I should be asking a more genital question lets give it a go:
Regards,
Ben
-- I am using terms like outside rails and stand alone but this may be the wrong terminology. It just seems logical to have access to rails infrastructure and put stuff there so there is not duplication, or I could use the rals infrastructure and have the batch processing classes separate, wit the scripts using both.
Maybe I should be asking a more genital question lets give it a go:
If you have a rails app you are primevally using to display data collected by web scraping scripts and want to leverage the rails framework (ActiveRecords and anything else that is useful) to batch import data. How should I do this architecturally. Is there any documentation/guides/howtos that discuss this. I am tuning data into information.
I already have classes I have written to-do this that have them working as stand alone .rb. It seems logical to move the methods into rails and extend the ActiviveRecord classes etc. This may involve restructuring the object model but using the methods.
Broadly speaking the three things
- I am doing is scraping the data from websites (and loading the data into arrays to represent the data on these websites).
- Processing this data (the arrays) from the formats the websites has to a standard format (i.e. turning the various ways times are from the website to HH:MM). Also spiting up data (often there's a string that has the date and time and this needs splinting up to date and time).
- Cleaning the data, removing white-space, trailing/leading spaces etc.
- Writing the data to a table.
rails runner has been suggested as part of the solution.
Ben
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-talk+unsubscribe@googlegroups.com.
To post to this group, send email to rubyonrails-talk@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/3974567e-9794-4e30-bddb-78f6f9ca384c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment