Ruby on Rails
Friday, September 1, 2017
This is going to be a bit of a long one;)
I have a bunch of stand alone scripts that I want to move into rails. They are batch scripts that update the database (run from cron). Where I live the music listings websites are very incomplete so I am using web scrapers to collect listings for a listings website. They are very gently, fetching a page every 15 seconds. There is a .rb script for each venue.
Basically I have three main classes, scrape, venue and event, the later 2 having corresponding tables. I have used scaffold to create the tables in a postgres database, now I need to work out where to put the methods. The scrape one does not need access to the database but the venue and events ones do. I was planing on punting scrape class in a module in lib, there will be other classes eventually but for now the module will probably only have the one.
This is basically how it works:
The Venue class has arrays for all the attributes needed for events (title, date, time, description...), it used the Scrape class to collect the raw data for events.
The venue arrays are then looped through and loaded into the Event class which interprets the raw data and writes the event record. It uses a lot of setters, so when, for example, the time is set it is interpreted into a HH:MM format (it can be 10am, 1030am, 1030, 10:30...). There are lots of other loose/fuzzy formatted data for events and this is all interpreted using setter. I am not saying this architecture is correct but it works.
As an aside I also clean the data as there are lots of white-space characters.
The first step is to move stuff into rails, at a later date I may change things but for now I just want to get it working in a smeller way to the currently functioning stand alone scripts. I will be using rails runner from cron to schedule the scripts.
So do I put the venue and event methods in to controllers or is there somewhere else. The venue and event classes map to tables which is why this seems the way to go. I want to put them into rails so I can use ActiveRecords, and also because it feels like the correct thing to do? I then need to work out how to change the setters in Active record, or maybe not, but at the moment I simply am trying to work out where the methods go.
-- I have a bunch of stand alone scripts that I want to move into rails. They are batch scripts that update the database (run from cron). Where I live the music listings websites are very incomplete so I am using web scrapers to collect listings for a listings website. They are very gently, fetching a page every 15 seconds. There is a .rb script for each venue.
Basically I have three main classes, scrape, venue and event, the later 2 having corresponding tables. I have used scaffold to create the tables in a postgres database, now I need to work out where to put the methods. The scrape one does not need access to the database but the venue and events ones do. I was planing on punting scrape class in a module in lib, there will be other classes eventually but for now the module will probably only have the one.
This is basically how it works:
The Venue class has arrays for all the attributes needed for events (title, date, time, description...), it used the Scrape class to collect the raw data for events.
The venue arrays are then looped through and loaded into the Event class which interprets the raw data and writes the event record. It uses a lot of setters, so when, for example, the time is set it is interpreted into a HH:MM format (it can be 10am, 1030am, 1030, 10:30...). There are lots of other loose/fuzzy formatted data for events and this is all interpreted using setter. I am not saying this architecture is correct but it works.
As an aside I also clean the data as there are lots of white-space characters.
The first step is to move stuff into rails, at a later date I may change things but for now I just want to get it working in a smeller way to the currently functioning stand alone scripts. I will be using rails runner from cron to schedule the scripts.
So do I put the venue and event methods in to controllers or is there somewhere else. The venue and event classes map to tables which is why this seems the way to go. I want to put them into rails so I can use ActiveRecords, and also because it feels like the correct thing to do? I then need to work out how to change the setters in Active record, or maybe not, but at the moment I simply am trying to work out where the methods go.
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-talk+unsubscribe@googlegroups.com.
To post to this group, send email to rubyonrails-talk@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/571a90ce-dba7-4af3-a200-5748a7ba1968%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment