Ruby on Rails

You won't have some friendly parsed javascript response. The javascript it's not information itself, it's a lot of command the will handle browser's DOM. That's why we're using a driver to get this informations.

If you get request and parse it, you'll get the raw html with javascript code, but if you use a driver, then it'll get the response and execute the loaded javascript. This is the key.

Em qui, 27 de dez de 2018 às 11:30, fugee ohu <fuge...@gmail.com> escreveu:

On Wednesday, December 26, 2018 at 6:14:19 PM UTC-5, Walter Lee Davis wrote:

> On Dec 26, 2018, at 5:32 PM, fugee ohu <fuge...@gmail.com> wrote:
>
>
>
> On Wednesday, December 26, 2018 at 2:15:37 PM UTC-5, Walter Lee Davis wrote:
>
> > On Dec 26, 2018, at 2:11 PM, fugee ohu <fuge...@gmail.com> wrote:
> >
> >
> >
> > On Wednesday, December 26, 2018 at 11:58:35 AM UTC-5, Rafael Belo wrote:
> > You've to include Capybara::DSL.
> >
> > ```
> > include Capybara::DSL
> > ```
> >
> > Em qua, 26 de dez de 2018 às 13:21, fugee ohu <fuge...@gmail.com> escreveu:
> >
> >
> > On Wednesday, December 26, 2018 at 10:25:33 AM UTC-5, Rafael Belo wrote:
> > Capybara has a friendly interface for your web drivers, you can integrate it with selenium, webkit, poltergeist and other.
> > Try to use it, I think you will like it.
> >
> > https://github.com/teamcapybara/capybara
> >
> > Em qua, 26 de dez de 2018 às 12:08, fugee ohu <fuge...@gmail.com> escreveu:
> >
> >
> > On Wednesday, December 26, 2018 at 9:49:28 AM UTC-5, Rafael Belo wrote:
> > Yes, if you are using capybara, you may use `visit 'http://myurl.com/goes-here'`
> >
> > Em qua, 26 de dez de 2018 às 11:21, fugee ohu <fuge...@gmail.com> escreveu:
> >
> >
> > On Wednesday, December 26, 2018 at 8:19:47 AM UTC-5, Rafael Belo wrote:
> > You are passing an URL instead of a script.
> > This function "execute_script" it's to execute sobre javascript "script".
> > You've to visit the page with the browser support.
> > If you're using Capybara, you shoul use the "visit" function passing the URL that you want to go.
> > But if not, you've to look which command your driver has to visit URL's.
> >
> > Em quarta-feira, 26 de dezembro de 2018 03:51:39 UTC-3, fugee ohu escreveu:
> >
> >
> > On Tuesday, December 25, 2018 at 6:40:23 PM UTC-5, Hassan Schroeder wrote:
> > On Tue, Dec 25, 2018 at 3:16 PM fugee ohu <fuge...@gmail.com> wrote:
> > >
> > > How do i scrape dynamic content from Struts framework with Ruby
> >
> > Same as any web source: send a request, parse the response. Is
> > there some particular issue you're encountering?
> >
> > --
> > Hassan Schroeder ------------------------ hassan.s...@gmail.com
> > twitter: @hassan
> > Consulting Availability : Silicon Valley or remote
> >
> > browser.execute_script('https://gpsfront.sitename.com/getI2iRecommendingResults.do?callback=jQuery18307882644047005491_1545806199753&currentItemList=32819755026&categoryId=200001521&shopId=2339135&companyId=238468932&recommendType=&scenario=pcDetailLeftTopSell&limit=6&offset=0&_=1545806304149')
> >
> > Selenium::WebDriver::Error::UnknownError: unknown error: Runtime.evaluate threw exception: SyntaxError: Unexpected end of input
> >   (Session info: chrome=71.0.3578.80)
> >   (Driver info: chromedriver=2.42.591071 (0b695ff80972cc1a65a5cd643186d2ae582cd4ac),platform=Linux 4.15.0-43-generic x86_64)
> >
> >
> >
> > Must I use the url?
> >  
> >
> > --
> > You received this message because you are subscribed to a topic in the Google Groups "Ruby on Rails: Talk" group.
> > To unsubscribe from this topic, visit https://groups.google.com/d/topic/rubyonrails-talk/CpOPHz-zFsc/unsubscribe.
> > To unsubscribe from this group and all its topics, send an email to rubyonrails-ta...@googlegroups.com.
> > To post to this group, send email to rubyonra...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/acaef991-80d3-4c6c-bccc-fdc017ef4734%40googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
> >
> >
> > --
> > Rafael Belo
> > Web Developer
> > Skype: rafaelrpbelo
> > Twitter: @rafaelrpbelo
> > Linkedin: rafaelrpbelo
> >
> > This is the first time I'm hearing Capybara recommended for web scraping Is this the preferred method for what I'm trying to do?
> >
> > --
> > You received this message because you are subscribed to a topic in the Google Groups "Ruby on Rails: Talk" group.
> > To unsubscribe from this topic, visit https://groups.google.com/d/topic/rubyonrails-talk/CpOPHz-zFsc/unsubscribe.
> > To unsubscribe from this group and all its topics, send an email to rubyonrails-ta...@googlegroups.com.
> > To post to this group, send email to rubyonra...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/6d8212f3-dae8-444c-9004-28a4c5b0b103%40googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
> >
> >
> > --
> > Rafael Belo
> > Web Developer
> > Skype: rafaelrpbelo
> > Twitter: @rafaelrpbelo
> > Linkedin: rafaelrpbelo
> >
> > I need to work in rails console and when I run `visit ...`  rails complains of no matching route
> >
> > --
> > You received this message because you are subscribed to a topic in the Google Groups "Ruby on Rails: Talk" group.
> > To unsubscribe from this topic, visit https://groups.google.com/d/topic/rubyonrails-talk/CpOPHz-zFsc/unsubscribe.
> > To unsubscribe from this group and all its topics, send an email to rubyonrails-ta...@googlegroups.com.
> > To post to this group, send email to rubyonra...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/2ae5f27c-eb96-4de2-bc2d-0974387b21c7%40googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
> >  
> >
> > --
> > Rafael Belo
> > Web Developer
> > Skype: rafaelrpbelo
> > Twitter: @rafaelrpbelo
> > Linkedin: rafaelrpbelo
> >  
> > require 'capybara/rails'
> > include Capybara::DSL
> >
> > No change I still get the same routing error  ActionController::RoutingError (No route matches [GET] "/getI2iRecommendingResults.do"):
> >
> >
>
> You'll probably get better answers if you show your work. Try writing a single script that demonstrates what you want to do, and post it as a Gist. Link it here, show what the output looks like, and see where that leads you. Often times, working in the constraints of making the example work in a single script forces you to reconsider the problem, or shows you a simple error you made while configuring something more complex.
>
> Walter
>
>
> I'd like to write a script after I get my commands down, for now I'm working in rails console It may not be the intended use of capybara to visit url's not defined in routes.rb Should I be using something else to make the request

Try using an API tool, like Faraday.

gem 'faraday'
require 'faraday'
response = Faraday.get('https://entire.url.of/your/api/data.json')

whatever_parsing_tool_you_want.parse(response.body)

Walter

>
> --
> You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-ta...@googlegroups.com.
> To post to this group, send email to rubyonra...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/ba70d205-f364-4c1e-ba25-b8edd4650c0f%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

I used Nokogiri::HTML.parse(response.body) It isn't converting the javascript response to something more friendly 

--
You received this message because you are subscribed to a topic in the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rubyonrails-talk/CpOPHz-zFsc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rubyonrails-ta...@googlegroups.com.
To post to this group, send email to rubyonra...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/96003e5a-48e6-41f8-922b-d67223602b27%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Rafael Belo

Web Developer

Skype: rafaelrpbelo

Twitter: @rafaelrpbelo

Linkedin: rafaelrpbelo