Tuesday, February 3, 2015
Hi all,
-- We use ruby 2.0.0 and rails 3.2.19 in production, with OOBGC. We are having problems understanding why our unicorn workers grow from around 400MB to 8-900MB in a number of hours after a deploy. We use New Relic and memory_tracker, but
We see that every so often, the memory_tracker log outputs lines with *** such as:
02-03 06:34:38 pid:17885 rss=320.28 vsize=649.02 *** {:total_allocated_object=>169444, :count=>0, :rss=>12.775423999999987, :heap_used=>2597, :in_use=>866139} /iphone_data/get_article
We aren't quite sure when memory_tracker chooses to print these lines, but they seem important. I think we can define them as showing the more problematic actions, that grow the heap the most. Here's an example output of processing the log, showing the number of times a given action triggers these *** lines over a day:
$ zgrep total_allocated_object log/memtracker_urls.log-20150203.gz|awk '{print $12}'|sed -r 's/[0-9]+(-[^/]+)?/:id/g'|sort|uniq -c|sort -n|tail -n20
20 /webapp/article/:id
22 /iphone_data/notification
24 /users/ajax_get_user_details_card
26 /articles/get_ajax_alternatives_by_symbol
28 /account/portfolio/holdings
31 /iphone_data/sync_portfolios
32 /account/ajax_authors_portfolio
34 /iphone_data/get_bookmarks
35 /news/:id
46 /user_comments_tracking/ajax_create_notification_cookies
47 /memcached:id/article_bottom/:id
57 /etf_hub
63 /account/portfolio/day_watch
96 /account/portfolio/news
106 /account/no_cookie_key
110 /api/v:id/notification_inbox/get_last_notifications
147 /article/:id
171 /instablog/:id/:id
419 /iphone_data/get_data
2519 /iphone_data/get_article
20 /webapp/article/:id
22 /iphone_data/notification
24 /users/ajax_get_user_details_
26 /articles/get_ajax_
28 /account/portfolio/holdings
31 /iphone_data/sync_portfolios
32 /account/ajax_authors_
34 /iphone_data/get_bookmarks
35 /news/:id
46 /user_comments_tracking/ajax_
47 /memcached:id/article_bottom/:
57 /etf_hub
63 /account/portfolio/day_watch
96 /account/portfolio/news
106 /account/no_cookie_key
110 /api/v:id/notification_inbox/
147 /article/:id
171 /instablog/:id/:id
419 /iphone_data/get_data
2519 /iphone_data/get_article
It is worth pointing out that in our sample of problematic (three-star) actions, the ratio (number of times action grows heap and prints *** to memory_tracker / number of times action called) is not much different. The actions get_article and get_data simply get called far more than other actions.
Are there any practical solutions to better-understanding what makes these actions grow the heap so much, apart from staring at the code? We feel we lack data / visibility in order to make improvements. Please let me know if this is too vague, and if more details can help provide better responses from the list.
Antony
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-talk+unsubscribe@googlegroups.com.
To post to this group, send email to rubyonrails-talk@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/b874f61e-f706-472c-a164-396cbce56724%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment