Mon 29 Jan 2018 6:29PM
Spikes of packages sent, results in inconsistent behaviour
Hugi Ásgeirsson
Public
Seen by 348
I'm running into some pretty strange problems. Once in a while, users start running into problems with threads not loading fully. When this happens, it's related to a spike in CPU and network traffic some period before. Restarting with "docker-compose down" and booting back up doesn't fix the problem, but it seems to resolve itself in a couple of hours. It's almost like the server has to "catch up" processesing some large amount of requests.
I'm uploading a screenshot of the spike of packages received. A similar spike appears in Packages sent and CPU usage. After this, certain long threads don't load fully for a couple of hours. Loading stops at the same comment regardless of weather one is logged in or not. This problem resolves fully in about two hours, and is now back to normal.
Here are the errors that appear during the period of the spike:
nginx_1 | nginx.1 | 2018/01/29 17:05:45 [error] 131#131: *150190 upstream prematurely closed connection while reading response header from upstream, client: [CLIENT IP], server: faye.talk.theborderland.se, request: "POST /faye HTTP/2.0", upstream: "http://172.17.0.3:3000/faye", host: "faye.talk.theborderland.se", referrer: "https://talk.theborderland.se/d/9EQT3dKL/fund-51-to-be-or-not-to-be-"
nginx_1 | nginx.1 | 2018/01/29 17:05:56 [error] 130#130: *150708 connect() failed (113: No route to host) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "GET /ahoy/messages/iRaevTBznbU2TNncyR7B9yGvIAPgo2vm/click?signature=8003a284649ed9ae5f4cab7467572e7a046807ec&url=https%3A%2F%2Ftalk.theborderland.se%2Fd%2FWUonBeaV%2Fboard-meeting-time-decision-thread%3Futm_campaign%3Dnew_comment%26utm_medium%3Demail%26utm_source%3Dthread_mailer HTTP/2.0", upstream: "http://172.17.0.9:3000/ahoy/messages/iRaevTBznbU2TNncyR7B9yGvIAPgo2vm/click?signature=8003a284649ed9ae5f4cab7467572e7a046807ec&url=https%3A%2F%2Ftalk.theborderland.se%2Fd%2FWUonBeaV%2Fboard-meeting-time-decision-thread%3Futm_campaign%3Dnew_comment%26utm_medium%3Demail%26utm_source%3Dthread_mailer", host: "talk.theborderland.se"
nginx_1 | nginx.1 | 2018/01/29 17:05:56 [error] 130#130: *150708 connect() failed (111: Connection refused) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "GET /ahoy/messages/iRaevTBznbU2TNncyR7B9yGvIAPgo2vm/click?signature=8003a284649ed9ae5f4cab7467572e7a046807ec&url=https%3A%2F%2Ftalk.theborderland.se%2Fd%2FWUonBeaV%2Fboard-meeting-time-decision-thread%3Futm_campaign%3Dnew_comment%26utm_medium%3Demail%26utm_source%3Dthread_mailer HTTP/2.0", upstream: "http://[SERVER IP]:3000/ahoy/messages/iRaevTBznbU2TNncyR7B9yGvIAPgo2vm/click?signature=8003a284649ed9ae5f4cab7467572e7a046807ec&url=https%3A%2F%2Ftalk.theborderland.se%2Fd%2FWUonBeaV%2Fboard-meeting-time-decision-thread%3Futm_campaign%3Dnew_comment%26utm_medium%3Demail%26utm_source%3Dthread_mailer", host: "talk.theborderland.se"
loomio_1 | ActionController::RoutingError (No route matches [GET] "/img/gradient.png"):
nginx_1 | nginx.1 | 2018/01/29 17:06:16 [error] 130#130: *150708 connect() failed (111: Connection refused) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "GET /api/v1/events?from=13&per=10&order=position&discussion_id=34&parent_id=196 HTTP/2.0", upstream: "http://[SERVER IP]:3000/api/v1/events?from=13&per=10&order=position&discussion_id=34&parent_id=196", host: "talk.theborderland.se", referrer: "https://talk.theborderland.se/d/WUonBeaV/board-meeting-time-decision-thread?utm_campaign=new_comment&utm_medium=email&utm_source=thread_mailer"
nginx_1 | nginx.1 | 2018/01/29 17:06:19 [error] 130#130: *150708 connect() failed (113: No route to host) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "GET /api/v1/events?from=13&per=10&order=position&discussion_id=34&parent_id=196 HTTP/2.0", upstream: "http://172.17.0.9:3000/api/v1/events?from=13&per=10&order=position&discussion_id=34&parent_id=196", host: "talk.theborderland.se", referrer: "https://talk.theborderland.se/d/WUonBeaV/board-meeting-time-decision-thread?utm_campaign=new_comment&utm_medium=email&utm_source=thread_mailer"
nginx_1 | nginx.1 | 2018/01/29 17:06:32 [error] 131#131: *150190 upstream prematurely closed connection while reading response header from upstream, client: [CLIENT IP], server: faye.talk.theborderland.se, request: "POST /faye HTTP/2.0", upstream: "http://172.17.0.3:3000/faye", host: "faye.talk.theborderland.se", referrer: "https://talk.theborderland.se/d/9EQT3dKL/fund-51-to-be-or-not-to-be-"
nginx_1 | nginx.1 | 2018/01/29 17:06:40 [error] 130#130: *150708 connect() failed (111: Connection refused) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "GET /api/v1/documents?poll_id=19 HTTP/2.0", upstream: "http://[SERVER IP]:3000/api/v1/documents?poll_id=19", host: "talk.theborderland.se", referrer: "https://talk.theborderland.se/p/9HkAtigf/lets-meet-to-follow-up-on-the-retreat-weekend-and-the-ap-s-it-spawned?utm_campaign=new_comment&utm_medium=email&utm_source=thread_mailer"
nginx_1 | nginx.1 | 2018/01/29 17:06:43 [error] 130#130: *150708 connect() failed (113: No route to host) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "GET /api/v1/polls/9HkAtigf? HTTP/2.0", upstream: "http://172.17.0.9:3000/api/v1/polls/9HkAtigf?", host: "talk.theborderland.se", referrer: "https://talk.theborderland.se/p/9HkAtigf/lets-meet-to-follow-up-on-the-retreat-weekend-and-the-ap-s-it-spawned?utm_campaign=new_comment&utm_medium=email&utm_source=thread_mailer"
nginx_1 | nginx.1 | 2018/01/29 17:07:05 [error] 131#131: *150774 connect() failed (113: No route to host) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "GET /dashboard HTTP/2.0", upstream: "http://172.17.0.9:3000/dashboard", host: "talk.theborderland.se"
nginx_1 | nginx.1 | 2018/01/29 17:07:05 [error] 131#131: *150774 connect() failed (111: Connection refused) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "GET /dashboard HTTP/2.0", upstream: "http://[SERVER IP]:3000/dashboard", host: "talk.theborderland.se"
db_1 | ERROR: duplicate key value violates unique constraint "visits_pkey"
loomio_1 | ActionController::RoutingError (No route matches [GET] "/img/gradient.png"):
nginx_1 | nginx.1 | 2018/01/29 17:07:18 [error] 131#131: *150774 connect() failed (111: Connection refused) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "GET /api/v1/documents?comment_id=837 HTTP/2.0", upstream: "http://[SERVER IP]:3000/api/v1/documents?comment_id=837", host: "talk.theborderland.se", referrer: "https://talk.theborderland.se/d/9EQT3dKL/fund-51-to-be-or-not-to-be-"
nginx_1 | nginx.1 | 2018/01/29 17:07:19 [error] 131#131: *150190 upstream prematurely closed connection while reading response header from upstream, client: [CLIENT IP], server: faye.talk.theborderland.se, request: "POST /faye HTTP/2.0", upstream: "http://172.17.0.3:3000/faye", host: "faye.talk.theborderland.se", referrer: "https://talk.theborderland.se/d/9EQT3dKL/fund-51-to-be-or-not-to-be-"
nginx_1 | nginx.1 | 2018/01/29 17:07:21 [error] 131#131: *150774 connect() failed (113: No route to host) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "GET /api/v1/documents?comment_id=838 HTTP/2.0", upstream: "http://172.17.0.9:3000/api/v1/documents?comment_id=838", host: "talk.theborderland.se", referrer: "https://talk.theborderland.se/d/9EQT3dKL/fund-51-to-be-or-not-to-be-"
nginx_1 | nginx.1 | 2018/01/29 17:07:24 [error] 130#130: *150708 connect() failed (111: Connection refused) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "GET /api/v1/drafts/poll/19? HTTP/2.0", upstream: "http://[SERVER IP]:3000/api/v1/drafts/poll/19?", host: "talk.theborderland.se", referrer: "https://talk.theborderland.se/p/9HkAtigf/lets-meet-to-follow-up-on-the-retreat-weekend-and-the-ap-s-it-spawned?utm_campaign=new_comment&utm_medium=email&utm_source=thread_mailer"
nginx_1 | nginx.1 | 2018/01/29 17:07:27 [error] 130#130: *150708 connect() failed (113: No route to host) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "GET /api/v1/drafts/poll/19? HTTP/2.0", upstream: "http://172.17.0.9:3000/api/v1/drafts/poll/19?", host: "talk.theborderland.se", referrer: "https://talk.theborderland.se/p/9HkAtigf/lets-meet-to-follow-up-on-the-retreat-weekend-and-the-ap-s-it-spawned?utm_campaign=new_comment&utm_medium=email&utm_source=thread_mailer"
nginx_1 | nginx.1 | 2018/01/29 17:07:29 [error] 131#131: *150774 connect() failed (111: Connection refused) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "GET /api/v1/memberships?group_key=05MAJOH8&per=10 HTTP/2.0", upstream: "http://[SERVER IP]:3000/api/v1/memberships?group_key=05MAJOH8&per=10", host: "talk.theborderland.se", referrer: "https://talk.theborderland.se/d/9EQT3dKL/fund-51-to-be-or-not-to-be-"
db_1 | ERROR: duplicate key value violates unique constraint "visits_pkey"
nginx_1 | nginx.1 | 2018/01/29 17:07:38 [error] 131#131: *150774 connect() failed (113: No route to host) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "GET /api/v1/translations?lang=en HTTP/2.0", upstream: "http://172.17.0.9:3000/api/v1/translations?lang=en", host: "talk.theborderland.se", referrer: "https://talk.theborderland.se/dashboard"
nginx_1 | nginx.1 | 2018/01/29 17:07:38 [error] 131#131: *150774 connect() failed (113: No route to host) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "POST /ahoy/events HTTP/2.0", upstream: "http://172.17.0.9:3000/ahoy/events", host: "talk.theborderland.se", referrer: "https://talk.theborderland.se/dashboard"
db_1 | ERROR: duplicate key value violates unique constraint "visits_pkey"
db_1 | ERROR: duplicate key value violates unique constraint "ahoy_events_pkey"
nginx_1 | nginx.1 | 2018/01/29 17:07:40 [error] 131#131: *150774 connect() failed (111: Connection refused) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "GET /api/v1/reactions?reactable_type=Discussion&reactable_id=61 HTTP/2.0", upstream: "http://[SERVER IP]:3000/api/v1/reactions?reactable_type=Discussion&reactable_id=61", host: "talk.theborderland.se", referrer: "https://talk.theborderland.se/d/9EQT3dKL/fund-51-to-be-or-not-to-be-"
loomio_1 | ActionController::RoutingError (No route matches [GET] "/img/gradient.png"):
nginx_1 | nginx.1 | 2018/01/29 17:07:41 [error] 130#130: *150708 connect() failed (111: Connection refused) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "POST /api/v1/drafts/poll/19 HTTP/2.0", upstream: "http://[SERVER IP]:3000/api/v1/drafts/poll/19", host: "talk.theborderland.se", referrer: "https://talk.theborderland.se/p/9HkAtigf/lets-meet-to-follow-up-on-the-retreat-weekend-and-the-ap-s-it-spawned?utm_campaign=new_comment&utm_medium=email&utm_source=thread_mailer"
nginx_1 | nginx.1 | 2018/01/29 17:07:45 [error] 130#130: *150708 connect() failed (113: No route to host) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "POST /api/v1/drafts/poll/19 HTTP/2.0", upstream: "http://172.17.0.9:3000/api/v1/drafts/poll/19", host: "talk.theborderland.se", referrer: "https://talk.theborderland.se/p/9HkAtigf/lets-meet-to-follow-up-on-the-retreat-weekend-and-the-ap-s-it-spawned?utm_campaign=new_comment&utm_medium=email&utm_source=thread_mailer"
nginx_1 | nginx.1 | 2018/01/29 17:08:00 [error] 131#131: *150774 connect() failed (111: Connection refused) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "GET /api/v1/documents?comment_id=797 HTTP/2.0", upstream: "http://[SERVER IP]:3000/api/v1/documents?comment_id=797", host: "talk.theborderland.se", referrer: "https://talk.theborderland.se/d/9EQT3dKL/fund-51-to-be-or-not-to-be-"
nginx_1 | nginx.1 | 2018/01/29 17:08:03 [error] 131#131: *150774 connect() failed (113: No route to host) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "GET /api/v1/documents?comment_id=793 HTTP/2.0", upstream: "http://172.17.0.9:3000/api/v1/documents?comment_id=793", host: "talk.theborderland.se", referrer: "https://talk.theborderland.se/d/9EQT3dKL/fund-51-to-be-or-not-to-be-"
nginx_1 | nginx.1 | 2018/01/29 17:08:05 [error] 131#131: *150190 upstream prematurely closed connection while reading response header from upstream, client: [CLIENT IP], server: faye.talk.theborderland.se, request: "POST /faye HTTP/2.0", upstream: "http://172.17.0.3:3000/faye", host: "faye.talk.theborderland.se", referrer: "https://talk.theborderland.se/d/9EQT3dKL/fund-51-to-be-or-not-to-be-"
db_1 | ERROR: duplicate key value violates unique constraint "visits_pkey"
loomio_1 | ActionController::RoutingError (No route matches [GET] "/img/gradient.png"):
nginx_1 | nginx.1 | 2018/01/29 17:08:19 [error] 130#130: *150708 connect() failed (111: Connection refused) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "POST /api/v1/drafts/poll/19 HTTP/2.0", upstream: "http://[SERVER IP]:3000/api/v1/drafts/poll/19", host: "talk.theborderland.se", referrer: "https://talk.theborderland.se/p/9HkAtigf/lets-meet-to-follow-up-on-the-retreat-weekend-and-the-ap-s-it-spawned?utm_campaign=new_comment&utm_medium=email&utm_source=thread_mailer"
nginx_1 | nginx.1 | 2018/01/29 17:08:22 [error] 130#130: *150708 connect() failed (113: No route to host) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "POST /api/v1/drafts/poll/19 HTTP/2.0", upstream: "http://172.17.0.9:3000/api/v1/drafts/poll/19", host: "talk.theborderland.se", referrer: "https://talk.theborderland.se/p/9HkAtigf/lets-meet-to-follow-up-on-the-retreat-weekend-and-the-ap-s-it-spawned?utm_campaign=new_comment&utm_medium=email&utm_source=thread_mailer"
nginx_1 | nginx.1 | 2018/01/29 17:08:37 [error] 130#130: *150708 connect() failed (111: Connection refused) while connecting to upstream, client: [CLIENT IP], server: talk.theborderland.se, request: "POST /ahoy/events HTTP/2.0", upstream: "http://[SERVER IP]:3000/ahoy/events", host: "talk.theborderland.se", referrer: "https://talk.theborderland.se/p/9HkAtigf/lets-meet-to-follow-up-on-the-retreat-weekend-and-the-ap-s-it-spawned?utm_campaign=new_comment&utm_medium=email&utm_source=thread_mailer"
nginx_1 | nginx.1 | 2018/01/29 17:08:52 [error] 131#131: *150190 upstream prematurely closed connection while reading response header from upstream, client: [CLIENT IP], server: faye.talk.theborderland.se, request: "POST /faye HTTP/2.0", upstream: "http://172.17.0.3:3000/faye", host: "faye.talk.theborderland.se", referrer: "https://talk.theborderland.se/d/9EQT3dKL/fund-51-to-be-or-not-to-be-"
```
I've had this happen twice now, so something seems to be going on. Is it that missing gradient.png file that is the culprit? Any ideas?
Hugi Ásgeirsson Mon 29 Jan 2018 6:49PM
Aha, good! Thanks. Spike is still pretty weird though, and has resulted in the same problem twice, so might be something to keep in mind...
Robert Guthrie Tue 30 Jan 2018 2:14AM
Hi @hugi we've updated the stable docker image, so please docker-compose pull, down, up and run a database migration. (full commands are in the loomio-deploy readme)
Good luck, keen to hear how you get on!
Hugi Ásgeirsson Tue 30 Jan 2018 2:28PM
Thanks! I very much appreciate your work. I've deployed the update. No problems so far, I'll let you know I run into more issues.
Robert Guthrie Wed 31 Jan 2018 1:08AM
Thanks for letting us know. It's great to hear from people running Loomio and to get some support in the form of this kind of feedback. Really appreciate it as we're a very small team.
Robert Guthrie · Mon 29 Jan 2018 6:41PM
Hi @hugi. Actually, this might be unrelated to the spike. We've had a bug (recently squashed) related to not being able to load the whole thread. We need to update the stable branch with the fix today, so you can docker-compose pull to update. I'll get back to you soon.