Delicious information for the vulpine.club community
This page knows of no reason why everything shouldn’t be working fine.
Page last rendered: 2020-03-12 14:03:49 +0000
Uploading media assets was failing (and receiving posts with media attachments was delayed in general) due to an upstream issue. This should be resolved but we are continuing to monitor. -rey
A host migration is in progress, to move us to a less-congested physical server. We’ll be back soon. (Update… we’re back)
High CPU Steal% is causing some performance issues. A ticket is open with our VPS provider to set up a migration to a less-congested host.
A lengthy database migration is in progress, as part of an upgrade to prod-20190903-01.
14:15: still deploying… there are new and exciting database indices undergoing creation at this time, causing some transient performance/availability issues
14:21: Load shedding in progress… I’ve turned off Sidekiq (background processing) and Streaming (the thing which feeds new posts directly into your eyeballs)
14:45: We’re back! Monitoring…
High CPU Steal% is leading to performance issues.
13:40: Scaled sidekiqs from 2 to 1 to reduce CPU pressure on web/streaming
13:45: Linode tkt 12012372 open
15:40: We’ve been moved to another host (live migration!!) and everything should be better.
upgrade to prod-20190115-01
during work to improve system reliability and monitoring, everything was broken for a short period of time.
09:35: web services (API, streaming) offline
10:00: web services back online, background processing (Sidekiq) still offline
10:45: background processing back online, handling backlog
11:00: everything back to real time!
deploying prod-20181229-01. this was about a ~10 minute outage for the reboot.
19:05: deployed prod-20181224-01. this broke docker and/or the kernel again. sigh. reboot is complete as of 19:20.
19:25: ugh, tried something to fix the previous problem, f’d the entire stack again. rebooting.
19:45: okay i think everything is okay for now… had to fix up a couple loose ends, but we’re… up! ish! kinda!!
still dunno what’s going on :( … i have upgraded to a very latest kernel, tho, let’s see if this fixes something.
22:50: deployed prod-20181212-01, which has a possible fix for bridge.joinmastodon.org not working w/ vulpine.club
23:05: that thing docker was doing last month is happening again, o no. system is back up and working good, will investigate further at some later time :)