Delicious information for the vulpine.club community
Starting at 2018-11-08 16:55:00 -0500, a production change was made to use CloudFront to serve the web UI front end code (particularly, the stuff under /packs). Unfortunately, the necessary magic to make cross-origin resource sharing (CORS) work was not configured on CloudFront. This resulted in members seeing a plain grey screen instead of the familiar web UI.
By 21:00, the proper configuration was in place, and most vulpine.club members should have been able to load the site properly.
At 21:40, a fresh build was deployed to invalidate references to possibly-broken cached files, and everything should be back to normal now.
tl;dr: I forgot to tell CloudFront to respect the Origin header and handle the OPTIONS method, and that made random parts of the web UI glitch out until it totally broke after a few hours. I hate computers.
16:55 - deployed prod-20181108-01 (6bdfc5f759ab)
17:00 - reverted to prod-20181105-01 (8ed6832fba82); URL was mis-specified in the configuration. doing a rebuild with a hotfix’d Dockerfile.
17:25 - re-deploying prod-20181108-01 (a247d59d6d7d)
17:30 - okay this looks good. resolving!
19:20 - observed that the placeholder avatar/header were not loading on public pages, because /avatars and /headers were not set up on the CloudFront side. fixed about ten minutes later.
19:30 - re-opening outage because of reports of multiple folks getting greyscreen on the web UI
19:38 - why i didn’t notice everything being broken: <rey> oh the fucking “What’s New” tab was hiding the console in chrome
19:40 - determined that the problem was CORS
19:55 - brief full-system outage (all containers) because of unclear documentation on a Docker command-line argument… oops
20:15 - rollback build started… the image without CDN_HOST had been expired so it had to be rebuilt. (narrator voice: she never actually remembered to deploy this image.)
20:30 - workaround in place to force things to work by injecting Options: urmom at the front-end proxy; cache invalidations begin
21:00 - CloudFront reconfigured to properly handle OPTIONS requests, the Origin header, and the __uncache invalidation parameter. props to @bea@glitch.social for the tips.
21:10 - i totally forgot to actually do the rollback to 807027078c54
<rey> wait did i not deploy the rollback build<rey> oh my god why am i doing this while high<tastymochafox> cuz who isnt high in 2018<rey> a point of valid21:40 - deployed a rebuilt prod-20181108-02 image (b6de4a35a2d2) to force an invalidation