This resolves that completely nonsensical memory leak situation. As far
as we can understand, the cause was a hodgepodge of the following:
- There is some buffer sharing going on deep in pgx
- Queries made with a cancellable but long-running context (like that
used for background jobs) would leave iterator-related goroutines
hanging
- These goroutines had a pgx `rows` object in their closures, preventing
the row stuff from being garbage collected
- If you look at a profile, it all appears to be caused by whatever
functions were doing the most database queries / reading the most from
Postgres. In fact those things were _allocating_ the most but not
retaining any of that data - it was being retained by these other
goroutines because of magic buffer sharing huzzah I love it
We could have solved this in approximately 30 minutes if Go could
actually tell us what is keeping things alive in the heap, instead of
just tracking allocations.
This includes the ability to "shadowban" new users who have not yet been
approved. We do not have UI for approving these users.
Migrate deserving users to new Approved status
Add post fetching helpers as well
The logic in the thread/post stuff is definitely getting redundant, but
I'm not sure I'm yet ready to try to abstract any of it away.
The next thing to do is probably to update blogs and other places that
fetch threads/posts, and delete the old helpers.
Move forums and blogs fully to new helpers
Use the helpers on the landing page too
that was easy!
Fix up some spots I missed
Check user status and use helpers on the profile page
We don't need to do this any more, since our database backups now
include the migration table. The seedfile command also no longer
requires the user to type a password.
This also prevents the Discord history bot from running when it has no
credentials.
Ben Visness
Asaf Gartner