Okay, I *thought* it wasn’t a problem (at the time), but it turns out, it just made it “less” of a problem.
After watching the server closely for a month or two I think I finally figured out what the hell is going on. mysqld is not really efficient with it’s memory (I finally figured it out with vmmap). To the point that I start getting 10,000+ different memory segments for mysqld after about 24 hours of use. On a server that has many gigs of unused/free memory (with MySQL only using about 200MB), that’s pretty sucky memory management if you ask me.
That explains why it gets slower and slower over time and also explains why pretty much removing the query cache (memory) made it less of a problem.
I opened a bug report with MySQL about it, so *hopefully* we can finally get it taken care of and I don’t have to restart the mysqld daemon every 24-48 hours.