So I think (and I hope) I managed to figure out the issues with my system. Here's an annotated side view of my quad:
In green is the antenna wire going up my PTFE tube, blue is the ground part coming back down. Now the end of the ground part always used to be bent like the yellow bit indicates (plus a bit inwards the quad, hard to show in 2D). This always resulted in my Taranis complaining about low or critical RSSI (with the standard thresholds) like there's no tomorrow. Even today. Then eventually for some unrelated reason, I bent it like the purple part indicates, my RSSI when I'm about to take off went from ~65 to ~73 on the OSD scale (whatever moon units that represents, but it seems significant), the Taranis stopped complaining about RSSI, and I didn't notice any more random short bouts of control loss.
My suspicion is that it picked up RF crap from the camera, that interfered with the receiver. The electrics themselves are relatively clean, because there's capacitance and filtering all over the place (altho I still want an oscilloscope eventually to tune this). And it used to happened randomly at hover, not just at load changes that cause lots of EMI.
Knocking on wood that this drama is over.
(Also, why the hell is this site still so slow after days?)