While searching for a solution to the elongated stars in my images online, I stumbled on a Cloudy Nights thread where a gentlemen realized that his QHY5 guiding camera had a loose circuit board inside the camera.
I went outside to get my Orion Starshoot Autoguider camera to investigate whether or not I might have the same issue. I realized that indeed, the circuit board inside this type of camera does suffer from some looseness. Mine was probably less worst then the mentioned gentleman (i.e. he had rattle), but moving my finger on the USB connector translated into movement of the circuit board, therefore the entire CCD was not fixed with respect to the camera casing.
I decided to glue the 2 connectors using cyanoachrylate glue and some plastic spacers. Here is the result. I hope to be able to test is soon and see if this was one of the source of flexure on my setup.