The usual issue here, apart from insufficient grounding, is that you're most likely getting a ground loop across the USB cable. You can usually tell if you measure current between the USB ground on your electronics to the USB ground on your PC.
This most likely means that the DC -VE on your PSU is wired directly to the AC Earth pin (in your electronics, and in your PC), which is probably not the best idea. It seems like a great idea from a protection point of view (so you don't get a shock) but it leads to an additional current path for your circuits, especially if the ground voltages differ by a small amount. This will cause current to flow across the USB cable ground.
Instead of wiring the DC -VE and AC Earth together, connect them via a reasonable valued resistor. This provides reasonable ground for AC, but blocks enough so that only a very small amount of DC will form a ground loop, if at all.
This most likely means that the DC -VE on your PSU is wired directly to the AC Earth pin (in your electronics, and in your PC), which is probably not the best idea. It seems like a great idea from a protection point of view (so you don't get a shock) but it leads to an additional current path for your circuits, especially if the ground voltages differ by a small amount. This will cause current to flow across the USB cable ground.
Instead of wiring the DC -VE and AC Earth together, connect them via a reasonable valued resistor. This provides reasonable ground for AC, but blocks enough so that only a very small amount of DC will form a ground loop, if at all.