I have seen quality like this, as well as better quality when more light is available.
Really you are dealing with the Low Light/Low Gamma (Gamut?) qualities of the Lens and Image Sensor.
Switch from the Red-Dot zoom, to the other, and try the same shot when it is nice and sunny, and you will see the difference.
However, the PS3Eye is only a webcam, and Image enhancement (like focus cleanups etc.) can only be done in software.
The Image will cleanup when stronger light is available. But only so much.
Another Option is to use a Better Lens (bigger/faster) in front of the Image sensor, and get the focus/aperture setup correct.
-sean