Light is refracted as it passes through the aperture, and if you study it you will find interference fringes. I suspect that is the "lint look" you are perceiving. And yes, the target spot will appear to move if your line of sight is not dead-nuts on the center of the aperture. The smaller the aperture, the worse it gets.
Best way to see this clearly is to shine a laser through a fine slit, say .050 wide, letting the beam fall on a piece of white paper. The fringes will be very distinct, because the laser light is monochromatic. Sunlight is a wide range of wavelengths, and each gets refracted through a slightly different angle, so you just see fuzz. This is one of the classical proofs that light is composed of waves.
http://labman.phys.utk.edu/phys222co...iffraction.htm
Other experiments prove that light is particles. If that makes your head hurt, take comfort. It does the same to every physicist who tries to resolve the paradox. Engineers ignore it, and use whichever model works best for the task at hand.