Image Processing

High-quality machine perception requires understanding image formation, anticipating imperfections, and applying suitable processing techniques.

1. Image Formation

• Geometric Primitives: Representation of points, lines, surfaces in 2D/3D.
• 2D Transformations: Translation, rotation, scaling, affine, projective.
• Projection Models: Orthographic, scaled orthographic, perspective projection.
• Camera Model: Extrinsic (rotation, translation) and intrinsic (focal length, image center) parameters; pinhole camera model.
• Lighting: Inverse square law, point vs. area sources, reflection types (diffuse, specular, Phong model).
• Digital Camera Pipeline: Optics → aperture/shutter → sensor → ADC → image processing (demosaic, white balance, compression).
• Factors Affecting Image Quality: Aperture size, shutter speed, sensor characteristics, ADC resolution, sampling and aliasing.

2. Image Imperfections & Artifacts

• Optical: Lens distortion (barrel, pincushion, fisheye), vignetting.
• Electronic: Sensor noise (thermal, quantization), pixel corruption (salt & pepper noise).
• Environmental: Motion blur, shadows, low contrast, incorrect white balance.
• Geometric Issues: Perspective distortion, occlusion.
• Mitigation Methods:
  • Geometric correction for distortion.
  • Median filtering for salt & pepper noise.
  • Histogram equalization for low contrast.
  • Shadow avoidance/removal.
  • White balance adjustment.
  • De-blurring algorithms.

3. Image Processing Techniques

3.1 Point Operators

• Pixel-based operations: Brightness/contrast adjustment, color correction/transformation, gamma correction.
• Histogram Equalization: Redistribute pixel intensities for improved contrast.

3.2 Linear Filtering

• Convolution with kernels (3×3, etc.); border handling strategies (zero-padding, replication, mirroring).
• Common Filters:
  • Moving average (box filter).
  • Gaussian filter (weighted smoothing).
  • Difference of Gaussian (edge detection).
  • Sobel operator (edge detection with noise smoothing).
  • Prewitt operator (simple edge detection).
  • Laplacian operator (second-order edge detection).
  • Wiener filter (deblurring with noise consideration).

3.3 Non-linear Filtering

• Median Filter: Effective for salt & pepper noise; replaces pixel with median of neighborhood.

3.4 Morphological Operations

• Applied to binary images using structuring elements.
• Dilation: Expands object boundaries.
• Erosion: Shrinks object boundaries.
• Applications: Hole filling, boundary extraction, skeletonization, thinning/thickening.

Key Takeaways

• Preprocessing improves data quality for later stages like feature extraction and object recognition.
• Choice of filtering or transformation depends on noise type, image content, and desired output.

---

Extra information on Sampling & Aliasing in Image Sensors

Core Idea

When light hits a camera’s image sensor, each pixel’s active sensing area collects photons, turns them into an electrical signal, and then digitizes it.

But — if:

• The fill factor (percentage of each pixel area that’s actually light-sensitive) is small, and
• The incoming light signal contains detail that’s too fine (too high in spatial frequency) for the sampling grid to capture properly,

you get aliasing — visual artifacts where fine detail is misrepresented as false patterns.

A Simple 1D Example

They use sine waves to illustrate:

• Imagine two sine waves:
  • One at frequency f = 3/4 (relative to some unit)
  • One at frequency f = 5/4
• If you sample them at f_s = 2 (two samples per unit), both produce identical samples — you can’t tell which one you had originally.

This is aliasing: high-frequency patterns get “folded” into lower frequencies, creating false structures in the sampled data.

Nyquist Theorem

• Shannon’s Sampling Theorem: To perfectly reconstruct a signal, the sampling rate must be at least twice the highest frequency in the signal.
• Nyquist frequency: half the sampling rate; the highest frequency you can represent without aliasing.
• Even with 100% fill factor, frequencies above the Nyquist limit still alias — although averaging over the pixel area (finite fill factor) attenuates them.

Why Aliasing Is Bad

• You lose information — the reconstruction can’t tell what the original high-frequency content was.
• Visually: moiré patterns, jagged edges, false colors.
• Downsampling with poor filters (e.g., box filter) makes aliasing worse, high-frequency details are undersampled and show up as wrong patterns.

Predicting & Measuring Aliasing

Aliasing potential is estimated using the Point Spread Function (PSF):

• PSF = combined blur from:
  • Lens optics
  • Pixel’s finite active area
  • Any anti-aliasing filter
    → Think of it as “what a perfect point of light looks like to the sensor.”
• Modulation Transfer Function (MTF) = Fourier transform of PSF → tells you how much each spatial frequency is preserved.
• Aliasing risk: area of the MTF curve beyond the Nyquist frequency.

Example:

• Slight defocus increases PSF size, which reduces high frequencies — lowering aliasing, but also reducing sharp detail.

Measuring PSF in Practice

• Lab test: photograph a tiny bright point (e.g., pinhole lit from behind) to get PSF.
• Limitation: pixel-resolution accuracy, misses sub-pixel shape.
• More advanced: use a slanted-edge pattern and resynthesize to sub-pixel precision for more accurate PSF and aliasing predictions.

Aliasing Beyond Acquisition

Aliasing isn’t just a camera problem:

• Any resampling step — scaling up or down, rotating, warping, can reintroduce aliasing.
• Solution: use proper low-pass (anti-aliasing) filters before resampling.

Key Takeaways

• Aliasing = high frequencies masquerading as lower ones.
• Nyquist limit = half the sampling rate, don’t let content exceed this without filtering.
• Fill factor reduces but doesn’t eliminate aliasing.
• PSF/MTF analysis predicts how much aliasing a system will produce.
• Anti-aliasing filters (optical or digital) are crucial before sampling or resampling.