#1058. Nyquist, MTF and the Impossible Hope of Pixel-Shifting

By pascaljappy | Opinion

Nov 09

It doesn’t work. Let’s be honest. Pixel shift is a brilliant, brilliant idea in a pantheon of brilliant, brilliant technical ideas. But it doesn’t work. And here’s why.

Aloof
 

As all clickbait titles, this one is a partial lie ๐Ÿ˜‰ Do you remember when Hasselblad showed their first pixel-shifting results using their large 100Mp sensor? Not only did it work, but it blew my mind.

Even as a non-repentent luddite, I absolutely adore the idea of pixel-shifting. Most of the time (99% of the time) there really is no need for very high resolutions in photography. They are a complete waste of time, natural resources and mental focus. But, on those rare occasions when the scene and intended photographic use warrant the big numbers, wouldn’t it be brilliant to be able to pull out the big gun and grab that perfect 200Mp photograph (think Candida Hรถfer, for instance)?

What a shame it doesn’t work, present Hassies excepted, then ๐Ÿ˜‰

Missed the mark
 

Now, before we jump into why it doesn’t work, you might challenge this claim. Particularly as my main camera doesn’t even have pixel shift, making my hands-on experience of it about as meaningful as western leaders experience at curbing a pandemic.

But I’ve tried it by proxy. Philippe did quite thorough testing with the Sony A7r4 and it really took some concentration to find differences between files made with or without pixel-shift. In a blind test, I would never had found the shifted files. And yes, he was using a tripod, a timer and a good lens.

And I’ve looked at many other files made with other pixel shift cameras, with equal lack of impression. Which doesn’t mean pixel-shift can’t be made to work. Only that, in real life conditions, it’s really difficult. Here’s why.

Shake, baby, shake
 

There are the obvious reasons everyone knows about :

  • No tripod, or a bad tripod, implies shake, so a large file of a blurry image.
  • A moving subject sees itself replicated 4 or more times at different places in the final file.
  • Bad focus means, again, a high-res file of a blurry photograph.

But it goes further. A moving referential (such as the subtly trembling ground of all large cities with cars, buses and underground trains) will also create more blur. And, of course, wind will deteriorate quality. As will a clunky shutter. And any slight loosening of anything in the imaging chain.

So, pixel shift can work, and does work, in lab conditions and careful post processing. Anything approaching real-life conditions very often stretches the very tight tolerances for success a step too far. But’s that’s still not why pixel-shift doesn’t work. Lenses just aren’t sharp enough!

Such a blur
 

Another way to put this is that pixels are too small. Here’s where Harry Nyquist and Claude Shannon come in handy to explain. And let me call Zeiss to illustrate when and how it can work.

Below is the MTF curve for the Zeiss Otus 100, possibly the closest to ideal lens Zeiss have ever made. The lines show the contrast with which finer and finer details are transmitted to the sensor by the lens. Just a reminder : Zeiss (and a couple of other manufacturers) measure these MTFs. Most other companies calculate them theoretically, as a theoretical lens with ideal tolerances and surfaces would produce.

Theoretically, a politician elected by the people and paid by the people’s taxes should have the people’s best interest at heart. In real life, just look around. Theoretically, if we all wore masks, Covid would be extinct in 3 weeks. In real life, it’s been around for more than a year and most predictions now stretch to 2023. In theory, trains run on time. In real life, I’ve seen people cry on the platform “no, not another cancelled train, I cannot lose another job”. Real life measurement have nothing to do with theoretical projections. Hence the selection of Zeiss measurement (for what is possibly their best measuring lens ever).

Zeiss Otus 100 MTF curve (c) Zeiss

Consider this to be the best performance money can buy. If there is a better measuring lens out there, it won’t be better by much (and please don’t point me to measurements done in a garage, a proper MTF measurement rig costs multiple 6 figures).

The bottom lines show us roughly 80% contrast transfer at 40 line pairs per second. At f/4. You can roughly extrapolate this by doubling the contrast loss for every doubling of the resolution. So possibly 60% contrast transfer at 80 lp/mm and 20% at 160 lp/mm. Those 20% contrast lines (starting with pure black and pure white lines in the chart) would be pretty close to the threshold of detail visibility for real life details. So, in a world without diffraction (see discussion about aperture, below), the Otus 100 would max out at about 160 lp/mm.

That’s roughly 3 micron pixels. At the best aperture on (possibly) the world’s best aperture.

Light is a wave
 

Enter sampling and the Nyquist-Shannon theorem. Simplified, the theorem states that the maximum frequency you can sample without aliasing with a sampling frequency of f is f/2. In sensor terms, this means you need 2 pixels (along each axis) to sample a detail the size of a pixel. So, to make use of the Otus 100’s 160lp/mm max resolution (3 micron), you need pixels half as large in each direction. So, 1.5 micron equivalent pixels. Considering most pixels fall in the 3-6 micron range, there’s a lot of room here to double the resolution and make the most of pixel shifting, in theory.

If you have the shooting ability to exploit pixels that small (no vibration at all, eg) then you see that, in theory, you can make pixel shift work with the Otus 100 even starting with the very small pixels of today’s sensors.

The optical illusion
 

You also see how excellent the lens has to be using current small-pixel sensors, and how perfect your technique has to be to maintain 1 micron-ish accuracy (vibrations …) throughout the multiple exposures, to get the best out it.

With less exceptional lenses, the real-life MTFs might max out at 80lp/mm. Meaning you can get away with lesser technique, but also that small pixels will start oversampling the details.

And that’s where the problem is. What we are seeing more and more are ultra high density sensors with tiny pixels and lenses that are nowhere near as good as that Otus, whatever manufacturers and fan clubs may claim. Using my 120 Macro on the large pixel X1D camera, files look every bit as sharp at 100% as they do at 20%. With other lenses, particularly older adapted lenses, 100% is nowhere near as percepetually sharp as the global view of the photograph.

The global view
 

In fact, that’s quite a good test to see for yourself whether a lens will be suitable for pixel-shifting. Look at a photograph made with that lens in your PP software. Then switch to 100%. If you notice a drop in sharpness, forget it. How could finer sampling of what the lens projects possibly be sharp if it isn’t perfectly sharp at the normal sampling?

And then, there’s aperture …

In theory again, the wider the aperture of a lens, the sharper in can be. In real life, aberrations always make full aperture significantly less sharp than closed down a few stops. But, let’s stay in the realm of theory for a while.

Interference and …
 

At f/2, a perfect lens has an airy disk of 2.6 microns. At f/8, a perfect lens has an airy disk of over 10 microns. And at f/22, a perfect lens has an airy disk of almost 30 microns.

The airy disk is the disk of light produced by a lens imaging a pinpoint light source (infinitely small, such as a star). The disk is surrounded by rings of descreasing intensity. Using the size of the airy disk at the smallest possible detail the lens can record on a sensor is a wildly optimistic hypothesis but let’s go with it.

You can see that at f/8, that smallest detail is already 10 microns wide. More realistically 15-20 microns. You can sample that as much as you want with as much pixel-shifting as technology will allow, you’re just oversampling a coarse detail (unless you statred with really mahussive 30 micron pixels ๐Ÿ˜‰ ๐Ÿ˜‰ ) So, if that pixel-shifted master-shot is a landscape and you were thinking of using a small aperture to maximise depth of field, well … ain’t happening.

… diffraction
 

So, there you have it: Nyquist was born in Sweden, which is why pixel shift only works on Swedish cameras such as the Hasselblads ! ๐Ÿ˜‰

More seriously, that Hasselblad shocker was made in a studio using a sensor with large pixels, a lens probably as good as that Otus and deliberate post processing. It can work, we’ve seen it happen. But it don’t work easy and it don’t work often.

As Jim Kasson has shown far more technically than me, pixel-shift has other benefits, mostly a reduction of aliasing in 16-frame executions. In very specific conditions, for very specific uses, it is a very useful technique. But for increasing resolution of landscapes or other spectacular scenes, forget it.

Cape Vignette
 

Sidenote: a related technique is used in astrophotography. It was invented by Nasa decades ago to compensate for the unsharp images of a Hubble camera. When the camera undersamples (ie has pixels too big to make use of all the information provided by the lens) the image projected by the lens or telescope (which is not the case with small pixels and average lenses), dozens or hundreds of individual photographs can be dithered (shifted and rotated very slightly – by subpixel amounts – in relation to one another), and stacked onto a layer with 4 times the original resolution (doubled on each axis) and the algorithm infers the missing data, producing a higher resolution final image (albeit with more noise).

The initial premise is that the lens has much higher resolution than the sensor can capture. The opposite of today’s usual operational conditions in the amateur photography world. When you see high-resolution pixel-shifted images from cameras with small pixels, particularly phones, you are in the presence of very clever inference algorithms, not free data ๐Ÿ˜‰

One stitch over, one stitch under
 

Me? I stitch. Works a treat.

 

โ€‹Never miss a post

โ€‹Like what you are reading? Subscribe below and receive all posts in your inbox as they are published. Join the conversation with thousands of other creative photographers.

  • Steve Mallett says:

    Pascal, there was a time when reading an article I didn’t understand, like this one, I’d have gone off and started reading to educate myself. Thankfully those days are over. Now I just look at the pictures (reverting to childhood maybe) and what pictures. Gorgeous.

    As for pixel shift, I have/had a series of Olys with the facility and have never used it. Other than to figure out how it worked and decide it was too much bother. Too many menus, too much to remember, just too much, period. These days getting the exposure and aperture about right and the horizon level is enough to be dealing with.

    And I absolutely love Interference and…

    Steve

    • pascaljappy says:

      Thanks so much, Steve. No, I don’t get the attraction either, 99% of the time. And stitching is so much easier (and creates a nicer look) that I fail to see pixel-shifting as anything more than an expensive marketing exercise. Cheers.

    • jean pierre (pete) guaron says:

      Wholeheartedly agree, Steve. Way over my head, but the photos are interesting.

      Pascal’s comments remind me of a “trial” I ran a while back, on half a dozen different programs promising to do “image sharpening”. When you bumped the magnification, what most of them did to the image was bloody awful. Standing way back from the image, you might have THOUGHT they all worked. But once you realised what they’d done to the image, there was no way you’d choose their version of “image sharpening”.

      And to be honest, most modern sensors & lenses have reached a point where we’re all better off concentrating on improving our technique. While some post processing can help, too much seems to have the opposite effect.

  • Frank Field says:

    Pascal —

    This is the first time I’ve seen a citation of Nyquist’s sampling theorem in regard to photography. Nyquist’s work in sampling along with Claude Shannon’s work in information theory are the basis for all modern communications where sampling is accomplished in the time domain. The same theory applies in the spacial domain and it is good to read your take on its application to photography. There are many, especially marketing departments, who argue in favor of “over-sampling” via ever more dense pixels. You have the story correct – there is no value in sampling beyond the limits of the lens and beyond the requirements of the smallest textural element to be preserved. Nyquist assures us that sampling at twice the rate of these limits insures we can fully reconstruct the image. Period.

    Best to you and let’s all steer well clear of COVID-19.

    Frank

    • pascaljappy says:

      Thank you Frank. I’m glad the angle spoke to you.

      Oversampling massively to then downsample and obtain a sharper image has long been a promise. And that, again, probably works superbly in ideal conditions. In real life, I’ve yet to see a phone compete in IQ with a lower-res camera that has a larger sensor. Data is data. And the whichcraft behind resising in anything but multiples of two is another area where I think a lot of marketing departments have very long wooden noses … ๐Ÿ˜‰

      • NMc says:

        Frank and Pascal
        For Audio recordings the sampling of microphone electrical voltage (analogue of air pressure) at higher frequencies is limited by the technology, basically you do not dilute the air pressure / microphone voltage. In camera sensors you are dividing the quantity of light into smaller โ€˜bucketsโ€™ that are harder to get accurate measurements in lower luminosity, so you have less dynamic range per pixel, perhaps like using a lower quality microphone with more noise in audio.

        So for me, the comparison between spatial and time sampling frequencies is not the apparent correct comparison. My understanding is that Audio equipment is not โ€˜perfectโ€™ at reproduction of the higher frequencies so high sample rates pushes the recording / playback artifacts to a frequency above audibility. It also gives more data for mixing and mastering so that the data losses from processing are not audibly detectable.

        With colour photography the different colour samples are offset and at different frequencies between green (1.4 pixel pitch diagonal or 50%) or the red /blue (2 pixel pitch orthogonal or 25% each). The primary benefit of Pixel shift is for coincident sampling of the colours, eliminating false colour interpolation errors, moirรฉ. The oversampling for the 16 x shifting is a much smaller improvement over the 4 x shift photos.

        Possibly the relatively recent improvements with raw processing, which is now much better at interpolation with less visible artifacts, are as significant as the increase in sensor resolution for modern image quality.

        Regards Noel

        • pascaljappy says:

          Hi Noel,

          I’m not referring to noise and well depth (hence signal/noise ratios), which would determine the accuracy of tone and colour, but to the resolution of the image. If the smallest detail a lens can resolve (with discernable contrast) is (say) 12 microns, then you can sample it well with 6 micron pixels and perfectly with 4 micon pixels (Nyquist-Sannon state a lower limit but there is a bit of improvement at slightly higher sampling, after which it is just waste).

          With pixel shift, what manufacturers are attempting to do is essentially to create smaller pixels. That great when you start off with large pixels and superb optics as with the larger Hasselblad bodies. But it is quite meaningless when using sensors with a tiny pitch and lenses which are not designed for that sort of use. Zeiss themselves state that their 5 grand lenses are suitable for 80Mp resolution. Trying to wring out more than that out of a 300 euros lens is … optimistic ๐Ÿ˜‰

          Cheers, Pascal

  • Claude Hurlbert says:

    Pascal, I cannot say much about pixel shifting as I have never had access to it to try it. All I know is what little I have read on the subject, and most of that underscores your take that as a technique it suffers severe limitations. On the other hand, your images in this post, which are obviously meant to be illustrative, are, as usual, impressive in their own right. The tonal quality of that wing in image 4 stopped me in my tracks. You may call the photo blurred, but I will call it beautifully soft. Finally, I do agree with you that “pixel shift,” while a genuine technology, feels an awful lot like a tool for marketing. And given how the big camera companies operate, who can help but be skeptical. Claude

    PS. I swear, the next time I fly, which will not be for a while, I will keep my camera close and not pack it away in the overhead!

    • pascaljappy says:

      That’s very kind of you, Claude, thank you. I dodged the wing a little bit to give it a measure of glow and am really happy that you noticed that ๐Ÿ™‚

      My wife and I were very lucky to squeeze out of France to visit our children in England between our lockdown and the UK’s (after three attempts, it must be said). Given that we were then quarantined in my son’s house, most of the photographs from this brief trip are actually from the flights ๐Ÿ˜‰ ๐Ÿ˜‰ ๐Ÿ˜‰

      All the best

  • Michael Ulm says:

    A stitch in time saves nine……or maybe itโ€™s 16 if I understand the math. Canโ€™t thank you enough for this article. It is a simple statement of the facts about this process and will become my go to reference when colleagues want to argue the state of this particular art. Thanks again, Michael

    • pascaljappy says:

      Hi Michael,

      it would be a lie to say stitching is totally without its problems ๐Ÿ˜‰ As David explains above, you need to rotate close to the nodal point of the lens otherwise you introduce parallax in the various frames, and that can make it hard for the software to stitch properly. I’m not overly careful of this while panning but do make sure I’m not swinging the camera at arm’s length, for example ๐Ÿ˜‰ ๐Ÿ˜‰

      But stitching is indeed a great way to obtain a higher resolution file if you need it. Use a longer lens and take two photographs side by side (with plenty of overlap and space aroud the edges) and the software will give you a file with roughly 50-60% more resolution. Use 4 frames or more and you can go way beyond that. You can do squares (often the case for me), panos … it’s great.

      Mainly, though, I use stitching because it allows me to get more 3D pop out of my photographs (because my lenses are not very fast and draw a little flat). Used to extremes, this gives you the Brenizer effect but 2 frames are usually enough for my purposes. The higher resolution is just the icing on the cake ๐Ÿ˜‰

      All the best, Pascal

  • JohnW says:

    Year’s ago I met Harold Merklinger, Director General at the Nation Research Council of Canada; a brilliant scientist and an avid photographer. That combination lead to what is probably the definitive treaties on camera “focus” – “About The INs and OUTs of FOCUS”. I tried reading it just to educate myself, but the only thing I ever got was a migraine. Same thing with Hawking’s “A Brief History of Time”. I came to the conclusion that my images are sharp enough for my taste and if the Universe decides to exterminate us … order a second extra tall double shot lactose free latte and enjoy the show.

    This article wasn’t as bad … a mere two Tylenol headache … and life goes on.

    ‘Nuff said.

  • philberphoto says:

    Very interesting, Pascal, but I am afraid I can’t make total sense out of it. Isn’t the Sony pixel shift aimed primarily at eliminating the Bayer matrix+de-bayering? Is this not a bit different from “merely” increasing resolution a la Olympus and Hassy? Either way, eve if it does/did work, the experimental conditions (stable tripod resting on a stable support, shooting at a stable target) are such that it feels like a very painful and limited/limiting way to achieve higher resolution.

    • pascaljappy says:

      I can’t either. Sony do say their pixel shift is there to eliminate Bayer. And that makes sense on paper. Why it isn’t borne out in real life is beyond my intellectual grasp ๐Ÿ˜‰ Cheers.

  • PaulB says:

    Pascal

    Itโ€™s too bad that we are an ocean and a continent apart, as this article has two points that we could use as subjects for a rousing discussion over an adult beverage.

    While I agree that in the real world pixel shift leaves a lot to be desired, I donโ€™t agree with the premise that our lenses are not good enough. In fact, I think the exact opposite, our lenses are better than we give them credit for and more resolution will only benefit them. Provided we can hold the image capture system (camera, lens, etc.) steady enough (as you have stated).

    Considering sensors and Nyquist all we need to do is look at current cameras. A 24 MP (4,000 x 6,000) full frame sensor has a resolution of 83 lp/mm. My Lumix G9 M43 camera uses a 20.3 MP sensor which scaled up to full frame would be 79.3 MP (7,273 x 10,970), or a resolution of 151 lp/mm. Our current lenses can do quite well at these resolutions, so a full frame sensor with a resolution of 160 lp/mm would only be 88.47 MP (7,680 x 11,520), which I doubt would be much of a challenge for our better lenses like an Otis.

    My personal thought about why pixel shift does not work, at least in smaller cameras is, motion. Or more specifically, in small cameras, like the G9 and Olympus Pen F, the IBIS system is too small to really control the sensor motion precisely enough. In my use with pixel shift I have gotten better results using slower shutter speeds than faster speeds. Which to my mind indicates that at fast shutter speed the IBIS systems may not have enough energy to fully counteract the momentum of moving the sensor. So the sensor is slightly out of position when the next exposure is made, and the pixel data does not line up quite right when the final image is put together. At slower shutter speeds the IBIS can reposition the sensor closer to the ideal pixel location giving a slightly better result. Larger cameras with larger IBIS systems should have larger magnets and better control over sensor position, which is probably why the Hasselblad system, and Lumix S cameras, are reported to produce better results than smaller cameras.

    Concerning viewing images at 100% to judge lens quality, I am afraid that this technique no longer works with newer monitors or all in one computers like the iMac and laptops. One reason for this is, as monitors and monitor/graphics software and firmware have improved, the artifacts that our eyes/brains accept as motion are being eliminated. Resulting in a projected image that is more like a static print. So when we enlarge an image to 100% our eyes see it at the native dot pitch of our monitor, which may not be accepted as sharp. During the evolution of digital photo printing the printer industry did a study to determine the minimum dot pitch a person with average vision would accept as โ€œsharpโ€, and the result was 300 dots per inch (dpi). The unfortunate result of the above is, in order to judge image sharpness we need to reduce our level of enlargement so the composite image is presented at the equivalent of 300 dpi on our monitor. For example, using my Mid-2012 generation iMac and its 110 dpi screen, I start to lose โ€œimage sharpnessโ€ after 33% enlargement, and on my new 16 inch PowerBook with its 220 dpi screen after 67% enlargement.

    A second reason is simply the high pixel counts contained in our sensors. With a 40+ MP sensor. 100% enlargement is looking at a very small area of the sensor compared to the area of the monitor. There is simply not enough detail to show. Which is the same problem we had if you tried to make enlargements using Kodak Ektar 25 film; even at 500x enlargement (20×30, with a 25x grain magnifier) there was no discernible grain.

    PS. When we get an opportunity to meet, I owe you a drink. ๐Ÿ˜‰

    PaulB

  • pascaljappy says:

    Ah, Paul, I would love that !! Where about are you ?

    You are right, the ability to make pixel shift work is linked to multiple factors. Stability, focus precision, lens qualitรฉ, vibrations, algorithms … we’re not there yet. It’s a bit sad that Sony haven’t implemented it on their A7S3. With a native resolution of 12Mp and huge pixels, that would have been the ideal candidate and such a versatile camera. I so don’t get Sony … why add pixel shift to a camera that already has 60Mp and not to one which is far better suited to it ?? It baffles me. Anyway … to that drink ๐Ÿ˜‰ Cheers

  • PaulB says:

    Pascal

    I am outside Seattle, in the US Pacific Northwest. Hopefully 2021 will provide for easier travel.

    Sony probably didโ€™t add pixel shift to the A7III Because they have video on the brain and did not think that far out of the box. A 16 image shift could make that sensor sing, at least for static shots.

    PaulB

  • >