X1D low light red bias

[rent]

I did some night time star shots the other day. What surprised me was the swath of red bias in the resulting images (and some green too). You can view some samples here: https://www.dropbox.com/sh/xl1in097x0w23ne/AABsx7Mmlup200nx-ZCaGt-qa?dl=0. It would be much more evident if you decrease contrast.

This was taken with a Zeiss APO T* Sonnar 2/135 at f/2, 2 second, ISO 3200 with electronic shutter. I have tried other lenses but the large swath of red bias vertically in the center and on the left corners remain.

I'm wondering if this is typical, or if this represents an abnormality in my specific unit. Would anyone kindly try this in a dark room with an exposure setting roughly at -4 EV (f/3.5, 6 sec, ISO 3200)?

Many thanks in advance!

Alex

[tcdeveau]

That could be a result of the electronic shutter.  Have you tried shooting the stars with an XCD lens, same settings but electronic shutter off?

[jwillson]

I did a little testing on mine to see if I get the same results. 

* ISO 3200
* Lens cap on
* f/32 (to minimize any light coming in through the lens cap--sometimes plastic allows a little IR to leak)
* 2s exposure

I took frames with electronic shutter enabled and with the regular mechanical shutter.

Exposures were essentially black when brought into LightRoom.  I then applied +5 EV of exposure compensation and a manual white balance to the image and noticed two interesting things:

* The electronic shutter image had significantly more amp glow (the lighter areas in the top left and bottom left corners) than the mechanical shutter image. 
* The strange magenta vertical band in the middle was only there in the electronic shutter images.  It was there in bias frames as well (1/2000 s exposure), not just in the longer exposures
* In addition to the magenta band, all of my exposures (whether dark frames or bias frames) had circular banding that, it turns out, is caused by the application of the embedded lens profile.  As soon as I turned that off, the rings went away. 

Here is what I suspect is going on, though I can't prove it:

Hasselblad, unlike most camera manufacturers, applies some calibration correction to each frame that is specific to every camera/chip.  I think it is basically a master bias subtraction that is ISO specific.  It's a good idea, in general, though it may be part of why the startup time is so long for the X1D--the need to load the calibration frames into memory.  This process reduces the need for high ISO noise reduction, which is nice. It won't take care of thermal noise, but it will take care of unwanted pattern noise in the read process.  Very cool.

I suspect that, in the case of electronic shutter, Hasselblad is skipping this calibration step.  If I compare bias frames made with and without the electronic shutter, I notice that the black point is lower with the mechanical shutter and signal to noise ratio is better. It really looks like they aren't applying the bias subtraction when you choose electronic shutter.  This would only be an issue for long exposures and, especially, for astrophotography where you are trying to pull details out of the dark shadows.

Looks like your results are normal.  I won't say they are "good" since I don't know why they are skipping the bias subtraction with electronic shutter, but your results are consistent with mine.  I'd recommend you use the mechanical shutter for astrophotography.  You'll still have noise in the shadows, but at least you'd get rid of that strange magenta band in the middle.  Also, if you are using Lightroom or Adobe Camera Raw for your raw conversion, make sure you don't apply the in-camera lens profile due to the circular banding pattern it introduces. Instead, make manual adjustments in Lightroom or Photoshop.  Distortion isn't an issue for astrophotography, so all you care about is the vignetting correction.  Better to do that yourself.

Ideally, for astrophotography you would use flat frames to address vignetting and dust on the sensor.  You would also use a master dark frame taken at the same sensor temperature as your light frames.  This may be overkill for what you are trying to accomplish, though, and is best done in some specialized software packages rather than Photoshop/Lightroom.  A reasonable first step is just to use the mechanical shutter.  Second step would be to mount the camera on a tracker so you can take longer exposures.  Third step would be to stack multiple images.  Then you could start worrying about bias frames and dark frames.

- Jared

[rent]

Hi Jared,

Thanks for the detailed report and explanation. I'm going to experiment some more and also shoot an email to Hasselblad for more information.

As for astrophotography, the next steps are 1) stacking, 2) use a tracker, 3) use longer, large aperture lenses (third party, so can't get away from e-shutter). I've been using Starry Landscape Stacker to some success. I am also in touch with the developer. Looks like the amp glow issue may be addressable (in theory at least) with dark frames.

Thanks again for your help!

Alex

[jwillson]

Personally, I wouldn't go too far down the stacking path till you are using a tracker simply because with 2s exposures read noise is still dominating over thermal noise and other sources of shot noise.  Might help a bit with suppressing chrominance noise on single pixel stars, but that's about it.  By all means try it out, though, since it doesn't require additional equipment and it's fun to find ways to improve one's technique.  I don't happen to have experience with Starry Landscape Stacker, but there are lots of programs out there so plenty of options.

As far as the amp glow and your requirement for a third party lens/electronic shutter... Yes, dark frames should address both issues.  The only catch is that the dark frames need to be of the same duration as your lights and the CMOS sensor temperature should be as close as possible to the temperature when you took the lights.  Use a bunch of dark frames, if possible.  And if you use electronic shutter for the lights, clearly you should use it for the darks as well.  Dark frames have bias embedded in them (all the read noise), so if I am right that Hasselblad isn't applying the calibration frame to the e-shutter frames, a good set of darks will get around this. 

Good luck!  I have been wondering whether I will want to try out the X1D on any of my telescopes.  The lure of a low-noise 50 megapixel medium format chip is astonishing.  And two of my telescopes produce an imaging circle that is large enough and flat enough to make this possible.  I've already got dedicated, cooled astronomy cameras that do a nice job, though, even if they are a touch smaller than full frame.

[rent]

I did another test with regards to this issue. With my XCD 45mm lens, I made the following shots at ISO 3200, f/3.5, with lens cap on, in a darkroom:

With leaf shutter: 2, 4, 8, 16, 32 seconds
With E-shutter: 2, 4, 8, 16, 32 seconds

The red/magenta glow in the large center vertical band, as well as the left top/bottom corners are clearly visible in the E-shutter 2 sec and 4 sec images. The corresponding 8, 16, 32 seconds shots using leaf or e-shutter are fairly comparable. These RAW files are here in case anyone is interested in taking a look: https://www.dropbox.com/sh/o7k0tntstr7l6ix/AAA6uerHcf1esZGNSaLvIoYWa?dl=0

The engineering aspect of sensor behavior is well beyond my knowledge. I'm happy to hear anyone who cares to comment on this. Going to get in touch with H as well.

Alex

[jwillson]

I'll see if I can duplicate the results on mine using the same shutter speeds.  I seems strange that the results with the electronic shutter would change when the exposure duration reaches eight seconds.  Weird.  Mine did show that strange magenta vertical band with bias frames (very short exposures) and with 2s exposures, but I didn't try anything longer than 2s.  I also didn't try other ISO's, though I believe the electronic shutter is limited to 3200 max.  I'll post when I get a chance to experiment a bit more.

By the way, as far as the overall magenta cast goes...  Normally I'd recommend using a master bias frame to correct it, but since Hasselblad does some sort of image calibration based on measured sensor statistics that may not be a good idea.  It might actually add some unwanted noise.  Might be better to insert a pedestal value for the green channel.

[rent]

Hi @jwillson,

Please do test as I'm very interested in confirming my observations.

The author of Starry Landscape Stacker suggests that this phenomenon could be related to how the sensor values are cleared depending on which mechanism (e-shutter vs mechanical) is used. And the e-shutter mechanism is not as effective for some reason.

I mentioned this to Ming, to which he vaguely concurred, and added that a lot of workarounds were done to make the e-shutter work, and also the reason ISO was limited to 3200 in order to maintain image quality for majority of exposure time. He also didn't think it would be an easy thing to calibrate out in future firmware.

Alex

[jwillson]

OK, here is what I did with my X1D:

- Set the aperture to f/32
- Set ISO to 100
- Set white balance to "daylight"
- Put the lens cap on
- Put the camera in a drawer

- Took pictures with mechanical shutter (leaf shutter) at 1/2,000s, 1s, 2s, 4s, 8s, and 16s
- Took pictures with electronic shutter at 1/2,000s, 1s, 2s, 4s, 8s, and 16s

Brought all of the above into Lightroom and added +5EV of exposure, +100 Shadows, +100 Blacks.  Turned off the lens profile correction to avoid artifacts from vignetting correction.

Things were pretty clean at ISO 100, for both the e-shutter and mechanical shutter images.  Blacks had moved off the left edge of the histogram, but not by much.  Some random hot pixels, but nothing serious.  So I brought the images into Photoshop for more stretching.

In Photoshop, I used a level adjustment to bring out whatever differences were there.  Specifically, I dragged the white point way down to around 83 ADU on each image.  This was enough to make the red/magenta cast in each image obvious. 

In the bias frame--the one taken at 1/2,000s shutter speed--I could just barely tell the two images apart, the one with the e-shutter and the one with the mechanical shutter.  There was just a hint of the central band of red/magenta.  Noise was higher in the e-shutter file (based on measuring standard deviation for a region of the image).  No amp glow was visible in either image, as you'd expect in a very short exposure.

When I looked at the 1s frame, the central bar of magenta became more obvious in the e-shutter image (though still very small), and amp glow become visible in the e-shutter image.  Both were absent in the mechanical shutter image.  Again, signal to noise ratio was better in the mechanical shutter image based on standard deviation measurements with an equalized black point.

Same effects in the 2s image only slightly more pronounced.  The central bar of magenta was still minimal, but amp glow was becoming more noticeable in the e-shutter image.  Again, noise was better controlled in the mechanical shutter image.

Same effects again in the 4s image.  Amp glow still getting worse in the e-shutter image with the central bar very minimal.  Neither was visible in the mechanical shutter image--just ordinary noise and some very minor horizontal banding.  Hot pixels from thermal signal were starting to become more obvious by the time the exposure length was hitting 4s.  Same pixels in both the 2s image and the 4s image, so it's inherent in the chip rather than being an effect from cosmic rays or radioactive decay in the cover glass (yes, those things really do matter and are often visible in narrow band astrophotography images). 

To this point, my contention is that for images made at and below 4s exposures, a master calibration frame is being subtracted from the mechanical shutter images, but not from the electronic shutter images.  This calibration frame clearly includes a master bias frame, and likely a master dark frame as well since it corrects amp glow as well as the bias pattern noise (the magenta bar in the middle of the frame).  It's likely a scaled, bias subtracted dark since that would require much less data be stored in the X1D's memory.  It won't/can't address all the hot pixels since the X1D isn't temperature controlled, so you still see some hot pixels that carry from frame to frame.  It's got to be a master calibration frame(s) since a single dark frame would result in lower SNR if fewer hot pixels and less pattern noise.  Good so far.

Now we get to the 8s and 16s images.  Suddenly, there is no difference between the e-shutter images and the mechanical shutter images.  Somewhere between 4s and 8s, the calibration frame that was being applied only to the mechanical shutter images is getting applied to both.  The central bar disappears.  The amp glow disappears.  The two images become equivalent. 

I repeated the exact same experiment at ISO 3200 (the highest gain where the e-shutter is enabled).  Exact same results.  Obviously, noise levels at ISO 3200 were vastly higher across the board, but the effects were identical.  You even see a hint of overcorrection from the bias calibration as the magenta bar in the middle becomes a (very slightly) darker area.  The mechanical shutter images all have a calibration file applied.  The e-shutter images have a calibration file applied to the 8s and 16s files, but not to the 4s and shorter exposures.

I have no idea why things are the way they are, but my recommendation is still the same.  Unless there is a reason you need the electronic shutter (such as a legacy lens), stick with the mechanical shutter.  Then you don't need to worry about what are "good" or "not good" shutter speeds to use.  Since calibration files are clearly being applied to mechanical shutter images--both master bias and a master dark, given the control of the amp glow--I'd recommend not doing dark frame or bias frame subtraction in the case of long exposures.  Instead, I'd build a bad pixel map for the few hot pixels that can't be fully corrected without a cooled sensor.  Your astronomy software can probably handle that task.  Then, you'll need some sort of pedestal applied to the blue and green channels to address the red/magenta cast to the overall image.  That should get you very good results out of this camera for astrophotography.  The attached file is a 16s dark frame made at ISO 3200 that has been stretched by 5EV and had shadows and blacks increased by the maximum 100% in Lightroom.  I didn't bother with the hot pixel map for a 1024 x 768 image, but I applied a pedestal offset to the green and blue channels.  Considering the ISO and the amount of stretching, this is actually a very good result--designed to show the remaining flaws.  It should make for an effective astronomy camera.  Even the thermal noise was well controlled at room temperatures with very little difference between a 1s exposure and a 16s exposure. 

[MomentsForZen]

I have made exactly the same observations regarding the vertical red stripe with the electronic shutter in low light conditions below 8 seconds exposure length.

As far as astrophotography is concerned, I have been very happy with the results that I have obtained with the 30mm f/3.5 XCD lens (using the mechanical shutter of course with this lens). Please see https://flic.kr/p/24jmhvp for an example.

I have to keep the exposure length at 8 seconds or above with adapted lenses (and the electronic shutter), which can result in star trails rather than point stars if the focal length is above 80mm.

... MomentsForZen (Richard)

[rent]

@jwillson @MomentsForZen,

Thank you for your reports and confirmation. Hopefully by summer I'm going to try the X1D on a tracker. Despite this little "defect" discovered by accident, still very happy with the camera overall.

Alex

[jwillson]

No problem The electronic shutter issues are a little strange, but the camera should actually do a nice job for astrophotography.  I haven't used mine for that yet but expect to.  Need to buy the 30mm lens to get the field of view I want.