NormalizeScaleGradient Script

[John Murphy]

Adam Block has just released an excellent new YouTube video on NormalizeScaleGradient 2.1.6

 

https://www.youtube.com/c/AdamBlock/videos

 

I would recommend that everyone watches it!

Regards, John Murphy

[TerryMcK]

I’ve just watched the video and you have put countless hours in developing the script John and it really shows. An excellent piece of work that we can all benefit from.

[zerolatitude]

Hi, 

 

I've been using NSG for quite some time now. Its a great script.

 

For my system / number of subs, PI registration and NSG takes a long time.

 

I've been experimenting with calibration and registration in Siril, and then using NSG from within PI on Siril-registered files. This is generally much faster, plus Siril has a neat command to do a degree one gradient removal on the raw subs, after calibration but before registration.

 

I am unable to see any significant difference in the output. If anything, Siril (registration plus BE_deg1)+NSG seems to have better fine detail than PI + NSG. And of course its faster.

 

Of course, this could very well be a result of user specific factors like data quality etc.

 

So two questions

1. Is there any theoretical reason why NSG would not work as well on files registered in any program other than PI? Any PI specific synergy?

2. is there any reason to avoid the degree1 BE in Siril if NSG is to be used as well?

 

Hope those are not too generic :-)

[John Murphy]

31 minutes ago, zerolatitude said:

I've been using NSG for quite some time now. Its a great script.

 

For my system / number of subs, PI registration and NSG takes a long time.

 

I've been experimenting with calibration and registration in Siril, and then using NSG from within PI on Siril-registered files. This is generally much faster, plus Siril has a neat command to do a degree one gradient removal on the raw subs, after calibration but before registration.

 

I am unable to see any significant difference in the output. If anything, Siril (registration plus BE_deg1)+NSG seems to have better fine detail than PI + NSG. And of course its faster.

 

Of course, this could very well be a result of user specific factors like data quality etc.

 

So two questions

1. Is there any theoretical reason why NSG would not work as well on files registered in any program other than PI? Any PI specific synergy?

2. is there any reason to avoid the degree1 BE in Siril if NSG is to be used as well?

(1) When calculating the image weights, NSG needs to determine the image noise. For greatest accuracy, the noise must be determined before registration (registration tends to smear the noise by variable amounts depending on the registration shift). PixInsight does this during calibration, and stores it in NOISExx headers. If these headers don't exist, NSG is forced to calculate the noise from the registered images. This will add extra uncertainty to the calculated image weights (NWEIGHT).

(2) It depends how the background extraction is done. Registered frames often have a black border due to registration shifts. NSG requires that these borders are completely black (level zero). If they look black, but actually have a small value, the NSG gradient correction will be adversely affected. If NSG is using stored NOISExx headers, it is important that the image are not multiplied or divided before using NSG. This would invalidate the stored noise values.

 

Hope this helps, John Murphy

[zerolatitude]

8 minutes ago, John Murphy said:

(1) When calculating the image weights, NSG needs to determine the image noise. For greatest accuracy, the noise must be determined before registration (registration tends to smear the noise by variable amounts depending on the registration shift). PixInsight does this during calibration, and stores it in NOISExx headers. If these headers don't exist, NSG is forced to calculate the noise from the registered images. This will add extra uncertainty to the calculated image weights (NWEIGHT).

(2) It depends how the background extraction is done. Registered frames often have a black border due to registration shifts. NSG requires that these borders are completely black (level zero). If they look black, but actually have a small value, the NSG gradient correction will be adversely affected. If NSG is using stored NOISExx headers, it is important that the image are not multiplied or divided before using NSG. This would invalidate the stored noise values.

 

Hope this helps, John Murphy

Hi,

The BE is before registration, so 2 wouldn't apply. But get that 1 would be an issue. Siril does calculate noise, but dont know if it stores it in those specific headers.

 

Thanks for the reply - its clear now.

[wichiepr]

I received the email with the license for the NSGXnml. I add it  to the PI repositories, How I register the license on PI?

 

 

[John Murphy]

2 hours ago, wichiepr said:

I received the email with the license for the NSGXnml. I add it  to the PI repositories, How I register the license on PI?

 

 

To activate the license:
PROCESS > Image Calibration > NSGXnml

[Andrew Bicos]

I just ran through the latest update to the script version 2.1.6 (and followed Adam's new YouTube tutorial) and when I ran the resulting populated ImageIntegration process I get this error:

*** Error: Incompatible local normalization data version. Expected >= 1, got 0: /Volumes/USB31FD/Astro Image Processing by Object/M106/2022 MarApr M106 EdgeHD925ASI6200/NSG/SII/M106_2164_L_2022-03-15_SII_Bin2x2_288s__-10C_c_cc_r_nsg_w100.xnml

<* failed *>

 

What am I doing incorrectly?

[John Murphy]

28 minutes ago, Andrew Bicos said:

I just ran through the latest update to the script version 2.1.6 (and followed Adam's new YouTube tutorial) and when I ran the resulting populated ImageIntegration process I get this error:

*** Error: Incompatible local normalization data version. Expected >= 1, got 0: /Volumes/USB31FD/Astro Image Processing by Object/M106/2022 MarApr M106 EdgeHD925ASI6200/NSG/SII/M106_2164_L_2022-03-15_SII_Bin2x2_288s__-10C_c_cc_r_nsg_w100.xnml

<* failed *>

 

What am I doing incorrectly?

I have added a new section to the NormalizeScaleGradient Reference Documentation:

1.4 NSG and DrizzleIntegration

This explains why this can happen, and what to do to fix it:

 

Changes were made to the '.xnml' format in PixInsight 1.8.9 and unfortunately these changes are not backward compatible. If '.xnml' files were created before PI 1.8.9 (either by LocalNormalization or NormalizeScaleGradient) they cannot be used in PI 1.8.9 or later. In these cases, PixInsight displays the following error message:

 

"*** Error: Incompatible local normalization data version. Expected >= 1, got 0:"

 

I have compiled NSGXnml for both PI 1.8.9 and the previous version PI 1.8.8-12 Make sure you install the correct version! The easiest way to ensure this is to use the NSG update repository, which will automatically install the correct version. See the NormalizeScaleGradient website for more details (see link above).

 

To check the NSGXnml version: PROCESS > Modules > Manage Modules... and select NSGXnml.

 

The API Version should be:

PixInsight 1.8.8-12: API version .... 0x172
PixInsight 1.8.9: API version .... 0x173

 

All downloads are available at:

https://sites.google.com/view/normalizescalegradient

 

Regards, John Murphy

[Andrew Bicos]

Hmmm?  I do have all the latest versions installed (PixInsight 1.8.9: API version...0x173) and all input files were just generated today using WBPP to get to the calibrated and registered inout subframes (_c_cc_r postfix).

[John Murphy]

2 hours ago, Andrew Bicos said:

Hmmm?  I do have all the latest versions installed (PixInsight 1.8.9: API version...0x173) and all input files were just generated today using WBPP to get to the calibrated and registered inout subframes (_c_cc_r postfix).

What version of NSGXnml are you using? The latest version is 1.0.2 You can find the version number using the same dialog that displays the NSGXnml API version.

[John Murphy]

It should look like this:

[rockenrock]

Sorry John, I just don't get it after study your Google sites.

I have been using NSG 2.1.6 via Repository (nsg.astropills.it) and it is working fine on PI 1.8.9. Windows 11.

 

I now purchased the NSGXnml, and received the email with License Key. 

How do I get NSGXnml to show up and activate it? Sorry to trouble you for my ignorance!

 

Roger

[rockenrock]

Hi John,

In the reference documentation PixelMath section is says not to use it for add, multiply or divide. The reasons seem logical.  What is your opinion of replacing the background (including black areas around a registered image) with a fixed value that is less than any nebula or fringes of a galaxy. This image would only be used as the reference image.

 

Should this invalidate the NSG calculations?

 

I tried this, and the results are an almost no-gradient image instead of a complicated gradient data set  (caused by my light polluted location).

 

This post is written in a different way than my original April 5 post above.

Thanks,

      Roger

[TerryMcK]

See John’s reply to another user earlier in the thread.

1 hour ago, rockenrock said:

Sorry John, I just don't get it after study your Google sites.

I have been using NSG 2.1.6 via Repository (nsg.astropills.it) and it is working fine on PI 1.8.9. Windows 11.

 

I now purchased the NSGXnml, and received the email with License Key. 

How do I get NSGXnml to show up and activate it? Sorry to trouble you for my ignorance!

 

Roger

 

[rockenrock]

1 hour ago, TerryMcK said:

See John’s reply to another user earlier in the thread.

 

Terry, I did not find the reply, but here is how I did it:

First set up the repository link to nsg.astropill.it, whereby you can see the NSG script in Scripts/Batch Processing after restart PI.

Purchase the NSGXnml license.

To activate NSGXnml start PI, go to Process / All processes/NSGXnml and click it. Then enter the registration information from the email John sends you when you buy the NSGXnml. Then presto it works.

Edited April 16, 2022 by rockenrock
Solution found.

[John Murphy]

2 hours ago, rockenrock said:

Hi John,

In the reference documentation PixelMath section is says not to use it for add, multiply or divide. The reasons seem logical.  What is your opinion of replacing the background (including black areas around a registered image) with a fixed value that is less than any nebula or fringes of a galaxy. This image would only be used as the reference image.

 

Should this invalidate the NSG calculations?

 

I tried this, and the results are an almost no-gradient image instead of a complicated gradient data set  (caused by my light polluted location).

 

This post is written in a different way than my original April 5 post above.

Thanks,

      Roger

Hi Roger,

You have thought of a very creative process, but unfortunately I would not recommend it. By creating a false background, you are likely to introduce sudden changes in the image gradients at the border of your objects. This could cause dark rings around the objects in your image.

 

DynamicBackgroundExtraction is a difficult process to master, but used well, it should be able to remove your background gradients (the gradients that were in your reference image) in your final stacked image. Email me a link to a stacked image, and I will give you some pointers on how to correct it with DBE. I should have some time after Tuesday.

 

Regards, John

[r.sarwar87]

Hi John Great tool. working great.

Re: drizzle, i seem to be having some issue. probably not doing things exactly right:

 

This is that i am doing- ImageCalibration -> CosmicCorrection -> Start Allignment (with generate drizzle files active)-> NSG (Normalization data  + drizzle data) -> ImageIntegration (using upated template icon). 

Image looks great.

 

On the second pass, i open up DrizzleIntegration to integrate the files and produced image is blurry, because of the underlining dithering which is showing up as if star allignment did not hold.

 

Was wondering what i may have missed

 

[John Murphy]

I think you are using the correct process. I think the drizzle parameters are critical. You also need a large number of high quality evenly dithered images.

[rockenrock]

On 4/16/2022 at 4:46 PM, John Murphy said:

DynamicBackgroundExtraction is a difficult process to master, but used well, it should be able to remove your background gradients (the gradients that were in your reference image) in your final stacked image. Email me a link to a stacked image, and I will give you some pointers on how to correct it with DBE. I should have some time after Tuesday.

Hi John,

You are too generous to offer to work on my DBE. i don't want to take you away from your current development work, and I take it as my responsibility for DBE. 

Actually I had fixed the image with DBE, before I tried my 'creative', and fast way, to create a false flat background reference image. When doing my DBE I studied Adam's Blocks videos again, and I think I was doing the right things. So after two times DBE it came out okay. 

My creative way did show the power of NSG in that in one time it flattened the image, but with some dark rings. I want to keep experimenting with reducing the rings.

All the best,

     Roger

 

 

[r.sarwar87]

22 hours ago, John Murphy said:

I think you are using the correct process. I think the drizzle parameters are critical. You also need a large number of high quality evenly dithered images.

I am using 80 clean shots. never had problems with drizzle before as i produce drizzle files on every pass by default.

if i leave out the NSG, it works just fine.

Edited April 17, 2022 by r.sarwar87

[John Murphy]

4 hours ago, r.sarwar87 said:

I am using 80 clean shots. never had problems with drizzle before as i produce drizzle files on every pass by default.

if i leave out the NSG, it works just fine.

NSG's job is to normalize the target images to the (brightness) scale and gradient of the chosen reference image. The only part it plays in the drizzle process is to provide the '.xnml' Normalization data files that tell DrizzleIntegration how to correct the scale and gradient. Since this is the only thing it creates for DrizzleIntegration, it cannot have any affect on the image sharpness or alignment.

 

But something clearly went wrong. The first thing I would try is selecting the 'Static data targets' in ImageIntegration and DrizzleIntegration. This forces them to use the full pathnames when matching files.

 

Another potential problem is if you did not integrate all the images in ImageIntegration. For example, the images with low weights at the end of the target list were deselected. If these images were then included in DrizzleIntegration, their drizzle data files would not have been updated by ImageIntegration. This would cause problems.

 

You might also want to check that the correct drizzle files were loaded into both ImageIntegration and DrizzleIntegration.

 

Regards, John Murphy

[rockenrock]

 

On 4/17/2022 at 5:40 PM, rockenrock said:

My creative way did show the power of NSG in that in one time it flattened the image, but with some dark rings. I want to keep experimenting with reducing the rings.

Hi John,

Regarding my PixelMath false background as the reference image study....

I further improved any dark rings showing near galaxies by adding more and larger rejections in sample generation, and reducing the gradient correction from -1.1 to -2.5. Overall my technique of creating the false background and manually adding the rejections was a lot more intuitive, and easier than the couple hundred sample points, and guessing, needed in DBE to get rid of the complex gradients I had. 

--- I am going to post process the original DBE flattened image and the false background NSG image. Then compare.

--- I will also compare the NWeights calculated for both runs of NSG I made. That means with and without false background.

--- If I had a site location where I did not have complex gradients, then I would not use a false background method.

 

Either method, false background or not, I really appreciate the NSG script, and the huge amount of information it can show in all the graphs you programmed into it. Thanks!

 

Roger

[rockenrock]

1 hour ago, rockenrock said:

--- I will also compare the NWeights calculated for both runs of NSG I made. That means with and without false background.

After compare the .nsg files weights of the two runs, I found the image order to be almost exactly the same. Only a few images got out of NWeight order by several positions. The NWeights for any image (after normalize to the same reference image) were between 0% and 3.6% different. 

So my conclusion is that creating the false background reference image (using pixelmath) is a valid way to use NSG on complex gradient backgrounds and eliminate the need for DBE.

[John Murphy]

Hi Roger, 

Really pleased to hear that your strategy is working well for you. A great result!

 

Roger has made this work. However, it is not the recommended approach. There are many potential problems that need to be carefully avoided:

• When using PixelMath to subtract the background, if any region ends up completely black, the process will not preserve star flux. The NSG scale measurement might manage to reject these outliers, but there is a risk that the measured scale will be wrong. This may result in artifacts around the astronomical objects (for example dark rings).
• The gradient may have sudden changes when the image is slightly above the PixelMath threshold value. This may cause problems for the NSG's gradient model. This may result in artifacts around astronomical objects (for example dark rings).
• If the NSG reference frame has any black borders due to registration, you must make sure that these regions remain at level zero after your processing. Looking black is not enough. Tiny offsets from zero can have a significant affect on the gradient model. Easily taken care of, but you need to be aware.
• Do not divide or multiply - this would invalidate the PixInsight noise estimates, which would then produce invalid weights.
• The corrected target images may end up with a significant fraction of pixels with negative values. This is OK if you are using the  Normalization data option, which does not truncate pixels to the zero to 1.0 range. But if you are not using this option, the negative values will end up as zero, and will be rejected by ImageIntegration. You will end up with too much noise in some background areas.

 

Roger has fixed these problems by using manual rejection circles. He has produced a good result, which just goes to show that NSG can be successfully used in many different ways. 

 

Regards, John Murphy

Edited April 19, 2022 by John Murphy