NormalizeScaleGradient Script

[John Murphy]

NormalizeScaleGradient V2.2.2 progress dialog.

 

This release now includes a progress dialog that is displayed while it is processing the target images. While progress information was always available from the console, it is much more convenient to see it within a progress dialog.

 

It shows accurate metrics on the most recently processed target image, progress and the estimated time remaining.

 

Enjoy!

John Murphy

https://sites.google.com/view/normalizescalegradient

Edited August 23, 2022 by John Murphy

[TerryMcK]

Thanks for all your fantastic work on this John.

[nmartin]

Hey John. Thanks for all of the work on this. I've been using NSG for a while now. I just processed a set of images after about a 6-month lull due to the weather. I'm loving some of the new improvements, but I've noticed it's running quite a bit slower. I'm getting times around 20-40 seconds per image on a 3700x. I've noticed that CPU usage is only around 30% while running NSG vs 100% when running any other pixinsight process. Is there a limitation preventing usage of all cores and threads for NSG? Are there any plans to improve the efficiency? This last stack took about an hour to just run NSG, and while that isn't a show stopper, it is a bit of a bummer.

[John Murphy]

On 9/7/2022 at 11:04 PM, nmartin said:

Hey John. Thanks for all of the work on this. I've been using NSG for a while now. I just processed a set of images after about a 6-month lull due to the weather. I'm loving some of the new improvements, but I've noticed it's running quite a bit slower. I'm getting times around 20-40 seconds per image on a 3700x. I've noticed that CPU usage is only around 30% while running NSG vs 100% when running any other pixinsight process. Is there a limitation preventing usage of all cores and threads for NSG? Are there any plans to improve the efficiency? This last stack took about an hour to just run NSG, and while that isn't a show stopper, it is a bit of a bummer.

Yes, I have been busy adding some great new functionality. These extra features have no impact on the performance. However, I have also made some changes that do affect performance. In most cases, these changes allow NSG to run faster, but in some cases they could slow it down:

1. In previous versions, if the image is very large, and contains thousands of stars, I used to automatically limit the star detection to a central rectangle. In the current version it will only limit the star detection area if the user requests it by setting the Photometry Region of Interest (see attached image).
2. The other time consuming task is to accurately determine the surface spline that models the gradient to be removed.  The time taken depends on the number of samples used. If the Gradient smoothness is very low (for example, -3.0), approximately 2000 samples will be used. When the Gradient smoothness is increased, the number of samples are reduced (less are needed to accurately model a smoothly changing gradient. At +4.0 it is extremely quick!). I have made changes in this area, increasing the number of samples used. In some cases, this may increase execution time.
3. Unfortunately JavaScript is single threaded. However, when the JavaScript code calls PixInsight C++ methods (for example, creating the surface spline) these methods can be multi threaded. I did consider porting the whole of NSG to C++, but this would require an enormous amount of work.

Regards, John Murphy

https://sites.google.com/view/normalizescalegradient

Edited September 13, 2022 by John Murphy

[nmartin]

4 hours ago, John Murphy said:

Yes, I have been busy adding some great new functionality. These extra features have no impact on the performance. However, I have also made some changes that do affect performance. In most cases, these changes allow NSG to run faster, but in some cases they could slow it down:

1. In previous versions, if the image is very large, and contains thousands of stars, I used to automatically limit the star detection to a central rectangle. In the current version it will only limit the star detection area if the user requests it by setting the Photometry Region of Interest (see attached image).
2. The other time consuming task is to accurately determine the surface spline that models the gradient to be removed.  The time taken depends on the number of samples used. If the Gradient smoothness is very low (for example, -3.0), approximately 2000 samples will be used. When the Gradient smoothness is increased, the number of samples are reduced (less are needed to accurately model a smoothly changing gradient. At +4.0 it is extremely quick!). I have made changes in this area, increasing the number of samples used. In some cases, this may increase execution time.
3. Unfortunately JavaScript is single threaded. However, when the JavaScript code calls PixInsight C++ methods (for example, creating the surface spline) these methods can be multi threaded. I did consider porting the whole of NSG to C++, but this would require an enormous amount of work.

Regards, John Murphy

https://sites.google.com/view/normalizescalegradient

Thanks for the reply, John. I have been using a smoothness between -1.0 and -2.0 so that probably has an impact, and the star detection is likely what has led to the increased execution time for me if it was automatically ignoring stars previously.

 

I completely understand that porting the code over would be a lot of work. Thanks for everything that you have put into this.

[John Murphy]

I have updated the NormalizeScaleGradient release notes, which are available on the NSG website:

https://sites.google.com/view/normalizescalegradient#h.4x03n7ou1ht1

 

I am currently testing the latest release, NSG 2.2.3, which will be released in early October. This will be a minor release with the following updates:

• The blink dialog is now available for 2 or more images. It used to require at least 3.
• The gradient graph dialog, plotted points: I no longer plot points that are within rejection circles (due to stars or manual rejection circles). Such points could be misleading when the user decides what level of smoothing to apply.
• Fixed JavaScript warnings.

Regards, John Murphy

https://sites.google.com/view/normalizescalegradient

Edited September 26, 2022 by John Murphy

[JDAstrophoto]

I just recently purchased NSG and have run it now about 7 times, testing it.  I believe I have it correct.

I run Weighted Batch Preprocessing Script and check everything BUT Image Integration.

Then I use the cc_c_r (registration) files and run NSG.  I add the drizzle files and the L. Norm files to Image Integration before I run it. 

Then I execute Image integration and the images are displayed in real time.

1. Is that correct sequence?

2. Is there a way to have the images from Image Integration saved automatically in a similar fashion WBPP does?  Or does it always have to be a manual save after Image Integration.

3. I read in the instructions that NSG can also be used with Subframe selection.  The order of WBPP is:

Subframe Weighting

Image Registration

Astrometric Solution

Local Normalization

 

So Subframe Weighting is the first one, so how do I used NSG to help with Subframe selection when the images are registered yet?

 

 

Edited September 26, 2022 by TerryMcK
Moved to relevant forum topic

[John Murphy]

16 hours ago, JDAstrophoto said:

I just recently purchased NSG and have run it now about 7 times, testing it.  I believe I have it correct.

I run Weighted Batch Preprocessing Script and check everything BUT Image Integration.

Then I use the cc_c_r (registration) files and run NSG.  I add the drizzle files and the L. Norm files to Image Integration before I run it. 

Then I execute Image integration and the images are displayed in real time.

1. Is that correct sequence?

2. Is there a way to have the images from Image Integration saved automatically in a similar fashion WBPP does?  Or does it always have to be a manual save after Image Integration.

3. I read in the instructions that NSG can also be used with Subframe selection.  The order of WBPP is:

Subframe Weighting

Image Registration

Astrometric Solution

Local Normalization

 

So Subframe Weighting is the first one, so how do I used NSG to help with Subframe selection when the images are registered yet?

 

 

When using NormalizeScaleGradient:

(1) In WBPP, you must deselect both Local Normalization and Image Integration. There is no point using NSG if you add WBPP's Local Normalization (.xnml) files to ImageIntegration. You need to use the .xnml files created by NSG.

(2) After WBPP has finished, run NSG. Make sure you select NSG's Normalization data option. This creates the .xnml normalization files. It is these files that instruct ImageIntegration how to apply the scale and gradient corrections that NSG calculated.

(3) After NSG has been run, on exit, it will display ImageIntegration, with all the correct settings for NSG. The LocalNormalization files (created by NSG) will automatically be added to ImageIntegration.

(4) NSG sets up the ImageIntegration Pixel Rejection sections based on the number of images. Check that the rejection settings are what you want.

(5) Run ImageIntegration. Look at the rejection maps it produces and check the console for the rejection percentage. You should aim for rejection of bright pixels between 0.5 and 3% (the ideal percentage depends on the number of hot pixels. Cameras without cooling need more rejection than a top end astronomical camera). If necessary, adjust the rejection parameters and run ImageIntegration again. It runs much more quickly after the first run.

 

NSG displays ImageIntegration instead of running it in the background. This allows the user to get the best results from ImageIntegration. Having to save the final stacked image manually is a small price to pay for this extra flexibility.

 

Although it is possible to use SubframeSelector after NSG, I would recommend using NSG's Image Rejection section instead. This rejects images based on the transmission (due to light clouds, humidity, dew) and the image weights (a combination of transmission and light pollution). Both the transmission and weights are calculated extremely accurately, which will produce the highest signal to noise ratio in the final stacked image. The sharpness of faint areas (galaxy spiral arms, nebula) have a greater dependency on the signal to noise ratio than tracking errors or seeing.

 

Personally, I would only use SubframeSelector if I was imaging a star cluster:

For star clusters / globular clusters, I would try to optimize the star resolution. I think SFS weights will often be the best way of doing this. Stars are very sensitive to 'seeing' and tracking. This is because they are bright point sources. Even a tiny percentage of the star light can cause a very noticeable star bloat.

For galaxies / nebulae, the resolution of the faint areas of an image are almost always limited by the shot noise. The tracking errors and atmospheric distortions are usually insignificant in comparison. I would therefore always use NSG to calculate the weights for these objects, and fix star profiles in post processing. ImageIntegration usually does a good job of rejecting the star bloat from the frames that had poor star profiles.

That said, I would probably still weed out frames with extremely bad star profiles. 

 

Regards, John Murphy

https://sites.google.com/view/normalizescalegradient

Edited September 27, 2022 by John Murphy

[JDAstrophoto]

Thank-you so much for the response.  I knew I was doing something wrong, because my "master lights" looked exactly the same whether I used WBPP fully, or inserted NSG in between.  

[JDAstrophoto]

I can't figure out why I am not getting Drizzle files.  I thought maybe it was a filename paths over 256 characters, but that isn't it.  

** WARNING: Missing drizzle '.xdrz' file: ...

[JDAstrophoto]

Never mind, I was using the wrong files.  

[ginyu2003]

I just updated NSG, then I attempted to use it like I always do and got this error.

 

**ERROR** NSGXnml cannot find reference file:

....

32.304 ms

Failed to normalize 'file' [62/314]

 

I picked the reference image from the NSG command box (after sorting the files based on noise). The file is there were it said... I'm not sure why it can't find it. 

 

Any idea's?

 

 

[John Murphy]

19 hours ago, ginyu2003 said:

I just updated NSG, then I attempted to use it like I always do and got this error.

 

**ERROR** NSGXnml cannot find reference file:

....

32.304 ms

Failed to normalize 'file' [62/314]

 

I picked the reference image from the NSG command box (after sorting the files based on noise). The file is there were it said... I'm not sure why it can't find it. 

 

Any idea's?

 

 

I need some more information. What version of NSG and NSGXnml are displayed in the console when you run NSG? It should look similar to this:

 

You need to be running the latest version of PixInsight (PI 1.8.9-1)

Note that if you use a Mac, you also need to update NSGXnml manually see

https://sites.google.com/view/normalizescalegradient#h.v2b2srsw8vn9

 

If you are already on the most up to date versions, please provide me with the full error message.

Regards, John Murphy

[ginyu2003]

11 hours ago, John Murphy said:

I need some more information. What version of NSG and NSGXnml are displayed in the console when you run NSG? It should look similar to this:

 

You need to be running the latest version of PixInsight (PI 1.8.9-1)

Note that if you use a Mac, you also need to update NSGXnml manually see

https://sites.google.com/view/normalizescalegradient#h.v2b2srsw8vn9

 

If you are already on the most up to date versions, please provide me with the full error message.

Regards, John Murphy

 

Well here is a screen shot of the error: 

 

https://drive.google.com/file/d/1gW1wFJw265Pc_Ys5r5L6F1H3D3EP50LS/view?usp=sharing

 

However there was a few pixinisght updates this morning... and its currently working so... If it happens again I try and get some more data. 

 

 

Thanks for reaching out,

Richard 

[Sedat]

Hello John,

 

I thought NSG needed the initial subframe weighting done by WBPP but probably I was wrong. Adam Block, disabled this SW box in his recent update video on NSG. I just wanted to make sure....

[John Murphy]

On 10/9/2022 at 6:20 PM, Sedat said:

Hello John,

 

I thought NSG needed the initial subframe weighting done by WBPP but probably I was wrong. Adam Block, disabled this SW box in his recent update video on NSG. I just wanted to make sure....

 

When using NormalizeScaleGradient, you should disable the following WBPP check boxes: Subframe Weighting, Local Normalization and Image Integration. The only pre-calculated data that NSG needs is the noise estimates, and these are always created by WBPP.

 

NSG creates its own '.xnml' Normalization data files which instruct ImageIntegration how to apply the scale and gradient corrections. This replaces Local Normalization.

 

NSG sets up ImageIntegration to use 'PSF Scale SNR' weights. This new PixInsight weighting method uses the same algorithm that NSG has always used (signal to noise ratio squared). However, the results when used with NSG is different to when used by PixInsight's Local Normalization. The reason for this is that the scale factor (signal) is passed to the weight algorithm via the '.xnml' data files. I spent a great deal of time and thought to make sure that the scale factor was calculated extremely accurately, and this can make a significant difference. 'PSF Scale SNR' only produces the same results as NWEIGHT if it is invoked with '.xnml' data files that were created with NSG.

 

I hope this helps clarify this area.

Regards, John Murphy

https://sites.google.com/view/normalizescalegradient

[Sedat]

All clear now, thanks.

 

Best regards

Sedat

[John Murphy]

NormalizeScaleGradient 2.2.3 has now been released.

• The blink dialog is now available for 2 or more images. It used to require at least 3.
• The gradient graph dialog, plotted points: I no longer plot points that are within rejection circles (due to stars or manual rejection circles). Such points could be misleading when the user decides what level of smoothing to apply.
• Fixed JavaScript warnings.

A big thank you to Marco for hosting the NSG repository.

Regards, John Murphy

https://sites.google.com/view/normalizescalegradient

[Spaced Out Bill]

Hello John,

 

I left a comment on Adam Block's video  NSG Script Update 2.2.3 -Part 1. You mentioned this Forum. Thank you for the comment and thank you in advance for helping!

My NSG results are really blotchy on this data. I am pretty much using the default settings in NSG.

Any ideas as to what I'm doing wrong would be greatly appreciated.

I have attached a link to the Final stacked image here.

https://drive.google.com/file/d/1DFztrRT4mBAFYdaWy0JIOuGTSSApYL8e/view?usp=sharing

As well as the Process Icons here.

https://drive.google.com/file/d/1KzN-IA6pHQ_oHn4BofqvU41zxnjThtuH/view?usp=sharing

https://drive.google.com/file/d/1kg12JqiQkyrVKWOgfsEselCVWgmuw7nr/view?usp=sharing

Here is what my untrained eyes are seeing. The first Image is only from Image Integration and the second is NSG and Image Integration. Both images are cropped the same and have the same DBE applied.

The blotchyness gets worse with color calibration also.

Any help would be appreciated understanding your awesome Script.

Thanks for all your hard work!

Bill

 

 

ImageIntegrationAndromeda.xpsm NSGAndromeda.xpsm

[John Murphy]

3 hours ago, Spaced Out Bill said:

Hello John,

 

I left a comment on Adam Block's video  NSG Script Update 2.2.3 -Part 1. You mentioned this Forum. Thank you for the comment and thank you in advance for helping!

My NSG results are really blotchy on this data. I am pretty much using the default settings in NSG.

Any ideas as to what I'm doing wrong would be greatly appreciated.

I have attached a link to the Final stacked image here.

https://drive.google.com/file/d/1DFztrRT4mBAFYdaWy0JIOuGTSSApYL8e/view?usp=sharing

As well as the Process Icons here.

https://drive.google.com/file/d/1KzN-IA6pHQ_oHn4BofqvU41zxnjThtuH/view?usp=sharing

https://drive.google.com/file/d/1kg12JqiQkyrVKWOgfsEselCVWgmuw7nr/view?usp=sharing

Here is what my untrained eyes are seeing. The first Image is only from Image Integration and the second is NSG and Image Integration. Both images are cropped the same and have the same DBE applied.

The blotchyness gets worse with color calibration also.

Any help would be appreciated understanding your awesome Script.

Thanks for all your hard work!

Bill

 

 

ImageIntegrationAndromeda.xpsm 71.46 kB · 1 download NSGAndromeda.xpsm 80.3 kB · 0 downloads

I am on holiday for a week. I will have a look when I get back.

Regards, John

[macbates]

I have a couple of questions when using drizzle with NSG. After running WBPP (using drizzling) I ended up with 264 xisf and xdrz files. I then ran NSG with “Drizzle data” checked, and ended up with 232 xnml files. No problem there; NSG nicely got rid of 32 files that fell below the weight and transmission levels. At the conclusion of the NSG run, it opened the integration process, which I ran. It did NOT open the drizzle integration though, which would have been handy since there were file mismatches.

 

When I run drizzle integration by hand, the problem is that there is a mismatch between the 264 xdrz files and the 232 xnml files. The problem is determining which 32 of the 264 xdrz files I need to exclude from drizzle integration. The integration process runs, but has lots of errors (not surprisingly, since there are 32 missing files). Deselecting the appropriate xdrz files by hand is a major amount of work, since the 32 missing files are scattered among the 264 input files.

 

So is there a way (hopefully automated) to do drizzle integration with NSG and include only the 232 xdrz files that have corresponding xnml files? Did I miss something in the processing steps?

 

Thanks in advance for any advice.

[John Murphy]

On 10/15/2022 at 8:39 PM, macbates said:

I have a couple of questions when using drizzle with NSG. After running WBPP (using drizzling) I ended up with 264 xisf and xdrz files. I then ran NSG with “Drizzle data” checked, and ended up with 232 xnml files. No problem there; NSG nicely got rid of 32 files that fell below the weight and transmission levels. At the conclusion of the NSG run, it opened the integration process, which I ran. It did NOT open the drizzle integration though, which would have been handy since there were file mismatches.

 

When I run drizzle integration by hand, the problem is that there is a mismatch between the 264 xdrz files and the 232 xnml files. The problem is determining which 32 of the 264 xdrz files I need to exclude from drizzle integration. The integration process runs, but has lots of errors (not surprisingly, since there are 32 missing files). Deselecting the appropriate xdrz files by hand is a major amount of work, since the 32 missing files are scattered among the 264 input files.

 

So is there a way (hopefully automated) to do drizzle integration with NSG and include only the 232 xdrz files that have corresponding xnml files? Did I miss something in the processing steps?

 

Thanks in advance for any advice.

When using Drizzle:

1. Set the 'Drizzle data; option
2. Run NSG
3. In the Image Rejection section, reject images based on transmission and weight.
4. In the Image Rejection section, use 'Move to ./NSG_Reject'. This moves the rejected .xnml and .xdrz files to the NSG_Reject folder, which then avoids any confusion when you come to use Drizzle Integration.

Regards, John Murphy

Edited October 17, 2022 by John Murphy

[macbates]

Thanks, John. An easy solution that I should have thought of myself. Thanks!

[GordonH]

I've using NSG for a while now. About a week ago I started getting a repository error whenever PixInsight booted. I assumed there was a temporary glitch, but, its persisted. The other sites seem to work fine. Is anyone else having this issue? Thoughts?

Edited October 18, 2022 by TerryMcK
Moved to relevant topic

[John Murphy]

19 hours ago, GordonH said:

I've using NSG for a while now. About a week ago I started getting a repository error whenever PixInsight booted. I assumed there was a temporary glitch, but, its persisted. The other sites seem to work fine. Is anyone else having this issue? Thoughts?

I think that this was a temporary problem. It looks to be accessible now.

Thanks again for Marco for providing the NSG repository. It's really good of him.

Regards, John Murphy