NormalizeScaleGradient Script

[Bruce D.]

Hi John
 

Good news.  It worked perfectly (settings below just to confirm).  So, my question:  Do I leave it at 8 and how does this affect the rest of PI processes?  Or do I need to switch it back and forth, using only 8 for NSG?

 

 

[John Murphy]

On 2/1/2023 at 6:09 PM, Bruce D. said:

Hi John
 

Good news.  It worked perfectly (settings below just to confirm).  So, my question:  Do I leave it at 8 and how does this affect the rest of PI processes?  Or do I need to switch it back and forth, using only 8 for NSG?

 

 

I would recommend trying to see how many cores you can use before the issue arises. Perhaps try 24 out of your 32 cores next time.

 

Another question to try to narrow down the root cause. Are you using networked or RAID storage?

Regards, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

 

Edited February 3, 2023 by John Murphy

[chrisastrophoto]

On 2/3/2023 at 11:46 AM, John Murphy said:

I would recommend trying to see how many cores you can use before the issue arises. Perhaps try 24 out of your 32 cores next time.

 

Another question to try to narrow down the root cause. Are you using networked or RAID storage?

Regards, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

 

Hi John, tried different numbers of processors ... and yes, it makes a difference.

Originally PI was configured to use all processors (12 physical cores, 24 logical processors in my case) ... on  a dataset of 336 registered files (300MB each) it crashed at about 150 files or so. Setting to 12 processors made it crash at about file 240. Setting it to 8 Processors it succeeded. But nevertheless private bytes of the PI Process indicate a small memory leak which seems to be smallest at 8 processors so far. Do not know, if it is NSG or PI itself ... Processor is a AMD Ryzen 9 5900X.

 

[Bruce D.]

On 2/3/2023 at 5:46 AM, John Murphy said:

I would recommend trying to see how many cores you can use before the issue arises. Perhaps try 24 out of your 32 cores next time.

 

Another question to try to narrow down the root cause. Are you using networked or RAID storage?

Regards, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

 

 

[Bruce D.]

OK- I will try 24-cores and see how it goes.  Interesting info from chrisastrophoto.  

 

Also, I have been updating folks about this issue on Adam Block's forum as this is where I first posted it and Adam recommended that I ask you about it.  

[John Murphy]

1 hour ago, chrisastrophoto said:

Hi John, tried different numbers of processors ... and yes, it makes a difference.

Originally PI was configured to use all processors (12 physical cores, 24 logical processors in my case) ... on  a dataset of 336 registered files (300MB each) it crashed at about 150 files or so. Setting to 12 processors made it crash at about file 240. Setting it to 8 Processors it succeeded. But nevertheless private bytes of the PI Process indicate a small memory leak which seems to be smallest at 8 processors so far. Do not know, if it is NSG or PI itself ... Processor is a AMD Ryzen 9 5900X.

 

Thank you for your extremely useful research.

If I could do another test for me, this would help narrow down where the problem is:

• Use all 24 logical processors
• Set Gradient smoothness to maximum smoothness (+4.0)

At maximum smoothness, NSG uses its own algorithm to fit a flat plane to the gradient surface (a 2D linear fit to the differences between the reference and target). At all other levels of smoothness, it uses PixInsight's surface spline algorithm.

 

Knowing whether this succeeds of fails will help narrow down the source of the problem

Thank you so much (it is so frustrating not to be able to reproduce the problem on my own hardware), John Murphy

[Bruce D.]

I just ran my complete dataset once again, only changing the gradient smoothness to +4.0.  I did not reduce the processors.  NSG ran perfectly normal and produced the same results I got by reducing the core processors like I did previously.

 

 

[John Murphy]

22 minutes ago, Bruce D. said:

I just ran my complete dataset once again, only changing the gradient smoothness to +4.0.  I did not reduce the processors.  NSG ran perfectly normal and produced the same results I got by reducing the core processors like I did previously.

 

 

Now that is interesting. But I will wait for a few more results to come in before I explain what I think might be happening.

Thank you so much for running this test.

John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

[chrisastrophoto]

On 2/5/2023 at 12:08 PM, John Murphy said:

Thank you for your extremely useful research.

If I could do another test for me, this would help narrow down where the problem is:

• Use all 24 logical processors
• Set Gradient smoothness to maximum smoothness (+4.0)

At maximum smoothness, NSG uses its own algorithm to fit a flat plane to the gradient surface (a 2D linear fit to the differences between the reference and target). At all other levels of smoothness, it uses PixInsight's surface spline algorithm.

 

Knowing whether this succeeds of fails will help narrow down the source of the problem

Thank you so much (it is so frustrating not to be able to reproduce the problem on my own hardware), John Murphy

Hi John

I can confirm the observation of @Bruce D.: using 24 processors and default settings for smoothness crashes PI after 129/250 frames. Setting smoothness to 4.0 (maximum) lets it get through successfully - and it seems that there is no memory leak in that case!

[John Murphy]

On 2/5/2023 at 12:46 PM, Bruce D. said:

I just ran my complete dataset once again, only changing the gradient smoothness to +4.0.  I did not reduce the processors.  NSG ran perfectly normal and produced the same results I got by reducing the core processors like I did previously.

 

 

From the data collected (thank you!) I conclude that:

• Reducing the number of processors PixInsight uses, decreases the risk of failure. I am not aware of any failures when using 8 processing thread.
• If NSG's Gradient Soothness is set to its maximum of +4.0, this either greatly reduces or eliminates the risk of failure. This is significant because at +4.0, NSG does not use PixInsight's Surface Spline method; it uses its own method to fit a flat plane to the data points. This may indicate that there is a bug in PixInsight's Surface Spline method.

Since I can only fix problems within NSG, I have worked hard to find a work around. The first solution is to limit PixInsight to using just 8 processing threads (in EDIT > Global Preferences):

 

I am also about to release NSG 3.0, which allows you to continue your run after a failure. It has been designed to work even if PixInsight crashes during the run. The code is finished, and I am currently updating the Reference Documentation. I hope to release it in about a weeks time.

 

 

If you would like early access to the beta version, let me know.

 

Regards, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

 

Edited February 9, 2023 by John Murphy

[Bruce D.]

John- I am happy to try the beta version.  Just to be clear, I can leave my processing cores at max with this new version- correct?  Also, if PI crashes, as expected, NSG remains open and running to completion- correct?  Then, I simply reopen PI to run the data in ImageIntegration or does that open on its own?

 

Bruce

[John Murphy]

On 2/10/2023 at 10:18 AM, Bruce D. said:

John- I am happy to try the beta version.  Just to be clear, I can leave my processing cores at max with this new version- correct?  Also, if PI crashes, as expected, NSG remains open and running to completion- correct?  Then, I simply reopen PI to run the data in ImageIntegration or does that open on its own?

 

Bruce

The tests have convinced me that the error is not within the NSG code. Unfortunately this means I can't fix it. So NSG 3.0 works around the problem. You now have the choice of:

1. In PixInsight's Global Preferences, limit the number of cores to 8.
2. Alternatively, use all the cores. If you have a large number of images, a crash might still occur. If so, you then need to restart PixInsight. Then start NSG. It will remember all your settings, and know which images have been successfully processed. You can then select 'Continue run' to complete the run.

Hopefully PixInsight will fix the underlying problem soon. Quite a lot of evidence points to the SurfaceSpline method, and NSG is not the only code to use this (although it does use it much more heavily than other processes).

 

Regards, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

[chrisastrophoto]

On 2/11/2023 at 7:35 PM, John Murphy said:

The tests have convinced me that the error is not within the NSG code. Unfortunately this means I can't fix it. So NSG 3.0 works around the problem. You now have the choice of:

1. In PixInsight's Global Preferences, limit the number of cores to 8.
2. Alternatively, use all the cores. If you have a large number of images, a crash might still occur. If so, you then need to restart PixInsight. Then start NSG. It will remember all your settings, and know which images have been successfully processed. You can then select 'Continue run' to complete the run.

Hopefully PixInsight will fix the underlying problem soon. Quite a lot of evidence points to the SurfaceSpline method, and NSG is not the only code to use this (although it does use it much more heavily than other processes).

 

Regards, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

Now it happend with 8 Processors ... ok, I had 718 files to process ... and it crashed on file 415 ... pretty annoying, have to try with 4 processors then. If I would know how to test the PI internal SurfaceSpline method without using NSG, I would do. But posting this now on th the PI forum is almost pointless, because they will tell me again that I did something wrong ... or Microsoft or somebody else ...

[John Murphy]

55 minutes ago, chrisastrophoto said:

Now it happend with 8 Processors ... ok, I had 718 files to process ... and it crashed on file 415 ... pretty annoying, have to try with 4 processors then. If I would know how to test the PI internal SurfaceSpline method without using NSG, I would do. But posting this now on th the PI forum is almost pointless, because they will tell me again that I did something wrong ... or Microsoft or somebody else ...

I have now finished NSG 3.0, it's ready to go! It should start to be offered as an update within the next few days. With this new release, you will be able to safely continue a run even if PixInsight crashes.

 

I would like to thank those that volunteered to test the beta versions. They found a few bugs, all of which have now been fixed. They also provided some great 'wish list' ideas. For example:

• Displayed images initially zoomed out to fit the window. While implementing this, I also improved the zoom functionality. Zoom button now zoom on the image center. The mouse scroll wheel zoom does a better job of centering the zoom position at the cursor.
• I had added the ability to display DrizzleIntegration on NSG exit. A request was made to allow a template DrizzleIntegration process icon to be specified. This has been added.
• Renamed the Integration section to: 'Display ImageIntegration/DrizzleIntegration after Exit'
• If PixInsight crashed during the previous run, I was writing the warning message to the console before the results summary. It was pointed out that if the user is processing hundreds of images, this message probably wouldn't be seen. It is now written after the summary text.
• There were a couple of bugs in the error handling that is used if PixInsight crashes. These have all been fixed.

So you can see, their input was extremely useful.

 

The Reference Documentation has been fully updated, and you should be able to find what you're looking for quickly by clicking on the content list hyperlinks. It has new sections on:

• Comet processing. Normalizing comet images provides a new set of problems due to the motion. NSG 3.0 has been updated to deal with this better, but the results also benefit from some simple user input.
• How to run NSG in a ProcessContainer. You can now process all your data, taken with different filters, overnight. When run in ProcessContainer mode, NSG will run ImageIntegration and DrizzleIntegration rather than just display them.
• How to run NSG in 'Apply' mode, which runs it as if it was in a ProcessContainer. It exits after the run has finished, and runs ImageIntegration and DrizzleIntegration.

Other notable changes, the target images now display gray, green or yellow spheres to indicate their status:

• Gray: Unprocessed
• Green: Processed
• Yellow: Processed, but with warning due to too few photometry star matches.

A new 'Continue run' button has been added, which is available not all images have been successfully processed. This might be because you have added extra images, or because the run was aborted, failed or PixInsight crashed.

 

A Results.nsg button has been added. This allows you to connect the currently displayed images with the results from a previous run. Provided the files still exist, and have not been modified, they will be marked as processed (green or yellow sphere). The Transmission, Weight graph will be available, and the 'Continue run' can be used.

 

Usually you won't need to use the Resutls.nsg button, because NSG saves the location of the results file in its settings. But it might occasionally be useful.

 

The full set of updates for this release can be found on my website, or directly from my google drive:

https://docs.google.com/document/d/1f7NfiYvio_hW-U1GDPcDX8GqtPRXezNi3EbogSTsJwI/edit?usp=share_link

 

Regards, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

[chrisastrophoto]

2 hours ago, John Murphy said:

I have now finished NSG 3.0, it's ready to go! It should start to be offered as an update within the next few days. With this new release, you will be able to safely continue a run even if PixInsight crashes.

 

I would like to thank those that volunteered to test the beta versions. They found a few bugs, all of which have now been fixed. They also provided some great 'wish list' ideas. For example:

• Displayed images initially zoomed out to fit the window. While implementing this, I also improved the zoom functionality. Zoom button now zoom on the image center. The mouse scroll wheel zoom does a better job of centering the zoom position at the cursor.
• I had added the ability to display DrizzleIntegration on NSG exit. A request was made to allow a template DrizzleIntegration process icon to be specified. This has been added.
• Renamed the Integration section to: 'Display ImageIntegration/DrizzleIntegration after Exit'
• If PixInsight crashed during the previous run, I was writing the warning message to the console before the results summary. It was pointed out that if the user is processing hundreds of images, this message probably wouldn't be seen. It is now written after the summary text.
• There were a couple of bugs in the error handling that is used if PixInsight crashes. These have all been fixed.

So you can see, their input was extremely useful.

 

The Reference Documentation has been fully updated, and you should be able to find what you're looking for quickly by clicking on the content list hyperlinks. It has new sections on:

• Comet processing. Normalizing comet images provides a new set of problems due to the motion. NSG 3.0 has been updated to deal with this better, but the results also benefit from some simple user input.
• How to run NSG in a ProcessContainer. You can now process all your data, taken with different filters, overnight. When run in ProcessContainer mode, NSG will run ImageIntegration and DrizzleIntegration rather than just display them.
• How to run NSG in 'Apply' mode, which runs it as if it was in a ProcessContainer. It exits after the run has finished, and runs ImageIntegration and DrizzleIntegration.

Other notable changes, the target images now display gray, green or yellow spheres to indicate their status:

• Gray: Unprocessed
• Green: Processed
• Yellow: Processed, but with warning due to too few photometry star matches.

A new 'Continue run' button has been added, which is available not all images have been successfully processed. This might be because you have added extra images, or because the run was aborted, failed or PixInsight crashed.

 

A Results.nsg button has been added. This allows you to connect the currently displayed images with the results from a previous run. Provided the files still exist, and have not been modified, they will be marked as processed (green or yellow sphere). The Transmission, Weight graph will be available, and the 'Continue run' can be used.

 

Usually you won't need to use the Resutls.nsg button, because NSG saves the location of the results file in its settings. But it might occasionally be useful.

 

The full set of updates for this release can be found on my website, or directly from my google drive:

https://docs.google.com/document/d/1f7NfiYvio_hW-U1GDPcDX8GqtPRXezNi3EbogSTsJwI/edit?usp=share_link

 

Regards, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

Thank you John! You are doing an incredible job! Very much appreciated! Sorry for not participating in the beta testing ... but I so busy currently ... Thanks again!

[AstroAv]

This is amazing John, thank you for all your work on this!

[macbates]

John: I just ran the latest NSG version using the process container method you described, and ran into a problem that I can't figure out. The attached image shows pretty severe 'blotching' in the NSG image, while the image that was run through normal WBPP processing appears much better. I re-ran NSG standalone 

(no process container), and the result was the same; pretty severe clumping throughout the image. I've done this with all default parameters, and the bad results with NSG remain while standard WBPP processing looks OK.

 

These are luminance images, but the results are the same for the RBG filters. This used to work just fine, so it's either something in the latest MNSG version (unlikely) or something I did (almost certainly). I can't figure out what I might have done to get such a terrible result, so any ideas you have would be greatly appreciated.

 

The attached images are screen shots from PI, both with a default STF applied and no processing at all.

 

Thanks in advance, and let me know if you need any more info.

 

 - Ken

 

[John Murphy]

17 hours ago, macbates said:

John: I just ran the latest NSG version using the process container method you described, and ran into a problem that I can't figure out. The attached image shows pretty severe 'blotching' in the NSG image, while the image that was run through normal WBPP processing appears much better. I re-ran NSG standalone 

(no process container), and the result was the same; pretty severe clumping throughout the image. I've done this with all default parameters, and the bad results with NSG remain while standard WBPP processing looks OK.

 

These are luminance images, but the results are the same for the RBG filters. This used to work just fine, so it's either something in the latest MNSG version (unlikely) or something I did (almost certainly). I can't figure out what I might have done to get such a terrible result, so any ideas you have would be greatly appreciated.

 

The attached images are screen shots from PI, both with a default STF applied and no processing at all.

 

Thanks in advance, and let me know if you need any more info.

 

 - Ken

 

The first thing I will need is the NSG log file:

.../NSG/NsgData/NSG_Log_<DATE>.txt

 

Regards, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

[macbates]

Thanks for looking at this, John. It's more than likely something I did wrong, but it has me baffled. The log file is attached.

NSG_Log_2023.02.21_15h59m25s.txt

[John Murphy]

2 hours ago, macbates said:

Thanks for looking at this, John. It's more than likely something I did wrong, but it has me baffled. The log file is attached.

NSG_Log_2023.02.21_15h59m25s.txt 454.96 kB · 1 download

OK, could you send me a link to some data via a message or email?

• The NSG and the WBPP stacked images
• Either all the registered images, or if that is too much to upload, a smaller subset that contains the best images, worst images and the reference image.

I will then look into it and get back to you.

Thanks, John Murphy

[John Murphy]

On 2/22/2023 at 2:54 AM, macbates said:

John: I just ran the latest NSG version using the process container method you described, and ran into a problem that I can't figure out. The attached image shows pretty severe 'blotching' in the NSG image, while the image that was run through normal WBPP processing appears much better. I re-ran NSG standalone 

(no process container), and the result was the same; pretty severe clumping throughout the image. I've done this with all default parameters, and the bad results with NSG remain while standard WBPP processing looks OK.

 

These are luminance images, but the results are the same for the RBG filters. This used to work just fine, so it's either something in the latest MNSG version (unlikely) or something I did (almost certainly). I can't figure out what I might have done to get such a terrible result, so any ideas you have would be greatly appreciated.

 

The attached images are screen shots from PI, both with a default STF applied and no processing at all.

 

Thanks in advance, and let me know if you need any more info.

 

 - Ken

 

 

I have had a look at your 10 best frames, reference image and worst 10 frames. The first gradient graph with the default -2.0 smoothness shows up the issue (see attached images below).

 

The graph line is following the noise, so the smoothness needs to be increased. The graph also shows up some clumps in the data. This is real, not a NSG artifact. Perhaps this is due to random noise clumping together, but I think it is more likely that the culprit was very thin tenuous clouds. These existed on the reference image (and perhaps on all the images). With a low level of smoothness, NSG replicates what is detected on the reference on to all the target images. The solution is to increase the level of smoothness. It will then no longer follow the noise, and the lumpiness from the individual images should average out. The second graph was created with a smoothness of 0.0

 

In this case, PixInsights Local Normalization's much higher level of smoothness has worked in its favor. It also demonstrates something important. NSG's default smoothness is one of the few parameters that does need user intervention. You should consider its default smoothness as a starting point. The default was chosen to suit 5 minute exposures. If you use short exposures, you will often need to use a smoother setting. 

I hope this helps, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

[Axel26]

John,

I did the automatic update of NormalizeScaleGradient 3.0 within PixInsight 1.8.9.1 under Win 11. According the Installed Updates the script is there. But I cannot find it in any of the Script menus.

Can you please help?

 

Axel

[John Murphy]

On 4/11/2023 at 5:12 PM, Axel26 said:

John,

I did the automatic update of NormalizeScaleGradient 3.0 within PixInsight 1.8.9.1 under Win 11. According the Installed Updates the script is there. But I cannot find it in any of the Script menus.

Can you please help?

 

Axel

Hi Axel,

After PixInsight has been restarted, the script should be installed at "Batch Processing" > "NormalizeScaleGradient".

 

 

Occasionally, PixInsight updates can fail to copy all the files. It assumes that the update was installed correctly and doesn't check to see if the files are really there. When this occurs, you need to force PixInsight to copy the update files again. To do this:

1. Delete the file 'updates.xri' from the PixInsight root folder to force the updates to install again. For example, on Windows delete: C:\Program Files\PixInsight\updates.xri
2. Check for updates, and then restart PixInsight.

 

Regards, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

Edited April 19, 2023 by John Murphy

[John Murphy]

Hi all,

PixInsight 1.8.9-2 has just been released. With this release, all C++ modules need to be recompiled to be compatible.

 

I have already compiled NSGXnml for both Windows and Linux. The NSG repository will automatically install NSG 3.0.1 and the correct NSGXnml version (1.0.4 for PI 1.8.9-1, 1.0.5 for PI 1.8.9-2)

 

I will be compiling the Mac OS version on Tuesday 15th August. Why the delay? My Mac machine is too old to run 1.8.9-2, and its compiler is not recent enough to compile for it either. Fortunately a friend has offered me some time on his up to date Mac, so all will be well 🙂

 

 

Regards, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

[chrisastrophoto]

Hi John,

We had contact a while ago concerning memory leak in PI resulting in NSG script crashing.

Of course the root cause of the problem has not been fixed with PI 1.8.9-2, because the developers of PI do not accept it as a bug ...

Now it is even worse: when the NSG script crashes, PI crashes in a way, which destroys the update history of scripts.

After the first event NSG was completely gone from the scripts menu. I could make it re-appear by adding the full scripts folder again under Feature Scripts. But when starting NSG it tells me that it has found a valid license but could not find the proper NSGxnml. The correct DLL is however in the bin folder of PI. Simply running a Check for  Updates does not help ... it does not find anything new - of course, because the system should be up to date. The only way out is to Reset Updates in PI and run all updates again. Then I can start NSG and continue ... until it crashes the next time. I have currently 500 files to stack and it crahes after about 135 files processed. This is really a pain now! Three times re-running updates just to create one stack ...

What can we do?

 

Regards

 

Chris