NormalizeScaleGradient Script

[rockenrock]

John,

Thank you for your reply. In my PixelMath false 'reference background' method I substituted all the background pixel values to a constant. The value was determined carefully set by examining the lowest pixel levels at the objects of interest, like edges of a galaxy. I compared this to background levels. It was a balance. Then  I used the expression:  iif($T<.245,0.245,$T), where the .245 is the selected background value for my image. This PixelMath image is only the reference for NSG. It is not used in image integration.

 

Since my background level is .245 you quickly know I am imaging in a light polluted urban area. I have to work much harder for my images, than those blessed with dark skies, low air pollution, and low humidity.

 

I can envision images where this method does not work, such as an image where an area of the background is lighter than the faint areas of the object of interest. Maybe I was just lucky!

Roger

[rockenrock]

Hi John,

Adam Block's 3 new videos on PI methods for image weighting, including your NSG script, are quite excellent. You, Adam, and Juan are all working so hard and making strides forward towards perfection.  Thanks!

Roger

Edited April 23, 2022 by rockenrock
Improve my description.

[Astroarg]

Hi John

Hope you are keeping well, first off thank you so much for the hard work with not only NSG, but also PhotometricMosaic.  Everytime I see posts online regarding mosaic issues using GMM, I point them towards PM 🙂

 

I've started to use the new LN tools in the latest PI, but still have your NSG script there to use.. this may be me, but I've noticed even on my upgraded rig (8 core, 16 thread) NSG will take a significant time compared to the LN tools to process when there are a fair few subs.

Is it because I create Normalized Images rather than just Normalized Data?  If I did the latter, wouldn't this just push additional processing time into ImageIntegration?

 

thanks for your efforts 🙂

 

Luke

[Apoplectic]

Hi John –

 

I’ve noticed that NSG 2.1.6 (more specifically NSGXnml) has trouble with image file names that contain two consecutive spaces. When I run NSG with:

a  b  c1.xisf

a  b  c2.xisf

a  b  c3.xisf

a  b  c4.xisf

 

NSGXnml says it can’t find the reference file (a  b  c4.xisf in this case). It looks like NSGXnml has replaced the double spaces with single spaces.

 

When I replace the double spaces with single spaces, everything works:

a b c1.xisf

a b c2.xisf

a b c3.xisf

a b c4.xisf

 

Not 100% sure but I think this worked in version 2.1.1. Today I updated from 2.1.1 to 2.1.6 under a fully updated PixInsight: PixInsight Core 1.8.9 Ripley (x64) (build 1544 | 2022-03-13) 

 

Thanks for taking a look.

 

Best –

                Dan

Edited May 1, 2022 by Apoplectic

[John Murphy]

Hi Dan

Yes, spaces in filenames have been the cause of much trouble... Unfortunately my original test data didn't include spaces in the filenames, so this did not get tested.

 

I am currently working hard on both NormalizeScaleGradient 2.1.7 and NSGXnml 1.0.3, and this should finally fix the space problems completely. I have just replicated your filenames, and it worked 🙂

 

I am currently testing NSG 2.1.7 with a pre-release copy of PixInsight 1.8.9-1. I intend to release NSG 2.1.7 very soon; probably at the same time as the immanent PI 1.8.9-1 release.

Regards, John

[John Murphy]

On 4/23/2022 at 8:49 AM, Astroarg said:

Hi John

Hope you are keeping well, first off thank you so much for the hard work with not only NSG, but also PhotometricMosaic.  Everytime I see posts online regarding mosaic issues using GMM, I point them towards PM 🙂

 

I've started to use the new LN tools in the latest PI, but still have your NSG script there to use.. this may be me, but I've noticed even on my upgraded rig (8 core, 16 thread) NSG will take a significant time compared to the LN tools to process when there are a fair few subs.

Is it because I create Normalized Images rather than just Normalized Data?  If I did the latter, wouldn't this just push additional processing time into ImageIntegration?

 

thanks for your efforts 🙂

 

Luke

Hi Luke,

Sorry, it has taken me some time to get back to you. I have been working very hard on the New NormalizeScaleGradient 2.1.7 / NSG 1.0.3 release.

 

I will describe the improvements in more detail when I make the release, but I can reveal that there have been some important performance improvements:

• If the previous NSG run used hundreds of images, the next time it is run, it could take a long time to read in the FITS headers for these previous headers. This could be very frustrating. I now cache the data that is needed for the target images table, which solves this problem completely. The cache records the target image's modified date and time, so it knows to read any modified files.
• When used with PI 1.8.9-1 (to be released very soon), NSG 2.1.7 will run faster when using the optional NSGXnml 1.0.3 C++ module. This is due to significantly less file I/O; these '.xnml' files are very small compared to the actual images. It also uses up much less disk space 🙂

Hope this helps, John Murphy

 

Edited May 2, 2022 by John Murphy

[Apoplectic]

2 hours ago, John Murphy said:

Hi Dan

Yes, spaces in filenames have been the cause of much trouble... Unfortunately my original test data didn't include spaces in the filenames, so this did not get tested.

 

I am currently working hard on both NormalizeScaleGradient 2.1.7 and NSGXnml 1.0.3, and this should finally fix the space problems completely. I have just replicated your filenames, and it worked 🙂

 

I am currently testing NSG 2.1.7 with a pre-release copy of PixInsight 1.8.9-1. I intend to release NSG 2.1.7 very soon; probably at the same time as the immanent PI 1.8.9-1 release.

Regards, John

Thanks John - Admittedly this case is a bit of an outlier and easily worked around. Thanks for looking and for all your great work! - Dan

[rockenrock]

Hi John,

When convenient, could you comment on below...

SBIG CMOS cameras have internal subs stacking without registration, then readout the stacked image. Their selling point is that the 12bit camera (stack of 16 internal subs) will download a 16 bit image (and save drive space).

Somewhat similarly, SharpCap does live stacking (with registration) in the program which can then be saved.

 

The question is if NSG should work correctly on stacked images after the normal calibration and registration?

 

Thanks, 

   Roger

Edited May 3, 2022 by rockenrock
Clarify sentence.

[zerolatitude]

Hi,

 

Two requests

 

1. Earlier versions had a slider to enable rejection of low quality images, which was removed. It was actually very useful for unattended runs especially overnight (in fact, the request for an option to integrate automatically came from that use case also). So right now, it is possible to have an unattended run, but not if you want to reject poor quality subs. Can this be brought back please? At worst it can be set to 0 if the user does not want any rejection.

 

2. As mentioned above, NSG is relatively slow, and takes only 25% odd CPU resources on my system. While this is great for doing other stuff alongside, it does take much more time. Can it be speeded up or maybe use the cores to the max?

 

Thanks  

 

[zerolatitude]

4 hours ago, rockenrock said:

Hi John,

When convenient, could you comment on below...

SBIG CMOS cameras have internal subs stacking without registration, then readout the stacked image. Their selling point is that the 12bit camera (stack of 16 internal subs) will download a 16 bit image (and save drive space).

Somewhat similarly, SharpCap does live stacking (with registration) in the program which can then be saved.

 

The question is if NSG should work correctly on stacked images after the normal calibration and registration?

 

Thanks, 

   Roger

Just to add - Sharpcap now has an option for gradient removal per sub before stacking. Its quite decent. That should improve the NSG compatibility, I suppose?

[rockenrock]

8 hours ago, zerolatitude said:

Just to add - Sharpcap now has an option for gradient removal per sub before stacking. Its quite decent. That should improve the NSG compatibility, I suppose?

Zero, Previously we were told not to do any DBE (or equivalent I guess) to background before running NSG. It would be great to have one sub with ideal background and then drive the others to have the same minimal gradient background. 

I hope John will comment again on your question.

     Roger

[jcdavis]

Hello - I am brand new to this board and to using NSG, just purchased and installed a few days ago on PI 1.8.9.

 

I recently ran the NSG on a stack of 1400 images and it ran fine (took 5+ hours though)...

 

Now I'm trying to open the NSG script dialog again - and it is taking a LONG time to re-open, I see the message "Reading FITS Headers" in the console with a % complete.... been running for about 10 minutes and is at 85%.

 

Was wondering if this is expected and if so - why it is reading the headers again before I interact with it.  What I was trying to do was take a much smaller subset of my images and run through again to try and determine why I didn't get any drizzle files after checking that option.  But that will be another question when and if I get to that point.

 

Thanks,

John D.

[zerolatitude]

1 hour ago, rockenrock said:

Zero, Previously we were told not to do any DBE (or equivalent I guess) to background before running NSG. It would be great to have one sub with ideal background and then drive the others to have the same minimal gradient background. 

I hope John will comment again on your question.

     Roger

True.

 

However, in practice, before the latest PI / NSG updates, I got my best results with Siril + per sub background extraction degree 1 + NSG 🙂

[jcdavis]

My 2nd question of the day - don't know how many I'm allowed 🙂

 

I ran NSG on my registered subs, with Drizzle Integration checked.  After NSG finished and I opened up the Image Integration process - I saw all the subs with the <n><d> in front.  I ran image integration which worked find and appeared to update drizzle files - and produced my integrated image.

 

However - I wanted to re-run image integration and I had deleted the process - so I needed to reload the images and drizzle files.  However I cannot figure out how I do that so that it is using the NSG files... If I just add the registered subs and drizzle files - it only shows the <d> in front of the image file reference - not the <n>, so I don't know if it will be integrating using the NSG files. 

 

I'm hoping that I do NOT have to re-run the NSG process again just to get Image Integration properly populated, I have 1440 subs and the NSG process ran for 5+ hours on my 8 core/16 thread Ryzen-7 computer with 64Gb of ram!

[John Murphy]

5 hours ago, jcdavis said:

Hello - I am brand new to this board and to using NSG, just purchased and installed a few days ago on PI 1.8.9.

 

I recently ran the NSG on a stack of 1400 images and it ran fine (took 5+ hours though)...

 

Now I'm trying to open the NSG script dialog again - and it is taking a LONG time to re-open, I see the message "Reading FITS Headers" in the console with a % complete.... been running for about 10 minutes and is at 85%.

 

Was wondering if this is expected and if so - why it is reading the headers again before I interact with it.  What I was trying to do was take a much smaller subset of my images and run through again to try and determine why I didn't get any drizzle files after checking that option.  But that will be another question when and if I get to that point.

 

Thanks,

John D.

Hi John

Yes, this is a known problem which will be fixed in 2.1.7, which will be released soon; I am currently testing it with a pre-release of PI 1.8.9-1 Until then, the work around is to remember to clear the target images before exiting NSG.

 

NSG does not create the drizzle files - that is done during registration. If the drizzle option is checked, it looks for the drizzle files, making the assumption that they are in the same directory as the target images. It then passes these onto ImageIntegration. If the files are else where, you can add them to ImageIntegration manually.

 

Hope this helps, John Murphy

[John Murphy]

6 minutes ago, jcdavis said:

My 2nd question of the day - don't know how many I'm allowed 🙂

 

I ran NSG on my registered subs, with Drizzle Integration checked.  After NSG finished and I opened up the Image Integration process - I saw all the subs with the <n><d> in front.  I ran image integration which worked find and appeared to update drizzle files - and produced my integrated image.

 

However - I wanted to re-run image integration and I had deleted the process - so I needed to reload the images and drizzle files.  However I cannot figure out how I do that so that it is using the NSG files... If I just add the registered subs and drizzle files - it only shows the <d> in front of the image file reference - not the <n>, so I don't know if it will be integrating using the NSG files. 

 

I'm hoping that I do NOT have to re-run the NSG process again just to get Image Integration properly populated, I have 1440 subs and the NSG process ran for 5+ hours on my 8 core/16 thread Ryzen-7 computer with 64Gb of ram!

The best practice is to save an ImageIntegration process icon (drag its small blue triangle onto the PI desktop). Provided the images, normalization data files and drizzle files are still in the same folder, you can then run ImageIntegration from the process icon.

 

If you did not save a process icon, you still might be able to get it from the History Explorer. Open up your integrated image, and then right click on the image tab on the left side of the image. Select History Explorer from the menu:

You can then drag the ImageIntegration icon from the left pane in the History Explorer onto the PI desktop to create a new process icon.

 

To add the '.xnml' files to ImageIntegration manually, use the ImageIntegration 'Add L.Norm. Files' button.

 

Regards, John Murphy

 

[John Murphy]

16 hours ago, zerolatitude said:

Earlier versions had a slider to enable rejection of low quality images, which was removed. It was actually very useful for unattended runs especially overnight (in fact, the request for an option to integrate automatically came from that use case also). So right now, it is possible to have an unattended run, but not if you want to reject poor quality subs. Can this be brought back please? At worst it can be set to 0 if the user does not want any rejection.

I am planning to add this functionality back in to NSG. Until then, you will have to run ImageIntegration again in the morning. ImageIntegration runs much faster the second time around because it caches a lot of data to disk.

 

Regards, John

[John Murphy]

20 hours ago, rockenrock said:

Hi John,

When convenient, could you comment on below...

SBIG CMOS cameras have internal subs stacking without registration, then readout the stacked image. Their selling point is that the 12bit camera (stack of 16 internal subs) will download a 16 bit image (and save drive space).

Somewhat similarly, SharpCap does live stacking (with registration) in the program which can then be saved.

 

The question is if NSG should work correctly on stacked images after the normal calibration and registration?

 

Thanks, 

   Roger

Provided that registration is NOT applied, stacking images before preprocessing will have no adverse affect on NSG.

 

Doing registration outside PixInsight will make the noise estimates less accurate. This is because registration smears the noise by random amounts per image. The calculated weights depend on the noise estimates, so these will also be less accurate.

 

Regards, John Murphy

[John Murphy]

16 hours ago, zerolatitude said:

Just to add - Sharpcap now has an option for gradient removal per sub before stacking. Its quite decent. That should improve the NSG compatibility, I suppose?

You should not apply gradient removal on registered images before NSG. This is because registered images will often have black margins around their image. These must be totally black (zero). If gradient removal makes these regions even slightly non zero, it might interfere with NSG's relative gradient model.

 

It is OK to apply gradient removal on the unregistered reference image, because these will not have any black margins. The reference image will look nicer while using NSG, but it doesn't really matter if you remove the gradient before, or after with DBE. DBE is very capable, but it has a steep learning curve. Once fully understood, it can produce amazing results.

 

There is absolutely no need to remove the gradients from the target images. This is completely unnecessary. NSG is extremely good at matching the target images to the reference.

 

Regards, John Murphy

Edited May 3, 2022 by John Murphy

[zerolatitude]

9 hours ago, John Murphy said:

I am planning to add this functionality back in to NSG. Until then, you will have to run ImageIntegration again in the morning. ImageIntegration runs much faster the second time around because it caches a lot of data to disk.

 

Regards, John

Thanks.

 

Till then, another related question on weights and drizzle.

 

In Image Integration after NSG, the files are sorted by weight (of course assuming that option is chosen). While using LN, an the weight of a file can be seen by hovering over the file name in the file list box. This helps discard the low weights files manually.

 

Similarly, for drizzle integration I would like to cull the files again based on the NSG weight. However, I cannot find any way to sort by weight or even see the weights.

 

Am I missing something?

 

Thanks

[zerolatitude]

Hi,

 

Another question - using up my quota fast 🙂

 

WBPP + LN + NSG (with LN) results in a band as in the attached image. This is the raw stack just out of Image Integration after NSG, with LN, and only STF applied.

 

Integration with NWEIGHT but no LN results in a clean image.

 

The data set is quite clean, and culled.

 

What could it be?

Edited May 4, 2022 by zerolatitude

[John Murphy]

41 minutes ago, zerolatitude said:

Hi,

 

Another question - using up my quota fast 🙂

 

WBPP + LN + NSG (with LN) results in a band as in the attached image. This is the raw stack just out of Image Integration after NSG, with LN, and only STF applied.

 

Integration with NWEIGHT but no LN results in a clean image.

 

The data set is quite clean, and culled.

 

What could it be?

If I have understood this correctly, you ran the PI LocalNormalization process, followed by NSG. The data was normalized twice using different software. You should never do this!

Regards, John

[John Murphy]

54 minutes ago, zerolatitude said:

Thanks.

 

Till then, another related question on weights and drizzle.

 

In Image Integration after NSG, the files are sorted by weight (of course assuming that option is chosen). While using LN, an the weight of a file can be seen by hovering over the file name in the file list box. This helps discard the low weights files manually.

 

Similarly, for drizzle integration I would like to cull the files again based on the NSG weight. However, I cannot find any way to sort by weight or even see the weights.

 

Am I missing something?

 

Thanks

This is why I removed the option from NSG to automatically cull low weight images. It works for ImageIntegration, but the user then has to do it manually for DrizzleIntegration. If an image has been rejected from ImageIntegration, it MUST also be removed from DrizzleIntegration; if not, DrizzleIntegration will fail. The hope was that if the user manually unticked the low weight images in ImageIntegration, they would remember not to add these images to DrizzleIntegration.

 

An option to automatically cull low weights would clearly be useful, but it also needs to work well with DrizzleIntegration. I am currently determining a strategy to achieve this.

 

Regards, John

[zerolatitude]

21 hours ago, John Murphy said:

If I have understood this correctly, you ran the PI LocalNormalization process, followed by NSG. The data was normalized twice using different software. You should never do this!

Regards, John

I did normalise with PI, but used the LN files generated by NSG.

 

Was under the impression that LN does not affect the registered files, but only generates additional data?

 

Okay, will try again w/o PI LN and see.

 

Thanks

[zerolatitude]

20 hours ago, John Murphy said:

This is why I removed the option from NSG to automatically cull low weight images. It works for ImageIntegration, but the user then has to do it manually for DrizzleIntegration. If an image has been rejected from ImageIntegration, it MUST also be removed from DrizzleIntegration; if not, DrizzleIntegration will fail. The hope was that if the user manually unticked the low weight images in ImageIntegration, they would remember not to add these images to DrizzleIntegration.

 

An option to automatically cull low weights would clearly be useful, but it also needs to work well with DrizzleIntegration. I am currently determining a strategy to achieve this.

 

Regards, John

Thanks.