NormalizeScaleGradient Script

[John Murphy]

The shorter answer to your question is below. The previous post provides a more detailed response.

Regards, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

Edited December 14, 2022 by John Murphy

[John Murphy]

On 12/13/2022 at 9:29 AM, zerolatitude said:

Question re using NSG on lights with different filters.

 

OSC capture, some with a UHC, some with an LPro. The LPro with much wider bandpass is obviously brighter and has more gradients than the UHC.

 

I usually integrate these all at once without any LN. Now I'm wondering what it would be like if I were to use NSG.

 

Since the background is so different, even a reference frame that works for all would seem difficult to get. I was thinking NSG separately for each filter set and then integrate.

 

Does that sound right? I will of course try it out, but wondered if there's any conceptual direction?

 

Thanks

The shorter answer to your question is:

Provided your OSC images don't contain any emission nebula, you can add all the images, taken with different light pollution filters, into NSG at once and ignore the red warning text. You can even mix unfiltered with light pollution filters. But never mix broadband filters with narrow band.

 

I don't use UHC filters, and their design between manufactures may vary. If it only passes emission lines (not a broadband light pollution filter), it would only really be useful with emission nebula, and you can't mix and match filters with emission nebula (see my previous more detailed answer).

 

For non OSC images, or for any other filters, read my previous answer

Regards, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

Edited December 14, 2022 by John Murphy

[Minusman]

Hello John, how do I proceed if I want to combine images with different pixel scales? Because apparently combining with Pixelmath does not lead to an optimal SNR result. 

[John Murphy]

On 12/16/2022 at 10:20 PM, Minusman said:

Hello John, how do I proceed if I want to combine images with different pixel scales? Because apparently combining with Pixelmath does not lead to an optimal SNR result. 

Unfortunately, combining images with different pixel scales is problematic because their PixInsight noise estimate will not be compatible. The resulting image weights will be wildly wrong. I did suggest a solution to PixInsight, were any operation that caused an image resize would update the noise estimate. However, this has not yet been addressed.

 

So unfortunately, I don't know of an easy way of doing this.

Regards, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

[Minusman]

Thanks for the info, John. I'll try to convince the Pixinsight people that something like this is needed.

[John Murphy]

On 12/12/2022 at 10:42 AM, Phillyo118 said:

Hi John,

 

Firstly thank you for the NSG script, I find it really useful and improves my images over the normal WBPP script even with the LN functionality included within WBPP.

 

However, for the last few weeks when I've tried to use NSG, Pixinsight has crashed midway through the script with the error "A breakpoint has been reached (Error 0x80000003)".

 

Any ideas what might be causing that? PI works in every other instance.

 

Thanks,

Phil.

One user has reported:
"I ran the tests which you linked and DISM found damaged system file data which it repaired. The check disk found file corruption on the NVME stripe which it also corrected. The NSG process now runs without incident."

 

In case this occurs again, the microsoft link is:

https://answers.microsoft.com/en-us/windows/forum/all/how-do-i-get-rid-of-the-error-exception-break/cb222a25-89c2-4b36-910e-9130bc686cba

Edited December 18, 2022 by John Murphy

[Bruce D.]

I ran NSG, followed by ImageIntegration and all went well. However, I have 2 issues:

1- The directory output folder I created to store the files is empty.

2- The final image was saved to be used later but when I went back to reopen it, only a starfield was present. However, if I re-ran II with the files still present in NSG, the image is there. I just can't save it.

[Astroarg]

check that folder from Windows Explorer.  You are accessing through PI and it's looking for images, but in NSG you are saving normalisation data files, not images.

 

your final screenshot, the stars is a jpg.. it'll look like that unstretched.  What are you trying to save as?  the pic on the left?  Save it as a .xisf file instead.

[John Murphy]

12 hours ago, Bruce D. said:

I ran NSG, followed by ImageIntegration and all went well. However, I have 2 issues:

1- The directory output folder I created to store the files is empty.

2- The final image was saved to be used later but when I went back to reopen it, only a starfield was present. However, if I re-ran II with the files still present in NSG, the image is there. I just can't save it.

Hi,

1. The reason you can't see the .xnml LocalNormalization files that NSG produced is that these are data files, not images. The file browser invoked by PixInsight is filtering for known image formats, so it does not display them. The .xnml data files instruct ImageIntegration how to normalize the images; the brightness scale factor to multiply the image by, and the gradient to subtract.
2. If you ever need to manually load the files into ImageIntegration yourself, you would load the registered but unnormalized files, and then use the 'Add L.Norm. Files' button to add the .xnml files. ImageIntegration displays a <n> in front of the filenames to indicate it has the .xnml data files.
3. After stacking, you need to stretch the image. For example, a HistogramTransformation. Tip: you can copy the STF stretch to HistogramTransformation by dragging the STF blue triangle onto the HistogramTransform status bar. Then apply it and cancel the STF.
4. The reason you could only see stars in the saved jpg image is that without a stretch, a jpg image does not have sufficient dynamic range to save all the data. The nebula will be lost.

Looks like a great image though!

I have recently added six YouTube videos:

https://www.youtube.com/@NormalizeScaleGradient

 

Hope this helps, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

Edited December 22, 2022 by John Murphy

[Bruce D.]

ah...OK

 

Thanks both!  I just went to Windows Explorer and found the files....phew.  I saved it as a .xisf file and it opened fine and then could save the stretched image as a jpeg and it opened fine.

 

Thx again..........and great program, John!

 

BTW- I posted this question on the PI forum and I got my wrist slapped with a response that I cannot ask questions on competing products!?  Really?  I know PI has LF but this is a script on PI and folks do ask questions about it.

 

 

Edited December 22, 2022 by Bruce D.
typo

[TerryMcK]

1 hour ago, Bruce D. said:

BTW- I posted this question on the PI forum and I got my wrist slapped with a response that I cannot ask questions on competing products!?  Really?  I know PI has LF but this is a script on PI and folks do ask questions about it.

I saw that too in your question over in the PI forum. The strange comment somebody had put didn't make sense to me either. Their forum rules do not mention anything about competing products. Oh well as you have had a solution to your question if I were you I would not worry about it.

[astra]

Hi John. 

I was using NSG with m42 with different exposures, 60sec, 120sec, 300sec. A reference is obviously 300sec image.

 

The issue I have is that after nsg I get core blown out. I added rejection circle, but it didn't help. And it is not even an integration

 

So my question is: is that correct that I can use different exposures within one nsg session? If yes, then how do I preserve core?

[John Murphy]

On 1/4/2023 at 11:23 PM, astra said:

Hi John. 

I was using NSG with m42 with different exposures, 60sec, 120sec, 300sec. A reference is obviously 300sec image.

 

The issue I have is that after nsg I get core blown out. I added rejection circle, but it didn't help. And it is not even an integration

 

So my question is: is that correct that I can use different exposures within one nsg session? If yes, then how do I preserve core?

In this situation, you need to choose a good reference image from the set with the shortest exposure. Include all sets within NormalizeScaleGradient. All images will then be normalized to match the reference image's dynamic range.

 

If you incorrectly set your reference image to one of the brighter images, the fainter images would be made brighter. Their pixel values would then exceed the allowed 0.0 to 1.0 pixel range. The pixel values have to be truncated back to 0.0 to 1.0 before they are saved to disk. This then causes M42's core to be blown out.

 

Note that if you use the NSGXnml add on, this problem no longer occurs. The brightness scale factor and the gradient to be removed are stored as instructions within .xnml data files. These instruct ImageIntegration how to normalize the images. Internally, ImageIntegration is not limited to the 0.0 to 1.0 pixel data range, so no truncation occurs.

 

However, you will still run into problems when you try to integrate. In this situation, data rejection will work against you - it will reject the brightest pixels, which is not what you want.

 

An alternative approach would be to stack each exposure set separately. Then run the resulting 3 stacks through HDRComposition.

 

Regards, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

Edited January 7, 2023 by John Murphy

[Anthony Quintile]

Hey John, (or anyone else who might know..)-

Do I need to add Local Normalization files to Drizzle Integration (1x, CFA), or is the normalization information already embedded into the drizzle files during the regular Integration?

(Sorry if this has already been answered, but this is a long thread.)

Thank you!

Anthony

[Anthony Quintile]

6 minutes ago, Anthony Quintile said:

Hey John, (or anyone else who might know..)-

Do I need to add Local Normalization files to Drizzle Integration (1x, CFA), or is the normalization information already embedded into the drizzle files during the regular Integration?

(Sorry if this has already been answered, but this is a long thread.)

Thank you!

Anthony

I found an answer to my question a page or two back...

 

"When NSG invokes ImageIntegration, it automatically adds these '.xnml' files to it. Once ImageIntegration has finished, you need to setup the DrizzleIntegration process. This time you need to manually add the '.xnml' files by using the  'Add L.Norm. Files' button.

"

Thanks, and glad I found this forum!

[John Murphy]

You might also find my video tutorials on NSG useful:

Regards, John Murphy

https://www.youtube.com/@NormalizeScaleGradient

https://www.normalizescalegradient.net/

 

[Anthony Quintile]

On 1/8/2023 at 3:47 AM, John Murphy said:

You might also find my video tutorials on NSG useful:

Regards, John Murphy

https://www.youtube.com/@NormalizeScaleGradient

https://www.normalizescalegradient.net/

 

John-

I have watched Adam Block's tutorials on NSG, but I haven't seen that he's done one on a workflow with NSG and CFA Drizzle. He does have a lot of stuff on his page, so maybe I missed it?

I will check out your videos as well.

Thank you for the response.

-Anthony

[John Murphy]

21 hours ago, Anthony Quintile said:

John-

I have watched Adam Block's tutorials on NSG, but I haven't seen that he's done one on a workflow with NSG and CFA Drizzle. He does have a lot of stuff on his page, so maybe I missed it?

I will check out your videos as well.

Thank you for the response.

-Anthony

Hi Anthony,

From the NSG Reference Documentation (help button at bottom left), section 1.4 NSG and DrizzleIntegration (but with a couple of extra screen shots):

To use NSG and DrizzleIntegration:

• In StarAlignment, check the Generate drizzle data option. It needs to create .xdrz drizzle data files. WBPP automatically does this step for you.
• In NSG, check both the Normalization data (creates '.xnml' data files) and the Drizzle data check boxes.

• After NSG has finished, ImageIntegration is displayed with the correct settings:

1. The Input Images will be prefixed with <d> and <n>.
2. The Generate drizzle data will be checked.
3. Normalization will be set to Local Normalization in both the Image Integration and Pixel Rejection (1) sections.

• Run ImageIntegration. ImageIntegration updates the drizzle data files with extra information that DrizzleIntegration will need.
• In DrizzleIntegration, add '.xdrz' data files (Add Files button) and '.xnml' data files (Add L.Norm. Files button).

Note the <n>, which indicates that NSG's Normalization data (.xnml files) have been added correctly (via the Add L.Norm. Files button).

 

Regards, John Murphy

https://www.youtube.com/@NormalizeScaleGradient

https://www.normalizescalegradient.net/

 

 

Edited January 12, 2023 by John Murphy
Noted that WBPP automatically creates drizzle files.

[bpaczkowski]

Message Deleted: Problem solved

 

 

Edited January 16, 2023 by bpaczkowski
problem solved

[John Murphy]

It's fantastic that PixInsight has an update system. However, after an update is installed, it doesn't check to see if the files are still there. It can therefore occasionally be necessary to force PixInsight to install the updates from scratch. This is true for all scripts and processes, not just NSG.

 

To force PixInsight to reinstall the updates, delete the file 'updates.xri' from the PixInsight root folder. For example, on Windows delete: C:\Program Files\PixInsight\updates.xri

 

Regards, John Murphy

https://www.youtube.com/@NormalizeScaleGradient

https://www.normalizescalegradient.net/

Edited January 16, 2023 by John Murphy

[chrisastrophoto]

Hi, I am having a strange memory problem with NSG on one particular computer.

The computer is running Windows 11 on a amd Ryzen 9 5900X, 96GB RAM and a RTX GeForce 3090.

I didn't use it with PixInsight for some weeks and when I try now to run NSG on about 250 registered files (300MB each) it crashes PI after about 150 files. Running several times, it does not crash on the same file. It does sometimes on reading a file and sometimes on writing a file. The common error is: 

 Error: File I/O Error: Unable to create file: Win32 error (1450): Insufficient system resources exist to complete the requested service.

Sometimes it continues a while and crashes finally with an access violation.

Process Explorer shows me that there must be a memory leak in the PI process: Private Bytes of the PixInsight Process increase after each processed image continuously until it reaches approximately 4GB. Then it suddenly jumps up to 7GB and PI crashes as described. (see Screenshot)

 

Doing exactly the same on the same registered files on a different computer with Windows 11 on a i7-8750H, but 16GB RAM and a somewhat smaller graphics adapter (a laptop) everything works fine. NSG comes to an end, there is NO observable memory leak in the PixInsight process ... but of course everything is much slower.

 

Both computers have the same patch level of Windows 11, same version of PI and NSG script.

 

The resolutions for this error proposed on the internet (tweaking the registry under in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management on PagedPoolSize and PoolSizeMaximum) do not help.

 

Any idea?

Edited January 21, 2023 by chrisastrophoto

[TerryMcK]

Hi Chris welcome to the forum.

You probably need a driver update for the graphics adapter as that appears to be the only difference other than different CPUs. Same OS, same PI installation, same NSG. Try turning off the GPU and try it again see link below. It will be slower as just using the CPU but if it still crashes then you have ruled out the GPU and can look elsewhere in the computer. Also are the computers running exactly the same processes i.e. is there something running in the background on the pc that crashes but is absent from the non crashing pc?

 

https://www.williamliphotos.com/starnet-cuda

 

[chrisastrophoto]

Did just that, installed latest driver from Nvidia, did a repair on PI, made sure that I use the "normal" tensorflow, not the CUDA enabled version, removed quite some other software (of course this PC is used for gaming ... because of its graphics adpater 😉), did a RAM Test (ok), checked the disks (OK) ... but of course there might be something running in the background ... still the same.

Rolled back the latest windows patch ... the same.

But it does not look like an interaction with a different process. It is a memory management problem ...

 

[chrisastrophoto]

2 hours ago, TerryMcK said:

Hi Chris welcome to the forum.

You probably need a driver update for the graphics adapter as that appears to be the only difference other than different CPUs. Same OS, same PI installation, same NSG. Try turning off the GPU and try it again see link below. It will be slower as just using the CPU but if it still crashes then you have ruled out the GPU and can look elsewhere in the computer. Also are the computers running exactly the same processes i.e. is there something running in the background on the pc that crashes but is absent from the non crashing pc?

 

https://www.williamliphotos.com/starnet-cuda

 

Sorry did not quote correctly on my latest post.

I am not sure, I cannot trace it back, but I think I did run NSG on this system at least once on a much larger number of files. But I cannot remember what has been installed on this computer since then ... I'll try to rollback, but this is a very strange behaviour ...

[TerryMcK]

You may have already seen this but googling seems to throw up a lot with that error https://answers.microsoft.com/en-us/windows/forum/all/how-to-resolve-error-1450-insufficient-system/ede7ec19-138a-4840-bf36-593e2b42b0d0