Jump to content
Welcome to Backyard Astronomy Space - please register to gain access to all of our features. Click here for more details. ×

NormalizeScaleGradient Script


John Murphy

Recommended Posts

John Murphy

NormalizeScaleGradient is designed to normalize the scale and gradient of registered images to that of a reference image. This is a crucial preprocessing step in image processing that makes a significant difference to the quality of the final stacked image.

 

This script is free, and it is also open source (GNU General Public License). I am very happy for the code to be used within any image processing software, free or commercial, and I am willing to provide any help and advice that might be needed. My aim has always been to create something that would help the amateur astronomy community. It is currently available as a PixInsight script.

 

How it works

When correcting images to match a reference, the gradient must be removed by subtraction, and the brightness scale factor must be corrected with multiplication. The problem is that it can be ambiguous how much of the correction should be applied as scale or offset. Any errors here might not be immediately apparent, but will cause a contrast difference between the images.

 

My innovation was to use stellar photometry to determine the brightness scale difference. Once this is accurately known, the relative gradient can be accurately removed. The accurate scale factor also makes it possible to accurately determine the correct weights to apply to the images when stacking them. This is vital in order to produce the optimum signal to noise ratio in the final stacked image. It is my hope that using stellar photometry during preprocessing will be copied widely, because when done correctly, it can produce extremely accurate results.

 

This thread's purpose

I would like to provide help and support to anyone using the current script, and help and advice to anyone that wishes to make use of the script's source code.

 

The script, and hence the JavaScript source code can be found here:

https://sites.google.com/view/normalizescalegradient

within NormalizeScaleGradient_2.1.1.zip

 

Regards, John Murphy

  • Like 2
Link to comment
Share on other sites

John Murphy

A user has reported the following error:

*** Error [000] ... /NormalizeScaleGradient.js, line xxxx: Error: ProcessInstance.launchInterface(): The caller process lacks entitlement to perform operation.

This occurs because PixInsight 1.8.9 prevents NSG from displaying the ImageIntegration process. To prevent this error, you need to upgrade to NormalizeScaleGradient v2.1.2

Regards, John Murphy

 

Link to comment
Share on other sites

I just installed NSG 2.1.2 and wanted to confirm a couple things. First can I use the Registered, Calibrated, Debayered and CosmeticCorrection images created by WBPP as my input images? These are from an OSC camera.

Second when I run NSG there is no column PSF Weight displayed as shown in the documentation.

How do I get that to display.  It does display noise, altitude, airmass and filter columns  but are all blank. Time column is correct. Do I need to run SubframeSelector and output Subframes?

Edited by DonPierce
Link to comment
Share on other sites

John Murphy
5 hours ago, DonPierce said:

I just installed NSG 2.1.2 and wanted to confirm a couple things. First can I use the Registered, Calibrated, Debayered and CosmeticCorrection images created by WBPP as my input images? These are from an OSC camera.

Yes, it is designed to be used with calibrated, registered images.

 

Second when I run NSG there is no column PSF Weight displayed as shown in the documentation. How do I get that to display?

PixInsight 1.8.9 no longer adds a PSF Signal FITS header entry, so NSG can only display PSF Weight data for images calibrated with PIxInsight 1.8.8-12

In NSG 2.1.2, if there are no PSF Signal values, I hide the PSFWeight column.

 

It does display noise, altitude, airmass and filter columns  but are all blank. Time column is correct.

If the data was calibrated using WBPP, I would have expected the Noise column to be populated. This column should provide a very good indication of the best reference image (least noise) and a good test image (maximum noise).

Do the images contain NOISE00, NOISE01, NOISE02 FITS headers?

 

Do I need to run SubframeSelector and output Subframes?

No, SubframeSelector will not help here

 

Regards, John Murphy

 

Edited by John Murphy
Fix typing mistake
Link to comment
Share on other sites

Thanks for the explanations.  I am sorry the Noise column does have values in them. BTW - I checked one FITS header and it contained all three Noise## entries. I will use them as instructed. I just wanted to be sure that doing debayer and CosmeticCorrection would not have any affect. I assumed not but just wanted to be sure.  Thanks for the excellent script and support on this.

Link to comment
Share on other sites

I loaded all my images and checked normalization Data, Drizzle Data, weight prefix and gradient images.

Set the Ref image and checked the ImageIntegration section.  It ran and produced the three images High and Low rejection and integration image.

For my data it appears to do a better job of matching the gradients than the latest LNorm process. That is good!

The only issue is it does not appear to have applied drizzle. It is always pretty clear to me when I drizzle integrate.

Do I need the Drizzle data from WBPP process in the same directory as the NSG output directory. That does not seem right.

In the attached image you see three images. The top left is the output from NSG, Top right is output from WBPP (Masterlight) and the bottom image is running DrizzleIntegration on the output from WBPP. I am not sure why the output from NSG has a much smoother background than the other two. I like it though! These three have had no post processing done. The Masterlight does not show as much pixelation when converted to jpg. Am I missing something?Picture1.jpg.efdfc0ad96f3ef8f2791ce72c330cabf.jpg

Link to comment
Share on other sites

John Murphy

DonPierce said:

I loaded all my images and checked normalization Data, Drizzle Data, weight prefix and gradient images.

Set the Ref image and checked the ImageIntegration section.  It ran and produced the three images High and Low rejection and integration image.

For my data it appears to do a better job of matching the gradients than the latest LNorm process. That is good!

The only issue is it does not appear to have applied drizzle.

 

NSG passes the '.xnml' normalization data files and the '.xdrz' drizzle files to ImageIntegration. The next step is to run ImageIntegration. This adds extra information to the '.xdrz' drizzle files. The final step is to run DrizzleIntegration. Use 'Add Files' to add the updated '.xdrz' files, and 'Add L.Norm.' to add the '.xnml' normalization data files.

 

Hope this helps, John Murphy

Edited by John Murphy
Clarify
Link to comment
Share on other sites

John Murphy
2 hours ago, DonPierce said:

I loaded all my images and checked normalization Data, Drizzle Data, weight prefix and gradient images.

Set the Ref image and checked the ImageIntegration section.  It ran and produced the three images High and Low rejection and integration image.

For my data it appears to do a better job of matching the gradients than the latest LNorm process. That is good!

The only issue is it does not appear to have applied drizzle. It is always pretty clear to me when I drizzle integrate.

Do I need the Drizzle data from WBPP process in the same directory as the NSG output directory. That does not seem right.

In the attached image you see three images. The top left is the output from NSG, Top right is output from WBPP (Masterlight) and the bottom image is running DrizzleIntegration on the output from WBPP. I am not sure why the output from NSG has a much smoother background than the other two. I like it though! These three have had no post processing done. The Masterlight does not show as much pixelation when converted to jpg. Am I missing something?Picture1.jpg.efdfc0ad96f3ef8f2791ce72c330cabf.jpg

 

I have answered this question here:

Let me know if you need any more assistance.

Regards, John Murphy

Link to comment
Share on other sites

I have gone back through the documentation and made a couple changes to the process. The main thing was to uncheck the auto exit. This way I can see what Image Integration is doing and then run DrizzleIntegration.  One thing I wanted to clarify, if I already ran NSG once and it updated the drizzle files when I run again will it overwrite that older update with the new updated information? I picked the wrong Ref frame last time so there will be differences.

 

Edited by DonPierce
Link to comment
Share on other sites

I've merged Drizzle question thread into this one as it is the appropriate place for discussion

  • Like 1
Link to comment
Share on other sites

John Murphy
6 hours ago, DonPierce said:

The main thing was to uncheck the auto exit. This way I can see what Image Integration is doing and then run DrizzleIntegration.  One thing I wanted to clarify, if I already ran NSG once and it updated the drizzle files when I run again will it overwrite that older update with the new updated information? I picked the wrong Ref frame last time so there will be differences.

 

If you use the 'auto exit', you can still see what ImageIntegration has done, then modify the settings and repeat. This can be done in two ways:

  • Use the History Explorer. In the 'integration' window, right click on the 'integration' tab. Select 'Load History Explorer'. Within History Explorer, expand '<Initial State>'. Now double click on its 'ImageIntegration' entry (or drag to PixInsight desktop to create a process icon).
  • Alternatively, before running NormalizeScaleGradient, create an ImageIntegration process icon. The contents don't matter - it's just a place holder for the process icon name and position on the desktop. Now run NormalizeScaleGradient, and in the ImageIntegration section, select the process icon in the 'Output icon' combo box (exactly the same process as when 'Auto exit and run ImageIntegration' is not selected).

NormalizeScaleGradient does not update the drizzle files, it just passes them on to ImageIntegration. NSG's job is to create the '.xnml' normalization data files. Hence it is perfectly safe to run NSG multiple times. It is also safe to run ImageIntegration multiple times. It will overwrite its last edit to the '.xdrz' data files.

Link to comment
Share on other sites

OK that helps. From history I know now where it grabs all the files so I can load manually. Yes I did use the Output Icon. Seems to work fine.  So I ran DrizzleInt with original WBPP xdrz files and NSG's xnml files and it worked like a dream.

Thanks for helping me through this. I got it now.

Edited by DonPierce
Link to comment
Share on other sites

This may already have been answered, if so, my apologies.

 

I have installed the new NSG and am unable to check normalization data - it is greyed out.  Guessing it's something fairly obvious that I'm missing.

 

Grateful for your help,

Marsha

Link to comment
Share on other sites

John Murphy
2 hours ago, MarshaW said:

I have installed the new NSG and am unable to check normalization data - it is greyed out.  Guessing it's something fairly obvious that I'm missing.

 

Grateful for your help,

Marsha

The ability to create '.xnml' normalization data files can only be done with a C++ module. To do this, NormalizeScaleGradient uses the optional NSGXnml module. The script runs this in the background to create these data files. For more details, see:

https://sites.google.com/view/normalizescalegradient

Link to comment
Share on other sites

I really hope I'm just doing something wrong, the same workflow worked fine for me before updating to PI 1.8.9 and NSG 2.1.2 but now the XNML files output by NSG cause errors trying to integrate. WBPP then NSG then running the ImageIntegration produced by the process icon: *** Error: Incompatible local normalization data version. Expected >= 1, got 0: Y:/PI/C7/in1/stack/LIGHT_Red_60.00s_-5.00C_R277.20_G56_B1x1_O25_High Gain Mode_USB0_2022-03-17_21-32-05_c_cc_r_nsg.xnml <* failed *>

spacer.png

Link to comment
Share on other sites

Well I followed the instructions for doing the repository and it did add the NSG script and process. But I lost WBPP.

I tried updating, ect and nowhere to be found. Is there a way to get it back from the Update menu?

Link to comment
Share on other sites

John Murphy
18 minutes ago, DonPierce said:

Well I followed the instructions for doing the repository and it did add the NSG script and process. But I lost WBPP.

I tried updating, ect and nowhere to be found. Is there a way to get it back from the Update menu?

Which OS are you using? We both tested the repository, but I will test again.

To get WBPP back:

SCRIPTS > Feature Scripts...

Select 'Add' and navigate to the PixInsight scripts folder. For example, on Windows this is:

C:\Program Files\PixInsight\src\scripts

It should find 1 script (WBPP)

Select 'Done'

Link to comment
Share on other sites

John Murphy
1 hour ago, OhmEye said:

I really hope I'm just doing something wrong, the same workflow worked fine for me before updating to PI 1.8.9 and NSG 2.1.2 but now the XNML files output by NSG cause errors trying to integrate. WBPP then NSG then running the ImageIntegration produced by the process icon: *** Error: Incompatible local normalization data version. Expected >= 1, got 0: Y:/PI/C7/in1/stack/LIGHT_Red_60.00s_-5.00C_R277.20_G56_B1x1_O25_High Gain Mode_USB0_2022-03-17_21-32-05_c_cc_r_nsg.xnml <* failed *>

spacer.png

PI 1.8.9 changed the '.xnml' data file format. The files are not backward compatible. Unfortunately, this means that if you created one or more of the '.xnml' files before 1.8.9, they cannot be used within 1.8.9 

 

I had to make a minor change to the NSGXnml C++ module (the API had changed as well). It is important that the correct NSGXnml version is installed. If this error occurred with data that you created in 1.8.9, I would recommend uninstalling NSGXnml, and reinstalling the PI 1.8.9 NSGXnml version.

 

To uninstall NSGXnml:

  • PROCESS > Modules > Manage Modules...
  • Scroll down to the NSGXnml entry and deselect it.
  • Press 'Done'.

To install it:

  • Unpack the compressed .zip / .tar.xz file.
  • Copy NSGXnml-pxm.dll (windows) / NSGXnml-pxm.dylib (mac) / NSGXnml-pxm.so (linux) to any folder (but don't use a PixInsight folder)
  • In the PixInsight PROCESS menu, select:
  • PROCESS > Modules > Install Modules...
  • Browse to the folder. Note that the folder contents are not displayed.
  • Press 'Search'. A dialog should display '1 additional PixInsight module(s) were found...
  • Press 'Install'

If you are installing on a Mac, See the Extra note for Mac OS on the https://sites.google.com/view/normalizescalegradient website before installing.

Link to comment
Share on other sites

18 minutes ago, John Murphy said:

PI 1.8.9 changed the '.xnml' data file format. The files are not backward compatible. Unfortunately, this means that if you created one or more of the '.xnml' files before 1.8.9, they cannot be used within 1.8.9

TLDR; I somehow mistakenly installed an old version of NSGXnml yesterday. Installing the current version fixed the issue. Thanks for your help and sorry I didn't notice this myself!

 

These are created on 1.8.9, I ran the WBPP on fresh data acquired last night and I'm even manually deleting the XNML files in my NSG output folder between runs. Looking at the XNML file output by WBPP there is a XML tag <LocalNormalization scale="512" version="1"> that the NSG output is missing the version field from. The XNML output during the NSG run looks like <LocalNormalization scale="128">

 

By manually editing the NSG XNML to be <LocalNormalization scale="128" version="1"> the error does not occur for that file.

 

I installed NSGXnml and re-registered it after installing PI 1.8.9, I did it again just now, perhaps I mistakenly installed an old version. I also updated NSG to 2.1.4 using the repo just now and running the current versions the error does not occur.

Link to comment
Share on other sites

Looks like NSG has just updated to 2.1.4.

It left the NSG2.1.2 version in the scripts list but I just updated using the Feature script routine

All seems fine on my Windows 10 platform.

Link to comment
Share on other sites

Mark de Regt

Hi,

 

I'm struggling a bit to get NSG working as it did before PI 1.8.9.

Specifically, I liked the way NSG automatically chose the clipping routine, and opened up image integration when done; then I had to tell image integration to start (but I could check settings first).

I can't figure out how to do exactly that now.  I can get it to go all the way through if I choose a process icon for it, but it won't stop.

 

Yes, I know that I can do the same thing manually, by writing the NSG files to a folder, and then opening up the image integration process; but I thought it was slick the way it used to be.

 

Any hints would be gratefully taken.

 

Thanks.

 

Mark

Edited by Mark de Regt
Link to comment
Share on other sites

John Murphy

Unfortunately changes were made to PI 1.8.9 to prevent NormalizeScaleGradient from displaying ImageIntegration.

 

The NSG 2.1.4 script automatically detects the PI version, and modifies its user interface accordingly; a new 'Output icon' combo box is added. To use this, start ImageIntegration and drag its blue triangle onto the PI desktop to create an ImageIntegration process icon. The contents of this process icon do not matter - all its data will be ignored. It is just a placeholder to specify the process icon's name and position on the desktop.

 

Now use NormalizeScaleGradient, and set the 'Output icon' to this placeholder ImageIntegration process icon. When you exit NSG, the contents of the ImageIntegration process icon will be replaced with all the correct settings. Now double click on the process icon to display ImageIntegration. Check the auto generated settings and then run it.

 

Alternatively, if you select 'Auto exit and run ImageIntegration', you can then use the 'History Explorer' to discover the settings used:

 

PI will display the stacked result in a new PI window. For example 'integration'. Right click on the 'integration' tab on the left margin of the window, and select 'Load History Explorer'. Expand the History Explorer tree, and double click on the 'ImageIntegration' entry. The ImageIntegration process is displayed.

 

I hope this helps,

John Murphy

Edited by John Murphy
Latest NSG version is v2.1.4
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...