NormalizeScaleGradient Script

[chrisastrophoto]

19 minutes ago, TerryMcK said:

You may have already seen this but googling seems to throw up a lot with that error https://answers.microsoft.com/en-us/windows/forum/all/how-to-resolve-error-1450-insufficient-system/ede7ec19-138a-4840-bf36-593e2b42b0d0

Saw that, but did not yet a repair upgrade of Windows. All other suggestions I found were tweaking the registry - which did not help.

Did a few more tests: Run NSG only half way just up to 120 files and aborted, but left PI running. Private bytes dropped a bit. Then started NSG again with the same set of files, private bytes went up again to the value before the abort and continued to go up. PI crashed after 30 files then. It happens exactly when PI reaches 4GB private bytes, the magic number for a 32bit OS. But I have a 64bit OS and PI is a 64 bit application, right? I did all checks for checking OS health, all came out with "no problems found". All other applications on this computer have no problem.

Are we back to Win95 era, when we had to repair and re-install Windows again and again?

This is ridiculous ... maybe it is not related to NSG but to PI in general, but as I said, a repair of PI did not help either.

[TerryMcK]

Might be something that could be asked on the PI forum. However as you may have seen earlier in the thread “we do not support 3rd party” seems to come up when you mention what they call non “in built” plugins/scripts.

[AstroAv]

6 hours ago, chrisastrophoto said:

Hi, I am having a strange memory problem with NSG on one particular computer.

The computer is running Windows 11 on a amd Ryzen 9 5900X, 96GB RAM and a RTX GeForce 3090.

I didn't use it with PixInsight for some weeks and when I try now to run NSG on about 250 registered files (300MB each) it crashes PI after about 150 files. Running several times, it does not crash on the same file. It does sometimes on reading a file and sometimes on writing a file. The common error is: 

 Error: File I/O Error: Unable to create file: Win32 error (1450): Insufficient system resources exist to complete the requested service.

Sometimes it continues a while and crashes finally with an access violation.

Process Explorer shows me that there must be a memory leak in the PI process: Private Bytes of the PixInsight Process increase after each processed image continuously until it reaches approximately 4GB. Then it suddenly jumps up to 7GB and PI crashes as described. (see Screenshot)

 

Doing exactly the same on the same registered files on a different computer with Windows 11 on a i7-8750H, but 16GB RAM and a somewhat smaller graphics adapter (a laptop) everything works fine. NSG comes to an end, there is NO observable memory leak in the PixInsight process ... but of course everything is much slower.

 

Both computers have the same patch level of Windows 11, same version of PI and NSG script.

 

The resolutions for this error proposed on the internet (tweaking the registry under in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management on PagedPoolSize and PoolSizeMaximum) do not help.

 

Any idea?

Thanks for posting this - I've been having the same problems today.

I haven't done much processing in January. I opened PI today, a bunch of scripts updated, and now NSG is crashing frequently.

Everything else works, and there was no problem before the updates when I last used PI+NSG a few weeks ago.

 

There is definitely a memory leak somewhere, I'm just not sure where because I've never tried profiling PI before.

[John Murphy]

I am investigating to see what can be done.

I am also creating a new NSG version:

• Improved error handling.
• Ability to continue a partially completed run. This can be used after adding extra images to the target list, after a user Abort, or after a failure.
• The file I/O code has changed from using ImageWindow to read the images, to lower level PixInsight methods - FileFormatInstance. On my data, this has provided a 10% performance boost.
• Reusing memory buffers to reduce the amount of work the memory garbage collector has to do.
• The ability to run NSG from a ProcessContainer. This would allow multiple NSG jobs (for example L,R,G,B) to be run overnight.

Regards, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

[chrisastrophoto]

15 hours ago, TerryMcK said:

Might be something that could be asked on the PI forum. However as you may have seen earlier in the thread “we do not support 3rd party” seems to come up when you mention what they call non “in built” plugins/scripts.

Yes I guess so, they are very picky with 3rd party  ... Thanks anyway for your suggestions!

[chrisastrophoto]

14 hours ago, AstroAv said:

Thanks for posting this - I've been having the same problems today.

I haven't done much processing in January. I opened PI today, a bunch of scripts updated, and now NSG is crashing frequently.

Everything else works, and there was no problem before the updates when I last used PI+NSG a few weeks ago.

 

There is definitely a memory leak somewhere, I'm just not sure where because I've never tried profiling PI before.

I am not the only one then ...

I pretty sure that it is an NSG thing. All other processes and scripts in PI run normally. In many cases PI uses far more than 4GB private bytes. Really annoying ... invested quite some money in a powerful computer and now I have to copy files to my old slow laptop to run NSG and copy the results back to my power machine ...

[AstroAv]

8 hours ago, John Murphy said:

I am investigating to see what can be done.

I am also creating a new NSG version:

• Improved error handling.
• Ability to continue a partially completed run. This can be used after adding extra images to the target list, after a user Abort, or after a failure.
• The file I/O code has changed from using ImageWindow to read the images, to lower level PixInsight methods - FileFormatInstance. On my data, this has provided a 10% performance boost.
• Reusing memory buffers to reduce the amount of work the memory garbage collector has to do.
• The ability to run NSG from a ProcessContainer. This would allow multiple NSG jobs (for example L,R,G,B) to be run overnight.

Regards, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

This is honestly amazing, I'm so excited!!

This addresses multiple things I've been wanting to ask for - resuming after failure, and parallelism. I just kept forgetting to ask 😄

 

FWIW after the crash I get, I do see it say that ImageWindow is undefined. I don't think this is the cause necessarily, I assume various objects are destroyed when this occurs and ImageWindow is one of them. Maybe your changes will bypass whatever it going on there.

 

This sounds amazing and I can't wait! Do you know when we should expect the new version?

[Roger Nichol]

Glad to see you are working on a fix for the new 'run out of system resources' error that crashes PI after running more than 90 or so images through NSG. Any idea of timeframe to fix and release?  

[John Murphy]

I have been working really hard on this, and a new version of NSG is nearly ready. It will probably be released in a couple of weeks time, as I have more testing to do, and a few extra improvements to add.

 

If you would like to try the beta version, send me a message.

 

Regards, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

 

[Bruce D.]

I've been using NSG for several months with no problems.  Tonight, I tried running 155 Ha registered frames through it but at the 100th frame, it stops and PI completely shuts down.  When I reopen PI, I can see the 99 files in the NSG directory and the 100th one was 84% being normalized before it crashed.  This happened 3 times and I did shutdown and reboot my computer.  I can't find anything wrong.  Any ideas what would cause this?

 

Bruce

[Bruce D.]

Here is the part in the log where it stops and shuts down.....

 

[Bruce D.]

I ran it a fourth time and again, it stop at the 100th frame but this time it gave an error before shutting down.....  

It says insufficiente resources, but I don't know why.  I have plenty of disk space and memory.  These are 155 full frame images.

 

 

Edited February 1, 2023 by Bruce D.

[John Murphy]

Hi Bruce,

I am still trying to determine what is causing this problem, but the next version of NSG will at least handle these failures much more gracefully. The error handling has been rewritten, which should prevent a crash. Then, after restarting PixInsight, NSG can continue the run. It knows to process all the failed and unprocessed images. It also processes the last successful result, even though it completed without any errors - just to be on the safe side. Not ideal, but it will work.

 

The new version of NSG will be available within a couple of weeks. If you would like to try the new version before that, let me know and I will send you a link.

 

Looking into this problem has been tricky because it is very hardware dependent. It appears to only affect high spec machines (which I can't afford!) So far:

• I think it only affects Windows. I am not aware of any failures on Linux or Mac.
• It is not due to memory exhaustion. NSG does not use much memory. The new, about to be released, version uses even less.
• I have completely rewritten the File I/O code within NSG, changing from using the PixInsight ImageWindow object, to using PixInsight's lower level FileFormatInstance object. This has prevented a crash, but has not eliminated the problem. Note that NSG does not read or save images directly - it always uses PixInsight methods to do this.
• I have been very careful to make sure that NSG does not leak file resources. For every file open, I am very careful to make sure there is also a close.
• The same error does very occasionally happen when running WBPP. This might indicate that the root cause is not NSG. See:

https://pixinsight.com/forum/index.php?threads/wbpp-crashes-pi-with-an-out-of-resource-error-message.20232/

 

Information that would be useful includes:

• What number of logical cores does the PC have?
• Global Preferences: How many 'Swap storage directories are you using? (I am using 4):

• Global Preferences: how many file reading and writing threads are you using?

 

Regards, John Murphy

[Bruce D.]

Hi John

 

Thank you for the explanation.  My PC has 32 cores and I am using 8 swap storage directories. 

 

AMD Ryzen Threadripper 3970X 32-Core Processor    3.70 GHz

 

Bruce

[John Murphy]

1 minute ago, Bruce D. said:

Hi John

 

Thank you for the explanation.  My PC has 32 cores and I am using 8 swap storage directories. 

 

AMD Ryzen Threadripper 3970X 32-Core Processor    3.70 GHz

 

Bruce

Thanks. How many file reading / writing threads are you using?

 

Regards, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

[Bruce D.]

I'm not sure how to locate that information. 

[John Murphy]

6 minutes ago, Bruce D. said:

I'm not sure how to locate that information. 

Hi Bruce,

It took me a while to locate it too!

• On the EDIT menu, select Global Preferences
• Then select Parallel Processing and Threads:

 

Thanks, John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient

Edited February 1, 2023 by John Murphy

[Bruce D.]

Sorry but I can't even find the EDIT menu.  I am looking under Settings/Systems.   

[TerryMcK]

Hi Bruce it is on the EDIT menu, select Global Preferences then go down to Parallel Processing and Threads

 

 

Here are my settings

[Bruce D.]

What is an EDIT menu? Where is it?

[John Murphy]

[Bruce D.]

OK- I completely misunderstood your request.  I was looking in Windows 10 Explorer, not in PI which was closed.

 

[John Murphy]

15 minutes ago, Bruce D. said:

OK- I completely misunderstood your request.  I was looking in Windows 10 Explorer, not in PI which was closed.

 

 

OK, that's interesting. That it failed for you when only using 1 thread for reading and writing files probably rules out this as being a contributing cause.

 

I wonder if it is related to the number of cores being used. As a test, would you be able to test to see if it still fails if you limit the number of cores used to 8?

 

I think you might need to use the blue circle 'Global apply' and then restart PixInsight to make sure the setting takes.

 

Thanks, John

Edited February 1, 2023 by John Murphy

[Bruce D.]

sure- I should be able to test it later today or this evening and get back to you

[John Murphy]

23 minutes ago, Bruce D. said:

sure- I should be able to test it later today or this evening and get back to you

Thanks, really appreciated.

 

John Murphy

https://www.normalizescalegradient.net/

https://www.youtube.com/@NormalizeScaleGradient