I often upload nzb files through remote admin software into the import folder when I’m away from home. When the connection is very slow sometimes UE will import the nzb file before the upload is complete resulting in an incomplete download.
I request a setting where one can tell UE how often to check the Import folder, like every minute, or every 5 minutes or 10 minutes, counting from on the hour.
Feature request: Import folder scanning interval setting
Re: Feature request: Import folder scanning interval setting
When you see the issue happening again - can you check in the "old" subfolder of the auto import folder, that the nzb file is complete? You can check by opening it in editor and looking at the end of the file or by importing it from there manually and verifying if anything new was added.
After UE imports the file - it moves it to the "old" subfolder. UE should not be able to move the nzb file while it is being uploaded, since it is open by another application.
So the nzb file should remain in the import folder and repeatedly being imported in such a case, there may be some very low probability sync issue, but the chance of it should be negligible (then you should see the complete nzb file in the "old" folder along with incomplete auto import before UE moved the file).
I can change it by adding a flag not to import if the file is open for writing by another application, unless it is already engaged, in any case there is no need to add a delay in import. I will check it before the next release or if I have your further feedback.
If the transfer was interrupted and the file wasn't uploaded completely (I think it is what likely happened) - you can just upload it again.
But let me know if there is some problem with the above or I missed something.
After UE imports the file - it moves it to the "old" subfolder. UE should not be able to move the nzb file while it is being uploaded, since it is open by another application.
So the nzb file should remain in the import folder and repeatedly being imported in such a case, there may be some very low probability sync issue, but the chance of it should be negligible (then you should see the complete nzb file in the "old" folder along with incomplete auto import before UE moved the file).
I can change it by adding a flag not to import if the file is open for writing by another application, unless it is already engaged, in any case there is no need to add a delay in import. I will check it before the next release or if I have your further feedback.
If the transfer was interrupted and the file wasn't uploaded completely (I think it is what likely happened) - you can just upload it again.
But let me know if there is some problem with the above or I missed something.
Re: Feature request: Import folder scanning interval setting
As soon as I get a bad connection again I will check what you asked.
The failed connection scenario seems unlikely as of a batch upload of little over half a dozen nzb's all of them were only partial imports and it was all uploaded in one slow upload.
The failed connection scenario seems unlikely as of a batch upload of little over half a dozen nzb's all of them were only partial imports and it was all uploaded in one slow upload.
Re: Feature request: Import folder scanning interval setting
Did you try to manually reimport those nzb files from the "old" subfolder to see if they are complete? They should be still there.
Re: Feature request: Import folder scanning interval setting
I ran some tests with very interesting results.
I uploaded large nzb files over a very slow connection into the import folder with UE running.
The nzb in the Import/Old folder is the same size as a copy of the original nzb, but it is incomplete as the majority of the imported nzb in the Import/Old folder is filled with zero's...
This confirms my hypothesis that UE imports the nzb while the nzb is still open for writing, imports what data is available and then already moves the (incomplete) nzb to the OLD folder. Though in fact it copies the nzb as it is available (with mostly zero's) to the Import/Old folder and tells Windows to delete it as soon as it will allow it. So Windows waits till the upload is finished and then deletes the nzb even though most of its data is not imported into UE.
I repeated this test 3 times with different downloads, 3 times the same result.
UE doesn't wait for the nzb upload to be completed, it imports what it has and moves (copies) the incomplete nzb tot the Import/Old folder and deletes the nzb once the upload is finished, losing much data in the process.
I uploaded large nzb files over a very slow connection into the import folder with UE running.
The nzb in the Import/Old folder is the same size as a copy of the original nzb, but it is incomplete as the majority of the imported nzb in the Import/Old folder is filled with zero's...
This confirms my hypothesis that UE imports the nzb while the nzb is still open for writing, imports what data is available and then already moves the (incomplete) nzb to the OLD folder. Though in fact it copies the nzb as it is available (with mostly zero's) to the Import/Old folder and tells Windows to delete it as soon as it will allow it. So Windows waits till the upload is finished and then deletes the nzb even though most of its data is not imported into UE.
I repeated this test 3 times with different downloads, 3 times the same result.
UE doesn't wait for the nzb upload to be completed, it imports what it has and moves (copies) the incomplete nzb tot the Import/Old folder and deletes the nzb once the upload is finished, losing much data in the process.
Re: Feature request: Import folder scanning interval setting
It is weird, since if you try to rename or move a file open for reading or writing you will get an error.
I'll just change permissions then, so it won't be able to open files which are open (if possible - only for writing).
I'll just give you a test version later to see if it works, I'll send you pm when it is ready.
I'll just change permissions then, so it won't be able to open files which are open (if possible - only for writing).
I'll just give you a test version later to see if it works, I'll send you pm when it is ready.
Re: Feature request: Import folder scanning interval setting
If UE is already set not to import files that are being written to then Windows might not see a file that is being written to through Remote Desktop as being 'open' in the the eyes of Windows.
Edit:
I just tried to do it myself. I can delete a file in Windows that is being written to/uploaded to it remotely (I use Radmin), Windows will NOT give an error message when I delete it, but when checking upon completion the file is still there and mostly zero's. So it DOES delete the contents of the file and fills it with zero's.
Edit:
I just tried to do it myself. I can delete a file in Windows that is being written to/uploaded to it remotely (I use Radmin), Windows will NOT give an error message when I delete it, but when checking upon completion the file is still there and mostly zero's. So it DOES delete the contents of the file and fills it with zero's.
Re: Feature request: Import folder scanning interval setting
If the file can be deleted, whatever flags are used it will not work.
It looks like radmin creates the file space and then uses physical writes into device bypassing the file system.
Is it possible to turn this "feature" of radmin off? It is very unusual to write the files without opening them.
The windows rdp client locks the files as expected. Maybe just to use it instead?
In UE settings you can clear the nzb import folder location, effectively disabling import temporarily, then you press "browse" and add it again, see the relevant settings.
Or you can upload to a different folder and then move the files into the nzb import folder. Does radmin allows you to issue a move command (or run a .bat file) after the upload completion?
The delay you proposed would be needed to be counted starting after the file has been discovered in the import folder, importing at an interval wouldn't be a solution.
It looks like radmin creates the file space and then uses physical writes into device bypassing the file system.
Is it possible to turn this "feature" of radmin off? It is very unusual to write the files without opening them.
The windows rdp client locks the files as expected. Maybe just to use it instead?
In UE settings you can clear the nzb import folder location, effectively disabling import temporarily, then you press "browse" and add it again, see the relevant settings.
Or you can upload to a different folder and then move the files into the nzb import folder. Does radmin allows you to issue a move command (or run a .bat file) after the upload completion?
The delay you proposed would be needed to be counted starting after the file has been discovered in the import folder, importing at an interval wouldn't be a solution.
Re: Feature request: Import folder scanning interval setting
RDP file transfer is only possible from the remote desktop afaik, the same desktop I (too often) cannot get into.
The problem is that internet connections over the Atlantic or through Africa or wherever I am (pilot) are too slow sometimes to RDP into my download PC, even with all the eye candy turned off and with minimal screen size. I am then not able to manipulate the desktop for downloading or moving nzb's around and thus can only hope to send nzb's to it remotely though programs like Radmin.
As it is now, sometimes I have to keep trying a long time to get into RDP just so I can exit UE so it won't scan the import folder while uploading nzb's. After the nzb's upload I (try to) RDP again so I can start UE again, but at times this is not possible, sometimes due to port blocking, and then I have to Radmin the nzb's and gamble which always results in the larger downloads failing due to this exact issue.
RDP file transfers are also ridiculously slow due to some encryption or overhead or whatever, slower than it has to be for mere nzb's, but that's a minor irritant. If I could RDP and use the desktop to RDP nzb's I could also just exit UE and Radmin the nzb's to it and this issue wouldn't exist. But the idea is to make large and fast downloads possible while completely removed far far away from my download PC, except for a thin internet straw to send nzb's through.
Radmin is perhaps old and very rudimentary (so much that it doesn't obey Windows rules I guess), but it works fast and well... just not with a thin internet straw and UE running. Since UE won't be able to see the nzb as being open, and an import interval is not in the cards, I'm looking into alternate software that can do the same as Radmin but locks the nzb as per Windows rules. Suggestions? (I'd prefer not to go FTP but keep it simpler).
The problem is that internet connections over the Atlantic or through Africa or wherever I am (pilot) are too slow sometimes to RDP into my download PC, even with all the eye candy turned off and with minimal screen size. I am then not able to manipulate the desktop for downloading or moving nzb's around and thus can only hope to send nzb's to it remotely though programs like Radmin.
As it is now, sometimes I have to keep trying a long time to get into RDP just so I can exit UE so it won't scan the import folder while uploading nzb's. After the nzb's upload I (try to) RDP again so I can start UE again, but at times this is not possible, sometimes due to port blocking, and then I have to Radmin the nzb's and gamble which always results in the larger downloads failing due to this exact issue.
RDP file transfers are also ridiculously slow due to some encryption or overhead or whatever, slower than it has to be for mere nzb's, but that's a minor irritant. If I could RDP and use the desktop to RDP nzb's I could also just exit UE and Radmin the nzb's to it and this issue wouldn't exist. But the idea is to make large and fast downloads possible while completely removed far far away from my download PC, except for a thin internet straw to send nzb's through.
Radmin is perhaps old and very rudimentary (so much that it doesn't obey Windows rules I guess), but it works fast and well... just not with a thin internet straw and UE running. Since UE won't be able to see the nzb as being open, and an import interval is not in the cards, I'm looking into alternate software that can do the same as Radmin but locks the nzb as per Windows rules. Suggestions? (I'd prefer not to go FTP but keep it simpler).
Re: Feature request: Import folder scanning interval setting
Ok I'll try to add the opening file delay in seconds edit field in the import settings.
Re: Feature request: Import folder scanning interval setting
Excellent! I had to renew my subscription anyway, so I did! 
You could also list it as a valued feature, to be able to download completely off-site.
In the future perhaps certain configuration settings might even be manipulated remotely with uploaded config files/switches.
But I'm just spitballing here.
Appreciate it.
You could also list it as a valued feature, to be able to download completely off-site.
In the future perhaps certain configuration settings might even be manipulated remotely with uploaded config files/switches.
But I'm just spitballing here.
Appreciate it.
Re: Feature request: Import folder scanning interval setting
Try this version if it works:
http://www.usenetexplorer.com/temp/ue56a32.zip
There is delay setting in the import properties per import folder, the default is zero.
The actual time may be up to 5 seconds more since the import folder scan is every 5 seconds.
http://www.usenetexplorer.com/temp/ue56a32.zip
There is delay setting in the import properties per import folder, the default is zero.
The actual time may be up to 5 seconds more since the import folder scan is every 5 seconds.
Re: Feature request: Import folder scanning interval setting
I’ll give it a try to see if the system works but just an extra 5 second delay is not going to make much difference. When the connection is really slow a UHD movie nzb of +-25MB might take more than 30 seconds to upload. Can’t you just make the delay up to a minute?
Given the good compressibility of nzb’s I do have another suggestion: allow nzb’s uploaded in the Import folder in zip or rar format. This will reduce the size by up to 90%.
Given the good compressibility of nzb’s I do have another suggestion: allow nzb’s uploaded in the Import folder in zip or rar format. This will reduce the size by up to 90%.
Re: Feature request: Import folder scanning interval setting
In the version above you can set any delay in seconds in edit menu->properties->search/import, nzb files, import folder, then select the import group which is associated with the file drop folder and edit the delay. If you want 2 minutes to be sure just enter 120.
What I meant UE scans import folders every 5 seconds, when it scans it will examine files taking into account the delays, so some files will get imported if their time came and some will be skipped to be imported later, since they were discovered in the folder less than the delay amount of seconds ago.
Just check it it works as advertised.
What I meant UE scans import folders every 5 seconds, when it scans it will examine files taking into account the delays, so some files will get imported if their time came and some will be skipped to be imported later, since they were discovered in the folder less than the delay amount of seconds ago.
Just check it it works as advertised.
Re: Feature request: Import folder scanning interval setting
I understand now and will get right on it.
Appreciate it!
Appreciate it!