Unstable download speed

[Terry]

I've been a NewsLeecher user for years. Good program but lacking decent filters. So right now I'm testing UE.
Filters work great. (Thanks for the great support by email Alex!)
I still have to get used to the UE interface but that's no problem.
What bothers me most right now is the download speed.
I don't have a very fast connection so every bit of extra speed is welcome.
UE and NewsLeecher the max download speed is the same.
In NewsLeecher my download speed was *very* stable. (1.2MB/s)
But unfortunately not so in UE. Quite often speed drops to 0.8MB/s. I've even seen extremes of 0.3MB/s.
I used 6 threads in Newsleecher and tested 6, 7 and 8 threads in UE (8=max allowed by my provider)
So I'm wondering what's going on.
a] UE speed counter isn't accurate.
b] Newsleecher speed counter isn't accurate.
c] UE configuration issue.

For testing purposes I've disabled my NOD32 virusscanner.

[Terry]

I've tested a few more things and none of them make any difference.

- Firewall on/off
- Anti-virus on/off
- SSL on/off
- Tried many different file sets
- 5-8 threads (8 is my maximum)
- Download sizes all > 700MByte



UE CPU usage: 1-2%

I'm not headering. I start a download using UE-search or NZB files.

[Terry]

I think I located/solved the issue.

So you can all stop replying now.
(just kidding)

[alex]

what it was? antivirus?

[Terry]

Not entirely sure...

I'm ripping a website with Teleport Pro. It uses only 50-100KByte/s so it can't account for 600KB/s speed drops.
But when I reduce the number of threads in Teleport the UE speed gets much more stable.
Although I can find nothing in my logs I think that possibly the threads of UE and teleport combined make the firewall "think" it's under attack.

I've seen that UE is possible of the max. speed so that's good.
And as expected because my max speed is only 1.3MByte/s.
Rather slow that's why I so desperately want to squeeze out the last bit of speed.


I vaguely remember Win XP had a registry hack to increase the possible number of connections (svchost?). Not sure Win 7 has similar hacks.....?

[Terry]

Hummm, I wrote a lot but didn't answer your question.


During tests I disabled NOD32. I also put the UE program files directory in the exclusion list.
I also configured UE as safe in my software firewall. Plus I switched it off.
So in theory they shouldn't interfere.
Do you have suggestions for other directories/processes I should exclude?

Anyway switching them on or off doesn't seem to make a lot of difference.

[alex]

There is an option edit menu->properties->tasks, at the middle bottom -
"Network threads - system priority" try to change it to above normal, check if there is a difference. Or to push it even more just to test you can increase the UE process system priority in windows task manager to High.

If not above edit menu->properties->general, you may try to uncheck keep connections alive, although normally better to keep it checked, also look if you notice any difference (e.g. slower but speed is more stable).

Maybe it makes sense to measure speed with external bandwith meter, especially if in UE you see spikes faster than the speed of your connection.

Is the total system CPU usage low?

[Terry]

UE task priority was at High or Realtime during all tests.

I'm currently monitoring speed with NetLimiter. That's the graph you see. I'll update this post as test progress.


The article not found errors make the speed drop to almost zero. Sounds obvious because with nothing to download there can't be downloading speed

ue1.gif (155.71 KiB) Viewed 20469 times

UE reports this set to be 96% complete.

ue2.gif (16.54 KiB) Viewed 20469 times

QuickPar shows a far worse completeness (I estimate 20%)

UE3.gif (82.71 KiB) Viewed 20469 times

To be continued

[alex]

If you reduce priority to normal with network threads also normal and the threads of the website download program running you see something different? If it doesn't work well UE normal and threads above normal does work?

Or it is just UE bandwidth counter, but actually it takes average from two intervals so it shouldn't be too jumpy.

[Terry]

The above pictures are taken without the website download program running.
I think it's best for testing; otherwise I never can be sure what causes the problems.
So during tests I won't have any download program running besides UE.

I did several downloads after I stopped using that program and UE's download speed is usually very stable and the maximum of my possible download speed.
The tiny spikes in the graph posted are missing articles I think.
The large gaps with almost no download speed is when UE is trying to download *many* previously failed articles.
I've just finished a series of downloads and have 4 sets that are extremely incomplete.
To make very sure the problem isn't with incomplete uploads on my news server I am going to do some test with another Usenet client, other nzb source, and using the UE internal search engine.
Then I'll report back.

[alex]

Missing articles should be slower, the news server does search for message-id, but also UE has clock which may contribute to slowness, I didn't specifically test, I suspect the server would contribute more of delay, as to UE it can be tested by increasing UE clock frequency in the code (need to recompile for that). On the screenshot if 8 tasks and I'm not mistaken the news server contributed >90% of delay.

[Terry]

The i-lynpcs6fpnf set I imported from an .nzb was so extremely damaged QuikPAR didn't even recognize the .par files.
I re-downloaded the files by using your search engine and everything was 100% complete. (no repair needed)
It shows the files are complete on my usenet server. And of course the quality of your search service. But you already knew that
Next test I did was downloading the exact same files using the same .nzb using NewsLeecher. An extremely incomplete result.
Next NewsLeecher test with an .nzb from another source. Again very incomplete.
Final NewsLeecher test with an .nzb fron yet another source. 100% complete.
Used that .nzb with UE and got a 100% complete.
So it seems I got two highly corrupted .nzb files....

It looks like I had extremely bad luck with many corrupt (this and several other file sets) .nzb files.
And using the search service is the easiest solution to fix the problem.

Question: In the above "repair scenario" does UE download every file found using the search service or does is only download the missing parts?
When I used NewsLeecher I sometimes really needed to "help" it with getting things complete. It doesn't have a very smart repair code.
So I usually went looking for other .nzb sources and only downloaded the damaged RAR files. And usually the files that are most damaged.
Doing things that way requires the least downloading because all that is needed is extra repair blocks for QuickPAR. Some files provide 0, some 2 and other maybe even 10 blocks.

[alex]

If you import from search service UE merges the result with the headers imported from the nzb file. If the same message-id or subject it will be shown as the same header.

So if you just import the result will be complete.

If it downloaded&saved and then deleted article bodies (when edit menu->properties->save, bypass database storage is enabled) it writes directly into file), UE will redownload those again.

If you want you set UE not to delete article bodies (edit menu->properties->save) after save, but then you need to delete them manually later, with the lower download speed the performance impact won't be very noticeable. Then it will be saving without downloading again. Then UE will keep copies of article bodies in its database, so watch its size (edit menu->bodies to see them all).

[Tha*Lunat!k]

With regard to the download speed, my guess is that the website ripping program opens a large number of simultaneous connections, and while it may not use much bandwidth overall it's still hogging network resources by employing a lot of inbound and outbound connections and connection attempts. Reminds me of when I am playing a game online on my Xbox, if my PC is downloading from torrents my game's latency will suffer greatly even when the torrents are using only a fraction of my bandwidth.