I am getting out of memory messages and crashes from UE. Either when getting new messages or when trying to compact and free binary newgroups.
I'm running Vista 64 bit with 8gb memory - so it's not a shortage of physical memory, but something in the way that UE is trying to allocate memory.
I also get them if I try and open too many newsgroups concurrently, when it seems to crash out at around "~900MB Allocated".
Is there any way to configure UE so that it can use more memory?
UE memory limit
I have no experience with UE + 64bit so my suggestion is generic.
The first suggestion is to make sure that any binary newsgroups are of type "Compact binary" so they use a minimum of memory.
Next suggestion is to limit retention that UE keeps for big newsgroups, also to minimize memory usage.
jonib
The first suggestion is to make sure that any binary newsgroups are of type "Compact binary" so they use a minimum of memory.
Next suggestion is to limit retention that UE keeps for big newsgroups, also to minimize memory usage.
jonib
The newsgroups are all set as "compact binary".
I don't want to cut the retention levels and don't believe I should have to as I believe that this is either a bug or design limitation in the current version of UE.
I am asking if there is any way to stop memory problems without reducing the retention.
It's my observation that older versions of UE used to suffer memory problems far less frequently than more recent versions - but its only anecdotal - no evidence to back it up.
Griff
I don't want to cut the retention levels and don't believe I should have to as I believe that this is either a bug or design limitation in the current version of UE.
I am asking if there is any way to stop memory problems without reducing the retention.
It's my observation that older versions of UE used to suffer memory problems far less frequently than more recent versions - but its only anecdotal - no evidence to back it up.
Griff
in 64 bit there is 4GB per process (not even 3GB as WinXP/Vista 32 bit), how much is memory usage of the UE process?
you should be able to open hundreds of millions of headers (with other programs the limit usually 5M to 12M per group when the program is still usable even if it is 64 bit compilation so UE is many times better in this regard).
you can cause out of memory just by opening groups without any other activity? (say if you disabled header overload option in properties->tasks and connection is very fast you can just fill memory with unprocessed headers).
how many of headers total, what retention like 240 days boneless?
did you limit the group retention at all (not natural?) ?
in UE the database capacity alone is 4GB so database cannot be the cause (the memory will run out first), with 4GB i think 3.6GB can be allocated which would give memory usage 2.7GB per heap memory allocations which exceeds 3 times the database size, somewhat strange.
when you start the program how many MB is allocated (UE status bar) and what is the process memory usage (in the windows task manager). then, say, you open a large (compact binary) newsgroup, right after opening how many MB allocated and what is the process memory usage then?
i checked here on smaller volumes (tens of millions of headers) it appears memory usave approximately is the double of allocation size (allocation size is included into memory usage). with 10 millions of headers taking around 46MB or 23MB allocation size thus 900MB allocation should be like 900/46*10=195M headers loaded, but in 64 bit system with 4GB per 32 bit process i would rather expect something like 1.5GB allocation size practical limit at least.
did you check if the behaviour is the same in WinXP?
i could recompile UE in 64 bit as well, for the search service i'm using 64 bit compilation (the server shares most of code with UE), i didn't do it because of QA considerations, it was more difficult to diagnose bugs there, not sure about now. 64 bit compilation would give full 4GB allocation space, i also have working 8GB allocation limit switch but didn't use it even for the search service.
you should be able to open hundreds of millions of headers (with other programs the limit usually 5M to 12M per group when the program is still usable even if it is 64 bit compilation so UE is many times better in this regard).
you can cause out of memory just by opening groups without any other activity? (say if you disabled header overload option in properties->tasks and connection is very fast you can just fill memory with unprocessed headers).
how many of headers total, what retention like 240 days boneless?
did you limit the group retention at all (not natural?) ?
in UE the database capacity alone is 4GB so database cannot be the cause (the memory will run out first), with 4GB i think 3.6GB can be allocated which would give memory usage 2.7GB per heap memory allocations which exceeds 3 times the database size, somewhat strange.
when you start the program how many MB is allocated (UE status bar) and what is the process memory usage (in the windows task manager). then, say, you open a large (compact binary) newsgroup, right after opening how many MB allocated and what is the process memory usage then?
i checked here on smaller volumes (tens of millions of headers) it appears memory usave approximately is the double of allocation size (allocation size is included into memory usage). with 10 millions of headers taking around 46MB or 23MB allocation size thus 900MB allocation should be like 900/46*10=195M headers loaded, but in 64 bit system with 4GB per 32 bit process i would rather expect something like 1.5GB allocation size practical limit at least.
did you check if the behaviour is the same in WinXP?
i could recompile UE in 64 bit as well, for the search service i'm using 64 bit compilation (the server shares most of code with UE), i didn't do it because of QA considerations, it was more difficult to diagnose bugs there, not sure about now. 64 bit compilation would give full 4GB allocation space, i also have working 8GB allocation limit switch but didn't use it even for the search service.
-
francky013
- Posts: 4
- Joined: Wed Jun 25, 2008 1:51 pm
i need to find stack trace code for x64 processors, then i could compile an official version.
speed won't change though (i checked it with server which as i already mentioned shares the engine with UE), just the program could load more headers, i estimate it is needed only for very few users since even as UE has already more than tenfold advantage as to header database capacity - because others didn't invest in changing framework to handle similar amounts even as for a binary downloader (without discussion threads support) it is a much easier task - thus i'm not sure many users need that.
speed won't change though (i checked it with server which as i already mentioned shares the engine with UE), just the program could load more headers, i estimate it is needed only for very few users since even as UE has already more than tenfold advantage as to header database capacity - because others didn't invest in changing framework to handle similar amounts even as for a binary downloader (without discussion threads support) it is a much easier task - thus i'm not sure many users need that.
I had UE simply closing itself a few minutes ago. No crash message, nothing in the log, nothing in the windows event log.
Before I enabled SSL - this showed the bandwith jumping to 20 MB/sec ( I have a 50 Mbit VDSL line).
I use 20 days retention for all groups, in task manager UE takes close to 1 GB RAM. Running on Windows 2003 X64.
Could it be it ran out of memory or overload because of bandwith?
EDIT: Ok I just watched it - it showed a tiny window "out of memory" and the closed.
EDIT Again: I limited it to 10mb/sec now and see what happens.
Now I got the headers down. Maximum speed seems to lead to a lot of memory usage for UE - so it will crash.
Before I enabled SSL - this showed the bandwith jumping to 20 MB/sec ( I have a 50 Mbit VDSL line).
I use 20 days retention for all groups, in task manager UE takes close to 1 GB RAM. Running on Windows 2003 X64.
Could it be it ran out of memory or overload because of bandwith?
EDIT: Ok I just watched it - it showed a tiny window "out of memory" and the closed.
EDIT Again: I limited it to 10mb/sec now and see what happens.
Now I got the headers down. Maximum speed seems to lead to a lot of memory usage for UE - so it will crash.
Alex, you have not flagged UE as "large address aware", so UE is limited to 2GB allocation *on any system*.alex wrote:in 64 bit there is 4GB per process (not even 3GB as WinXP/Vista 32 bit), how much is memory usage of the UE process?
And taking into account typical memory fragmentation (address space), effective limit is ~1.2GB, depending on what kind of allocation strategy you have engineered.
Re-compiling as LAA would enable UE to use 4GB on a 64bit OS (and 3GB on a 32bit, if /3GB kernel parameter given). (making the typical effective limit on a 64bit OS to ~2.0-2.5GB depending on address space fragmentation)
Of course, 64bit compile would remove all mem limits.
(but 64bit compile would also use MORE memory due to bigger pointers etc - not sure how much UE would be affected, but if there is no speed advantage also, it probably would be kind of pointless)Personally, I ran into allocation issues with SSL enabled (compression was good) and scanning big groups headers, but otherwise I've had no problems (and I could circumvent those scanning issues with "a strategy") with ~120d retention even on bigger groups, as long as I don't try to open everything at the same time.
Bigger issue for me is "poor" threading strategy (UI really should be preferred to be completely in its own thread, maybe some others should be separated too, and mutex (or whatever communication strategy preferred) should be ensured to be in sync) which can cause long GUI congestion, especially when SSL+autoload enabled and all speed restrictions dropped. But even that is a minor issue overall and easily overcome.
you are right, i forgot to add this flag in the UE linker options.
i added it only for the server code years ago when i was testing 32 bit server version with /3GB when there was no 64 bit computer for the server yet available and i apparently thought it was the case for the client as well.
the memory fragmentation is not significant, since for most of memory intensive operations i don't use regular heap.
as to threading strategy (loading newsgroups in the main thread) it is to simplify synchronization, just with NewsPro i had looser model, but in the end the synchronization degraded to the same efficiency with the difference adding new view reflected features on the top (like collections) would be extremely difficult. when a group is being loaded, because of cross-posting or virtual groups other windows have to be updated (the storage is shared among all newsgroups).
here is the link to recompiled version with LAA enabled:
http://www.netwu.com/ue/UE_LAA.rar
the version in about box / file version number is 2.3.1.3, it will be also enabled from now on in subsequent releases, no rush for urgent release since it is not a bug and even without the option it was adequate and relatively superior than others, now will be like 2 times more for 64 bit system.
header overload shouldn't be disabled for fast connections, but you can try to set higher threshold for faster speed looking at the process memory so it won't grow too fast depending on your configuration, since when the option is not effective it may lead to excessive memory fragmentation and swapping with detrimental or at best the same performance.
i added it only for the server code years ago when i was testing 32 bit server version with /3GB when there was no 64 bit computer for the server yet available and i apparently thought it was the case for the client as well.
the memory fragmentation is not significant, since for most of memory intensive operations i don't use regular heap.
as to threading strategy (loading newsgroups in the main thread) it is to simplify synchronization, just with NewsPro i had looser model, but in the end the synchronization degraded to the same efficiency with the difference adding new view reflected features on the top (like collections) would be extremely difficult. when a group is being loaded, because of cross-posting or virtual groups other windows have to be updated (the storage is shared among all newsgroups).
here is the link to recompiled version with LAA enabled:
http://www.netwu.com/ue/UE_LAA.rar
the version in about box / file version number is 2.3.1.3, it will be also enabled from now on in subsequent releases, no rush for urgent release since it is not a bug and even without the option it was adequate and relatively superior than others, now will be like 2 times more for 64 bit system.
header overload shouldn't be disabled for fast connections, but you can try to set higher threshold for faster speed looking at the process memory so it won't grow too fast depending on your configuration, since when the option is not effective it may lead to excessive memory fragmentation and swapping with detrimental or at best the same performance.
Thank you for the comments, Alex.
Yes, I gathered there might be some inherent issues with threading making it inefficient/difficult to implement in another way, and that's why I put the word "poor" in quotes (can't really criticize others' code, without even seeing the code
).
My comments were based on observations with minimal thread analyzing and it just seemed inefficient as it is.
It doesn't even bother me too much anymore, as I've used to it by now and have adjusted the way I use UE so the effects of threading are minimal and do not disrupt me much. UE still is the most efficient binary reader I've used.
I was worried about it more when I was evaluating UE and made me consider if I even wanted to purchase it, but I'm glad I did.
There are studies (not sure if public) which show how much a negative image on software even minimal UI congestion gives, so it might be worthwhile (in commercial sense) to see if it can be made more fluid in user's perspective, even if it would be more inefficient overall.
Yes, I gathered there might be some inherent issues with threading making it inefficient/difficult to implement in another way, and that's why I put the word "poor" in quotes (can't really criticize others' code, without even seeing the code
).My comments were based on observations with minimal thread analyzing and it just seemed inefficient as it is.
It doesn't even bother me too much anymore, as I've used to it by now and have adjusted the way I use UE so the effects of threading are minimal and do not disrupt me much. UE still is the most efficient binary reader I've used.
I was worried about it more when I was evaluating UE and made me consider if I even wanted to purchase it, but I'm glad I did.
There are studies (not sure if public) which show how much a negative image on software even minimal UI congestion gives, so it might be worthwhile (in commercial sense) to see if it can be made more fluid in user's perspective, even if it would be more inefficient overall.
i may check the bottlenecks, loading asynchronously will take more processor time, the indications i had when a newsgroup is loaded then unloaded and loaded again (so it will loaded from system file cache) - the loading time was about the same, so if the bottleneck is not disk accesses and just incorporating records into database not much can be done.
with the LAA limit it was more complicated than it looked in the beginning, the program was designed in 2GB RAM per process in mind (the framework was written before 2005 when 4GB memory was expensive, then as i used the code for the server in 2005 i extended the limit, but left the client unchanged (RAM was still expensive), because it saved little RAM in the process address space, so now i had to use some code from the server and check all precompiler directives related to server/client difference.
with the LAA limit it was more complicated than it looked in the beginning, the program was designed in 2GB RAM per process in mind (the framework was written before 2005 when 4GB memory was expensive, then as i used the code for the server in 2005 i extended the limit, but left the client unchanged (RAM was still expensive), because it saved little RAM in the process address space, so now i had to use some code from the server and check all precompiler directives related to server/client difference.