Page 1 of 1
Extremely low diskspace when headering huge newsgroup
Posted: Thu Oct 12, 2006 11:53 am
by erikb
Hi,
I ran 1.7.2 and had the following happen to me:
HD partition had about 3 GB free. I downloaded all headers for alt.binaries.hdtv with about 500 kbyte/sec. Database 29.3 MB allocated / 43.5 MB reserved. Disk 684KB free. Memory usage was about 80 MB of 320 total. In explorer the disk still had 3 GB free.
Is this a bug or am I missing something?
Still a great program.
Posted: Sat Oct 14, 2006 5:37 am
by alex
i don't understand
what you see how much disk space is available is returned by a system windows function, did you refresh the status in explorer?
the database usage shown is related to headers in RAM only (it is also deducted from the disk space but it is just a unimportant detail), if you downloaded bodies in the same time and didn't save them it could be explanation why the space has been consumed, check the size of the bodies directory.
now, if indeed the group is huge even if it takes ten MB in RAM - on the disk it may take much more in the Newsgroups directory (open map.txt and check the size of the directory which contains the newsgroup using the number from map.txt as the reference).
ue can handle newsgroups ten times larger than other newsreaders but the information about every part should be still kept somewhere, thus it is on the disk
Posted: Wed Oct 18, 2006 5:54 pm
by erikb
alex wrote:i don't understand
what you see how much disk space is available is returned by a system windows function, did you refresh the status in explorer?
I did not do a refresh in explorer, I did not think of that.
alex wrote:the database usage shown is related to headers in RAM only (it is also deducted from the disk space but it is just a unimportant detail), if you downloaded bodies in the same time and didn't save them it could be explanation why the space has been consumed, check the size of the bodies directory.
This was not the case. The "Disk 748KB free" message was only during the downloading of the headers of the ab.hdtv newsgroup. Before UE started downloading the headers for the newsgroup ab.hdtv and after it was finished downloading them, the message was something like "Disk 2,93 GB free". This was shown on the bottom bar in UE, to the right of "Bandwith/Database/allocated/reserved". My computer is slow and has low internal ram and the group is huge, so it took a while so I could observe it well.
alex wrote:now, if indeed the group is huge even if it takes ten MB in RAM - on the disk it may take much more in the Newsgroups directory (open map.txt and check the size of the directory which contains the newsgroup using the number from map.txt as the reference).
map.txt states: " 9 alt.binaries.hdtv (regular, compact binary)" and the size of map 9 has:
cnt.dat 48.003KB
cnt.sdb 5 KB
mex.sdb 0 KB
mit.sdb 8 KB
ng.sdb 15.049 KB
Size in RAM for UE in Windows Task Manager during header download was 80 MB.
alex wrote:ue can handle newsgroups ten times larger than other newsreaders but the information about every part should be still kept somewhere, thus it is on the disk
Okay, do the numbers I give you make any sence?
Posted: Thu Oct 19, 2006 1:05 am
by alex
disk/ram usage fluctuates during header download, maybe fluctuations will be a bit more if connection is very fast, although it cannot be several GBs, the mapped area to the disk cannot exceed 1.6GB in principle in 32 bit systems but it doesn't go there at all in the first place with compact binary newsgroup destination type.
do you have system swap file on the same partition? no other programs that may mess with UE database files (antiviruses, etc)
Posted: Thu Oct 19, 2006 3:48 pm
by erikb
alex wrote:disk/ram usage fluctuates during header download, maybe fluctuations will be a bit more if connection is very fast, although it cannot be several GBs, the mapped area to the disk cannot exceed 1.6GB in principle in 32 bit systems but it doesn't go there at all in the first place with compact binary newsgroup destination type.
do you have system swap file on the same partition? no other programs that may mess with UE database files (antiviruses, etc)
I do not run an antivirus program and the swapfile is on the other partition. I tried to reproduce the issue but I did not succeed, so I guess the cause stays unknown. My issue is non-reproducable, so I'm happy