cnt.dat file size limit

[MikusR]

while getting headers for alt.bin.hdtv.x264 (about 180880819) when completion is about 60% ue.exe freezes, it's memory jumps to 3+GB and exits. After starting again group has only about 10 000 000 headers. While cnt.dat for that newsgroup is 3.99 GB (4,291,697,629 bytes).
Vista x64, 4GB RAM, Astraweb ssl+compressed headers.

Code: Select all

UE2500R internal error (76E9E124/E06D7363)
OS Info: MajorVersion[6] MinorVersion[0] Build[6002] Platform[2] CSDVersion[Service Pack 2]
Call stack:
Address   Frame     Logical addr  Module
76E9E124  4F0BFCE8  0001:0000E124 C:\Windows\syswow64\kernel32.dll
005F8332  4F0BFD28  0001:001F7332 C:\Soft\ue\ue.exe
0069656C  4F0BFD3C  0001:0029556C C:\Soft\ue\ue.exe
006A3120  4F0BFD60  0001:002A2120 C:\Soft\ue\ue.exe
00512C59  4F0BFD7C  0001:00111C59 C:\Soft\ue\ue.exe
0051B0E1  4F0BFD9C  0001:0011A0E1 C:\Soft\ue\ue.exe
0051C188  4F0CFE54  0001:0011B188 C:\Soft\ue\ue.exe
0051D65C  4F0CFEA0  0001:0011C65C C:\Soft\ue\ue.exe
0051D6E0  4F0CFECC  0001:0011C6E0 C:\Soft\ue\ue.exe
0051D7AB  4F0CFED4  0001:0011C7AB C:\Soft\ue\ue.exe
006A5368  4F0CFF54  0001:002A4368 C:\Soft\ue\ue.exe
005FD6D8  4F0CFF88  0001:001FC6D8 C:\Soft\ue\ue.exe
76F0ECCB  4F0CFF94  0001:0007ECCB C:\Windows\syswow64\kernel32.dll
77D5D24D  4F0CFFD4  0001:0006D24D C:\Windows\SysWOW64\ntdll.dll
77D5D45F  4F0CFFEC  0001:0006D45F C:\Windows\SysWOW64\ntdll.dll

[alex]

what retention did you set for this group?

i think the limit for cnt.dat is 4GB (there is separate cnt.dat for every group), i don't think though it checks for the limit anywhere, so after 4GB the behaviour is not predictable.

180M headers is pretty lot for one group, if you check with other newsreaders you cannot get even 10 times less and have newsgroup loaded afterwards in reasonable time.

i think better you limit retention in properties->newsgroups for this group, in the next version i'll try to check for the limit so it will give warning and even automatically adjust retention or stop further getting headers.

what would be reasonable to implement is to set some trigger at a safe distance from 4GB, so it will adjust retention when the threshold is hit.

i could also to easily extend the file size limit if to change db record format, not sure it is very actual.

it is preliminary reply, i'll check for details later, i didn't touch this part since 2005 i think, collections, unpack and other changes since the first UE release are completely different code, thanks for exhaustive description.