Something seriously wrong with ngroups

[Maccara]

Can someone verify?

Also, HOW can the following happen:
1. clean install (deleted bodies/newsgroups folders, using only old reg.dat)
2. filters disabled
3. reseted all headers
4. "get new headers" -> newest post ~118d old
5. "rescan headers" -> gets some new headers ~20h old
6. "get new headers" -> new posts GONE and newest again ~118d old

I'm testing 4+5+6 on a few small text groups for now, which I know have frequent posts (f.e. comp.soft-sys.math.mathematica).

Can someone please explain, how is it possible that if I get new headers, next get will REMOVE those headers?? I tried with "header kill filter" Supersedes:* on and off, in case there was something fishy serverside, but that didn't make a difference.

I thought I knew how usenet works, but what is happening above (steps 4+5+6) shouldn't be possible, unless UE suddenly broke (no, I haven't done any upgrades, or other changes on my system, for that matter - so it must be something new from server side; UE can't just break itself suddenly).

How can I debug this? More importantly, how can I fix this? How is it possible to get new headers which delete other new headers in UE?

Edit: this seems an US server issue - EU works ok now. nGroups is looking into possible server issues. However, I would still like to know how it is possible to disappear headers which have already been retrieved by UE. Something to do with erratic header ranges from server? Should UE catch this and maybe prompt the user "server must be fucked up" or something similar?

[alex]

Ngroups switched provider they resell from highwinds to readnews and they switched maybe half of customers to readnews. In fact I had someone complaining about corrupted downloads from EU server few days ago.

Try to reset the newsgroup then in workspace right click the newsgroup then advanced->header range properties, try to copy here header ranges on every stage for US and EU servers to compare, maybe will be more clear for us then what is going on there I could contact ngroups as well.

[Maccara]

For the moment EU servers seem to work again, and I'm currently re-downloading headers from the past week, so I'm not going to do any additional testing at the moment (maybe later if EU server hickup again).

Suffice to say, any amount of resetting headers did not help on US servers. (I even deleted and re-added server definition to get server ranges reseted too)

My suspicion without further debugging: depending on which server DNS round robin returned, at least one of them returns fubared header ranges / article numbers and UE gets extremely confused and purges already downloaded headers. Plausible?

Btw, I contacted ngroups also and they at least said a tech is going to take a look (as I was able to show US vs EU differences, so problem is not only at this end).

[alex]

What newsgroup type are you using compact binary or article numbers? (edit menu->properties->newsgroups, when you select newsgroup the type is shown to the right of the selected newsgroup name in the "current newsgroup" area below the list control)

What are retention settings (edit menu->properties->retention and headers per server) and rescan settings in properties->general?

When you get new headers unlikely UE will remove any old headers unless it is article number type which allows full synchronization with the server.

If the server returns sporadic article numbers UE has some checks in place, but it is not possible to handle any input, you can use reset header range in the advanced submenu in workspace if you see any problem with input, usually server article numbers don't change at all, if they change usenet provider should let users know about it.

[Maccara]

What newsgroup type are you using compact binary or article numbers? (edit menu->properties->newsgroups, when you select newsgroup the type is shown to the right of the selected newsgroup name in the "current newsgroup" area below the list control)

comp.soft-sys.math.mathematica is article numbers (I've used that for text groups)
binary groups I use compact binary.

What are retention settings (edit menu->properties->retention and headers per server) and rescan settings in properties->general?

retention 35d (normally, but during testing I used 7d now), headers 5000000
I also had "Natural" for retention dates for text groups originally, but all is the same now.

Rescan last 500000 & "only redownload" checked (what does the optimize do? I've had that on and off and haven't noticed any difference)

When you get new headers unlikely UE will remove any old headers unless it is article number type which allows full synchronization with the server.

That might explain why I initially noticed the problems with text groups, as I have them set for "article number".

Binary groups I didn't test this "disappearing headers" - they were just missing new headers which prompted me in the first place to look for issues.

If the server returns sporadic article numbers UE has some checks in place, but it is not possible to handle any input, you can use reset header range in the advanced submenu in workspace if you see any problem with input, usually server article numbers don't change at all, if they change usenet provider should let users know about it.

Yup, that's what I thought. That's why I suspect there's some server in group which is returning wrong article numbers.

I suppose I could try to check from directly the servers instead of whatever the DNS happens to return (I see a t least 2 servers from DNS). Hmm... Can't remember the nntp protocol so I could do it with telnet (need to go google).

[alex]

Then it is all clear as to the UE behaviour.

It is old code and NewsPro was article numbers only, so the newsgroup article number type code is very well refined, any possible discrepancy UE will rectify it by deleting "expired" (in the sense they don't exist on the server) articles, so you see pretty exact picture reflecting what headers the news server returns.

With other newsgroup types UE doesn't keep article numbers so it cannot delete older headers by article number, it is only the retention setting which matters and the retention is not applied when getting headers, only on newsgroup loading/opening.

[Maccara]

Indeed, that explained it nicely. I didn't come to think of that "side-effect" from using article numbers. Makes perfect sense, though.

Btw, nGroups confirmed there indeed was a problem with one of the US servers and have fixed it. And their EU servers are still a bit flaky.

I now changed a bit how I use UE with them (at least until they get their stuff working more reliably again - they mentioned they're still doing changes): I get headers from both US & EU servers and I changed text groups to use message-id (binaries still compact binary).

I believe that minimizes the chances of missed headers and doesn't purge already retrieved headers in case one of the servers starts returning bogus article numbers again.

One confirmation: I used to purge most headers "daily" to keep everything nice and tidy. Now that I'm using 2 servers I think I should instead delete them and keep a little longer, so that I don't see the same headers twice when they arrive on EU servers too? (I think this should work nicely at least with "message-id" type groups, but does it work on compact binary too?)

[jimzak]

I had to reset my headers for the binary groups I follow on Ngroups today.

Inexplicably, until I reset, I could not download headers more recent than 1.5 days ago yet network activity continued endlessly when I selected to download new headers.

[alex]

Did headers range or newsgroups reset completely resolved the problem and the issue did not resurface again?

It appears ngroups abandoned highwinds completely and switched all users to readnews.

To me it looks as something more serious, I saw older headers resurfacing and well maybe after newsgroup reset, it downloaded old headers indefinitely for the search service.

Please give me as more information as possible, I'm cleaning the search service database first, then I will be checking this issue if the feed is not ok, there are still 2 other feeds left, but better to have 3 than 2, so if ngroups (e.g. readnews) feed is now broken beyond repair I maybe will look for a replacement feed from a different third source.

[jimzak]

The header reset completely resolved the problem