Re:retention

03 Oct 23: One server has been relocated, the server currently doesn't have a public IP address in the meantime. I wrote some additional tunnel code running separately to handle this. When the server is engaged the connection is 5600+ days uniform (since around 24 July 2008). In the unlikely case I disengaged the server because of some problem the retention is 1200-3800 days depending on newsgroup. If you experience any issue please let me know.

29 Nov 23: Because of encrypted and obfuscated posts flood mass posted by a few nzb indexing websites using usenet servers as private storage for their members to download the posts - the situation with the content is pretty chaotic since their purpose is to post in such a way, so users must use their website exclusively, when the website disappears - the encrypted posts just eat usenet providers' hard disk space uselessly. If you can't find something specific please let me know at alexbirj at gmail dot com what exactly you can't find for me to check how it is possible to retain the posts. Non-obfuscated posts shouldn't be affected at all, let me know if you notice anything missing.

29 Nov 23: The search protocol had to be upgraded to extend 3 byte (16M+) limit on the server side for number of sets per instance, so at least the version 5.8.3 is needed to access it, otherwise should be no any difference.

21 Jun 24: The search protocol had to be upgraded again to be able to properly process the trial version searches on the tunneled server (see the first note above).



Post Reply
Killer Skunk
Posts: 12
Joined: Sun Apr 02, 2006 1:00 pm

Re:retention

Post by Killer Skunk »

"November 20: The server affected by the ransomware attack had the system reinstalled and it has been upgraded (RAM, disk space). I'm in the process of copying data from another server. Even as there is more RAM, for a subset of groups I will try to run mid retention range in the compact mode using sets as the server then starts several times faster and consumes twice less RAM, we'll see how it goes."

Does this mean long time retention will come back later or not at all?
With NZBKing being gone too, the long time retention suddenly has become a real issue.
I've had a hdd crash a few month ago and lost all my movies, i've been havily lending on ue, nzbking to get the older items back.

Anyways i hope eventually the retention will be as how it has been.
alex
Posts: 4518
Joined: Thu Feb 27, 2003 5:57 pm

Re: Re:retention

Post by alex »

As far as I can see there is no loss of header data, I'm just copying it first before trying anything else as for some of data I have only one copy on another server.

There was backup disk, but it was plugged in when the server was compromised.

The issue is unlikely to happen again, as the server rdp port is no more directly accessible, the server was running on windows xp as consumer grade later windows have problem with handling memory for such kind of application, it is now upgraded to much later windows server. Windows XP was reinstalled in September, the 2019 rdp patch was applied, but it maybe somehow it wasn't effective, otherwise I don't see how it could happen.

I'll try to engage the second server within a week after it will start downloading headers for all groups.

There was also hard drive loss on the working server short time before the other server was compromised, otherwise I would have two copies on the functioning server, so I'm just being careful. For example I don't want to download headers with high intensity hard drive load and at the same time copy terabyte of data to the same drive. It will cause some delay, but as I will have more backup there will be less likelihood losing header data.

So I kept 4 copies of data, it is the first time when it was reduced to 1 for some of it which is somewhat interesting to learn.

As to retention I'm planning to run full retention, in addition I'll try to use the compact mode on the server side (sets) for subset of groups so the server will use less RAM and as retention grows there will be enough RAM left to index more data.

Right now the server which is being restored doesn't download headers for the largest subset of groups, but the file needed for that is currently being copied and within a day there should be enough there to run the server as backup.
alex
Posts: 4518
Joined: Thu Feb 27, 2003 5:57 pm

Re: Re:retention

Post by alex »

the search service is now working at full retention, it is better than before even with the remaining mid range retention gap for some groups as the prior retention was less than full 5250 days.

the mid-gap will be filled maybe after 24 hours (but i think in those years most posts which were cryptic already are not referenced from any website, so they are mainly just dead weight) and copying process will be finished maybe within one week from now so there will be enough backup.
Killer Skunk
Posts: 12
Joined: Sun Apr 02, 2006 1:00 pm

Re: Re:retention

Post by Killer Skunk »

Am getting very good result again Alex, just grabbed several things from over 4500 days and it just finds it without a heartbeat. Thanx for all the hard work.
Post Reply