Compressed headers support
Compressed headers support
Will be non-SSL compressed headers support added in the forseable future?
yes i'll look into it, should take a day to implement maybe if i do.
UE also supports SSL zlib which theoretically can bring the same compression ratio.
astraweb (also ngroups, newshosting and occationally UNS) support SSL zlib as well but i didn't check it on a fast connection, you can try (UE engages the zlib automatically), just sometimes there is bottleneck on server side i don't know why which doesn't bring benefit on fast connections, for slower connections the benefit is about 10 times speed increase.
actually SSL zlib is adequate, SSL doesn't bring overhead if connections are kept alive and noone can spy on user downloads (e.g. ISP). also it is possible to make selective zlib within SSL since zlib supports non-compressed blocks within the same session, here the attempt is made to shift the processing to the newsreading client, those in short my reservations. still the approach is better than giganews accelerator.
try to check SSL zlib do you see header acceleration when using UE (astraweb server icon should have yellow color loc - not dark cyan - it means zlib is engaged). it should be yellow since i saw they support zlib. what speed increase do you see when downloading headers only?
UE also supports SSL zlib which theoretically can bring the same compression ratio.
astraweb (also ngroups, newshosting and occationally UNS) support SSL zlib as well but i didn't check it on a fast connection, you can try (UE engages the zlib automatically), just sometimes there is bottleneck on server side i don't know why which doesn't bring benefit on fast connections, for slower connections the benefit is about 10 times speed increase.
actually SSL zlib is adequate, SSL doesn't bring overhead if connections are kept alive and noone can spy on user downloads (e.g. ISP). also it is possible to make selective zlib within SSL since zlib supports non-compressed blocks within the same session, here the attempt is made to shift the processing to the newsreading client, those in short my reservations. still the approach is better than giganews accelerator.
try to check SSL zlib do you see header acceleration when using UE (astraweb server icon should have yellow color loc - not dark cyan - it means zlib is engaged). it should be yellow since i saw they support zlib. what speed increase do you see when downloading headers only?
your right i have a yellow lock icon on my server... and when i download headers using SSL the speed is much faster, my network card is reporting ~400KB, and UE, is reporting ~4MB (it varies a lot) but it clearly faster
but i had problems use the SSL servers to download headers of big groups (A.B.boneless A.B.cores etc) never figured out the problem i just stop using SSL for headers. i had no ideal that the ssl server was using compression
i will do some more tweaking to see if i can find the right setting that will allow me to download headers using SSL and remain stable (i believe was too processor intensive)
but i had problems use the SSL servers to download headers of big groups (A.B.boneless A.B.cores etc) never figured out the problem i just stop using SSL for headers. i had no ideal that the ssl server was using compression
i will do some more tweaking to see if i can find the right setting that will allow me to download headers using SSL and remain stable (i believe was too processor intensive)
the speed should vary and there will be no difference whether you use SSL+zlib or those commands.
it is the same processor intensive for any method.
the question is if there is optimization in place to use selective compression option within the same SSL connection (if it is bottleneck at all, if not they can compress all traffic as well without any problem).
i'm using SSL customization quite a lot, but it appears on the server side it is a stumbling point for their programmers, i don't know why, the optimization would be very simple to implement (so not to compress bodies as well if it creates CPU bottleneck).
if problems with SSL it is some server issue, SSL is TCP based protocol so should be no suprises or lesser instability or even download speed comparing to not using SSL.
but again after some hesitations i'm rather leaning towards adding this option, at least it is generic, i don't like highly custom solutions which would only serve for the benefit of one usenet provider and with other simple things like downloading by message-id problems with providers persisted for years so you don't know how long it will take until SSL will work properly with zlib giving maximal acceleration, so even if i personally would prefer SSL+zlib combo it is also ok to have support for this option as well.
with downloading headers in large groups it appears then they are using tornado (highwinds) server software, i also observed this, but maybe it was fixed a month or two ago, since afterwards i saw no or only few header task retries instead of constant retries when using SSL, but i still observed some kind of bottleneck on high speed connections, maybe it is fixed by now as well.
it is the same processor intensive for any method.
the question is if there is optimization in place to use selective compression option within the same SSL connection (if it is bottleneck at all, if not they can compress all traffic as well without any problem).
i'm using SSL customization quite a lot, but it appears on the server side it is a stumbling point for their programmers, i don't know why, the optimization would be very simple to implement (so not to compress bodies as well if it creates CPU bottleneck).
if problems with SSL it is some server issue, SSL is TCP based protocol so should be no suprises or lesser instability or even download speed comparing to not using SSL.
but again after some hesitations i'm rather leaning towards adding this option, at least it is generic, i don't like highly custom solutions which would only serve for the benefit of one usenet provider and with other simple things like downloading by message-id problems with providers persisted for years so you don't know how long it will take until SSL will work properly with zlib giving maximal acceleration, so even if i personally would prefer SSL+zlib combo it is also ok to have support for this option as well.
with downloading headers in large groups it appears then they are using tornado (highwinds) server software, i also observed this, but maybe it was fixed a month or two ago, since afterwards i saw no or only few header task retries instead of constant retries when using SSL, but i still observed some kind of bottleneck on high speed connections, maybe it is fixed by now as well.
I got 17Megabytes headers download speed, my normal max is under 2Megabytes.
it is well possible ssl+zlib will have the same speed as their zlib commands, but on the astraweb side it will be double processor usage, because they will be compressing compressed headers again on their SSL end.
technically i'll add it here shortly (maybe by monday) but then i'll try to clarify those details before publishing it for all users, if you are not subcribed for next version preview area subscribe to get access to the version.
astraweb is running their own server software, so the question also whether others will be ready to add this behaviour to their server end.
technically i'll add it here shortly (maybe by monday) but then i'll try to clarify those details before publishing it for all users, if you are not subcribed for next version preview area subscribe to get access to the version.
astraweb is running their own server software, so the question also whether others will be ready to add this behaviour to their server end.
ok maybe i'll finish it tomorrow, managed to find time only today to add it.
astraweb is using zlib in non-standard way, took an hour to figure it out, i don't know why they are trying to save on zlib block headers, they could at least mention it clearly instead of a very vague hint.
basically it works but i need to do something with bandwidth counter, since in this case it doesn't show the accelerated bandwidth.
but my impression is it is not faster than SSL+zlib standard combo, so maybe waste of time, i'll check it for sure tomorrow.
i'm also not sure it is a good way to use zlib if they want it to be a standard (using undocumented zib initializations).
astraweb is using zlib in non-standard way, took an hour to figure it out, i don't know why they are trying to save on zlib block headers, they could at least mention it clearly instead of a very vague hint.
basically it works but i need to do something with bandwidth counter, since in this case it doesn't show the accelerated bandwidth.
but my impression is it is not faster than SSL+zlib standard combo, so maybe waste of time, i'll check it for sure tomorrow.
i'm also not sure it is a good way to use zlib if they want it to be a standard (using undocumented zib initializations).
I would probably favor a system that allowed for compressed headers and non-SSL bodies, without having to start another session (with different server connection details) whenever switching between them.
Since a lot of providers (mainly Highwinds) started offering free SSL (about 2 years ago) as a new service, there's been a lot of complaints from users about slow SSL download speed. As for myself, I have found that on SSL the speed might be fast one day and slow another day. I am assuming this may be due to server overload. Using 20 connections on SSL might get me the same speed as 1 connection using non-SSL -- if I was lucky. So I usually used non-SSL -- sometimes even when connecting from public wi-fi hotspots (even knowing that anyone running a sniffer could have easily stolen my passwords -- So another thing I have often wished for is the ability to use SSL encryption to login to the server but nothing else.)
SSL download speeds have been much better for me the last 6 months or so (and Giganews never seemed to have this SSL-slowdown problem)
So while having compressed headers can be a huge bonus, the problems I've had with SSL connections tends to make me want the option to avoid encryption when downloading files -- but still get compressed headers. I could of course do this by setting up two separate server login details, but the ability to do this on a single login would be a nice convenience.
Since a lot of providers (mainly Highwinds) started offering free SSL (about 2 years ago) as a new service, there's been a lot of complaints from users about slow SSL download speed. As for myself, I have found that on SSL the speed might be fast one day and slow another day. I am assuming this may be due to server overload. Using 20 connections on SSL might get me the same speed as 1 connection using non-SSL -- if I was lucky. So I usually used non-SSL -- sometimes even when connecting from public wi-fi hotspots (even knowing that anyone running a sniffer could have easily stolen my passwords -- So another thing I have often wished for is the ability to use SSL encryption to login to the server but nothing else.)
SSL download speeds have been much better for me the last 6 months or so (and Giganews never seemed to have this SSL-slowdown problem)
So while having compressed headers can be a huge bonus, the problems I've had with SSL connections tends to make me want the option to avoid encryption when downloading files -- but still get compressed headers. I could of course do this by setting up two separate server login details, but the ability to do this on a single login would be a nice convenience.
Considering the massive amount of spammed trojans hitting newsgroups lately - if things get much worse, users might someday be forced to give up using headers entirely.so maybe waste of time
ok i added proper bandwidth counter here so i can see the difference.
on a slow connection i get SSL+zlib without those commands about 30% faster than using XZVER (using SSL+zlib+XZVER or without SSL - all the same). i'm wondering whether they are using the same compression level for XZVER as in SSL zlib or it is lower.
i think they need to switch to standard inflate() instead of omitting zlib packets checksums, it is not realistic to make use on yenc crc32 because of such large volume of headers (to redownload maybe millions of headers if error in crc32), so zlib should abort if there is any error.
i'm wondering if they had some problem which caused them to not checksum zlib packets.
when i have the release i'll test it on two fast connections, then i'll give link to the release in the next release preview area and maybe here plus the results of the test here.
i'll publish it most likely today, need to add the code in 3 more places where i'm using XOVER and XHDR and clean it up (but actually it is one small function), until it is clarified as to the odd way of zlib use on the server side which forces clients to use it in the same way or fail otherwise - it won't be an official release.
we need a kind of check box to compare different ways, i thought to make it automatic but i see now i need to add some switch somewhere.
if it is slower on fast connections as well - there is no rationale to use it when SSL performs adequately.
on a slow connection i get SSL+zlib without those commands about 30% faster than using XZVER (using SSL+zlib+XZVER or without SSL - all the same). i'm wondering whether they are using the same compression level for XZVER as in SSL zlib or it is lower.
i think they need to switch to standard inflate() instead of omitting zlib packets checksums, it is not realistic to make use on yenc crc32 because of such large volume of headers (to redownload maybe millions of headers if error in crc32), so zlib should abort if there is any error.
i'm wondering if they had some problem which caused them to not checksum zlib packets.
when i have the release i'll test it on two fast connections, then i'll give link to the release in the next release preview area and maybe here plus the results of the test here.
i'll publish it most likely today, need to add the code in 3 more places where i'm using XOVER and XHDR and clean it up (but actually it is one small function), until it is clarified as to the odd way of zlib use on the server side which forces clients to use it in the same way or fail otherwise - it won't be an official release.
we need a kind of check box to compare different ways, i thought to make it automatic but i see now i need to add some switch somewhere.
if it is slower on fast connections as well - there is no rationale to use it when SSL performs adequately.
on a fast connection the speed is about the same, on a slower connection like 384kbps as i noted earlier ssl+zlib with normal xover is 30% faster. speed boost on normal connection like 25mbps is around 4 times, far from 15 they claim, the same slowdown like in SSL zlib it means need to do something about zlib input bufferization, but then maybe CPU load on their servers will be higher.
but in short i don't see advantage of using those commands versus normal ssl with zlib enabled (also xzver zlib packet checksums are not added comparing to ssl zlib), but if their SSL is slow it makes sense then.
here is this version:
http://www.netwu.com/ue/test/ue2311.rar
let me know whether you see it working any faster comparing to ssl.
i need to add several formal changes to make it official release grade code but for all practical purposes it works.
but in short i don't see advantage of using those commands versus normal ssl with zlib enabled (also xzver zlib packet checksums are not added comparing to ssl zlib), but if their SSL is slow it makes sense then.
here is this version:
http://www.netwu.com/ue/test/ue2311.rar
let me know whether you see it working any faster comparing to ssl.
i need to add several formal changes to make it official release grade code but for all practical purposes it works.
to make things more complicated - latest build of highwinds servers also supports those commands but they don't work properly, it appears i hit different servers one returned bad data and one end of stream both results are wrong.
i see the commands are supported but not working on UNS, then ngroups which also runs on highwinds servers (just different farm) they are not supported yet at all ("500 unsupported command" reply).
since it appears more servers support it i'm cleaning the code towards official release level, need to clarify with highwinds what is the mess there.
i see the commands are supported but not working on UNS, then ngroups which also runs on highwinds servers (just different farm) they are not supported yet at all ("500 unsupported command" reply).
since it appears more servers support it i'm cleaning the code towards official release level, need to clarify with highwinds what is the mess there.
this is official release grade code:
http://www.netwu.com/ue/test/ue2312.rar
still no switches, if you try it on UNS for example because the command is there but it always fails later - the version can only used only to test on astraweb along with news servers which support or don't support the command alltogether (not those with buggy implementation).
i'm trying to clarify with highwinds, what appears they still using zlib in the non-documented way like astraweb, but it doesn't work properly further as well, i checked the both possible zlib modes (negative/positive windowBits).
i didn't want to add another per server switch but because of highwinds maybe it is the only way to go.
it appears the feature is still in its infancy on the server side, maybe i'll manage to persuade astraweb to use zlib in the way which doesn't contradicts zlib RFCs then highwinds would do the same.
ideally should be no switches regarding those commands, it should be default which may be supported or not on the server side, when not the program would use regular XOVER or XHDR instead.
but again for those connections i checked with astraweb (two different locations) at best there is no difference between ssl with zlib and those commands also astraweb single connection appears to be with limited bandwidth about 600KB/s (maybe other users can confirm that, at least one user from Germany did), which is lower comparing to e.g. highwinds servers.
http://www.netwu.com/ue/test/ue2312.rar
still no switches, if you try it on UNS for example because the command is there but it always fails later - the version can only used only to test on astraweb along with news servers which support or don't support the command alltogether (not those with buggy implementation).
i'm trying to clarify with highwinds, what appears they still using zlib in the non-documented way like astraweb, but it doesn't work properly further as well, i checked the both possible zlib modes (negative/positive windowBits).
i didn't want to add another per server switch but because of highwinds maybe it is the only way to go.
it appears the feature is still in its infancy on the server side, maybe i'll manage to persuade astraweb to use zlib in the way which doesn't contradicts zlib RFCs then highwinds would do the same.
ideally should be no switches regarding those commands, it should be default which may be supported or not on the server side, when not the program would use regular XOVER or XHDR instead.
but again for those connections i checked with astraweb (two different locations) at best there is no difference between ssl with zlib and those commands also astraweb single connection appears to be with limited bandwidth about 600KB/s (maybe other users can confirm that, at least one user from Germany did), which is lower comparing to e.g. highwinds servers.