Trials-shack Archiving

[Simon]

In an effort to reduce bandwidth to allow trials-shack to stay open all month once again, i am proposing "The archive".

Scenario one for "Last Modification";

Basically, archiving will take place every 24 hours, at 3am when traffic is at the lowest. The script will scan every directory and every supported file type, to see if it was last modified over a certain amount of days. For example, if i set the limit at any files over 120 days last modified, then ~410 files fall into that option. Just under half of the total files on trials-shack

Scenario two for "Last Downloaded";

This again would scan every 24 hours at 3am but would only scan files in the database, where as scenario one scans every supported type (for reference, there are around 40 supported files not in the database). However, the time limit would be compared against the last download time pulled from the database. So for example, if i set last download limit to 30 days, then 143 files will be archived.

There are advantages for both scenarios. They are however both as in-secure as eachother. Its very easy to fake either, one easier then the other, but both just as easy.

Now, what actually is the archive?

The archive is a big list of all files that can be downloaded, but only with a password, which will be retrievable from the user area. The password will be randomly generated when the file is archived. If users know the password, they may download an archived file.

Any questions, dont be afraid to ask, i can understand this being hard to understand in places.

Vote/discuss away.

Also, as you may of guessed, trials-shack will be closing early once again this month, hence new ideas and stripping down of the site, which some have noticed.

[tank_rider]

can i ask what the advantages of removing files that havent been downloaded in ages is anyway? surely if they havent been downloaded in quite some time the likelyhood of them being downloaded is small anyway. I imagine its a bandwidth limit problem you are having rather than a storage limit problem, or does that somehow contribute to the bandwidth?

Anyways im not too bothered either way, that reminds me time for another donation (Y)

[Simon]

can i ask what the advantages of removing files that havent been downloaded in ages is anyway?  surely if they havent been downloaded in quite some time the likelyhood of them being downloaded is small anyway.  I imagine its a bandwidth limit problem you are having rather than a storage limit problem, or does that somehow contribute to the bandwidth?

Anyways im not too bothered either way, that reminds me time for another donation (Y)

←

Ah sorry, i didn't make this clear.

The files are not removed, simply removed from the real database. There not changed on the server at all, there simply moved from one database to another.

Also very true, and also why i may add another limit to the files.

Idea is that the latest files are obviously the latest files, and therefor have a higher priority then some old file that is downloaded once every week. You may think once a week isn't much, but once a week times 147 files is quite a lot over a month.

Just suggesting ways i can save the bandwidth. Every little helps heh, especially if over 140 files are effected, or 410.

[tank_rider]

Ah sorry, i didn't make this clear.

The files are not removed, simply removed from the real database. There not changed on the server at all, there simply moved from one database to another.

Also very true, and also why i may add another limit to the files.

Idea is that the latest files are obviously the latest files, and therefor have a higher priority then some old file that is downloaded once every week. You may think once a week isn't much, but once a week times 147 files is quite a lot over a month.

Just suggesting ways i can save the bandwidth. Every little helps heh, especially if over 140 files are effected, or 410.

←

in that case you need to change the criteria ^^^^^^ up there for last time downloaded. If the majority of the files are being downloaded once a week then surely they will never reach the 30 day limit, and those that have reached the 30 day limit are highly unlikely to be downloaded much/again as 30 days is quite some time when compared to a week.

Maybe having an archieve feature for those with an ftp account, so that we can shift our older vids into the archieve. I know personally that im only really fussed about my latest couple of vids being easily available for download, and that if people really want the older ones then they can put in the effort to log in/register for the privalige (Y)

[Brian Bleech]

I like the idea of archiving to keep trials-shack open, not entirely sure how it work, need to re-read a few times!

Obviously the most recent files are the most important.

what about say if a file hasn't been downloaded for 45(just plucked that from thin air) days it gets archived.

then if someone tries to access it, they can't download it, but it possible to say if they pm/email you to get a password (maybe keep the passwords random, one use only) and they can download it! And obviosuly if loads of people want that vid again, it can be brought out of archive and can be downloaded normally.

i dunno much (anything) about the coding or work involved to do this and obviously i don't wanna give you loads more work.

is this sorta like what you were saying above? i am still confused

mike

[Simon]

I like the idea of archiving to keep trials-shack open, not entirely sure how it work, need to re-read a few times!

Obviously the most recent files are the most important.

what about say if a file hasn't been downloaded for 45(just plucked that from thin air) days it gets archived.

then if someone tries to access it, they can't download it, but it possible to say if they pm/email you to get a password (maybe keep the passwords  random, one use only) and they can download it! And obviosuly if loads of people want that vid again, it can be brought out of archive and can be downloaded normally.

i dunno much (anything) about the coding or work involved to do this and obviously i don't wanna give you loads more work.

is this sorta like what you were saying above? i am still confused

mike

←

Basically yes. Except passwords i won't be able to access, well i can obviously, but passwords will be randomly generated for each and every video that goes into the archive, so each file has a different password.

The password can then be got from the user area, or it will e-mailed to the user.

Coding part isn't hard at all. I already have "shells" coded for each scenario, and days can be set to any amount. Just need to see what people think.

[munkee]

how much bandwidth do you estimate this will save? i mean what percentage do you think of the videos downloaded are actually older than a few weeks? Just wondering really because to me it seems the bandwidth is getting rinsed by the huge videos like tm1 sizes and not these smaller videos which look like are being targeted. It seems a good idea in ways.. but i dunno im gonna start donating because the problem wont be solved till a new server can be bought. How much will that all cost by the way? to transfer all the videos over and new server? Could do with one of those counters on ts that shows how much is needed and how much has been donated so far (with a list of donators possibly)

[tank_rider]

would a plea to those with ftp accounts to delete the old vids they dont need help, i just went through my account and deleted several vids that i no longer want up there. Also i have one waiting to go into the archieve too (Y) (not sure if its been downloaded recently, just not one of my two most recent ones)

[Matt_Tupman]

I put 150 days but after fully reading this topic I think 30 days would be far better

not many vids are sirculating after that long so they may aswell be archived (Y)

top job your doing

[Simon]

This is also why i may be imposing a download limit for items over a certain amount. For example, a 60+ MB video may only be downloaded 300 times a day, and a 70+MB video may be downloaded 250 times a day. (only for example, figures and will more then likely change)

Archiving is just one of the steps im trying to take to preserve the bandwidth.

Spacemunkee; Im honestly not sure how much it will save, but im trying to save where ever i can at the moment, hence removing the banners from the site, and re-coding it so its faster and less bandwidth intensive. A new server to cope would be an extra £50 a month and to transfer data would be £40/45. I can also do a list of donaters, as well as a how much is needed and how much is left, but not sure if its possible to do it dynamically.

Currently im set to use ~1350GB, with my limit being 1200GB, which means closing around 2/3 days before end of month.

I also don't think deleting is the way, as i've seen on a few occasions where new members, and old have remembered an old video and ask for it to be hosted, to find thats it on trials-shack, just really far down the list. Or ask a old deleted video to be re-hosted. Archiving solves this problems, as well as helping with the bandwidth, if only a little bit, it still helps.

[delusional]

Did you have any luck with Coral Cache in the end?

Also, it might be worth thinking about the way videos are downloaded, with the current system from what I can tell php is parsing the file from disk and handing it out alongside a webpage. I'm guessing this is to stop direct linking? However this not only restricts your use of Coral Cache, but might also be obstructing external proxy caching, which is something that could have a big impact on your bandwidth.

Off the top of my head, you may be able to work around this by forwarding any requests from external referrers to files with video extensions to a php page where the link is available (in terms of useability, nil difference to the current site, except the title bar would change slightly). This may cause some slightly unexpected behaviour from proxy servers though (i.e. people downloading from a caching proxy will probably get the actual file served up to them rather than being forwarded to the html page).

Am I right about the direct linking prevention? It creates some rather interesting complications with bandwidth limiting by doing that :)

[Simon]

Did you have any luck with Coral Cache in the end? 

Also, it might be worth thinking about the way videos are downloaded, with the current system from what I can tell php is parsing the file from disk and handing it out alongside a webpage. I'm guessing this is to stop direct linking? However this not only restricts your use of Coral Cache, but might also be obstructing external proxy caching, which is something that could have a big impact on your bandwidth.

Am I right about the direct linking prevention? It creates some rather interesting complications with bandwidth limiting by doing that (Y)

←

Yeah, honestly, i didn't fancy the coral cache, so didn't look further into it. Again if im honest, i would prefer to use all in-house scripts. Still not looked into the torrent option yet though, and i keep meaning to do so.

Basically (totally revealing how trials-shack works here :S ) the file is uploaded via FTP as you would normally. In the parent public_html directory is a .htaccess which rewrites how supported file types are handled. This is why you see a normal page when loading a video, and not the actual video. Because of the .htaccess, if i simply then linked to the file, you would only download a html page. To get around this, i "parse" the file using a mass of around 15/18 header(" "); statements, which sort of "fake" the file being there, and brings up the save dialog box, in most cases anyway :D

Its not easy to look at ways to stop bandwidth via the file parser, but its not to hard. However, its a lot easier limit things before the file parser, such as the archive and big files per-day download limit.

Off the top of my head, you may be able to work around this by forwarding any requests from external referrers to files with video extensions to a php page where the link is available (in terms of useability, nil difference to the current site, except the title bar would change slightly). This may cause some slightly unexpected behaviour from proxy servers though (i.e. people downloading from a caching proxy will probably get the actual file served up to them rather than being forwarded to the html page).

In reference to forwarding, thats basically what i do, but because of the way .htaccess works, i can't simply link to the file, and have to do it the way i explained.

[Dr. Nick Riviera]

couldnt you just set a 3 per day download limit?

[Brisa]

You really deserve a medal, the effort you put into trials-shack and dedication you have for it is unreal. BIG (Y)

Like some people have said, to solve bandwidth problems why not make users register. I know people complain about eengoddee (sp?) but the majority that do are usually lazy newbies.

It does not take long for people to sign up for an account and with IP logging you could make sure people don't rip the piss and make loads of accounts. This would almost definately save a lot on bandwidth.

A download limit of even 5 vids a day would almost certainly still save you a lot of bandwidth as there usually isn't even 5 videos a day to download so I can't see people having a problem with that.

This is just my view on it, im sure the selfish people who hate the hassle of registering will argue me down. Anyway, like people say, 'don't look a gift hourse in the mouth'

[Simon]

couldnt you just set a 3 per day download limit?

←

For files or for users?

Which ever one, its a bad idea. 3 downloads a day per file is stupidly low, considering there around 4000 downloads a day (ish). 3 downloads a day per user is basically eeng' and is one of the reasons users don't like eeng anymore.

If i do limit downloads, it will be on a per file basis, and will be a lot more then 3 per day (Y)

[Brisa]

Could you even limit the amount of downloads per day on certain files such as TM1 and other files of similar size?

[munkee]

to be fair i think limiting may be the only option at the minute. at the moment i only download new vids that come out rarely an old find. some limit at around 10 would be fine or you could give each user a weekly quota on bandwidth allowed to be used. Of course they will need to register.. but it may help some