Internal workings

[solunvar]

Right, this one hasn't got anything to do with the content of the site, but with the engine supporting the archive.

I understand plans are in the make and/or already started to program a better/optimized engine for the archive. That piece of knowledge, together with the 'fact' that the server's apparently under a lot of strain , got me thinking about ways things could be made more efficient.

At the moment (and for the foreseeable future), I haven't a clue how things are handled (on the php-side), but I have noticed several aspects that could lessen the load somewhat. For instance, merely browsing through the site implies that you load every single page completely. I'd think that, especially where the stories and their texts are concerned, a more asynchronous approach could be more beneficial. The only parts that need renewal on those pages are after all the story body and navigation, not?

I'm not sure whether it's like this here, but fanfiction.net implemented it a while back: a waiting queue for stories (and other information) to appear in the main indexes and search parts of the site. Basically, what they do (I think) is make an index file of the archived stories every ten minutes or so (=one query per category every x minutes) and show those in their browse section. Chapters added in that time frame can still be accessed, since that's a separate query for the database to process.

There's probably several other things that I or other users might think of. I just figured: might as well offer my suggestions - who knows, maybe something comes out of it.

Aside from all of that, if you need people helping out, I'm always willing. (I study Applied Informatics, though I'm still in my first year, so any PHP know-how comes from personal tinkering rather than official lessons I've had.)

[DemonGoddess]

Firstly, the archive engine itself is currently coded in php2 and php3, roughly half and half. With very little php4, as 2 & 3 don't work well together with 4 or 5. That is because the language for the scripting itself, how the queries are structured and whatnot are quite different between php4 and up, and anything php3 and below. There was a MAJOR rewrite of how the queries work and are written for php4.

This I think shows you the need for the rewrite in the first place. As this software is no longer supported by their original writer (I believe the company is defunct, if memory serves), we have to do something to keep our dataset without losing it.

We've looked at the possibilities of perhaps using and heavily modifying some applications that are out there, but they simply do not support a database the size of what we have. We tried that offsite already, to see if things could be possibly sped up in that way. Needless to say, the experiment did not work.

The navigation is a global file, as is the wrapper which shows the top level categories, and the latest listing. So, what draws on your screen when you click a link is the content you're going after, rather than the entire thing all at once, all over again. So, say you click the next link in the latest story listing, what redraws is the story listing, not the menu at top, or the side menu of the top level categories. Also, there's an autorefresh that's set in, but it runs I think only every ten minutes or so (I'd have to look to be sure, don't remember).

Yes, there has been significant restructuring done already, in an effort to reduce the query load itself. In fact, there is much being done on that end to continue to reduce it, and make it run more efficiently, and also to prep it for the new stuff. There are other structure hierarchies we'll be putting in place as well once the software is in place which will support it.

In the meantime, we work within the limits of what we have, and continue doing the other stuff in the backfield.

That's really all that can be done for now with the archive, unfortunately; until the rewrite is complete.

Something to think about, for example, is how the vast difference in language structure from php3 to php4, and also mysql3 to mysql4, caused most php driven applications to have to be entirely rewritten to accommodate those changes.

That's what made for phpbb3, IPB 2.x, and so forth. They HAD to be rewritten, in order to function properly with the new language structures both for the scripting language and the database query language. With the release of php5, you're seeing this again. I expect we'll be seeing an announcement from phpbb3 that they're developing the next level, but I bet that doesn't happen for quite some time, and won't even be ready for beta for quite awhile either. IPB 3.0 has been in development since php5 was released. They're getting ready to release it within the next few months, I think it is, which will mean yet another big forum upgrade. However, I have to do some reading to be certain that it will be supported on earlier server versions. We'll see. If it's not, that upgrade will have to wait.