Backup / Restore

[Zottel]

When restoring the database in the ACP, it will not work with larger databases, because the maximum allowed execution time for a php-script will be exceeded.

Off course, max_execution_time could be set higher to avoid this. But this would not be possible for everyone, depending on the hoster, and it would just move the problem (to higher DB-sizes), not really solve it.

Is this a bug, or is it this way "by design", because people are expected to do backups/restores on another way (phpmyadmin, mysql itself, etc), and the backup/restore in the ACP is just for starters which are new to phpBB and have a small database?
Not sure about this, so I didnt report this in the bug tracker.

[ChrisRLG]

This is the same as phpBB2 - the answer was to use myphpadmin or a cron job on the server.

[Highway of Life]

phpMyAdmin will still have an upload limit of 2MB, which still defeats large board databases.
cron jobs are not possible for anyone that has less than a Private Virtual Server or Dedicated Server.

Best solution is to use bigDump.
But it would be great if phpBB3 used some sort of function that would split the work up for large databases and be able to pick up where it left off.
OR... set the max execution time a LOT higher... but there are problems if there is an error, so it still would be best to pick up at a certain point or where it was stopped.

[Paul]

phpMyAdmin will still have an upload limit of 2MB, which still defeats large board databases.
cron jobs are not possible for anyone that has less than a Private Virtual Server or Dedicated Server.

Best solution is to use bigDump.
But it would be great if phpBB3 used some sort of function that would split the work up for large databases and be able to pick up where it left off.
OR... set the max execution time a LOT higher... but there are problems if there is an error, so it still would be best to pick up at a certain point or where it was stopped.

Weird, i has 2 different hostings, and with both i can create cronjobs (Cpanel and the other with ssh )

[Zottel]

Ok, so the backup/restore is not meant to really be used, except for people with a small database?

I think then there should be a notice at the backup-tool which explains that.
The danger thing here for people who dont know this is, that the backup may work fine, while the restore may not (happened to me while testing: backup was fine, and restore was impossible due to the timeouts).
Would be nice if the users of olympus would know that the restore may not work (even if the backup works), before something happens where they are in a situation where they need a restore

So a big fat notice explaining that in the backup tool wouldn't be a bad idea i guess

[code reader]

first a small correction:

This is the same as phpBB2 - the answer was to use myphpadmin or a cron job on the server.

well, olympus restore is not at all like phpbb2.
in v2, the restore is performed from a file on the user's pc, and the main issue is the limitation php puts on upload file size.
in olympus the restore is performed from a local file on the server, so the upload file size is not relevant.

however, the database restore has (at least) 2 very serious issues:
1) unlike some other operations (including backup) which might be very long, there is no call to set_time_limit(). this causes timeouts with large enough files.

2) much more serious is the fact that the restore logic is to read the whole file into a (string) variable, then run several string operation on this variable, and finally, serving it to the db layer.
note that because php by default pass all variables by copy, this (potentially) very large variable exists at certain points in up to 4 copies.
this will exhaust server memory budget even for less-than-huge files.
this strategy is clearly limited to not-too-big databases.

both issues can (and should) be solved relatively easily.

the first one is a no-brainer: add set_time_limit(0) to the restore operation.

the 2nd one requires a little more work.
currently, the logic is this:

Code: Select all

1) read the whole file into a single variable.
2) if this is a compressed file, uncompress it (at this point you have, temporarily, 3 copies of the data)
3) clean it (at this point you have temporary 3 copies of the data)
4) split it to individual statements (at this point you temporarily have 4 copies of the data)
5) feed the individual statements to the db layer

the "correct" way of executing the restore, imo, is either of the following ways:
A) find a way to feed the file directly to the db layer without going through a php variable. this can be achieved if exec() or system() are enabled.

or,

B) use the following process:

• open the file. if the file is compressed, use gzopen(), bzopen() of zip_open(), otherwise use fopen().
• loop through the following steps:
  1. read a chunk from the file. the chunk can easily be several megs, but not the whole thing. if the file is compressed, use gzread(), bzread() or zip_entry_read(), otherwise use fread().
  2. clean the chunk.
  3. split the chunk to individual statements. note that the first and last items of the split array may not be whole queries.
  4. if there is a remainder from previous round, prepend the remainder to the first fragment, thus making it a whole query.
  5. test the last item to see if it's a broken query. if it is, save it for next round
  6. feed the whole queries to the db
  7. if you know how (i dont) update a progress indicator on the user's browser
• close the file

[Highway of Life]

Weird, i has 2 different hostings, and with both i can create cronjobs (Cpanel and the other with ssh )

If you have ssh, it's not considered shared hosting anymore. -- unless you have a souped up plan --
I repeat... shared hosting plans (most, if not all) will not be able to create cronjobs.

A while back, I had requested an option to use extended and/or complete inserts as a backup option, but the request was rejected... I now create manual backup files from Navicat that have complete and extended inserts, and am able to retain much larger backups due to both those options being available.

[Cheater512]

Does someone want a mod to support everything a decent host can do when backuping/restoring?

[Highway of Life]

I'm sure it would be popular... there seem to be quite a few free-hosts, or cheap-hosts that could use it.

[Handyman]

A while back, I had requested an option to use extended and/or complete inserts as a backup option, but the request was rejected... I now create manual backup files from Navicat that have complete and extended inserts, and am able to retain much larger backups due to both those options being available.

I have a bug report on that… its set to review later because it is an issue with IE and forums anywhere, including localhost.
It also makes a huge difference in the size of the backup.