Backup / Restore

Discussion of general topics related to the new version and its place in the world. Don't discuss new features, report bugs, ask for support, et cetera. Don't use this to spam for other boards or attack those boards!
Forum rules
Discussion of general topics related to the new release and its place in the world. Don't discuss new features, report bugs, ask for support, et cetera. Don't use this to spam for other boards or attack those boards!
User avatar
Zottel
Registered User
Posts: 9
Joined: Tue Nov 28, 2006 11:58 pm

Backup / Restore

Post by Zottel » Wed Feb 07, 2007 7:38 pm

When restoring the database in the ACP, it will not work with larger databases, because the maximum allowed execution time for a php-script will be exceeded.

Off course, max_execution_time could be set higher to avoid this. But this would not be possible for everyone, depending on the hoster, and it would just move the problem (to higher DB-sizes), not really solve it.

Is this a bug, or is it this way "by design", because people are expected to do backups/restores on another way (phpmyadmin, mysql itself, etc), and the backup/restore in the ACP is just for starters which are new to phpBB and have a small database?
Not sure about this, so I didnt report this in the bug tracker.

User avatar
ChrisRLG
Registered User
Posts: 160
Joined: Wed Oct 11, 2006 9:47 am
Contact:

Re: Backup / Restore

Post by ChrisRLG » Wed Feb 07, 2007 9:32 pm

This is the same as phpBB2 - the answer was to use myphpadmin or a cron job on the server.

User avatar
Highway of Life
Registered User
Posts: 1399
Joined: Tue Feb 08, 2005 10:18 pm
Location: I'd love to change the World, but they won't give me the Source Code
Contact:

Re: Backup / Restore

Post by Highway of Life » Wed Feb 07, 2007 10:24 pm

phpMyAdmin will still have an upload limit of 2MB, which still defeats large board databases.
cron jobs are not possible for anyone that has less than a Private Virtual Server or Dedicated Server.

Best solution is to use bigDump.
But it would be great if phpBB3 used some sort of function that would split the work up for large databases and be able to pick up where it left off.
OR... set the max execution time a LOT higher... but there are problems if there is an error, so it still would be best to pick up at a certain point or where it was stopped.
Image

Paul
Infrastructure Team Leader
Infrastructure Team Leader
Posts: 367
Joined: Thu Sep 16, 2004 9:02 am
Contact:

Re: Backup / Restore

Post by Paul » Thu Feb 08, 2007 2:24 pm

Highway of Life wrote: phpMyAdmin will still have an upload limit of 2MB, which still defeats large board databases.
cron jobs are not possible for anyone that has less than a Private Virtual Server or Dedicated Server.

Best solution is to use bigDump.
But it would be great if phpBB3 used some sort of function that would split the work up for large databases and be able to pick up where it left off.
OR... set the max execution time a LOT higher... but there are problems if there is an error, so it still would be best to pick up at a certain point or where it was stopped.

Weird, i has 2 different hostings, and with both i can create cronjobs (Cpanel and the other with ssh :))

User avatar
Zottel
Registered User
Posts: 9
Joined: Tue Nov 28, 2006 11:58 pm

Re: Backup / Restore

Post by Zottel » Thu Feb 08, 2007 7:23 pm

Ok, so the backup/restore is not meant to really be used, except for people with a small database?

I think then there should be a notice at the backup-tool which explains that.
The danger thing here for people who dont know this is, that the backup may work fine, while the restore may not (happened to me while testing: backup was fine, and restore was impossible due to the timeouts).
Would be nice if the users of olympus would know that the restore may not work (even if the backup works), before something happens where they are in a situation where they need a restore :)

So a big fat notice explaining that in the backup tool wouldn't be a bad idea i guess :)

code reader
Registered User
Posts: 653
Joined: Wed Sep 21, 2005 3:01 pm

Re: Backup / Restore

Post by code reader » Thu Feb 08, 2007 11:12 pm

first a small correction:
ChrisRLG wrote: This is the same as phpBB2 - the answer was to use myphpadmin or a cron job on the server.
well, olympus restore is not at all like phpbb2.
in v2, the restore is performed from a file on the user's pc, and the main issue is the limitation php puts on upload file size.
in olympus the restore is performed from a local file on the server, so the upload file size is not relevant.

however, the database restore has (at least) 2 very serious issues:
1) unlike some other operations (including backup) which might be very long, there is no call to set_time_limit(). this causes timeouts with large enough files.

2) much more serious is the fact that the restore logic is to read the whole file into a (string) variable, then run several string operation on this variable, and finally, serving it to the db layer.
note that because php by default pass all variables by copy, this (potentially) very large variable exists at certain points in up to 4 copies.
this will exhaust server memory budget even for less-than-huge files.
this strategy is clearly limited to not-too-big databases.

both issues can (and should) be solved relatively easily.

the first one is a no-brainer: add set_time_limit(0) to the restore operation.

the 2nd one requires a little more work.
currently, the logic is this:

Code: Select all

1) read the whole file into a single variable.
2) if this is a compressed file, uncompress it (at this point you have, temporarily, 3 copies of the data)
3) clean it (at this point you have temporary 3 copies of the data)
4) split it to individual statements (at this point you temporarily have 4 copies of the data)
5) feed the individual statements to the db layer
the "correct" way of executing the restore, imo, is either of the following ways:
A) find a way to feed the file directly to the db layer without going through a php variable. this can be achieved if exec() or system() are enabled.

or,

B) use the following process:
  • open the file. if the file is compressed, use gzopen(), bzopen() of zip_open(), otherwise use fopen().
  • loop through the following steps:
    1. read a chunk from the file. the chunk can easily be several megs, but not the whole thing. if the file is compressed, use gzread(), bzread() or zip_entry_read(), otherwise use fread().
    2. clean the chunk.
    3. split the chunk to individual statements. note that the first and last items of the split array may not be whole queries.
    4. if there is a remainder from previous round, prepend the remainder to the first fragment, thus making it a whole query.
    5. test the last item to see if it's a broken query. if it is, save it for next round
    6. feed the whole queries to the db
    7. if you know how (i dont) update a progress indicator on the user's browser
  • close the file

User avatar
Highway of Life
Registered User
Posts: 1399
Joined: Tue Feb 08, 2005 10:18 pm
Location: I'd love to change the World, but they won't give me the Source Code
Contact:

Re: Backup / Restore

Post by Highway of Life » Fri Feb 09, 2007 6:51 am

paulus wrote: Weird, i has 2 different hostings, and with both i can create cronjobs (Cpanel and the other with ssh :))
If you have ssh, it's not considered shared hosting anymore. -- unless you have a souped up plan --
I repeat... shared hosting plans (most, if not all) will not be able to create cronjobs.

A while back, I had requested an option to use extended and/or complete inserts as a backup option, but the request was rejected... :x I now create manual backup files from Navicat that have complete and extended inserts, and am able to retain much larger backups due to both those options being available.
Image

User avatar
Cheater512
Registered User
Posts: 245
Joined: Thu Mar 23, 2006 1:29 am
Location: Brisbane, Australia
Contact:

Re: Backup / Restore

Post by Cheater512 » Fri Feb 09, 2007 10:31 am

Does someone want a mod to support everything a decent host can do when backuping/restoring? ;)

User avatar
Highway of Life
Registered User
Posts: 1399
Joined: Tue Feb 08, 2005 10:18 pm
Location: I'd love to change the World, but they won't give me the Source Code
Contact:

Re: Backup / Restore

Post by Highway of Life » Fri Feb 09, 2007 4:20 pm

I'm sure it would be popular... there seem to be quite a few free-hosts, or cheap-hosts that could use it. ;)
Image

User avatar
Handyman
Registered User
Posts: 522
Joined: Thu Feb 03, 2005 5:09 am
Location: Where no man has gone before!
Contact:

Re: Backup / Restore

Post by Handyman » Fri Feb 09, 2007 7:03 pm

Highway of Life wrote: A while back, I had requested an option to use extended and/or complete inserts as a backup option, but the request was rejected... :x I now create manual backup files from Navicat that have complete and extended inserts, and am able to retain much larger backups due to both those options being available.

I have a bug report on that… its set to review later because it is an issue with IE and forums anywhere, including localhost.
It also makes a huge difference in the size of the backup.
My phpBB3 Mods || My Mod Queue
Search Engine Friendly (SEO) URLs || Profile link on Avatar and/or Username || AJAX Chat
Display Posts Anywhere || CashMod || AJAX Quick Edit || AJAX Quick Reply

Image

Post Reply