[RFC] Migrations

These requests for comments/change have lead to an implemented feature that has been successfully merged into the 3.1/Ascraeus branch. Everything listed in this forum will be available in phpBB 3.1.
Post Reply
User avatar
naderman
Product Manager
Product Manager
Posts: 1727
Joined: Sun Jan 11, 2004 2:11 am
Location: Karlsruhe, Germany
Contact:

[RFC] Migrations

Post by naderman » Wed Aug 03, 2011 1:47 am

There are currently two RFCs covering the integration of a modified form of UMIL, the unified MOD installation library into phpBB:
[RFC] UMIL
[RFC] UMIL design proposal: Data providers

Goal of this RFC
This RFC aims to satisfy these proposals through an overhaul of the phpBB database updater with the use of so-called migrations. A migration generally defines a set of schema changes as well as a set of instructions that modify data in the database to accomodate the schema changes or to simply add or modify existing information. The current db_tools class provides all functionality required to make schema changes and it is already used by the database updater. Further functionality for setting configuration values, modifying, adding or deleting permissions, altering control panel modules, etc. will be largely taken from UMIL.

Rather than creating dedicated installation scripts, migrations describe the changes within methods of a class. A generic installer & updater can then run all the necessary changes for phpBB itself as well as all installed MODs. The database updater in the install folder will no longer contain significant code that will be missing for MODs later. This simplifies MOD installation scripts and allows us to create a unified interface that will be used by all MODs. At the same time MODs shipping with their own version of UMIL will continue to work, but will be discouraged.

Other Implementations of Migrations
Traditional implementations of migrations typically apply migrations linearly. Migrations are executed in order of their (often numeric) names or simply in chronological order. Since we work with feature branches in git, it becomes unfeasible to keep a database up to date with these migrations: Consider two branches A and B. Branch A was created before branch B. Both branches contain a migration. Branch B is now merged into develop. A developer updates their board, the migration from branch B is applied. Now branch A is merged. Since branch A's migration chronologically precedes the migration in branch B, it will not be applied. The same problem can be found simply with bugfix releases of multiple concurrently developped feature releases. Someone could either update from 3.0.9 to 3.0.10 to 3.1.1 or from 3.0.9 to 3.1.0 to 3.1.1. Either one or both of these paths would not apply some of the migrations if the are executed sequentially. We could maybe get this to work by merging and modifying old entries in the database updater, but that would be risky. An example:

Code: Select all


3.0.9: database_update.php:
	from 3.0.8: set_config('foo', 1);

3.1.0 database_update.php:
	from 3.0.8: set_config('foo', 1);
	from 3.0.9: set_config('foo', $config['foo'] + 2);

3.0.10 database_update.php:
	from 3.0.8: set_config('foo', 1);
	from 3.0.9: set_config('foo', $config['foo'] + 4);

3.1.1 database_update.php
	from 3.0.8: set_config('foo', 1);
	from 3.0.9: set_config('foo', $config['foo'] + 2);
				set_config('foo', $config['foo'] + 4); (merged from 3.0.10)
	from 3.1.0: set_config('foo', $config['foo'] + 8);

value of foo:
3.0.9 -> 3.1.0 -> 3.1.1
1        3        11

3.0.9 -> 3.0.10 -> 3.1.0 -> 3.1.1
1        5         9        17
Using timestamps to sort migrations could potentially work even in Git or across phpBB versions. However they do not allow a MOD to easily work across different phpBB versions. A MOD then cannot supply migrations that only apply to a different feature release of phpBB.

This is an example of two simple linear migrations, the first creates a config table with integer ids and the second adds a boolean dynamic flag to it. Similar to the config table in phpBB.
migrations-1.png
A simple linear migration
migrations-1.png (12.08 KiB) Viewed 11192 times
Solution: Migrations Graph
Instead I propose a graph of migrations, in which every migration defines its immediate dependencies. All migrations that can be found will be applied, as long as their dependencies are satisfied. Recursively this will apply all migrations in an acceptable order, regardless of the current database state.

This example shows an additional migration that changes the config table ids from integers to strings. This migration is independent of the migration adding a dynamic column. Each migration is added in a separate branch. They can be applied in either order.
migrations-2.png
Migrations in two concurrent branches
(20.68 KiB) Downloaded 7317 times
Release/Version Migrations
phpBB versions are represented by migrations that have neither schema nor data changes. They depend on all the leaf-migrations in the current code, allowing later migrations and MODs to simply refer to the version migration instead of particular other migrations.
migrations-3.png
Release migrations without schema and data changes
(24.74 KiB) Downloaded 7317 times
MOD Migrations
As a byproduct we can easily use migrations for MODs. If updated MOD migrations are available at the time of a phpBB update they will be updated together. Migrations in MODs encode which phpBB versions they run on. So if you update your MOD to a new version which requires a newer phpBB version, it will not apply the migration until phpBB has been updated.

The following picture shows a MOD called Foo which creates its own configuration table for 3_0_RC3 because it uses string identifiers. After version 3.0.0 is released the MOD first copies all of its own configuration data to the global phpBB configuration table, which now uses string identifiers, and then drops its own config table.
migrations-4.png
MOD migrations
(43.37 KiB) Downloaded 7317 times
Installation & Removal
In addition to regular migrations I propose that we continue to provide a direct installer since some parts of the update process can be rather lengthy. This is a special migration that lists all the migrations it contains. For phpBB itself this will continue to be the regular installer. MODs will have the option of providing a special installer or simply relying on the application of migrations. Migrations can optionally provide a rollback mechanism. This is useful in development and for the uninstallation of MODs. A MOD can either provide a special purge migration or rely on its migrations to be reversible.

Ticket: http://tracker.phpbb.com/browse/PHPBB3-9737
Sub-task Ticket: http://tracker.phpbb.com/browse/PHPBB3-9738
Patch (in progress): https://github.com/phpbb/phpbb3/pull/1067
Last edited by imkingdavid on Mon Dec 17, 2012 2:19 pm, edited 1 time in total.
Reason: Updated PR link

User avatar
naderman
Product Manager
Product Manager
Posts: 1727
Joined: Sun Jan 11, 2004 2:11 am
Location: Karlsruhe, Germany
Contact:

Re: [RFC] Migrations

Post by naderman » Wed Aug 03, 2011 3:32 pm

Updated the RFC a bit:
  • rollback support
  • phpBB install
  • MOD install/purge
  • clarified that timestamps would work for the basic case, but not for MODs supporting multiple phpBB versions

User avatar
Derky
Extension Customisations
Extension Customisations
Posts: 17
Joined: Sat Oct 08, 2005 7:50 pm
Location: Netherlands
Contact:

Re: [RFC] Migrations

Post by Derky » Tue Aug 16, 2011 11:55 am

naderman wrote:Migrations in MODs encode which phpBB versions they run on. So if you update your MOD to a new version which requires a newer phpBB version, it will not apply the migration until phpBB has been updated.
So that means the php code is updated and the database is not. That will crash your board with an SQL error.
naderman wrote:The following picture shows a MOD called Foo which creates its own configuration table for 3_0_RC3 because it uses string identifiers. After version 3.0.0 is released the MOD first copies all of its own configuration data to the global phpBB configuration table, which now uses string identifiers, and then drops its own config table.
migrations-4.png
If you have 2 version: mod_foo_1.0.1 and mod_foo_1.0.2 both for 3_0_RC3 where do they go in that drawing?
Above Migration: release_3_0_RC3 or just below or ? Could you make a drawing of that as well?

User avatar
naderman
Product Manager
Product Manager
Posts: 1727
Joined: Sun Jan 11, 2004 2:11 am
Location: Karlsruhe, Germany
Contact:

Re: [RFC] Migrations

Post by naderman » Tue Aug 16, 2011 6:16 pm

Derky wrote:
naderman wrote:Migrations in MODs encode which phpBB versions they run on. So if you update your MOD to a new version which requires a newer phpBB version, it will not apply the migration until phpBB has been updated.
So that means the php code is updated and the database is not. That will crash your board with an SQL error.
This obviously only applies to a MOD that has PHP code that works on either version. If you have a MOD that only works on the new version you would have to update it together with phpBB or disable it until you have the update available. And you would not install the newer version of the MOD before the newer phpBB version.
Derky wrote:
naderman wrote:The following picture shows a MOD called Foo which creates its own configuration table for 3_0_RC3 because it uses string identifiers. After version 3.0.0 is released the MOD first copies all of its own configuration data to the global phpBB configuration table, which now uses string identifiers, and then drops its own config table.
migrations-4.png
If you have 2 version: mod_foo_1.0.1 and mod_foo_1.0.2 both for 3_0_RC3 where do they go in that drawing?
Above Migration: release_3_0_RC3 or just below or ? Could you make a drawing of that as well?
Just below. If they are for 3_0_RC3, they obviously depend on 3_0_RC3.

User avatar
Derky
Extension Customisations
Extension Customisations
Posts: 17
Joined: Sat Oct 08, 2005 7:50 pm
Location: Netherlands
Contact:

Re: [RFC] Migrations

Post by Derky » Tue Aug 16, 2011 7:37 pm

naderman wrote:Just below. If they are for 3_0_RC3, they obviously depend on 3_0_RC3.
But it would be impossible to install it on 3_0_RC4 when that comes out if I understand correctly.

User avatar
naderman
Product Manager
Product Manager
Posts: 1727
Joined: Sun Jan 11, 2004 2:11 am
Location: Karlsruhe, Germany
Contact:

Re: [RFC] Migrations

Post by naderman » Tue Aug 16, 2011 8:06 pm

Then you must have misunderstood something. There is nothing in my RFC that would suggest that.

User avatar
Derky
Extension Customisations
Extension Customisations
Posts: 17
Joined: Sat Oct 08, 2005 7:50 pm
Location: Netherlands
Contact:

Re: [RFC] Migrations

Post by Derky » Tue Aug 16, 2011 9:09 pm

Then it's okay. :P

User avatar
naderman
Product Manager
Product Manager
Posts: 1727
Joined: Sun Jan 11, 2004 2:11 am
Location: Karlsruhe, Germany
Contact:

Re: [RFC] Migrations

Post by naderman » Fri Jan 06, 2012 3:48 pm

I've added the WIP pull request https://github.com/phpbb/phpbb3/pull/527

What's missing is utility functions from UMIL to be used in migrations. Changing migrations so that schema data is returned as an array similar to current updates, so one can potentially compute schema changes across migrations (e.g. for the installer). After that we need to change database_update.php to use migrations, integrate migrations with extensions, and port all current update steps to migrations (I'd say one migration per version should be enough).

User avatar
MichaelC
Development Team
Development Team
Posts: 889
Joined: Thu Jan 28, 2010 6:29 pm

Re: [RFC] Migrations

Post by MichaelC » Tue Feb 14, 2012 1:38 am

What happens when an extension logs data to a log using a language variable, then the extension is disabled. Will it remove the log entries (I'm guessing not). So there needs to be a way to be able to add instructions that are specific to un-installing (allowing it to remove any data it has added to other tables it did not add).
Formerly known as Unknown Bliss
psoTFX wrote: I went with Olympus because as I said to the teams ... "It's been one hell of a hill to climb"
No unsolicited PMs please except for quotes.

User avatar
imkingdavid
Registered User
Posts: 1050
Joined: Thu Jul 30, 2009 12:06 pm

Re: [RFC] Migrations

Post by imkingdavid » Tue Feb 14, 2012 2:32 am

Unknown Bliss wrote:What happens when an extension logs data to a log using a language variable, then the extension is disabled. Will it remove the log entries (I'm guessing not). So there needs to be a way to be able to add instructions that are specific to un-installing (allowing it to remove any data it has added to other tables it did not add).
When it's disabled, no it shouldn't remove the log entries. And for that, I think we could keep it looking through the extension's language directory, perhaps. But when an extension is purged, it should remove related log entries (although really that's up to the extension author to implement, I think?).
I do custom MODs. PM for a quote!
View My: MODs | Portfolio
Please do NOT contact for support via PM or email.
Remember, the enemy's gate is down.

Post Reply