[RFC] Migrations

[MichaelC]

Are there any plans to create a develop script that will be able to generate migrations by comparing DB schema differences from the current db schema and quickly being able to apply all migrations not yet applied (a CLI interface for the updater essentially).

Also, why not just have the Migrations called UNIXTIMESTAMP.php? They don't really need names. Migrations should be generated one for every change made during development, not just 1 per release so there should be no need for names.

Also, why do you have migrations relying on others? Surely it would make sense to just rely on the timestamps?

[Oleg]

Are there any plans to create a develop script that will be able to generate migrations by comparing DB schema differences from the current db schema and quickly being able to apply all migrations not yet applied (a CLI interface for the updater essentially).

Script that applies migrations can be built. Comparing schemas is/will be unnecessary as all schema changes (will) originate as migrations. The migrations simply need to be applied.

Also, why not just have the Migrations called UNIXTIMESTAMP.php?

This will work as long as you have a single migration. Beyond a single migration, not so much.

They don't really need names.

If migrations are classes, they need names otherwise they can't be referenced.

Also, why do you have migrations relying on others? Surely it would make sense to just rely on the timestamps?

Optional dependencies for extensions and extension compatibility with multiple phpbb versions. Somewhat exotic stuff. For the core there is no need for dependencies. I would also consider cutting dependencies out of 3.1.

[nickvergessen]

For the core there is no need for dependencies.

Yes there is, either global announcements must be located previous to softdelete update or the otherway round, because of the changed column names

[Oleg]

Timestamps will provide sequencing.

Just about the only exception I can think of is parallel development *and* editing of migrations:

1. Alice starts working on a feature A, creates a migration.
2. Alice realizes there needs to be a feature B for feature A to work.
3. Bob implements feature B, creates a migration.
4. Alice edits original migration in feature A (timestamped before migration in B) to use what B creates.

This is solved by either renaming the migration in A or creating an additional migration rather than editing the original one. The general rule with migrations is any edits to them must be carefully scrutinized.

[nickvergessen]

Btw is there any special reason why we discuss this? Is that the only missing part? I guess it's already done?

And Oleg, no. Sequence does not solve dependencies. F.e. there could be extensions that only work for a special phpbb-db version. With migrations you could use dependencies to check it. So yes dependencies is a must have on such a feature.

[Oleg]

I am trying to cut functionality to arrive at a subset that is sufficient for 3.0/3.1 and does not have a 10,000 line diff to review.

My idea of this subset:

1. One file per migration stored in install/migrations.
2. One class per migration file.
3. Migration classes have two methods: one to retrieve schema changes and one to perform data changes. Same as existing database updater.
4. Neither method takes any arguments. Use globals for everything. Same as current database updater. This will allow 3.0 to use migrations also.
5. Require migration files to be timestamped as I mentioned.
6. Migrator simply obtains a list of all existing migration files, pulls all applied migration names out of the database, applies the remaining files in timestamp order. Then it writes a row to migrations table to signify migration completion. Steps, state, etc. are all cut. This migrator will work in 3.0 as well.
7. The migration interface can be extended in 3.1/3.2 to include steps and state.

The idea is to allow expressing what is currently possible to do in database updater via migrations. Nothing more.

[MichaelC]

I am trying to cut functionality to arrive at a subset that is sufficient for 3.0/3.1 and does not have a 10,000 line diff to review.

My idea of this subset:

1. One file per migration stored in install/migrations.
2. One class per migration file.
3. Migration classes have two methods: one to retrieve schema changes and one to perform data changes. Same as existing database updater.
4. Neither method takes any arguments. Use globals for everything. Same as current database updater. This will allow 3.0 to use migrations also.
5. Require migration files to be timestamped as I mentioned.
6. Migrator simply obtains a list of all existing migration files, pulls all applied migration names out of the database, applies the remaining files in timestamp order. Then it writes a row to migrations table to signify migration completion. Steps, state, etc. are all cut. This migrator will work in 3.0 as well.
7. The migration interface can be extended in 3.1/3.2 to include steps and state.

The idea is to allow expressing what is currently possible to do in database updater via migrations. Nothing more.

+1

Are there any plans to create a develop script that will be able to generate migrations by comparing DB schema differences from the current db schema and quickly being able to apply all migrations not yet applied (a CLI interface for the updater essentially).

Script that applies migrations can be built. Comparing schemas is/will be unnecessary as all schema changes (will) originate as migrations. The migrations simply need to be applied.

Also, why not just have the Migrations called UNIXTIMESTAMP.php?

This will work as long as you have a single migration. Beyond a single migration, not so much.

They don't really need names.

If migrations are classes, they need names otherwise they can't be referenced.

Also, why do you have migrations relying on others? Surely it would make sense to just rely on the timestamps?

Optional dependencies for extensions and extension compatibility with multiple phpbb versions. Somewhat exotic stuff. For the core there is no need for dependencies. I would also consider cutting dependencies out of 3.1.

Many existing migrations scripts offer generation as generally the developer will apply db changes, then get it working how they want it, then create a migration. Also generated migrations are less likely to break.

As for timestamp.php, how will this not work for more than 1 migration? Again, this is what existing migrations libs do and the only problem would be is when two migrations are made at the exact same second which is very unlikely but easy to resolve.

They can be called their timestamps. I'm saying that instead of timestamp_name.php and the class being timestamp_name it should just be timestamp.php and class being timestamp.

[Oleg]

Many existing migrations scripts offer generation as generally the developer will apply db changes, then get it working how they want it, then create a migration.

This is more complicated, error prone and only covers transitions that are accounted for. Which means, this is also a reduction in functionality compared to what is currently possible.

Also generated migrations are less likely to break.

What would be an example of database updater "breaking"?

As for timestamp.php, how will this not work for more than 1 migration?

Which of the following files contain changes for pruning users?

1354877513.php
1354711242.php
1354308639.php
1354108598.php
1354748196.php
1354522012.php
1354375160.php
1354826681.php
1354836491.php
1354842171.php
1354091588.php
1354517488.php
1354325118.php
1354612444.php
1354208440.php
1354360076.php
1354479350.php
1354836279.php
1354581861.php
1354306151.php
1354314469.php
1354843508.php
1354114481.php
1354329646.php
1354675411.php
1354687288.php
1354570385.php
1354214827.php
1354868628.php
1354598616.php
1354126757.php
1354428804.php
1354499330.php
1354441865.php
1354815033.php
1354103679.php
1354857156.php
1354861770.php
1354916617.php
1354490787.php

Again, this is what existing migrations libs do

I am yet to see a migration library that uses timestamp only for identifying migrations. I would definitely be interested in seeing such a beast.

The only time I can imagine it would be acceptable is for write-only code, which phpbb is generally not.

and the only problem would be is when two migrations are made at the exact same second which is very unlikely but easy to resolve.

It's more likely than you think, because people copy paste. As a matter of fact, I would not use a timestamp, especially one with a second resolution, to uniquely identify anything.

They can be called their timestamps. I'm saying that instead of timestamp_name.php and the class being timestamp_name it should just be timestamp.php and class being timestamp.

Last time I checked 1354441865 was not a legal class name in php.

[callumacrae]

Which of the following files contain changes for pruning users?

1354877513.php
1354711242.php
1354308639.php
1354108598.php
1354748196.php
1354522012.php
1354375160.php
1354826681.php
1354836491.php
1354842171.php
1354091588.php
1354517488.php
1354325118.php
1354612444.php
1354208440.php
1354360076.php
1354479350.php
1354836279.php
1354581861.php
1354306151.php
1354314469.php
1354843508.php
1354114481.php
1354329646.php
1354675411.php
1354687288.php
1354570385.php
1354214827.php
1354868628.php
1354598616.php
1354126757.php
1354428804.php
1354499330.php
1354441865.php
1354815033.php
1354103679.php
1354857156.php
1354861770.php
1354916617.php
1354490787.php

ack 'prune'

[brunoais]

As for timestamp.php, how will this not work for more than 1 migration?

Which of the following files contain changes for pruning users?

1354877513.php
1354711242.php
1354308639.php
1354108598.php
1354748196.php
1354522012.php
1354375160.php
1354826681.php
1354836491.php
1354842171.php
1354091588.php
1354517488.php
1354325118.php
1354612444.php
1354208440.php
1354360076.php
1354479350.php
1354836279.php
1354581861.php
1354306151.php
1354314469.php
1354843508.php
1354114481.php
1354329646.php
1354675411.php
1354687288.php
1354570385.php
1354214827.php
1354868628.php
1354598616.php
1354126757.php
1354428804.php
1354499330.php
1354441865.php
1354815033.php
1354103679.php
1354857156.php
1354861770.php
1354916617.php
1354490787.php

I think that can be solved with something like...
[meaningfulNameOfMigration]_[TIMESTAMP].php