[RFC|Replaced] UMIL

[igorw]

This RFC was replaced by [RFC] Migrations

I already posted an RFC for some design aspects of an UMIL-like system. I'm going to take this a step further and suggest including such a system in phpBB 3.1. This proposal topic is based on one I posted in January 2010 in one of the private forums. naderman and A_Jelly_Doughnut have stated their support.

What is UMIL

UMIL is a library/framework for MOD installation and updates. It wraps around phpBB's API to provide a unified interface for installation and update scripts. This includes db_tools, set_config, auth, modules and many other parts. It works a lot like phpBB 3.0's database updater. There is UMIL auto which takes a large array of version numbers and associated actions + arguments. Additionally it's possible to include the files and make the according calls manually.

One of the reasons why UMIL was created was to remove the need of SQL in MOD install files. And (as described later) to allow automation and different user interfaces for running the actions.

The abstraction provided by UMIL makes MOD installation and updates very easy. It makes it easier for the mod writer and (in theory) the person installing the MOD.

Status Quo

Currently UMIL is maintained by the MOD team which allows them to release new versions any time. The problem however is distribution. Because they don't want end users to have to maintain their own UMIL installs the MOD authors must include the current UMIL with the MOD. This is bad because a) bugs aren't fixed, b) MOD sizes rise, c) overwriting existing UMIL when extracting new MODs.

The amount of MODs using UMIL is still relatively low. Many MODs still use custom install scripts.

Proposal

The phpBB 3.x core should include UMIL or something comparable. This would resolve packaging issues and provide a standard API for installation scripts to all MOD authors. Ideally phpBB would re-use this system for it's own installation. It could be used to easily create in-memory sqlite databases for unit tests.

There would be some changes required to make this fully possible though. The user interface would need some better abstraction to allow automated calls of the install file. As a possible alternative to plain PHP install files, the instructions could be specified in alternative format, such as XML. Another potential addition would be the possibility to revert certain actions on uninstallation (eg using the Command pattern).

The system should be pluggable, allowing MOD authors to create their own installation "actions". If certain custom actions are highly demanded and were implemented for a MOD, they be contributed back upstream and included in the core.

Some ideas for implementation details can be found in: [RFC] UMIL design proposal: Data providers.

Ticket: http://tracker.phpbb.com/browse/PHPBB3-9737

[imkingdavid]

I agree with this idea fully, especially for the point you made where older version of UMIL overwrite newer versions in MODs that are validated before the new release of UMIL. I think it should be a standard feature of 3.1.

[MattF]

I agree with this idea too. I have a MOD which uses UMIL to perform three simple SQL queries... It just seems like a lot of overhead to need to provide the entire UMIL package with my MOD, keep up to date with it, and ask end users to install it - all for the sake of a very tiny SQL query.

I like the way UMIL makes it simpler for end users to "install, update, uninstall" a MOD which is why I use it. If it were an innate part of phpBB that would be just awesome, speaking as a MOD author and MOD user

[Oleg]

A strong opinion I have is that the db altering part of umil and phpbb installer should be included into permanent phpbb install.

Current situation is that during initial install or phpbb update the installer runs lots of queries. These queries are specified either in db-agnostic form or with multiple versions for supported databases. The packaging of code however is such that it cannot be reused elsewhere. After install or update is done, the code is deleted (it's in install directory).

Later prospective mod author wants to write a mod. The mod requires some db changes. phpbb installer could do them, but installer code is not available during mod installation and is not in a usable structure. There is umil, but it does not support everything phpbb installer supports (I think, and maybe vice versa too).

I ended up writing my own db migration code, since neither umil nor phpbb installer were usable as is.

The database part of umil requires no management; it's just a bunch of code that needs to be merged with phpbb installer db code and properly factored to be usable by mods, phpbb installer, and custom scripts (think integration with third-party software). Perhaps it would be worthwhile to split db part from the rest of umil for the purposes of this discussion.

[naderman]

The basis for all the db operations is already included in db_tools.php in the regular phpBB distribution. I agree that rather than bringing a copy of that with it in database_update.php, phpBB should ideally use that file. It's noteworthy that the updater is built in this way so that bugs in the files used by the updater cannot prevent the updater from updating. However that should only ever be an issue with the files and not the schame if done right, and even that should not usually occur. If something as bad as this ever happened one could deliver a special updater for the problematic version. So I agree that at the very least the database altering and module installation parts of current UMIL and database_update need to come with phpBB and be used by MODs and phpBB itself alike. I'd be happy to see a more detailed proposal / an actual implementation in the form of a patch that we can start with.

[ToonArmy]

It's very simple, if new/includes/db_tools.php exists in the updater package, use that.

[nickvergessen]

It's very simple, if new/includes/db_tools.php exists in the updater package, use that.

We also do this many other times, f.e. the language/en/install, the includes/diff/ adm/style/install_* etc.

[igorw]

We could call this system "migrations", since what it does is migrate the database to a new state/version.

[Jhong]

I agree that UMIL should be in the core.

I don't think it is very natural for users installing MODs to have to potentially overwrite the UMIL directory with each MOD -- it is not consistent with any other file copy action they will have to perform. Just another layer of complexity/opacity for the end user. UMIL in the core would fix problem.

[DavidIQ]

UMIL in the core would fix problem.

And also reduce its size. Right now pretty much any existing UMIL language is included from which you'd potentially use only 1 of those.