Including external projects in phpBB

[wGEric]

Please start a discussion on area51 about including external projects in phpBB, cause we won't be commiting the entirety of ckeditor into the phpBB source tree. We need to find a general solution for reusing 3rd party libraries both on the PHP and js level.

Making users download external programs to use them in phpBB isn't an option IMO so they would need to be included in the release packages.

Would the best option be using a submodule in git to add the project and then adding config files into phpBB so that the files of the external tool are not modified?

Reason for this discussion is because I've started implementing CKEditor.

[naim]

Like how drupal works?

[bantu]

This problem also has to be solved for PHP libraries.

For example: Because of "[RFC|Merged] No db storage of stylesheets and templates" we want to be able to write to the filesystem using SFTP and for SFTP we want phpseclib but there is currently no proper strategy for including third-party libraries in phpBB.

[RFC|Merged] No db storage of stylesheets and templates: viewtopic.php?f=84&t=33456
[RFC] SFTP support: viewtopic.php?f=84&t=39000

[A_Jelly_Doughnut]

Any reason the strategy couldn't be the same as you have now for jQuery?

[Oleg]

Goutte got bundled recently, needs to be addressed as well.

[Ger]

Any reason the strategy couldn't be the same as you have now for jQuery?

Good logic. Perhaps the new setting for local/CDN jQuery should be converted to a more general setting for "Hosting 3rd party libraries locally or using a CDN". Doing it that way makes it more easy to expand.

Of course this will not work for PHP libraries. It seems to me that these should be included in the download package from phpBB.com. When the work on a maintaince release starts (e.g. when the previous maintaince release is released) the most recent version of the library should be merged into the phpBB package. Any function that breaks with the new version of the library should be resolved before releasing the new phpBB version.

[A_Jelly_Doughnut]

Any reason the strategy couldn't be the same as you have now for jQuery?

Good logic. Perhaps the new setting for local/CDN jQuery should be converted to a more general setting for "Hosting 3rd party libraries locally or using a CDN". Doing it that way makes it more easy to expand.

Sorry, I think you've got the wrong discussion in mind. Including jQuery came before there was talk of using a CDN

See Igor's post here for the jQuery inclusion strategy: viewtopic.php?p=227721#p227721

[Oleg]

The requirements for a solution are:

1. It must address how developers would install the package(s) into their development tree.

2. It must bundle the package(s) with phpbb when phpbb packages are built, so that our users do not need to download/install anything extra.

3. It must have an update policy: when we update our copy of the external project, which versions of the external project we are tracking in which versions of phpbb, how do we handle the case when a security issue is found in the external project and it is not fixed in the version that we are tracking (i.e. we must upgrade to a higher major version to get the security fix).

Additionally we have the following nice to haves:

1. We would like to not have huge amounts of third-party code in our tree.

2. We should consider whether it is possible for the user to run a different version of the third-party package (i.e. newer, older or patched) with phpbb, and how this would work.

3. If it is possible to have the third-party package installed systemwide, phpbb should be able to use the systemwide copy.

[naderman]

After thinking about this for a while, I think Eric's original proposal of using git submodules is the most suitable for now. It doesn't require an external tool and gives us all the features we need. The only downside is that we will have to maintain git mirrors of third party repositories if they don't use git, but that's easy enough to do. It simultaneously also allows us to use patched versions of third party libraries if necessary, since we would control the git repository ourselves. One thing to keep in mind with git submodules is that one can only use an entire repository as a submodule and not some subdirectory of another repository like svn:externals. However if we really only want to include a subdirectory of a third party repository we might as well maintain a subtree split branch of that directory (see http://help.github.com/subtree-merge/).

The requirements for a solution are:

1. It must address how developers would install the package(s) into their development tree.

git submodule init
git submodule update

2. It must bundle the package(s) with phpbb when phpbb packages are built, so that our users do not need to download/install anything extra.

Packages will continue to be built the same way, as long as git submodule update was run before building, all checked out third party code will be included in the package.

3. It must have an update policy: when we update our copy of the external project, which versions of the external project we are tracking in which versions of phpbb, how do we handle the case when a security issue is found in the external project and it is not fixed in the version that we are tracking (i.e. we must upgrade to a higher major version to get the security fix).

This policy will have to be decided on a case-by-case basis. Git submodules make it clear which version is tracked in which revision of phpbb code, so all we need to do for each library is determine when/how we want to update to which version.

Additionally we have the following nice to haves:

1. We would like to not have huge amounts of third-party code in our tree.

With git submodules we won't have third party code in our tree.

2. We should consider whether it is possible for the user to run a different version of the third-party package (i.e. newer, older or patched) with phpbb, and how this would work.

Yes would work as explained in the beginning.

3. If it is possible to have the third-party package installed systemwide, phpbb should be able to use the systemwide copy.

I disagree with this one, because I think it's more important to have a consistent phpBB system with precisely the correct version than to save a few bytes of disk space to reuse some system wide installed library. The git submodules solution wouldn't offer this, and you already only marked it as "nice to have". So I think skipping this one should be ok.

[callumacrae]

git submodules <3

What would happen with jQuery? https://github.com/jquery/jquery

It needs to be built.