phpbb Testing Architecture

[igorw]

It is still very limited. I intend to add more features such as creating the db, copying vanilla board files, file-based configuration with mappings, maybe automod (easy now since it uses UMIL). Basically a client-side CLI equivalent of quickinstall. If you have suggestions as to what it could do, feel free to tell me.

[naderman]

We do have continuous integration now: http://bamboo.phpbb.com/ This is still very much work in progress. It's also integrated with the tracker: http://tracker.phpbb.com/browse/PHPBB3- ... issue-tabs We intend to set up bamboo "agents" on various platforms running the tests on different database systems, operating systems, PHP versions, etc.

[Oleg]

We do have continuous integration now: http://bamboo.phpbb.com/

I see it tests trunk, what about branches that are not yet merged into trunk and/or are under (active) development?

[naderman]

Currently it only tests develop-olympus. It'll be set up to test the other main branches too. I'm not entirely sure how we can reuse the infrastructure for testing during development yet. I don't think that's something for continuous integration. But I do think the infrastructure we have to set up for testing on different systems should be available for anybody who needs to run development tests, too.

[Oleg]

Perhaps you could state your target audience and use cases a bit more clearly. Unit testing vs manual testing. Is this something for the QA team? How exactly would the person using it use it? I'm just trying to get a better idea here.

Restrictively defined, the goal of phpbb proving ground (working name) is to allow developers to rapidly provision phpbb environments for testing code under development.

Conceivably testers (including QA team) can also provision phpbb environments for testing code after it is "finished", but barrier to entry may be too high given constraints described in the first post.

By "phpbb environment" I mean a combination of the following:

1. Code. Any branch of any fork of any version of phpbb (phpbb3/phpbb4) specifically is to be supported in initial release. Subversion checkouts are possible under the architecture but are not a priority until requested. Said differently, any developer should be able to deploy any code derived from phpbb source tree for testing.

2. Database. All databases supported by phpbb and requested by developers are to be available at all times.

3. Data. Ppg allows defining sets of data that can be rapidly loaded into the database. The goal is to make it easy to obtain states that are normally not easily obtainable. Datasets are independent from code, databases (as much as possible) and other factors, and are expected to be shared among developers. Using a dataset built by someone else is to be trivial.

4. Web server and php execution model. As with data the goal is to make it easy to obtain unusual combinations, such as apache+fastcgi.

ppg and manual testing

Until there is much more unit test coverage, ppg is expected to be used primarily for manual testing. Here "manual testing" refers only to the actual testing part; preparations such as setup and data loading are to be automated and are the scope of ppg.

ppg vs unit testing

Unit testing requires having a database configured, writable directories arranged, and perhaps some initial data loaded. Ppg takes over these processes. Once code is deployed to ppg, unit tests can be executed using whichever unit test framework/runner are in use.

Ppg allows in this case:

1. Unit testing on databases developer does not have installed (vs testing locally)
2. Unit testing code that is not merged into main development branch(es) (vs testing in continuous integration tool)

Ppg makes it possible to test things that are hard to write unit tests for:

1. Performance testing, including actually using the board in the state being tested
2. Testing things other than php code, for example mail submission and/or delivery

ppg vs continuous integration

Continuous integration usually works on a small set of branches, using code that is "finished". It is not possible to test an arbitrary source tree, and code which is not finished (under active development) is not submitted to continuous integration tools.

Unlike bamboo, ppg is free software.

Use cases

For manual testing, I don't think it makes much sense to deploy into several different environments. This is something that should be covered by unit tests.

Let's suppose you are developing a feature like this one: http://tracker.phpbb.com/browse/PHPBB3-9173 You can write a unit test, but do you have firebird and postgres from 7.4 through 8.4 installed on your system? Even if all these databases are supported by CI tool eventually CI only tests code committed to main branch and does not help you during development.

With ppg you only need to commit the changes to your branch and then create 5 different configuration files (~5 lines each) and a dataset for required conditions (specifically the current attachment size sum should be large), and ppg will give you 5 boards that you either 1) run unit tests on or 2) login to, post attachment and check new file size (what would have happened presently).

A short time after the administrators try to send a mass PM to all the users on the board, some pages don't load due to a maximum execution error on the cache/queue.php.
The forum obviously tries to send an email to all the users to notify about the new PM, and for some reason the queue system fails.

http://www.phpbb.com/community/viewtopi ... &t=2042715

Unit testing is not helpful here because to start, the problem is not known. To even begin you need a board with a large number of users to spam, and you obviously don't want to spam real users so you need to specify special email addresses that you own for everyone. Ppg can address this use case by providing an MTA that delivers mail locally, so it's a real MTA as far as phpbb can tell but perfectly harmless, and providing a dataset consisting of 2000 users with appropriately configured email addresses.

[RFC] Modular cron

viewtopic.php?f=84&t=32772

This is a large chunk of code with nontrivial changes. Testing it requires resetting the board to a particular state where none of the cron tasks have run recently, there are forums that need to be pruned and not pruned, both forums should have topics in certain time ranges. Running the cron tasks destroys the state, and getting it back gets old very quickly. Ppg addresses the issue by storing the state in a format that allows it to be loaded with a single command.

How exactly would the person using it use it?

I take documentation for this project seriously, essentially developing it documentation-first. There isn't much documentation yet because there isn't much code written, but here is a start that should give some idea: http://svn.bsdpower.com/webtools/phpbb- ... kflows.txt

[igorw]

Thanks for the thorough explanation.

It seems like quite an effort, and I am a little skeptical as to if it will pay off. The idea sounds great. Where I see the most potential is for the QA team. Their job is to verify bug fixes. These bugs have not yet been merged, etc. So the best thing imo would be an interface that is really easy. Possibly a web-based interface.

The source would default to github and the repo would default to phpbb3. All the QA team member would have to do is select the user and branch (eg. ticket id) from a menu. The system would then create an instance of phpBB (my or chris' script may be of use) and redirect the user to the appropriate URL, allowing him to test it.

Developers can also use it, of course, but they are more likely to test it locally.

[Oleg]

The source would default to github and the repo would default to phpbb3. All the QA team member would have to do is select the user and branch (eg. ticket id) from a menu. The system would then create an instance of phpBB (my or chris' script may be of use) and redirect the user to the appropriate URL, allowing him to test it.

Agreed. A web frontend is probably required in order for the system to be usable by qa. I'm open to the idea of implementing one, but not without some commitment from potential users to use it.

Developers can also use it, of course, but they are more likely to test it locally.

I see. If developers only work on features that they can test on their local boards they have no need for other configurations.