phpbb Testing Architecture

[Oleg]

Overview

Testing phpbb code requires too much repeated effort which can be automated. This RFC proposes to create a test system that will automate much of the repetitive tasks done during testing.

Intended Audience and Prerequisites

This testing architecture is something I would like to use myself for testing phpbb code. I suspect there may be wider demand for it or something similar, which is what this RFC is intended to determine. Still I need to start somewhere and limit the scope, so here are constraints that I am happy with. If there is sufficient interest they of course may be amended or removed.

* Unix console only. These tools are for people who can use the shell and run console scripts. There are no plans for windows support or gui of any kind. I will try to support cygwin as much as I can.
* Free choice of programming languages. This means shell, php, python and ruby. Deployment will be done via capistrano, which is written in ruby. Scripts will be written in whichever one of shell, php or python is most convenient.

Goals

Make it possible to easily configure a board with any combination of the following properties:

* Vanilla released version
* Experimental code from github (any github user/branch)
* Any feature version (3.0/3.1)
* Any supported database system and version
* Any data set (publicly available, test systems should not be trusted)

The following are explicitly excluded:

* Any modifications that must be installed through a manual process (currently, all of them)

Process

Alice wants to test feature snazzy on postgres 8.0.

1. Alice obtains a user account on test system.
2. Alice specifies which code to deploy (alice@github:feature/snazzy)
3. Alice specifies which database to use (postgres/8.0)
4. Alice specifies which data set to import (none - she will install the board herself)
5. Alice puts all of these settings into a config file, and runs provisioning script
6. Alice is given a url of her board ( http://alice-1234.testing.com)
7. Alice goes to http://alice-1234.testing.com/install/, installs the board
8. Alice runs provisioning script, instructing it to finalize the installation (remove install dir, fix permissions)
9. Alice uses installed board

Custom setup

There will be provisions to load data or execute code to configure an installation. The following options are to be provided:

1. Import sql dump
2. Run php code (in phpbb environment)

The idea here is to store code/data required separately from phpbb code, so that the setup may be used for different versions or different purposes, and may be shared between users. Some examples of custom setups:

* Huge forum count: import 1k forums.
* Huge board: import 100k posts with associated threads and users.
* Active board: create 1k user sessions (f.e. for testing who's online behavior)
* Broken styles: set board to use a non-existent style

It is expected that custom setups will be stored in a version control system (e.g. git/github) as opposed to being directly uploaded to test server.

Extras

Alice will have shell access to the test server. The board will be installed under her user account.

Authentication is to be done via ssh keys.

Comparison with Existing Alternatives

Manual Installation

Besides copying/extracting the code in a subdirectory under webroot, the following must be done:

* Create database
* Grant privileges to database to user
* Set directory permissions

Running multiple database engines on one system is not necessarily something everyone wants to do. There is a total of 15 free database engines currently supported (sqlite, mysql 4, 4.1, 5.0, 5.1, 5.4, 5.5, postgres 7.3, 7.4, 8.0, 8.1, 8.2, 8.3, 8.4, firebird 2.1). The test server will either have all of them running at all times or will provide an option to start and stop the less commonly used ones easily if they turn out to need too much memory.

phpbb Quick Install

Does not support installing different code versions. Does not actually work for me. Other issues?

Concerns

The testing architecture as envisioned essentially allows random people to execute arbitrary code on the server. This is going to be addressed as follows:

1. Every user will use their own user account. Initially every user must apply for their own user account. Later there may be a public guest account, and people who want to have their own account may apply for one.
2. Administrative tasks will be performed via wrapper scripts that will enforce good behavior. E.g. users can trigger database creation, but database creation script will ensure that database name does not conflict with other users (f.e. prefixed with user name of the user who requested it).
3. Every user's code will be executed under a separate user account. I plan to achieve this either by running php scripts via cgi or using persistent fastcgi processes per user (starting with one, more if necessary).
4. The test system will be kept up-to-date.
5. The test system will be isolated from other systems and will run no unrelated services.

Irrespective of these measures, the test system is not to be trusted. Any password used on the test system is to be considered publicly exposed. Fortunately, phpbb source code is public so this is not expected to be a significant issue.

Initially I was going to run the test system as a freebsd jail on a dedicated server. Unfortunately postgres does not play nice with jails in this application. In order to support postgres and achieve the level of security I require the test server must be a dedicated piece of hardware. Currently that means a machine in my apartment on its own network. Depending on demand I may install a test system supporting all other database engines on a dedicated box.

[Oleg]

I'm gathering demand for such a system. Would you:

* use this system if I administered it?
* use it if phpbb administered it?
* use it if you yourself administered it?
* use it if it was implemented differently?

If you think this is a pointless undertaking please post too.

[ToonArmy]

I am working on off on a script to build up virtual machines on Ubuntu of various configurations. It'd be very nice to somewhere down the line have a test infrastructure built up and even have people contributing CI nodes to run our test suites.

[Oleg]

Any code? More details?

It sounds like our ideas are somewhat orthogonal. My proposal is basically about deploying a large number of different application packages into a stable server environment. Your proposal is about creating a variety of server environments for one or few application packages.

FWIW my proposal does not require particular os type or version, in fact it may end up being deployed on debian first.

[ToonArmy]

I'm trying to take some time out so it was a very brief reply, I got the link and everything but forgot about pasting it: http://github.com/cs278/phpbb-vm-builder

While we're not talking about the same thing, they're related and certainly both fall under the scope of the topic title.

[Oleg]

No interest at all, huh? Is this because everyone is happy with how they test the code, or because no one tests much?

Who runs development code (trunk) on a board they actually use?

[DavidIQ]

Who runs development code (trunk) on a board they actually use?

*raises hand*

The dev team has a pretty good testing procedure now. I'll let them talk about that.

[igorw]

phpbb Quick Install

Does not support installing different code versions. Does not actually work for me. Other issues?

I haven't touched it for ages, so I'd assume it's not working.


I am not quite sure if your suggestion is for manual testing or for unit tests. For manual testing, I don't think it makes much sense to deploy into several different environments. This is something that should be covered by unit tests.

Manual installation is a pain. I have built a CLI script based on Chris' VM builder, that does automated installation. You can find it on github. It's a PHP-based HTTP client for the phpBB installation with error handling etc. I intend to add more features (like quickinstall had).

Perhaps you could state your target audience and use cases a bit more clearly. Unit testing vs manual testing. Is this something for the QA team? How exactly would the person using it use it? I'm just trying to get a better idea here.

[DavidIQ]

phpbb Quick Install

Does not support installing different code versions. Does not actually work for me. Other issues?

I haven't touched it for ages, so I'd assume it's not working.

Oh I didn't notice this before. QI is being actively worked on by tumba25. Will be provided to the MOD community in the coming weeks/months.

[Oleg]

I have built a CLI script based on Chris' VM builder, that does automated installation. You can find it on github.

Are you happy with it? Does it do everything you need? It seems fairly limited in scope.