[RFC] Phpbb4 Extensions Repo organization

Note: We are moving the topics of this forum and it will be deleted at some point

Publish your own request for comments/change or patches for the next version of phpBB. Discuss the contributions and proposals of others. Upcoming releases are 3.2/Rhea and 3.3.
code reader
Registered User
Posts: 653
Joined: Wed Sep 21, 2005 3:01 pm

[RFC] Phpbb4 Extensions Repo organization

Post by code reader » Mon Dec 21, 2009 6:46 pm

This is an RFC for the organization of extensions repo.
it does not speak of the way extensions are constructed, installed, removed, enabled, disabled, configured, queried etc.
it also does not try to fully define the operation of the "package manager" on the client side, although it may describe parts of it to illustrate why certain things are desired or undesired.

if you are interested in this discussion please response with constructive or nonconstructive criticism.
by this i mean, either suggest improvement, or explain why this whole thing should not be done, or suggest a completely different approach to solve the problem.
if/once there is a sense that consensus is reached, this should be transferred to the wiki. maybe i'll do it and maybe someone else will: if you feel like creating the wiki page, please inform us on this thread with a link to this page.


Chapter 0: Extension types
This RFC is not meant to discuss the extensions themselves, but in order to talk about other stuff later on, it makes sense to list the extension types.
  • interface: this extension does not actually have to add anything to the code. basically an interface is a declaration of a list of supported services. any extension that wants to declare itself as "implementing an interface" will have to adhere to these requirements.
  • just "extension". this extension adds functionality by adding some code, possibly visual components, possibly some new language strings. this extension can declare to implement any number of interfaces. an extension can support any number of languages.
  • style. this extension does not add functionality, but rather adds a new "skin" or "style". if "style inheritance" will be supported, a style can declare to inherit another style. a style can be "full" or "partial". for any given session, exactly one "full" style is expected, with zero or more "partial" styles. a style can add new language strings. a style can require an extension to be present.
  • translation: every translation define a language and optional modifiers, e.g. Gender (for gender-sensitive languages) or formality (eg Sie/du). a translation can be "full" or "partial", so, for instance, the "du" translation can override some of the strings in the "Sie" translation while leaving anything else in place. a partial translation can define which full translation it depends on.
  • translation-for-extension: if an extension or style were released *with* language strings but without support for a specific language, a separate "extension translation" can be later released.
Dependencies:
each of the following can depends on minimum kernel version.
-- an Interface depends on zero or more other interfaces.
-- an extension depends on zero or more interfaces, and zero or more other extensions. an extension that depends on an interface requires in order to install and work, at least one other extension that claims to implement this interface. it can also depend on some php extension (e.g. "Calendar"), as well as external services. it can depend on specific DB backend,
-- a style that inherits another style depends on its parent. a style depends on zero or more interfaces, and zero or more extensions.
-- a partial translation depends on a full translation in the same language.
-- an extension translation depends on the extension it translate


Chapter 1: Overview
a phpbb4 extension repo is a site that contain
-- A digest file which serves as an index to all the extensions currently on this site
-- The extensions themselves, where each extension is contained in *a single file* using either .zip, .tar.gz or .tar.bz2 formats.
The digest file always has the same name: phpBB4_extensions.digest
All the extension files are located in relative paths corresponding to the information in the digest, and relative to the location of the digest file, which by definition is considered to be the repo root.
the "name" a phpbb4 repo is exactly the path to the digest file. it may use http://, https:// or ftp:// protocols. the same protocol used to retrieve the digest file should be used to retrive the extensions themselves.

Chapter 2: The Extension Manifest - overview
Each Extension file will include in its root directory a file named phpBB4_extension.manifest

This manifest is written in YAML 1.2 3rd edition ( http://www.yaml.org/spec/1.2/spec.html ), using schema TBD.

The digest, in essence, is an aggregation of all the manifests of all the extensions currently available in the repo, with the addition of the (relative) path to the actual file containing each extension.

in addition, the Digest can contain additional information about each extension which is not supplied by the extension author.
examples are things like "User ratings", "Editors ratings", "Awards and recognition", "Known vulnerabilities", new incompatibilities that were discovered after the release, comments like "Same functionality achieved by more modern/better written/better supported package Y".
in short, any relevant information about the extension that appeared after the release, or that the repo operator wishes to add on top of the author supplied manifest.

The client's ACP will pull the digests from all the repos it configured to poll, combine them to one "master" digest, and will present the results to the user who will then be able to select extensions to be pulled and installed.

Chapter 3: The Extension Manifest - a little more details
The extension manifest will contain the following information:
  • Some unique identifier, guid-like (Question: Should an extension retain the quid across versions?)
  • The Extension name
  • The Extension type (see chapter 0)
  • The extension Category. an extension can have multiple categories. the category will be either from a predefined list, or of type "Other(XXX)" (e.g. "administration", "moderation" "posting", "search", "library" or "Other(telephony)")
  • Author with details: name, nick at phpbb.com, homepage, email, contact info etc.
  • Description: What does it do, why is it worth doing
  • Alternative Descriptions containing the same information in other languages
  • Demo sites using this extension
  • Version
  • Dependencies: Minimum version of phpBB4
  • Dependencies - other extensions (including minimum version of each)
  • Dependencies - PHP libraries (ditto)
  • Dependencies - other (e.g.: this extension only works if you use mysql)
  • Known incompatibilities (i.e., other extensions that are known to be incompatible with this one)
  • Validation info (was it validated by phpbb? someone else?)
  • Does it contain DB Schema changes?
  • List of supported languages (i.e. languages contained in the package)
  • Size: file count and total size of files contained in extension
  • Expected effect on performance
  • (if it's a style:): Does it play nice with RTL?
  • (if it's a style): Accessibility related information
  • (if it's a style): Special device compatibility (iphones, other special

Chapter 4: Tools
There are only one or two tools to help maintain a repo:
-- digest builder
-- mirror creator
The Digest builder scans a list of extensions, and create the .digest file.
A mirror builder pulls the Digest from another site, and knows enough about the Digest structure to locate any extension described in the digest not already present in the mirror. it then pulls all those "new" extensions to the mirror, and concludes by rebuilding the mirror's digest.


Chapter 5: That's all, folks!
as stated above, this is an RFC.
please feel free to comment, including well reasoned arguments explaining why the whole thing is stupid or wrong, how it can or should be made better, pointing to an existing standard that already describe how such a repo should behave, why a specific item described above is wrong or unnecessary, what items are missing etc.
if you feel we reached perfection, or at least "good enough" status (this may happen after certain changes were discussed, or immediately, or never) please say so so we can begin look for volunteer to create the wiki.



peace.


[EDIT, 2009 12 23]:
-- incorporated suggestions by naderman (Thanks!): specifically change the manifest format from ini to YAML, add "interface" extension type.
-- added chapter 0 (extension types)
-- added dependency precedence.
-- added a small paragraph about optional contents in the digest not taken from the extension manifest.
[/EDIT]
Last edited by code reader on Thu Dec 24, 2009 7:11 pm, edited 1 time in total.

Phil
Registered User
Posts: 185
Joined: Sun Mar 11, 2007 3:20 am
Contact:

Re: [RFC] Phpbb4 Extensions Repo organization

Post by Phil » Wed Dec 23, 2009 12:28 am

I don't see anything wrong after a quick glance but TBH I think it is far too early to start commenting/debating RFCs with regard to features that will not be relevant for at least a year or more.

Perhaps we should defer this until an "extension" architecture is written/Titania is finished? Things like packaging format will probably be quite dependent on how "extensions" (have we decided what to call them yet?) are implemented.
My phpbb.com account
Note that any of my opinions expressed in RFC topics are my own and not necessarily representative of the opinion of the phpBB Team.

User avatar
bantu
3.0 Release Manager
3.0 Release Manager
Posts: 557
Joined: Thu Sep 07, 2006 11:22 am
Location: Karlsruhe, Germany
Contact:

Re: [RFC] Phpbb4 Extensions Repo organization

Post by bantu » Wed Dec 23, 2009 1:28 am

Sounds good so far. :)

I'm curious how other package repositories are organised, but I didn't have the time to look at those. I guess there is a lot we can learn from the Linux repositories like apt. Did you look at any of those before compiling the list?

Phil
Registered User
Posts: 185
Joined: Sun Mar 11, 2007 3:20 am
Contact:

Re: [RFC] Phpbb4 Extensions Repo organization

Post by Phil » Wed Dec 23, 2009 1:34 am

IIRC apt functions much like code reader's RFC described.

Out of curiosity code reader, how would you suggest naming be handled? I know there is potential for two different "extensions" to have the same name, which would create a conflict if two repositories were added that contained these separate extensions. I could also see an issue with dependency support.

One of the suggestions that came up within Titania development included extensions generating a special key when packaged from some central location (phpbb.com?) to prevent this from happening (I believe another project does this but cannot remember which one). Any other ideas?
My phpbb.com account
Note that any of my opinions expressed in RFC topics are my own and not necessarily representative of the opinion of the phpBB Team.

code reader
Registered User
Posts: 653
Joined: Wed Sep 21, 2005 3:01 pm

Re: [RFC] Phpbb4 Extensions Repo organization

Post by code reader » Wed Dec 23, 2009 2:01 am

Actually, i think that the repo format can be quite independent of the actual structure/architecture of the add-ons themselves.
the neat thing about it is that the "librarian" piece is almost entirely independent of the "installer" piece, so once the repo format is agreed on, someone (team member and/or community member) can actually begin working on the librarian completely independently of the installer.

it is pretty easy to figure how work on the installer can commence without even mentioning the librarian: after all, we've been doing it for years, ever since the first version of easymod begun development.
but realizing that you can *also* do the other part without having to wait for the definition and/or architecture of the add-on themselves and the installer is kinda cool.

i posted in the past an outline of how i think extensions can be implemented, but the response was that there is already some more sophisticated architecture in discussions.
until some details about these discussions become public, i see little point in discussing the extensions themselves: we want to work *with* the dev-team, not to second-guess them.

i think it is actually good if we can break down the big picture into as-independent-as-reasonable pieces, and discuss specs for each such component independently. this will allow developers and community members to present work-in-progress, and we will have a spec to measure this WIP against.

so in conclusion, i do not think it is premature to present an RFC, especially in the case of an almost independent component.
i think that if other people would do the same for other components, and if we begin giving and receiving egoless critique, constantly evolving those specs it's a great way to push the wagon uphill.

it is also true that an RFC describing the architecture from a bird's view will also be helpful.
such an RFC will list the different components and (some of) the relations between them.
other RFCs that are waiting to be written are:
--objectives (for instance, i would love seeing "performance" listed among the goals)
--design principles (e.g.: how much OO do we really want to see? [if you ask me - not too much...] )
--technology framework (here i think we do have good understanding, but no formalization: straight PHP with no extra dependencies, and support for wide variety of DB)
--level of expected compatibility to phpbb 3.0 or at least 3.1 (something like: "we will keep the current template syntax, while extending it to support X")

even though the practical implications of each such RFC may express themselves at different stage of the work, i do not think that any one of them would be premature if someone bothered to present one now.




unfortunately, so far there was no specific feedback to help either remove this RFC as wrong direction or advancing it forward... :(
if no feedback will come, this can never mature from "RFC" to "spec". that's the whole point of open development.


peace.

User avatar
naderman
Product Manager
Product Manager
Posts: 1727
Joined: Sun Jan 11, 2004 2:11 am
Location: Karlsruhe, Germany
Contact:

Re: [RFC] Phpbb4 Extensions Repo organization

Post by naderman » Wed Dec 23, 2009 3:30 am

iWisdom wrote:I don't see anything wrong after a quick glance but TBH I think it is far too early to start commenting/debating RFCs with regard to features that will not be relevant for at least a year or more.
In fact this is exactly the kind of stuff we should be doing now to get phpBB4 moving.
iWisdom wrote:Perhaps we should defer this until an "extension" architecture is written/Titania is finished?
I don't really see Titania becoming a tool for phpBB4.
code reader wrote:it may use http://, https:// or ftp:// protocols. the same protocol used to retrieve the digest file should be used to retrive the extensions themselves.
Do you have a particular reason for requiring them to be the same? On the other hand I suppose the user would select a method in the ACP and that would be used for everything. So I'm fine with assuming this, just curious.
code reader wrote:phpBB4_extension.manifest
I'm not sure having phpBB4 in the name makes sense for either this or the digest file. I don't see why the 4 is relevant here.
code reader wrote:This manifest has the following structure: it contains sections, using the .ini format, i.e., each section begins with a section name in square bracket, and then a list of "key=value" items.
I'd prefer to use YAML instead which is gaining interest in the PHP community. Read this Interesting blog post summarising the state of YAML in PHP if you'd like to learn more about it. I'd suggest to simply use the PHP5.3 based YAML component.
code reader wrote:Some unique identifier, guid-like (Question: Should an extension retain the quid across versions?)
Yes it should. The extension name should not be used to identify the extension since conflicts could arise. On updating you would otherwise have to keep around a list of "ancestor guids" to find out what the newest version is. Maybe we should simply use a hash over the package contents rather than a guid?
code reader wrote:The Extension type (e.g., plug-in, style, translation)
Would you mind explaining in a bit more detail how you envision the plug-in, style and translation dependencies to work? What results from this extension type?
code reader wrote:The extension Category (e.g. "administration", "moderation" "posting", "search", "library")
One category only or tags? The later might make more sense since e.g. a library could still be search related.
code reader wrote:Known incompatibilities (i.e., other extensions that are known to be incompatible with this one)
When I was presenting an idea like this one to the MOD team before I also thought about a simple way to report back conflicts that arise during installation. Obviously back then it was all about MODs that edit files which easily results in conflicts. Do you think this would still make sense?
code reader wrote:List of supported languages (i.e. languages contained in the package)
I thought language packages were a separate extension type?
code reader wrote:Expected effect on performance
How would you measure / specify that?

User avatar
naderman
Product Manager
Product Manager
Posts: 1727
Joined: Sun Jan 11, 2004 2:11 am
Location: Karlsruhe, Germany
Contact:

Re: [RFC] Phpbb4 Extensions Repo organization

Post by naderman » Wed Dec 23, 2009 3:34 am

Oh and as an addition I suggest some kind of extension interface specification. Basically I'd like multiple libraries/extensions to be able to specify that they implement a particular interface. Another extension can then depend on that interface rather than on a particular other extension. I remember the point system vs. cash mod from phpBB2 where something like this would have helped a lot.

bobtheman
Registered User
Posts: 63
Joined: Sat Dec 19, 2009 4:00 pm

Re: [RFC] Phpbb4 Extensions Repo organization

Post by bobtheman » Wed Dec 23, 2009 3:49 am

until some details about these discussions become public, i see little point in discussing the extensions themselves: we want to work *with* the dev-team, not to second-guess them.
either lack of direction, lack of community involvement, or "secrecy" has hindered the development of phpbb in the past... in the past the community wasn't properly informed until it was to late for any type of input, assuming the input would of been taken anyways. im hoping this has come to an end.

code reader
Registered User
Posts: 653
Joined: Wed Sep 21, 2005 3:01 pm

Re: [RFC] Phpbb4 Extensions Repo organization

Post by code reader » Wed Dec 23, 2009 2:25 pm

bobtheman wrote:
until some details about these discussions become public, i see little point in discussing the extensions themselves: we want to work *with* the dev-team, not to second-guess them.
either lack of direction, lack of community involvement, or "secrecy" has hindered the development of phpbb in the past... in the past the community wasn't properly informed until it was to late for any type of input, assuming the input would of been taken anyways. im hoping this has come to an end.
Please do not be too harsh.
even in an open development environment, it does not make sense to require that everyone will present everything at every stage.
for instance, when i work on something, i prefer to edit it to the point that i can be proud to show it to others, or at the least to the point i am not embarrassed to show it.
getting something to this stage may take some time.
it still make sense for me to say: "hey community, i am currently working on XYZ and plan to present it shortly".
someone else might say "i know this guy, his work is no good and i can do XYZ better" and continue to develop a separate spec/implementation/documentation that will compete with mine, or they might think: "let's see what code reader come up with and *than* slam it".

either way, i think it's kosher to tell the community "i am working on XYZ" before actually exposing it.

of course, saying something like this and then disappearing for a long time is not good, because it leaves the community in limbo (like we were when the "community projects" of summer 2005 were announced and then commenced with no visibility, progress reports or even termination notice).

peace.

code reader
Registered User
Posts: 653
Joined: Wed Sep 21, 2005 3:01 pm

Re: [RFC] Phpbb4 Extensions Repo organization

Post by code reader » Wed Dec 23, 2009 3:29 pm

naderman wrote:In fact this is exactly the kind of stuff we should be doing now to get phpBB4 moving.
thanks for the encouragement and support.
and even more, thanks for the critique.
after posting this i will update the original post of this thread to reflect changes/refinements made per your response.

following are my views on the points you raised.
naderman wrote: code reader wrote:it may use http://, https:// or ftp:// protocols. the same protocol used to retrieve the digest file should be used to retrive the extensions themselves.


Do you have a particular reason for requiring them to be the same? On the other hand I suppose the user would select a method in the ACP and that would be used for everything. So I'm fine with assuming this, just curious.
in my mind it makes sense. a certain repo may be visible to a certain phpbb installation through http but not https or vice versa, e.g. as a result of technology (maybe the php installation is configured to block https:// file retrieval) or, say, some firewall in the way.
by using the same protocol you practically guarantee that if you successfully pulled the manifest you'll be able to pull extensions.
(and also, the protocol is effectively a part of the URI used to identify the repo....)
naderman wrote:I'm not sure having phpBB4 in the name makes sense for either this or the digest file. I don't see why the 4 is relevant here.
"4" in itself maybe not, but something that identify the product being served does make sense, so if someone feeds the librarian a URI that's really serves some other product (say, phpbb5) it will not get confused and get the wrong manifest.
however, i do not feel strongly about this point.
naderman wrote:I'd prefer to use YAML instead which is gaining interest in the PHP community. Read this Interesting blog post summarising the state of YAML in PHP if you'd like to learn more about it. I'd suggest to simply use the PHP5.3 based YAML component.
makes sense. adding to spec.
naderman wrote:Yes it should. The extension name should not be used to identify the extension since conflicts could arise. On updating you would otherwise have to keep around a list of "ancestor guids" to find out what the newest version is. Maybe we should simply use a hash over the package contents rather than a guid?
avoiding name conflict was the main reason i added this, so one will be able to say: "depends on extension X" without worry that there may be a completely different extension that also chose X for its name.
i agree that this identifier should be unchanged for any extension: the "full" identifier for an extension will be this unique one plus the version.
naderman wrote:Would you mind explaining in a bit more detail how you envision the plug-in, style and translation dependencies to work? What results from this extension type?
to some extent the different types and how they behave depends on the yet-to-be-discussed extension architecture. i will elaborate more below, when discussing dependencies.
naderman wrote:One category only or tags? The later might make more sense since e.g. a library could still be search related.
no strong opinion here. the idea is to give some "sorting, filtering and statistics sugar".
i think i will not touch this part because i did not put enough thought into it. if someone wishes to elaborate i'll try to incorporate it into the spec, or once it's in the wiki hopefully someone will make it useful.
naderman wrote:When I was presenting an idea like this one to the MOD team before I also thought about a simple way to report back conflicts that arise during installation. Obviously back then it was all about MODs that edit files which easily results in conflicts. Do you think this would still make sense?
the question of detecting incompatibilities, and hopefully automating it is important and interesting, but is not relevant to the repo spec.
we should move this question to another area: maybe open a new thread?
(btw: the whole huge subject of testing and verification was not even mentioned yet anywhere that i saw. if there is a community members with expertise in the area, please open a thread)
naderman wrote:I thought language packages were a separate extension type?
what i meant is that a language pack for the core product can be packaged as an extension (what we call now language pak).
i think it might be awkward to require that each extension will have to supply its language strings in a separate package.
however, it should be possible to publish a "language pak" just for a specific extension.
i will try to explain by an example:
let's say someone publishes an extension, and let's say i operate a phpbb board in Hungarian. i really like the extension, so i take it, translate its 12 strings to Hungarian, and use it.
then i decide to contribute back to the community and pack those 12 strings in Hungarian as a separate package, and publish this package as "Hungarian translation for extension X"
i also inform the original author.
so, by the next version of this cool package, the author packages the Hungarian strings in the extension, so people who get the newer version will not have to download *my* package.
in order for this to work smoothly, it's best if the manifest can list the languages included in the package.


naderman wrote: code reader wrote:Expected effect on performance
How would you measure / specify that?
this is the same issue as "known incompatibilities": i do not have an answer to the question "how was the information obtained", but this should not preclude us from defining how the information is presented.
i specifically thought this might be useful for extensions whose main purpose is to *improve* performance. think of the Xcache plugins of phpbb3: it would be nice if they could claim what kind of performance gain should be expected.
it may also be useful for extensions that add functionality with some known performance cost - e.g., let's imagine "print as pdf" or "support LaTex" extensions. it is conceivable that the author can specify expected performance cost, so people with not-strong-enough servers can save the trouble of installing it just to discover their server can't support it, or at least they will know to stress-test after installation.
i expect the vast majority of extensions to ignore this field (or just state "none".
naderman wrote:Oh and as an addition I suggest some kind of extension interface specification. Basically I'd like multiple libraries/extensions to be able to specify that they implement a particular interface. Another extension can then depend on that interface rather than on a particular other extension. I remember the point system vs. cash mod from phpBB2 where something like this would have helped a lot.
i think this is brilliant. i will try to incorporate it into the RFC.



it might take me a while to edit the original post, though, but it should not be later than EOD tomorrow.


peace.

Post Reply