Asset management

General discussion of development ideas and the approaches taken in the 3.x branch of phpBB. The next feature release of phpBB 3 will be 3.3/Proteus.
Forum rules
Please do not post support questions regarding installing, updating, or upgrading phpBB 3.2.x. If you need support for phpBB 3.2.x please visit the 3.2.x Support Forum on phpbb.com.

If you have questions regarding writing extensions please post in Extension Writers Discussion to receive proper guidance from our staff and community.
User avatar
callumacrae
Former Team Member
Posts: 1046
Joined: Tue Apr 27, 2010 9:37 am
Location: England
Contact:

Re: Asset management

Post by callumacrae »

only works if you're literally just doing minification and concatenation. doesn't work if you want stuff like autoprefixer, also doesn't work if you want the build process to pull in libraries from npm instead of including the jquery source in the repo.
Made by developers, for developers!
My blog

Nicofuma
3.2 Release Manager
3.2 Release Manager
Posts: 299
Joined: Sun Apr 13, 2014 1:40 am
Location: Paris

Re: Asset management

Post by Nicofuma »

For the recall, that's what I think:

Goal: We need to be more attractive to frontend developers and contributors

Assumptions:
  • Frontend contributors wants to use modern tools: gulp, less/sass, npm etc...
  • They want to use tools they are used to
  • frontend code === js/css
  • backend code === php
  • glue code (nor backend or frontend) === templates
  • A frontend dev shouldn't have to deal with backend code and tools (just once to install the board)
  • A frontend dev shouldn't have to know anything about php
  • A backend dev shouldn't have to deal with frontend code and tools (never)
  • Manually embedding the libs in the repo is a bad practice
  • It should still be possible to make minor contributions to the frontend without having to deal with frontend tools
  • Major contributions to the frontend must use frontend tools
  • The end user must be able to tweak the css/js
To sum-up: to install phpBB from the main repo the only thing you should have to do is to have php/apache/mysql (or any other web server and database) installed

And for me it means:
  •  We can use frontend tools using nodejs
  • A ready to use js and css must be committed in the repo
  • The contributor should embed the generated files in his PR if possible. If not someone else will have to take his PR over and send another one with the generated files. (for this point, the contributors used to frontend tools will have to be nice :) @brunoais @callumacrae I can do it too if we prepare a ready to use docker image)
  • We should provide a docker image (and maybe a vagrant box) with clear instructions on how to run the tools (for merger mainly)
  •  What I mean by ready to use docker image is that the only thing you need to do is to run from phpBB root: docker run --rm -ti -v $PWD:/phpbb phpbb/frontend-tools generate or something like that)
But we still have a few issues, mainly for the end user:
  • How can an extension replace a css or js file? for me it's unsupported. An extension adds what is needed for itself. To change anything in the design create a new style
  • How to give the end user the ability to tweak the design? For me, If we use minified files the work should be done by INCLUDEJS and INCLUDECSS: search for a minified file including the requested one; if found: add the minified file to the files to load; if not add the requested file. This way the admin just have to remove the minified files. (it could even be done automatically by embedding a checksum of the individuals file: if the cheksum of one of the requested file have changed; don't load the minified one).
  • It means that we have to ship: minified files and individual generated files (prettified). If the user has enough knowledges with frontend tools he could crab the source files from the repo. (I don't think that we should ship the less/sass/whateveryouwant source files because it means we will also have to ship dev dependencies and frontend tools config file).
Member of the phpBB Development-Team
No Support via PM

User avatar
callumacrae
Former Team Member
Posts: 1046
Joined: Tue Apr 27, 2010 9:37 am
Location: England
Contact:

Re: Asset management

Post by callumacrae »

Imo, front-end developers contributing to phpBB should also be able to modify the front-end tooling.
Made by developers, for developers!
My blog

User avatar
Elsensee
Development Team
Development Team
Posts: 42
Joined: Sun Mar 16, 2014 1:08 pm
Location: Hamburg, Germany
Contact:

Re: Asset management

Post by Elsensee »

So we allow forum admins to change the js by removing the *.min version. What about updates? Changes in the js would mean a new *.min.js version as well so the custom javascript would be useless until the *.min.js is removed again. I don't want to let the support team deal with tons of questions about this since we could somehow solve this issue, can't we?

User avatar
brunoais
Registered User
Posts: 964
Joined: Fri Dec 18, 2009 3:55 pm

Re: Asset management

Post by brunoais »

@Elsensee
Just use a file with the metadata related to that. No worries ;). Just a file that the template system resorts to to know what relates to what for names.

Nicofuma
3.2 Release Manager
3.2 Release Manager
Posts: 299
Joined: Sun Apr 13, 2014 1:40 am
Location: Paris

Re: Asset management

Post by Nicofuma »

Nicofuma wrote:it could even be done automatically by embedding a checksum of the individuals file: if the cheksum of one of the requested file have changed; don't load the minified one
Member of the phpBB Development-Team
No Support via PM

User avatar
callumacrae
Former Team Member
Posts: 1046
Joined: Tue Apr 27, 2010 9:37 am
Location: England
Contact:

Re: Asset management

Post by callumacrae »

I don't see how revisioning the filename would solve the problem
Made by developers, for developers!
My blog

Nicofuma
3.2 Release Manager
3.2 Release Manager
Posts: 299
Joined: Sun Apr 13, 2014 1:40 am
Location: Paris

Re: Asset management

Post by Nicofuma »

I'm talking about a checksun of the file content (or filename + mtime)
Member of the phpBB Development-Team
No Support via PM

User avatar
nickvergessen
Former Team Member
Posts: 733
Joined: Sun Oct 07, 2007 11:54 am
Location: Stuttgart, Germany
Contact:

Re: Asset management

Post by nickvergessen »

callumacrae wrote:
Nicofuma wrote: [*] In any case we will ever add a required dependency to a frontend dev tool. So you need to propose something that make these tools optional for the contributor. (You should be able to install a dev board without having to deal with these tools).
The only way for that to be possible is to commit the compiled code. That's possible, but it's a bit ugly, and if a malicious contributor makes a change to the compiled code, we probably wouldn't notice (although it would be overwritten the next time the build tool was ran).
In case it has not been set yet. We had that issue with the database schema files as well. We can easily check that on travis and add it to the release procedure.
Member of the Development-TeamNo Support via PM

User avatar
brunoais
Registered User
Posts: 964
Joined: Fri Dec 18, 2009 3:55 pm

Re: Asset management

Post by brunoais »

This is my current proposal. It is built from all the ideas and proposals given above, specially by callumacrae, Nicofuma and CHItA. It is as follows:
Note: Unless explicitly mentioned as being specific to JS or CSS, the information applies to both.

Add 2 metadata files (the location is TBA) which stores information about the minimization process.
File 1:
  • Static information.
  • Information about file groups, their location and exclusion lists.
  • Final destination for each file group (includes file name).
  • Extensions can modify the information later parsed from this file (indirectly, using events, a file in their own extension, etc... (TBD)).
  • Is versioned in the repository.
File 2:
  • Dynamic information (no human should manually modify it).
  • Contains the metadata required for the correct management of the minimized files.
  • The actual metadata that is fully useful is TBD but I already have some ideas (read below).
  • Is not available in the repository.
Create a gulp.js file.
  • Reads from file 1 mentioned above.
  • Writes to file 2 mentioned above.
  • Contains definitions to run jsHint (optional; real below).
  • Contains definitions to run fixmyjs (optional; real below).
  • Contains definitions to run UglifyCSS and UglifyJS.
    • They minify and generate sourcemaps.
    • Metadata of the process goes to File 2.
  • Create an HTTP-compatible gzip version of the file(s) (if such exists).
Metadata for file 2:
  • Store JSON key-value object.
  • Store, for each minified file,
    • which files it is sourced from.
    • Is there a gzip version of it.
Include an asset manager in phpBB:
  • Runs on 2 components:
    1. component:
      • Reads from file 1 and file 2 mentioned above.
      • Control which file to deliver between gzip and non-gzip at request phase.
      • Deliver appropriate HTTP Cache headers.
      • Calls the 2nd component mentioned below, when appropriate.
      • Used by the template system (related to INCLUDEJS and INCLUDECSS' internals)
      • Is "Recompile stale style components" (or a new option) aware and acts accordingly.
      • Knows how to translate an origin URL into the sprite URL and CSS rule.
    2. component:
      • Reads from file 1 mentioned above.
      • Writes to file 2 mentioned above.
      • It runs on a best-effort with available tools to concatenate and minimize files with sourcemaps.
      • Uses ClosureCompiler to minimize javascript if so chosen.
      • Creates gzip versions of files not gziped.
      • Creates sprited icons.
Both gulp and the asset manager must not collide with each other (unless abused, ofc).
To deal with concurrency, gulp.js should contain code to generate a hidden file. While the file is there, it is working and the 2nd component of the asset manager must not execute. I already found a flawless way in windows (creating and unlinking right after) and an almost flawless way in windows and in GNU/linux (handling all handlable signals related to terminating the process).
The gulp.js script should include code for a watcher for convenience (and not terminate immediatly) which can be turned off (specially for Travis) with a command line argument.


Important:Travis is supposed to contain all the required and optional elements of the gulp.js file for testing purposes and best output helping output text.
---------------------------------------------------------------------------------------------

Main advantages:
  • Gulp user or not, you can get the assets managed, minimized and gziped for HTTP requests.
  • The assets can be delivered in gzip regardless of server settings (the same file doesn't have to be minimized multiple times as the server drops previous minimized versions).
  • Extensions can advantage from this as the Asset Manager can converge all CSS and all JS from all sources into a single file, if so desired (although this not being the optimal way).
Main disadvantages:
  • Although the minimization tools are ready to use on php, they are not as powerful as the ones written in javascript.
  • Writing or merging the asset manager will take some work. The proposed Assetic can't do this.
Extra: If Node and gulp are installed in the server, it can take full advantage of that... but only users in VPS or dedicated phpBB servers will ever be able to install gulp and I don't think that most ever will.

I believe I could write the code for this but I won't be able to have it ready for phpBB 3.2.

Post Reply