Asset management
Forum rules
Please do not post support questions regarding installing, updating, or upgrading phpBB 3.3.x. If you need support for phpBB 3.3.x please visit the 3.3.x Support Forum on phpbb.com.
If you have questions regarding writing extensions please post in Extension Writers Discussion to receive proper guidance from our staff and community.
Please do not post support questions regarding installing, updating, or upgrading phpBB 3.3.x. If you need support for phpBB 3.3.x please visit the 3.3.x Support Forum on phpbb.com.
If you have questions regarding writing extensions please post in Extension Writers Discussion to receive proper guidance from our staff and community.
- callumacrae
- Former Team Member
- Posts: 1046
- Joined: Tue Apr 27, 2010 9:37 am
- Location: England
- Contact:
Re: Asset management
only works if you're literally just doing minification and concatenation. doesn't work if you want stuff like autoprefixer, also doesn't work if you want the build process to pull in libraries from npm instead of including the jquery source in the repo.
Re: Asset management
For the recall, that's what I think:
Goal: We need to be more attractive to frontend developers and contributors
Assumptions:
And for me it means:
Goal: We need to be more attractive to frontend developers and contributors
Assumptions:
- Frontend contributors wants to use modern tools: gulp, less/sass, npm etc...
- They want to use tools they are used to
- frontend code === js/css
- backend code === php
- glue code (nor backend or frontend) === templates
- A frontend dev shouldn't have to deal with backend code and tools (just once to install the board)
- A frontend dev shouldn't have to know anything about php
- A backend dev shouldn't have to deal with frontend code and tools (never)
- Manually embedding the libs in the repo is a bad practice
- It should still be possible to make minor contributions to the frontend without having to deal with frontend tools
- Major contributions to the frontend must use frontend tools
- The end user must be able to tweak the css/js
And for me it means:
- We can use frontend tools using nodejs
- A ready to use js and css must be committed in the repo
- The contributor should embed the generated files in his PR if possible. If not someone else will have to take his PR over and send another one with the generated files. (for this point, the contributors used to frontend tools will have to be nice @brunoais @callumacrae I can do it too if we prepare a ready to use docker image)
- We should provide a docker image (and maybe a vagrant box) with clear instructions on how to run the tools (for merger mainly)
- What I mean by ready to use docker image is that the only thing you need to do is to run from phpBB root:
docker run --rm -ti -v $PWD:/phpbb phpbb/frontend-tools generate
or something like that)
-
How can an extension replace a css or js file?for me it's unsupported. An extension adds what is needed for itself. To change anything in the design create a new style - How to give the end user the ability to tweak the design? For me, If we use minified files the work should be done by INCLUDEJS and INCLUDECSS: search for a minified file including the requested one; if found: add the minified file to the files to load; if not add the requested file. This way the admin just have to remove the minified files. (it could even be done automatically by embedding a checksum of the individuals file: if the cheksum of one of the requested file have changed; don't load the minified one).
- It means that we have to ship: minified files and individual generated files (prettified). If the user has enough knowledges with frontend tools he could crab the source files from the repo. (I don't think that we should ship the less/sass/whateveryouwant source files because it means we will also have to ship dev dependencies and frontend tools config file).
Member of the phpBB Development-Team
No Support via PM
No Support via PM
- callumacrae
- Former Team Member
- Posts: 1046
- Joined: Tue Apr 27, 2010 9:37 am
- Location: England
- Contact:
Re: Asset management
Imo, front-end developers contributing to phpBB should also be able to modify the front-end tooling.
- Elsensee
- Former Team Member
- Posts: 42
- Joined: Sun Mar 16, 2014 1:08 pm
- Location: Hamburg, Germany
- Contact:
Re: Asset management
So we allow forum admins to change the js by removing the *.min version. What about updates? Changes in the js would mean a new *.min.js version as well so the custom javascript would be useless until the *.min.js is removed again. I don't want to let the support team deal with tons of questions about this since we could somehow solve this issue, can't we?
Re: Asset management
@Elsensee
Just use a file with the metadata related to that. No worries . Just a file that the template system resorts to to know what relates to what for names.
Just use a file with the metadata related to that. No worries . Just a file that the template system resorts to to know what relates to what for names.
Re: Asset management
Nicofuma wrote:it could even be done automatically by embedding a checksum of the individuals file: if the cheksum of one of the requested file have changed; don't load the minified one
Member of the phpBB Development-Team
No Support via PM
No Support via PM
- callumacrae
- Former Team Member
- Posts: 1046
- Joined: Tue Apr 27, 2010 9:37 am
- Location: England
- Contact:
Re: Asset management
I don't see how revisioning the filename would solve the problem
Re: Asset management
I'm talking about a checksun of the file content (or filename + mtime)
Member of the phpBB Development-Team
No Support via PM
No Support via PM
- nickvergessen
- Former Team Member
- Posts: 733
- Joined: Sun Oct 07, 2007 11:54 am
- Location: Stuttgart, Germany
- Contact:
Re: Asset management
In case it has not been set yet. We had that issue with the database schema files as well. We can easily check that on travis and add it to the release procedure.callumacrae wrote:The only way for that to be possible is to commit the compiled code. That's possible, but it's a bit ugly, and if a malicious contributor makes a change to the compiled code, we probably wouldn't notice (although it would be overwritten the next time the build tool was ran).Nicofuma wrote: [*] In any case we will ever add a required dependency to a frontend dev tool. So you need to propose something that make these tools optional for the contributor. (You should be able to install a dev board without having to deal with these tools).
Member of the Development-Team — No Support via PM
Re: Asset management
This is my current proposal. It is built from all the ideas and proposals given above, specially by callumacrae, Nicofuma and CHItA. It is as follows:
Note: Unless explicitly mentioned as being specific to JS or CSS, the information applies to both.
Add 2 metadata files (the location is TBA) which stores information about the minimization process.
File 1:
To deal with concurrency, gulp.js should contain code to generate a hidden file. While the file is there, it is working and the 2nd component of the asset manager must not execute. I already found a flawless way in windows (creating and unlinking right after) and an almost flawless way in windows and in GNU/linux (handling all handlable signals related to terminating the process).
The gulp.js script should include code for a watcher for convenience (and not terminate immediatly) which can be turned off (specially for Travis) with a command line argument.
Important:Travis is supposed to contain all the required and optional elements of the gulp.js file for testing purposes and best output helping output text.
---------------------------------------------------------------------------------------------
Main advantages:
I believe I could write the code for this but I won't be able to have it ready for phpBB 3.2.
Note: Unless explicitly mentioned as being specific to JS or CSS, the information applies to both.
Add 2 metadata files (the location is TBA) which stores information about the minimization process.
File 1:
- Static information.
- Information about file groups, their location and exclusion lists.
- Final destination for each file group (includes file name).
- Extensions can modify the information later parsed from this file (indirectly, using events, a file in their own extension, etc... (TBD)).
- Is versioned in the repository.
- Dynamic information (no human should manually modify it).
- Contains the metadata required for the correct management of the minimized files.
- The actual metadata that is fully useful is TBD but I already have some ideas (read below).
- Is not available in the repository.
- Reads from file 1 mentioned above.
- Writes to file 2 mentioned above.
- Contains definitions to run jsHint (optional; real below).
- Contains definitions to run fixmyjs (optional; real below).
- Contains definitions to run UglifyCSS and UglifyJS.
- They minify and generate sourcemaps.
- Metadata of the process goes to File 2.
- Create an HTTP-compatible gzip version of the file(s) (if such exists).
- Store JSON key-value object.
- Store, for each minified file,
- which files it is sourced from.
- Is there a gzip version of it.
- Runs on 2 components:
- component:
- Reads from file 1 and file 2 mentioned above.
- Control which file to deliver between gzip and non-gzip at request phase.
- Deliver appropriate HTTP Cache headers.
- Calls the 2nd component mentioned below, when appropriate.
- Used by the template system (related to INCLUDEJS and INCLUDECSS' internals)
- Is "Recompile stale style components" (or a new option) aware and acts accordingly.
- Knows how to translate an origin URL into the sprite URL and CSS rule.
- component:
- Reads from file 1 mentioned above.
- Writes to file 2 mentioned above.
- It runs on a best-effort with available tools to concatenate and minimize files with sourcemaps.
- Uses ClosureCompiler to minimize javascript if so chosen.
- Creates gzip versions of files not gziped.
- Creates sprited icons.
- component:
To deal with concurrency, gulp.js should contain code to generate a hidden file. While the file is there, it is working and the 2nd component of the asset manager must not execute. I already found a flawless way in windows (creating and unlinking right after) and an almost flawless way in windows and in GNU/linux (handling all handlable signals related to terminating the process).
The gulp.js script should include code for a watcher for convenience (and not terminate immediatly) which can be turned off (specially for Travis) with a command line argument.
Important:Travis is supposed to contain all the required and optional elements of the gulp.js file for testing purposes and best output helping output text.
---------------------------------------------------------------------------------------------
Main advantages:
- Gulp user or not, you can get the assets managed, minimized and gziped for HTTP requests.
- The assets can be delivered in gzip regardless of server settings (the same file doesn't have to be minimized multiple times as the server drops previous minimized versions).
- Extensions can advantage from this as the Asset Manager can converge all CSS and all JS from all sources into a single file, if so desired (although this not being the optimal way).
- Although the minimization tools are ready to use on php, they are not as powerful as the ones written in javascript.
- Writing or merging the asset manager will take some work. The proposed Assetic can't do this.
I believe I could write the code for this but I won't be able to have it ready for phpBB 3.2.