Asset management

General discussion of development ideas and the approaches taken in the 3.x branch of phpBB. The next feature release of phpBB 3 will be 3.3/Proteus.
Forum rules
Please do not post support questions regarding installing, updating, or upgrading phpBB 3.2.x. If you need support for phpBB 3.2.x please visit the 3.2.x Support Forum on phpbb.com.

If you have questions regarding writing extensions please post in Extension Writers Discussion to receive proper guidance from our staff and community.
Nicofuma
3.2 Release Manager
3.2 Release Manager
Posts: 299
Joined: Sun Apr 13, 2014 1:40 am
Location: Paris

Re: Asset management

Post by Nicofuma » Tue Jun 02, 2015 8:41 pm

CHItA wrote:I guess it would be on deployment
Member of the phpBB Development-Team
No Support via PM

User avatar
brunoais
Registered User
Posts: 964
Joined: Fri Dec 18, 2009 3:55 pm

Re: Asset management

Post by brunoais » Tue Jun 02, 2015 10:02 pm

1 question, then.

How is testing made between the original js used at development stage (multiple files) and the one after deployment? Is there file merging? If so, how are the different assets handled?
If files were gzipped for browsers that support it, loading times can be heavily reduced. Are you going to have that working? If so, how? Studies from multiple sources in the interwebz show that gzipping reduces the minimized file size by around 50% (around 70% for the non-minimized version).

User avatar
VSE
Extension Customisations
Extension Customisations
Posts: 670
Joined: Mon Mar 08, 2010 9:18 am

Re: Asset management

Post by VSE » Wed Jun 03, 2015 5:08 am

Gulp, FTW...
Has an irascible disposition.

User avatar
callumacrae
Former Team Member
Posts: 1046
Joined: Tue Apr 27, 2010 9:37 am
Location: England
Contact:

Re: Asset management

Post by callumacrae » Wed Jun 03, 2015 9:06 am

The sourcemaps issue is still not resolved. It's resolved for CSS by a LESS parser which only supports an old subset of LESS, but it isn't resolved for JS.

EDIT: I was wrong, the sourcemaps issue is resolved! It's resolved by using Gulp.

Gzipping is usually done by the server or CDN. There are also gulp plugins to handle it.
Made by developers, for developers!
My blog

CHItA
Development Team
Development Team
Posts: 159
Joined: Thu Mar 12, 2015 1:43 pm
Location: Budapest, Hungary

Re: Asset management

Post by CHItA » Wed Jun 03, 2015 10:58 am

Possible idea for a solution that would address all issues here:

My assumptions (I don't know gulp, I don't know if everything here is actually implementable):
  • All CSS would go into a single CSS file xy.min.css
  • All JS would go into a single JS file yz.min.js
How the solution would look like:
  • Have simple backend code for checking if *.min.* files are exist
  • Have IF blocks in template, that wraps the asset include HTML blocks, and using minified versions if they are available, and the "normal" versions if they are not.
  • Only store the non-minified versions in the repo
  • Run the tool selected by our front-end devs on assets before packaging
  • Add CI tests for assets, so noone can commit changes to them, which would break the minification process (or whatever that is called)
What we possibly gain by this:
  • Forum admins can edit their stuff as before, they just need to remove *.min.* file from their directory
  • Backend devs get the assets
  • If we would have minified versions of assets in the repo, wouldn't that be possible, that I do some harmless thing in the "normal" version of the JS and add some malware install code into the minified one? (That's might be a non-issue, just asking)
  • Front-end devs can just use the tool they decide on, and phpBB automatically switch to the minified version, they can test that if they would like to
Do you think that this system could work? If not, there is some points that should be changed to fix the flow, or just the whole idea is nonsense?

User avatar
callumacrae
Former Team Member
Posts: 1046
Joined: Tue Apr 27, 2010 9:37 am
Location: England
Contact:

Re: Asset management

Post by callumacrae » Wed Jun 03, 2015 11:02 am

Yeah, thats basically what I was going to do
If we would have minified versions of assets in the repo, wouldn't that be possible, that I do some harmless thing in the "normal" version of the JS and add some malware install code into the minified one? (That's might be a non-issue, just asking)
It can be an issue, but it's a pretty easy one to work around. My preferred approach would just be to run the build task when building a release of phpBB; people can malware the devs, but they can't malware the users, because anything they add will be removed upon release. Also, it would be removed whenever anyone else ran gulp.

Basically, it can be an issue, but it wouldn't be.
Made by developers, for developers!
My blog

CHItA
Development Team
Development Team
Posts: 159
Joined: Thu Mar 12, 2015 1:43 pm
Location: Budapest, Hungary

Re: Asset management

Post by CHItA » Wed Jun 03, 2015 11:12 am

I guess the only real difference to your proposal is that it would not be required to commit the compiled code to the repo. If I understand correctly, the only drawback of that would be that the repo code is slower if you don't run gulp or whatever yourself, and IMO that is pretty irrelevant if you not testing the performance on slow connections, in which case you should run gulp yourself as I mentioned.

So if you want to contribute front-end code, you don't need any extra tools, but you have the option to use it.

User avatar
callumacrae
Former Team Member
Posts: 1046
Joined: Tue Apr 27, 2010 9:37 am
Location: England
Contact:

Re: Asset management

Post by callumacrae » Wed Jun 03, 2015 11:15 am

It should 100% be required to commit the compiled code to the repo, we don't want people who don't want to make front-end changes to have to install and run gulp. I never suggested otherwise—it's not a great solution, but it's the best we have.
Made by developers, for developers!
My blog

CHItA
Development Team
Development Team
Posts: 159
Joined: Thu Mar 12, 2015 1:43 pm
Location: Budapest, Hungary

Re: Asset management

Post by CHItA » Wed Jun 03, 2015 11:22 am

It is my suggestion to not to commit compiled code. So back-end devs would run the slower JS and CSS but back-end devs IMO would not really need the performance as they working locally and from a computer I guess. Also, back-end devs I imagine are more interested in the performance of the back-end, so I guess the JS, CSS performance difference is only a concern for those who work on the front-end.

This would solve the back-end dev needs to touch some JS still don't need gulp, but the release would be optimized.

I just try to explain what I've proposed, I'm not saying that my way is the best way (only IMO).

User avatar
brunoais
Registered User
Posts: 964
Joined: Fri Dec 18, 2009 3:55 pm

Re: Asset management

Post by brunoais » Wed Jun 03, 2015 11:44 am

@CHItA
That is a quite good solution it would work perfectly if there could only be a single template for a single language at a time. It is a good step-stone to start on, though.
I think we don't need the minimized versions in the repo. We do need in the RC and final deployment releases, though.
Here's the main gotchas:
  1. There's language-specific CSS and language-agnostic CSS for each template.
  2. There are general-purpose template-agnostic js and there's template-specific js.
  3. If you join everything, where do the files go, then?
  4. Where are the css and js file lists (or directories) listed?
  5. If an extension wants to replace a file, how does it work?
For 1., I'd do 2 different files instead of joining it all into a single file. I'd use 2 <link> tags 1 for each one (basically, what I'd do is to inline the @include directives, minimize and gzip the css files).

For 2., I'd do 1 js file for all the general js, 1 js file for each template-specific file. They'd be minimized and gziped. That, instead of your suggestion.

By doing both of the above, 3. should be as easy as placing the minimized files in the assets or the template depending where they originated from.

For 4. unless we make a standard for the location of the files, we do need to make a definition somewhere about what goes and what doesn't go to which pack.

For 5.... I have no idea.

@callumacrae
It would be much easier if you'd answer my questions above. There's already a decent backlog of questions you didn't answer yet and you still insist your solution is the best for phpBB without actually argumenting in favor of your option with determent of the alternatives being presented to you.
You are still not mentioning why your idea is better, you are just mentioning what your idea allows to do.

Post Reply