Subfolder system for attachments

General discussion of development ideas and the approaches taken in the 3.x branch of phpBB. The next feature release of phpBB 3 will be 3.2/Rhea followed by 3.3.
Forum rules
Please do not post support questions regarding installing, updating, or upgrading phpBB 3.1. If you need support for phpBB 3.1 please visit the 3.1.x Support Forum on phpbb.com.

If you have questions regarding writing extensions please post in Extension Writers Discussion to receive proper guidance from our staff and community.
BadBlock2
Registered User
Posts: 9
Joined: Wed Sep 13, 2017 6:09 am

Subfolder system for attachments

Post by BadBlock2 » Wed Sep 13, 2017 12:47 pm

Referring to this archived topic: viewtopic.php?f=108&t=42329

Ok, implemented the change on my phpBB 3.0 board earlier, now porting it to my 3.2 dev instance.
My board has already reached 500k attachments, so over a million files in one folder (including thumbnails) would be... I'm not sure if even possible for the filesystem, and would certainly affect the performance of the site.

So, subfolders.

Method of storage: 2-level folder structure based on the first 4 symbols of the attachment hash.
E.g. for hash 7b485f3c846e35334bc28c61c586162a its files are stored as follows:

Code: Select all

files/7b/48/234_7b485f3c846e35334bc28c61c586162a
files/7b/48/thumb_234_7b485f3c846e35334bc28c61c586162a
As my modification was not part of core, but applied as an add-on patch, I tried to make as few changes as possible.
So, upload works as usual (files placed to "files/"), but on download, the new location in subfolder is checked and files get moved if they are not present in their respective subfolder.
The benefit of this method is that it provides seamless backward compatibility with the existing method of storage and does not require any conversion; files get moved to subfolders gradually, one by one, as they are downloaded by users.
Also, I had to amend the attachment deletion code.
Files changed:

Code: Select all

forum/includes/functions_download.php
forum/phpbb/attachment/delete.php
The diff for phpBB 3.2 is here: http://files.sarov.info/2017/attachment ... lders.diff
It does not fully comply with coding guidelines ATM, just as a proof of concept.
BTW, in my patch, when an image file is moved to subfolder, its orientation also gets fixed to prevent some browsers from showing portrait photos as landscape; you can just ignore this piece of code for now.

Question: is this a feasible solution? Is it possible to get it implemented in core?
Last edited by BadBlock2 on Wed Sep 13, 2017 4:50 pm, edited 3 times in total.

User avatar
sitesplat
Registered User
Posts: 8
Joined: Wed Dec 10, 2014 4:28 pm

Re: Subfolder system for attachments

Post by sitesplat » Wed Sep 13, 2017 2:16 pm

- FLATBOOTS - phpBB Flat Design - Bootstrap3
- Contact me Via PM or e-mail for Custom work
- Deploy an SSD cloud server in 55 seconds. Sign up and get $10 in credit.

CHItA
Development Team
Development Team
Posts: 128
Joined: Thu Mar 12, 2015 1:43 pm
Location: Budapest, Hungary

Re: Subfolder system for attachments

Post by CHItA » Wed Sep 13, 2017 4:30 pm

It is possible to implement it. However, probably you'll need to make two different PRs for 3.2 and master (or just do it for master, and make it an extension for 3.2). I'm not sure if a folder per user is such a good idea but i have nothing against having attachments in multiple subfolders in general.

BadBlock2
Registered User
Posts: 9
Joined: Wed Sep 13, 2017 6:09 am

Re: Subfolder system for attachments

Post by BadBlock2 » Wed Sep 13, 2017 4:44 pm

> However, probably you'll need to make two different PRs for 3.2 and master (or just do it for master, and make it an extension for 3.2).

I'm new to extensions actually, couldn't find events that I could use for this purpose.
Attachment filepath generation all seems to be hardcoded in core phpBB.

> I'm not sure if a folder per user is such a good idea

I'm using the first four symbols from the hash, not "folder per user".
So, if the hash is 7b485f3c846e35334bc28c61c586162a, then subfolder for the attachment and thumbnail will be files/7b/48.

BadBlock2
Registered User
Posts: 9
Joined: Wed Sep 13, 2017 6:09 am

Re: Subfolder system for attachments

Post by BadBlock2 » Wed Sep 13, 2017 4:47 pm

{duplicate}

BadBlock2
Registered User
Posts: 9
Joined: Wed Sep 13, 2017 6:09 am

Re: Subfolder system for attachments

Post by BadBlock2 » Wed Sep 13, 2017 4:59 pm

What about the concept itself?
Should I leave it as-is to provide backward compatibility and seamless gradual conversion of the file storage?

CHItA
Development Team
Development Team
Posts: 128
Joined: Thu Mar 12, 2015 1:43 pm
Location: Budapest, Hungary

Re: Subfolder system for attachments

Post by CHItA » Wed Sep 13, 2017 5:30 pm

BC is great, i would probably go for a folder structure by upload date probably or something like that (theoretically this way you can have unlimited number of folders and not "just" 256*256 which sounds a lot, though i'm not convinced that the attachments get distributed equally among them).

For the extension part, you could add events, that shouldn't be that hard to do actually.

BadBlock2
Registered User
Posts: 9
Joined: Wed Sep 13, 2017 6:09 am

Re: Subfolder system for attachments

Post by BadBlock2 » Wed Sep 13, 2017 5:52 pm

CHItA wrote:
Wed Sep 13, 2017 5:30 pm
BC is great, i would probably go for a folder structure by upload date probably or something like that (theoretically this way you can have unlimited number of folders and not "just" 256*256 which sounds a lot, though i'm not convinced that the attachments get distributed equally among them).
This method (4 symbols from hash, 256*256 subfolders) is widely used, e.g. as filestore in Moodle LMS, which I'm quite familiar with, and over years proved robustness on very big numbers, e.g. installations with hundreds of thousands of users and millions of files in filestore. In the worst case scenario (say, 30 million attachments!) we will be talking about a few hundreds of files in one directory, which is comparable with the initial 256 and for sure is not a big deal.

Trust me, files get distributed more or less evenly (total evenness is hardly even necessary). :)
If required I can write a script to collect statistics (min and max files in one dir) from my board with 500k+ attachments which has already been running attachments in subfolders for years on phpBB 3.0 using exactly this folder structure.

User avatar
rubencm
Registered User
Posts: 15
Joined: Sun Mar 12, 2017 8:30 pm

Re: Subfolder system for attachments

Post by rubencm » Thu Sep 14, 2017 10:38 am

I think this is partially solved in 3.3 with the storage system, since some storages allows unlimited files in one "virtual" folder.

BadBlock2
Registered User
Posts: 9
Joined: Wed Sep 13, 2017 6:09 am

Re: Subfolder system for attachments

Post by BadBlock2 » Thu Sep 14, 2017 11:48 am

rubencm wrote:
Thu Sep 14, 2017 10:38 am
I think this is partially solved in 3.3 with the storage system, since some storages allows unlimited files in one "virtual" folder.
Hmm... Can you please provide a bit more information on the matter?

Post Reply