Subfolder system for attachments

[BadBlock2]

Referring to this archived topic: viewtopic.php?f=108&t=42329

Ok, implemented the change on my phpBB 3.0 board earlier, now porting it to my 3.2 dev instance.
My board has already reached 500k attachments, so over a million files in one folder (including thumbnails) would be... I'm not sure if even possible for the filesystem, and would certainly affect the performance of the site.

So, subfolders.

Method of storage: 2-level folder structure based on the first 4 symbols of the attachment hash.
E.g. for hash 7b485f3c846e35334bc28c61c586162a its files are stored as follows:

Code: Select all

files/7b/48/234_7b485f3c846e35334bc28c61c586162a
files/7b/48/thumb_234_7b485f3c846e35334bc28c61c586162a

As my modification was not part of core, but applied as an add-on patch, I tried to make as few changes as possible.
So, upload works as usual (files placed to "files/"), but on download, the new location in subfolder is checked and files get moved if they are not present in their respective subfolder.
The benefit of this method is that it provides seamless backward compatibility with the existing method of storage and does not require any conversion; files get moved to subfolders gradually, one by one, as they are downloaded by users.
Also, I had to amend the attachment deletion code.
Files changed:

Code: Select all

forum/includes/functions_download.php
forum/phpbb/attachment/delete.php

The diff for phpBB 3.2 is here: http://files.sarov.info/2017/attachment ... lders.diff
It does not fully comply with coding guidelines ATM, just as a proof of concept.
BTW, in my patch, when an image file is moved to subfolder, its orientation also gets fixed to prevent some browsers from showing portrait photos as landscape; you can just ignore this piece of code for now.

Question: is this a feasible solution? Is it possible to get it implemented in core?

[sitesplat]

Also: https://tracker.phpbb.com/browse/PHPBB3-10119

[CHItA]

It is possible to implement it. However, probably you'll need to make two different PRs for 3.2 and master (or just do it for master, and make it an extension for 3.2). I'm not sure if a folder per user is such a good idea but i have nothing against having attachments in multiple subfolders in general.

[BadBlock2]

> However, probably you'll need to make two different PRs for 3.2 and master (or just do it for master, and make it an extension for 3.2).

I'm new to extensions actually, couldn't find events that I could use for this purpose.
Attachment filepath generation all seems to be hardcoded in core phpBB.

> I'm not sure if a folder per user is such a good idea

I'm using the first four symbols from the hash, not "folder per user".
So, if the hash is 7b485f3c846e35334bc28c61c586162a, then subfolder for the attachment and thumbnail will be files/7b/48.

[BadBlock2]

{duplicate}

[BadBlock2]

What about the concept itself?
Should I leave it as-is to provide backward compatibility and seamless gradual conversion of the file storage?

[CHItA]

BC is great, i would probably go for a folder structure by upload date probably or something like that (theoretically this way you can have unlimited number of folders and not "just" 256*256 which sounds a lot, though i'm not convinced that the attachments get distributed equally among them).

For the extension part, you could add events, that shouldn't be that hard to do actually.

[BadBlock2]

BC is great, i would probably go for a folder structure by upload date probably or something like that (theoretically this way you can have unlimited number of folders and not "just" 256*256 which sounds a lot, though i'm not convinced that the attachments get distributed equally among them).

This method (4 symbols from hash, 256*256 subfolders) is widely used, e.g. as filestore in Moodle LMS, which I'm quite familiar with, and over years proved robustness on very big numbers, e.g. installations with hundreds of thousands of users and millions of files in filestore. In the worst case scenario (say, 30 million attachments!) we will be talking about a few hundreds of files in one directory, which is comparable with the initial 256 and for sure is not a big deal.

Trust me, files get distributed more or less evenly (total evenness is hardly even necessary).
If required I can write a script to collect statistics (min and max files in one dir) from my board with 500k+ attachments which has already been running attachments in subfolders for years on phpBB 3.0 using exactly this folder structure.

[rubencm]

I think this is partially solved in 3.3 with the storage system, since some storages allows unlimited files in one "virtual" folder.

[BadBlock2]

I think this is partially solved in 3.3 with the storage system, since some storages allows unlimited files in one "virtual" folder.

Hmm... Can you please provide a bit more information on the matter?