[RFC] Attachments in subfolders

[Oleg]

Following your syntax, we will have a flood of new folders on the first attachments added to the board with only one or two files in it.

Yes and I think that's fine.

As a default, i will build a function that converts the unique attachment id (which is mediumint)

The problem with this is on boards that have thousands of attachments the attachment id may very well have been expanded to a full 32-bit field.

into a hex string (maximum "ffffff" for mediumint unsigned), and use every two bytes as a subdirectory name, so there is a maximum of two levels (the last two byte are stripped):
dec(1) -> hex(1) -> folder "00/00"
dec(100) -> hex(64) -> folder "00/00"
dec(255) -> hex(ff) -> folder "00/00"
dec(256) -> hex(100) -> folder "00/01"
dec(12345) -> hex(3039) -> folder "00/30"

This I'm having a hard time understanding and I also don't understand why you need two levels of directories for 64k attachments. 256*256=64k, if you have 256 directories on the top level each will have no more than 256 files in it.

board owners only need to download the contents of the newest folder(s) for frequently backups via ftp because every 256 attachments a folder is "complete"

To create a backup you would list, not download, every directory on the ftp server and only download the files you don't have.

So, every 256 attachments a new subfolder is built,

If this is what you want don't bother with hex, just divide the id by 256 and use the result as the directory name.

[canonknipser]

The problem with this is on boards that have thousands of attachments the attachment id may very well have been expanded to a full 32-bit field.

So, those boards are modded, and the subfolder-attachment function has also the need be modded like in other cases board owners decide to change default database setup (i'm not sure if dbal can figure out the type of fields, so it can be done automatically)
mediumint, by the way has a maximum of 16,777,215 (16M)

This I'm having a hard time understanding and I also don't understand why you need two levels of directories for 64k attachments. 256*256=64k, if you have 256 directories on the top level each will have no more than 256 files in it.

it can be up to 514 file in each subfolder (256 attachments, 256 thumbs for attachments, .htaccess and (empty) index.htm to prevent folder browsing like in every other folder on pbpBB), but this is a guaranteed maximum. And also the number of subfolders in a folder will never go over 256, so we don't break the directory list maximum of many ftp server settings.

Yes, for the first 64k attachments, all files will go into folder "00", with subfolders up to "ff".

To create a backup you would list, not download, every directory on the ftp server and only download the files you don't have.

Right, but it's much easier to list only the latest folders and not all subfolders. Eg.:
Last backup has shown a folder 00/0e, actual the board shows 00/10 as highest subfolder.
So, i have only to list folder 00/0e (because on last backup, it was not "full"), 00/0f (i don't have to list, because it is new since last backup and i can download it complete) and 00/10 (the same like 00/0f).
The next time, i can start at folder 00/10.

Following your logic for building folder names, every time i want to make a backup via ftp i have to check all 256 folders for new files.

It don''t know how many attachments exist on phpbb.com. Our board has 33,793 attachments at the moment (nearly all with thumbnails, so a total of > 67,000 files), which means our current subfolder in above syntax would be 00/84, the board is online since 2 1/2 years, i expect the 64k-border broken in the middle of 2013, so subfolder would change to 01/00.

My preferred subfolder syntax is by date (Y/m/w), so a board will have a three level subfolder syntax, but only 84 (12 month * 7 weekdays) folders a year. Smaller board can use "Y/m", having 12 subfolders per year.

[Oleg]

I think your proposal will work fine.

[DavidIQ]

Just bumping this. There is an RC MOD on phpBB.com that might be useful:
https://www.phpbb.com/community/viewtop ... &t=2130170

Bringing this up again because the number of attachments seems to be an issue for the forum in use at Joomla! and they've asked for this to be looked into. According to their ACP they have 58,482 attachments. For comparison I went to our own phpbb.com forum and see that it reports that we have 19,039 attachments. I shelled in however to check the number of actual files we have and it's a bit higher: 31,367. This is because the count in the ACP does not include thumbnails and would mean that in the case of Joomla! the number of files in the files directory is actually over 100,000. This is causing them some performance issues.

[bantu]

We should certainly address this issue. For the Joomla forum I would however suggest using another filesystem such as XFS or ext4 with proper configuration instead of fiddling with the board software.

[Fit]

Hello,
I'm afraid to install the MOD. It will feature Attachments in subfolders included in the new version of phpBB 3.1? I think it's a very important thing! Does anybody have any information? Thank you.

[EXreaction]

3.1 is feature froze and will not include this feature.

[Fit]

The most important thing is not included... After all, it's a big problem phpbb. It is a great pity .

[keith10456]

The most important thing is not included... After all, it's a big problem phpbb. It is a great pity .

Very useful and needed, yes it is. The "most important", no it's not

[Fit]

Ok, so maybe in the next release of phpBB. For me it's important. People upload a lot of photos on the forum. I'm afraid to install MOD for compatibility with new versions. Moreover, I am not a programmer.