[RFC] Attachments in subfolders

Note: We are moving the topics of this forum and it will be deleted at some point

Publish your own request for comments/change or patches for the next version of phpBB. Discuss the contributions and proposals of others. Upcoming releases are 3.2/Rhea and 3.3.
Oleg
Posts: 1150
Joined: Tue Feb 23, 2010 2:38 am
Contact:

Re: [RFC] Attachments in subfolders

Post by Oleg » Sat Dec 24, 2011 7:22 pm

Following your syntax, we will have a flood of new folders on the first attachments added to the board with only one or two files in it.
Yes and I think that's fine.
canonknipser wrote: As a default, i will build a function that converts the unique attachment id (which is mediumint)
The problem with this is on boards that have thousands of attachments the attachment id may very well have been expanded to a full 32-bit field.
canonknipser wrote: into a hex string (maximum "ffffff" for mediumint unsigned), and use every two bytes as a subdirectory name, so there is a maximum of two levels (the last two byte are stripped):
dec(1) -> hex(1) -> folder "00/00"
dec(100) -> hex(64) -> folder "00/00"
dec(255) -> hex(ff) -> folder "00/00"
dec(256) -> hex(100) -> folder "00/01"
dec(12345) -> hex(3039) -> folder "00/30"
This I'm having a hard time understanding and I also don't understand why you need two levels of directories for 64k attachments. 256*256=64k, if you have 256 directories on the top level each will have no more than 256 files in it.
canonknipser wrote: board owners only need to download the contents of the newest folder(s) for frequently backups via ftp because every 256 attachments a folder is "complete"
To create a backup you would list, not download, every directory on the ftp server and only download the files you don't have.
So, every 256 attachments a new subfolder is built,
If this is what you want don't bother with hex, just divide the id by 256 and use the result as the directory name.

User avatar
canonknipser
Registered User
Posts: 59
Joined: Mon Sep 19, 2011 4:42 am
Location: Germany

Re: [RFC] Attachments in subfolders

Post by canonknipser » Sat Dec 24, 2011 8:12 pm

Oleg wrote:The problem with this is on boards that have thousands of attachments the attachment id may very well have been expanded to a full 32-bit field.
So, those boards are modded, and the subfolder-attachment function has also the need be modded like in other cases board owners decide to change default database setup (i'm not sure if dbal can figure out the type of fields, so it can be done automatically)
mediumint, by the way has a maximum of 16,777,215 (16M)
Oleg wrote:This I'm having a hard time understanding and I also don't understand why you need two levels of directories for 64k attachments. 256*256=64k, if you have 256 directories on the top level each will have no more than 256 files in it.

it can be up to 514 file in each subfolder (256 attachments, 256 thumbs for attachments, .htaccess and (empty) index.htm to prevent folder browsing like in every other folder on pbpBB), but this is a guaranteed maximum. And also the number of subfolders in a folder will never go over 256, so we don't break the directory list maximum of many ftp server settings.

Yes, for the first 64k attachments, all files will go into folder "00", with subfolders up to "ff".
Oleg wrote:To create a backup you would list, not download, every directory on the ftp server and only download the files you don't have.
Right, but it's much easier to list only the latest folders and not all subfolders. Eg.:
Last backup has shown a folder 00/0e, actual the board shows 00/10 as highest subfolder.
So, i have only to list folder 00/0e (because on last backup, it was not "full"), 00/0f (i don't have to list, because it is new since last backup and i can download it complete) and 00/10 (the same like 00/0f).
The next time, i can start at folder 00/10.

Following your logic for building folder names, every time i want to make a backup via ftp i have to check all 256 folders for new files.

It don''t know how many attachments exist on phpbb.com. Our board has 33,793 attachments at the moment (nearly all with thumbnails, so a total of > 67,000 files), which means our current subfolder in above syntax would be 00/84, the board is online since 2 1/2 years, i expect the 64k-border broken in the middle of 2013, so subfolder would change to 01/00.

My preferred subfolder syntax is by date (Y/m/w), so a board will have a three level subfolder syntax, but only 84 (12 month * 7 weekdays) folders a year. Smaller board can use "Y/m", having 12 subfolders per year.
Greetings, Frank

english is not my native language

Oleg
Posts: 1150
Joined: Tue Feb 23, 2010 2:38 am
Contact:

Re: [RFC] Attachments in subfolders

Post by Oleg » Sun Dec 25, 2011 2:38 am

I think your proposal will work fine.

User avatar
DavidIQ
Customisations Team Leader
Customisations Team Leader
Posts: 1731
Joined: Thu Mar 02, 2006 4:29 pm
Location: Earth
Contact:

Re: [RFC] Attachments in subfolders

Post by DavidIQ » Fri Nov 09, 2012 1:38 pm

Just bumping this. There is an RC MOD on phpBB.com that might be useful:
https://www.phpbb.com/community/viewtop ... &t=2130170

Bringing this up again because the number of attachments seems to be an issue for the forum in use at Joomla! and they've asked for this to be looked into. According to their ACP they have 58,482 attachments. For comparison I went to our own phpbb.com forum and see that it reports that we have 19,039 attachments. I shelled in however to check the number of actual files we have and it's a bit higher: 31,367. This is because the count in the ACP does not include thumbnails and would mean that in the case of Joomla! the number of files in the files directory is actually over 100,000. This is causing them some performance issues.
Image

User avatar
bantu
3.0 Release Manager
3.0 Release Manager
Posts: 557
Joined: Thu Sep 07, 2006 11:22 am
Location: Karlsruhe, Germany
Contact:

Re: [RFC] Attachments in subfolders

Post by bantu » Fri Nov 09, 2012 1:43 pm

We should certainly address this issue. For the Joomla forum I would however suggest using another filesystem such as XFS or ext4 with proper configuration instead of fiddling with the board software.

Fit
Registered User
Posts: 3
Joined: Mon Jun 03, 2013 8:53 am

Re: [RFC] Attachments in subfolders

Post by Fit » Mon Jun 03, 2013 9:00 am

Hello,
I'm afraid to install the MOD. It will feature Attachments in subfolders included in the new version of phpBB 3.1? I think it's a very important thing! Does anybody have any information? Thank you.

User avatar
EXreaction
Registered User
Posts: 1555
Joined: Sat Sep 10, 2005 2:15 am

Re: [RFC] Attachments in subfolders

Post by EXreaction » Mon Jun 03, 2013 2:00 pm

3.1 is feature froze and will not include this feature.

Fit
Registered User
Posts: 3
Joined: Mon Jun 03, 2013 8:53 am

Re: [RFC] Attachments in subfolders

Post by Fit » Mon Jun 03, 2013 3:35 pm

The most important thing is not included... After all, it's a big problem phpbb. It is a great pity :(.

User avatar
keith10456
Registered User
Posts: 523
Joined: Sat Apr 22, 2006 10:29 pm
Contact:

Re: [RFC] Attachments in subfolders

Post by keith10456 » Mon Jun 03, 2013 4:06 pm

Fit wrote:The most important thing is not included... After all, it's a big problem phpbb. It is a great pity :(.
Very useful and needed, yes it is. The "most important", no it's not ;)

Fit
Registered User
Posts: 3
Joined: Mon Jun 03, 2013 8:53 am

Re: [RFC] Attachments in subfolders

Post by Fit » Mon Jun 03, 2013 8:41 pm

Ok, so maybe in the next release of phpBB. For me it's important. People upload a lot of photos on the forum. I'm afraid to install MOD for compatibility with new versions. Moreover, I am not a programmer.

Post Reply