Post/Reply by Email - Integration With Email Lists

General discussion of development ideas and the approaches taken in the 3.x branch of phpBB. The current feature release of phpBB 3 is 3.3/Proteus.
Forum rules
Please do not post support questions regarding installing, updating, or upgrading phpBB 3.3.x. If you need support for phpBB 3.3.x please visit the 3.3.x Support Forum on phpbb.com.

If you have questions regarding writing extensions please post in Extension Writers Discussion to receive proper guidance from our staff and community.
Tecktron
Registered User
Posts: 2
Joined: Thu Jun 18, 2015 12:20 am

Re: Post/Reply by Email - Integration With Email Lists

Post by Tecktron »

I can across this thead looking for this exact same thing (a way to interact with the board via email).
I was wondering that, since this is almost a year old, if anything has become of this?

If it has then I've probably written all this for nothing (which is good!) and hopefully I can be pointed in the right direction. Maybe I even have some thoughts here to help improve it.

Next, I don't know about the whole argument of if this should be core or extension, I imagine that we can start as the latter and become the former (if that's even possible, IDK).

I'd like to help put something like this into place. However, although I can write most of the actual code, I'm none familiar with the codebase of phpbb, so I'll eventually need some help when it comes to integration.

Anyway I wanted to share what I had in mind for how this might work. I'm open to ideas, suggestions, criticisms, gotcha's in the phpbb codebase I'm not be aware of (which as I stated, I'm not familiar with outside of being a user), etc. Please try to keep the comments constructive.

mail server:
  • Catch all email address receives all incoming email
phpbb:
  • Each user has a hash associated with them (like a git commit). We can use something like the first six or so of a sha1 of their email address.
  • Each board then has an associated hash, as do the topics and posts.
  • When you receive and email, it's 'reply-to' address is a combination of these hashes user+board+topic+comment (up to 64 characters total, as by spec)
  • To create a topic, a user must post the board name and the user's own hash (coming from their own email address). The system would then send another confirmation back to the user's address on file just to be sure (this could be optional). If they reply to that then it's posted.
  • Collection of emails headers could also happen and if the headers appear the same over time, the reply method could be deemed unnecessary by the system (later version)
incoming mail processor:
  • Run as both a pass to script (mail server to forward the mail directly to it) or as a cron jon that runs every minute or so.
  • As Cron, this script grabs all the emails form the catch all account.
  • For each email, the address is split into it's parts, and the hash's matched.
  • The user hash is matched to the user and the from address much match also.
  • If the parts don't match or add up then the message is discarded.
  • Otherwise, everything is posted and the view count increased.
  • As mention earlier, if a post is deemed valid, headers could be saved in a per user basis (probably a later version).
Opt-in/opt-out methods for the user:
  • Option to never use email in user settings (this will ignore anything sent in via this user, basically discard their hash)
  • A general setting to auto opt-in to everything you comment on. Maybe prefill an opt-in checkbox in the post options at the bottom, would this maybe be tied to the 'Notify of reply' checkbox?
  • If following a topic via email, sending 'unsubscribe' as the subject opts out.
  • 'opt-in' button/checkbox at top of topic pages.
  • Hash's are sent via the digest emails as well.
  • If you post via email to a board, you're auto-opted in.
Some things to consider:
  • Obviously as an admin, whether to enable any of this is optional.
  • Sending mails as a separate process (via system call to script) or even another cron job.
  • The email server should probably be just for the board, otherwise the catchall can quickly get filled with junk that would require more processing.
  • The would need to be db enhancements to hold the hash's.
  • How are emails sent (multipart HTML Vs PlainText Vs HTML as PlainText), probably user optional
  • Parsing/stripping the HTML from emails
  • Looking into mail headers and see what can be used, such as RFC-2369, etc.
  • To hold the downloaded mail while processing (mostly for cron), a temp table might need to be created (it will be emptied at end of job).
  • Attachments, whether to allow them or not (in a later version).

Thanks!

User avatar
DavidIQ
Customisations Team Leader
Customisations Team Leader
Posts: 1904
Joined: Thu Mar 02, 2006 4:29 pm
Location: Earth
Contact:

Re: Post/Reply by Email - Integration With Email Lists

Post by DavidIQ »

No this has not been considered for inclusion into the core so no work has been done for it. An extension sort of exists for it but does not work in both directions.
Image

Tecktron
Registered User
Posts: 2
Joined: Thu Jun 18, 2015 12:20 am

Re: Post/Reply by Email - Integration With Email Lists

Post by Tecktron »

Well, given my ideas above, could you comment on if such an undertaking as an extension that would also allow for system upgrades (the extension maintenance aside) would be possible?

I'd like to start actually developing this, but if it's not possible given the current state of the codebase, I'd rather not waste the time on it and move on. Nothing worse than writing a bunch of code only to find out it can't be integrated.

Thanks!

User avatar
DavidIQ
Customisations Team Leader
Customisations Team Leader
Posts: 1904
Joined: Thu Mar 02, 2006 4:29 pm
Location: Earth
Contact:

Re: Post/Reply by Email - Integration With Email Lists

Post by DavidIQ »

An extension is indeed possible:
https://github.com/DavidIQ/phpBB-MailingList

But what it doesn't allow for is replying to the mailing list email and posting the reply to the forum. However this last part is indeed possible through the use of the cron tasks and events available there for adding additional cron tasks that would possibly trigger a check of the email account where the replies would go to and process those, but it was not a requirement when I built that extension. For that to work properly however one would have to set up an actual server cron instead of the built-in cron that runs off of the loading of the site.
Image

deej-
Registered User
Posts: 1
Joined: Thu Aug 27, 2015 2:40 am

Re: Post/Reply by Email - Integration With Email Lists

Post by deej- »

For a number of years I ran several Mailman mailing lists, and a few years back I converted to PHPBB2 and Mail2Forum so that I could offer the members a web interface in addition to email. That combination worked great, but as the code base was old I started looking around for something to replace it. The simple fact is that the majority of my members use email to read and reply to posts.

I finally gave in earlier this year and converted back to Mailman mailing lists when I upgraded the OS because I could not find a suitable upgrade path from the old PHPBB2 and M2F software. If PHPBB3 supported email integration (read and reply), I'd move back to it in a heartbeat.

The only alternative I can find that offers this at present is Yahoo groups, and frankly the web interface is horrible, although the email interface seems to work fairly well.

From my perspective, porting the Mail2Forum software or offering an alternative with similar functionality using current PHPBB versions would be ideal.

Earlier in this thread it was asked "Why can't people just click on a web link to reply", and the fundamental answer is that isn't what a lot of people want. They want to use their email client of choice, and not a web browser. Many of them archive messages of interest and use the email client search tools for everything. They can have all of their messages from several different mailing lists in one consistent format and location rather than having to go to several different websites. "It is just more convenient in my email" is a recurring theme I hear.

On the other hand, we do have some that would use the web interface if available. It would be great to be able to make both types of members happy.

Why can't we have both and not force people into a particular mold?

TXGRunner
Registered User
Posts: 2
Joined: Fri Jan 22, 2016 3:43 am

Re: Post/Reply by Email - Integration With Email Lists

Post by TXGRunner »

I joined this forum just to ask this question:

Since phpBB won't support mailing list and web forum integration, what competing products will?

---
Our group (~2000) users is currently split between a Yahoo Groups and a BB web forum. The mailing list is very active, but the web forum is not.

Yahoo Groups is universally despised and we want to consolidate and a large number of people prefer to interact primarily via email.

Good, bad, or indifferent, this is what they want. Please spare the comments about educating them and RSS feeds, this is their requirement and they are the customer. We just need a solution that will let users receive new posts and replies via email and respond using reply functionality.

Although we don't want spam, security is not a huge concern...it's hobby related, not PII or banking information.

Thanks,
V/r
Brian

omarf
Registered User
Posts: 2
Joined: Thu Mar 28, 2013 5:51 pm

Re: Post/Reply by Email - Integration With Email Lists

Post by omarf »

TXGRunner wrote: Fri Jan 22, 2016 3:49 am Since phpBB won't support mailing list and web forum integration, what competing products will?
Discourse
Last edited by DavidIQ on Mon Jan 25, 2016 3:12 pm, edited 1 time in total.
Reason: Remove spam

TXGRunner
Registered User
Posts: 2
Joined: Fri Jan 22, 2016 3:43 am

Re: Post/Reply by Email - Integration With Email Lists

Post by TXGRunner »

Thanks. That looks like a great solution.

Cheers,
Brian

Post Reply