Search System Discussion

Discussion of general topics related to the new version and its place in the world. Don't discuss new features, report bugs, ask for support, et cetera. Don't use this to spam for other boards or attack those boards!
Forum rules
Discussion of general topics related to the new release and its place in the world. Don't discuss new features, report bugs, ask for support, et cetera. Don't use this to spam for other boards or attack those boards!
User avatar
EXreaction
Registered User
Posts: 1555
Joined: Sat Sep 10, 2005 2:15 am

Re: Search System Discussion

Post by EXreaction » Fri Aug 29, 2008 1:46 am

Good point, I hadn't noticed that but it would be a great idea. :)

User avatar
Highway of Life
Registered User
Posts: 1399
Joined: Tue Feb 08, 2005 10:18 pm
Location: I'd love to change the World, but they won't give me the Source Code
Contact:

Re: Search System Discussion

Post by Highway of Life » Fri Aug 29, 2008 7:06 am

EXreaction wrote:Just the parsed text and everything prepared for outputting to the template.
Perhaps unparsed text from the DB would even be better than the parsed text. The processing is not heavy for parsing the text such as BBCodes and Smileys, if you’re just trying to lighten the load on the Database Server.

The point is that you’d just be trying to cache the results of the search, if I understood you. And that makes sense, but don’t go too far with overcaching data or parsed text. :)
Image

User avatar
naderman
Consultant
Posts: 1727
Joined: Sun Jan 11, 2004 2:11 am
Location: Karlsruhe, Germany
Contact:

Re: Search System Discussion

Post by naderman » Fri Aug 29, 2008 9:56 am

code reader: I agree with most of those points, and it simply wasn't done for 3.0 due to lack of time.

As for Sphinx, it should actually support in topic search, on the version that's in svn on code forge it worked for me locally, but apparently people have problems with that, too. I just still haven't properly fixed that.

User avatar
EXreaction
Registered User
Posts: 1555
Joined: Sat Sep 10, 2005 2:15 am

Re: Search System Discussion

Post by EXreaction » Fri Aug 29, 2008 6:27 pm

Highway of Life wrote:
EXreaction wrote:Just the parsed text and everything prepared for outputting to the template.
Perhaps unparsed text from the DB would even be better than the parsed text. The processing is not heavy for parsing the text such as BBCodes and Smileys, if you’re just trying to lighten the load on the Database Server.

The point is that you’d just be trying to cache the results of the search, if I understood you. And that makes sense, but don’t go too far with overcaching data or parsed text. :)
That is a possibility, but then we have to send each back to it's system for parsing.

Perhaps it could go like this:
  1. Search to find all proper item id's and types.
  2. Send each subsystem the correct id's depending on it's type for a permission check. Return an array of allowed id's.
  3. Take the allowed ids and cache it for this search for the next 30 minutes or so.
  4. Figure out what id's need to be shown on this page.
  5. Go back to each subsystem and tell it to grab all the required information for each of it's allowed id's that will be shown on this page.
  6. Go through each id to be shown in a loop and tell the subsystem for it to parse the information, then output it.
Or it could also go like this:
  1. Search to find all proper item id's and types.
  2. Send each subsystem the correct id's depending on it's type for a permission check. Return an array of allowed id's.
  3. Take the allowed ids and cache it for this search for the next 30 minutes or so.
  4. Figure out what id's need to be shown on this page, check with the data cache to see if any data needs to be gotten from the database for any of the id's (and if it does, have it grab the next 50). If it does not need to get data for any more id's, skip 2 steps.
  5. Go back to each subsystem and tell it to grab all the required information for each of it's allowed id's that will be shown on this page.
  6. Cache all of the data from each subsystem.
  7. Go through each id to be shown in a loop and tell the subsystem for it to parse the information, then output it.

Either way will work fine. The second is more complex, but it would be more efficient without having to get all of the data for all the displayed items each time. Also, with the permission check, there could be two ways a subsystem can check for permissions.

1. Build part of a query that can be used in the initial search. I do not think this would be too useful for the current parts of phpBB3, but perhaps mods could use it. The only time I can think of where it would be useful is for anonymous users (for example, they obviously can not search PM's so every PM type could be ignored).
2. The normal way of going through every available ID and seeing if the current user is allowed to see it or not.

User avatar
EXreaction
Registered User
Posts: 1555
Joined: Sat Sep 10, 2005 2:15 am

Re: Search System Discussion

Post by EXreaction » Sun Aug 31, 2008 5:51 am

Alright, I threw together a very quick concept for it.

A lot of things don't work, like new posts won't be indexed with this yet (have not gotten that far) and the normal search input page isn't shown (to test it you have to build the search terms query manually). But it does seem to work very nicely so far.

The biggest trouble I ran into was with the sorting. It made things a bit more difficult and complex, but I have it setup to work on a basic level (or so I hope).

To give it a try just replace/add the files from the zip package and in the search word match table, rename post_id to id and add a new column named type (and set all current search items to 'post').

Yes, it is definitely a quick hack and quite a mess, but it kinda works and perhaps will give some ideas on how this could be done. :P
Attachments
Search Concept.zip
(5.31 KiB) Downloaded 323 times

User avatar
JimA
Community Team Leader
Community Team Leader
Posts: 66
Joined: Sun Aug 24, 2008 2:29 pm

Re: Search System Discussion

Post by JimA » Sun Aug 31, 2008 7:43 am

EXreaction wrote:Alright, I threw together a very quick concept for it.

A lot of things don't work, like new posts won't be indexed with this yet (have not gotten that far) and the normal search input page isn't shown (to test it you have to build the search terms query manually). But it does seem to work very nicely so far.
Sure? I was searching on the word "Alright", and I found your post at the top of the Search results. The only way the search function does not work is when you converted from phpBB3, then you only have to rebuild the Search tables.
Image Jim Mossing Holsteyn - Moderator Team Member
Knowledge Base | Documentation | Board rules

No support here on Area51, please visit the Support forums over at phpBB.com

User avatar
ElbertF
Registered User
Posts: 583
Joined: Fri Dec 03, 2004 4:35 pm
Location: tracing..
Contact:

Re: Search System Discussion

Post by ElbertF » Sun Aug 31, 2008 8:56 am

He was talking about his own code, not the search function here. :|

Post Reply