Poor man’s full text search for SuiteCRM

Poor man’s full text search for SuiteCRM

The poor man’s CRM, a.k.a SuiteCRM, is great (… was great for its time!); that’s another story.

One of the features provided is “unified search”, as the geeks call it; some may call it global/universal search across many tables and records in the database. What it does is simply as follows:

• Loop over a list of modules (e.g. accounts, contacts, opportunities, calls, meetings, notes, etc.)
• Search for records matching a phrase (entered by user) by looking at all the fields
• List the records found under each module

From technical point of view, their solution includes the following logic (e.g. using contact module).

1. Create main query.
(source: include/ListView/listViewData.php function getListViewData())

2. Create list count query based on main query by removing ORDER BY and LIMIT clauses.
(source: data/SugarBean.php function create_list_count_query())

3. Execute list count query and get the total number of records that would be retrieved by main query.

4. Execute main query by using ORDER BY and LIMIT clauses.
(source: include/database/MysqlManager.php function limitQuery())
Loop over the records found, and prepare for display, which may require running additional queries to load related data such as user name of the user who created a selected record.

There are two main issues with this common approach:

A. Too many columns are selected unnecessarily from the tables.

B. We execute more or less same query twice. So-called database abstraction layer and the logic that comes with it prevents us from using a powerful feature of MySQL:

function found_rows()

We can modify the main query and inject the key SQL_CALC_FOUND_ROWS; e.g.

After running this version, we can simply extract the total number of records found:

Obviously, this is very specific non-standard feature of MySQL. Still, I think the database abstraction layer should handle this requirement, and implement for each database management system supported.

Next, I want to explain my implementation of global full text search with a simple scoring feature. Let’s start with a very simple example (‘john smith’ is search phrase):

The idea is this: for each word in the search phrase, create an expression for each column in both main table and custom table; if every match is 1 point; and we simple add the points to calculate an overall score. Then, we can order the records by score calculated in descending order. Also, we can favour the recently modified records when they have same score.

Here is a real example when we search for ‘jo smi ya 18’:

 Full Text Search MySQL