Loading...
 
Skip to main content

History: Search and List from Unified Index

Preview of version: 115

Unified Index

This applies to the search capabilities in Tiki, such as those used by PluginList or the search at tiki-searchindex.php which rely on a search index.

Tiki can support multiple search engines internally. Each of those will have different capabilities and limitations. The default engine should provide capabilities good enough for small and medium sites. Larger sites may need additional infrastructure to get the most performance. Please see: Unified Index Comparison

Fields


Below is a matrix between the fields and the object types.

Legend:

X - Available

/ - Static value

? - Depends on the data

*) Tokenized - as in decomposed in words for full text search

Tracker Fields

In general, tracker fields are indexed as tracker_field_PERMNAME/ID. PERMNAME/ID is your tracker field permanent name or ID. However, many tracker field types have additional useful variants of the main field (see below) that are indexed for each field.

The indexing for tracker fields will vary depending on the field type. As a general rule, tracker_field_PERMNAME/ID will be used as the field and will be sortable. However, there are a few exceptions:

  • Image and File fields are not indexed
  • TextArea is not sortable

Multilingual fields are indexed as multiple fields

  • The main one (tracker_field_PERMNAME/ID) contains all languages
  • tracker_field_PERMNAME/ID_lang contains one language only (tracker_field_12_fr for example)

  • tracker_field_PERMNAME/ID contains the average
  • tracker_field_PERMNAME/ID_sum contains the vote totals
  • tracker_field_PERMNAME/ID_count contains the number of votes

  • tracker_field_PERMNAME/ID_text contains the text instead of the IDs of the linked/listed items

Language of the tracker item

  • If a language field is set for the tracker item, that language is indexed as the item language, i.e. the language field.


Some used in buildQuery/tiki-searchindex.php (need explanation on whether these are real fields or just helpers):
type: refers to object_type
deep: if this is set, categories will be considered deep-categories
autocomplete: Will search for items with title starting with this

Rebuild search index

The index is stored in temp/unified-index/. While the rebuild is occuring, a directory temp/unified-index-new/ will appear (This is to permit the existing index to be used until the new one is ready). If temp/unified-index-new/ doesn't disappear after the indexing, something must have gone wrong. You can delete it and try the re-indexing again. You may want to run sh setup.sh to make sure the permissions are OK.

From the Tiki interface

You can visit this url: tiki-admin.php?page=search&rebuild=now and search index will be rebuilt if the site is small. For medium to high load sites, you can do that from the command line.

From the command line

The search index can be rebuilt from the command line, and since Tiki9 can be run using a Cron job where the server runs the command automatically - see Cron Job to Rebuild Search Index.

Below are the commands that may be used to rebuild the index.

 Note
Lucene Engine: Important note These commands need to be run as the web server user when using the Lucene implementation, often www-data or nobody depending on your server setup, otherwise the web server cannot update and maintain the index. This means prepending the following commands with "sudo -u www-data ".

 Note
All commands below assume you are already in the Tiki root directory.


You can also rebuild it using the unified console.php command, with the appropriate parameters. For example:

Basic command

Copy to clipboard
php console.php index:rebuild <-- or --> php console.php i:r
 Note
The syntax "i:r" is equivalent to "index:rebuild".

Multitiki sites

For multitiki sites, you can rebuild with commands like:

Copy to clipboard
php console.php index:rebuild --site=site1.example.com php console.php index:rebuild --site=site2.example.com ...

Successful rebuild

If the rebuild is successful a message like the following will be produced (for cron jobs, this can usually be sent to you via email as part of automatically running the command):

Copy to clipboard
Started rebuilding index... Indexation wiki page: 150 forum post: 67 blog post: 412 article: 61 file: 1294 trackeritem: 196 comment: 0 Rebuilding index done X-Powered-By: PHP/5.5.8 Content-type: text/html; charset=utf-8

Troubleshooting

If the rebuild is unsuccessful, instead of the above message you may get a message that indicates there has been an internal server error, or it may say "Rebuild in progress." This may be because the rebuild process uses more memory or takes more time than allowed by the server's php settings. Such settings can be changed on the fly as part of the rebuild command - examples of how to do this are shown below.

Not enough memory

Increase memory limit
One way to increase memory is to change the memory_limit php setting as follows (this example changes the memory limit to 4 gigabytes while the rebuild process is running):

Copy to clipboard
php -dmemory_limit=4G console.php i:r --log

You could also direct php to use a specific php.ini file, where there may be a higher memory limit setting or no limit. In this case you would use the -c parameter followed by the path to the php.ini file, as in the example below:

Copy to clipboard
php -c /etc/php5/cli/php.ini console.php i:r --log

Timeout error

Increase maximum execution time
Getting an internal server error may indicate the rebuild process takes longer than the max_execution_time php setting. That can be increased as part of the command as shown below where the max execution time is set to 300 seconds, or 5 minutes. (This command is also increasing the memory limit as described above):

Copy to clipboard
php -dmemory_limit=4G -dmax_execution_time=300 console.php i:r --log

Force rebuild

When the rebuild is unsuccessful with a "Rebuild in progress" message, this usually means that the rebuild failed previously in the middle of the process, leaving a temporary folder called temp/unified-index-new on the server. When a new rebuild is started and the program sees this folder, it thinks there is a rebuild already in progress and will stop. You can either delete this folder before rebuilding again or include the --force parameter in the rebuild command as follows:

Copy to clipboard
php -dmemory_limit=4G -dmax_execution_time=300 console.php i:r --force --log

Log to review possible problems with content
Copy to clipboard
php console.php i:r -p --log

Several index_ files in the database

Typically happening when using the same database over and over you may end with several indexes in your database.
We provide a developer tool to clean properly this issue.
You need to be the web user and use bash to run the script.

Copy to clipboard
bash doc/devtools/clear_mysql_search_indexes.bash


You need to rebuild your index after the operation.

MySQL limits for very big numbers of tracker fields

How to detect

If Unified Search is configured with the MySQL Full Text Search engine, you might encounter a failure to reindex with the following symptoms:

  • php console index:rebuild stops without displaying the list of indexing statistics:
Copy to clipboard
php console index:rebuild [14 mai 2020 09:26 EDT] Started rebuilding index... Unified search engine: MySQL, version 10.1.45-MariaDB (it takes a very long time and in the end nothing shows up)


If you investigate with producing a log it ends this way:

Copy to clipboard
php console.php index:rebuild --log [14 mai 2020 16:10 EDT] Started rebuilding index... logging to file: temp/Search_Indexer_mysql_database_name_console.log Unified search engine: MySQL, version 10.1.45-MariaDB (it takes a very long time and in the end nothing shows up) tail temp/Search_Indexer_mysql_database_name_console.log … 2020-05-14T16:21:47-04:00 ERR (3): Indexing failed while processing "2512" (type trackeritem) with the error "Could not perform index modification: Too many columns" 2020-05-14T16:21:47-04:00 ERR (3): WARNING: PDO::query(): SQLSTATE[HY000]: General error: 1117 Too many columns {"code":null,"file":"/path/to/tikiroot/lib/core/TikiDb/Pdo.php","line":104} 2020-05-14T16:21:47-04:00 INFO (6): addDocument trackeritem 2513 {"memoryUsage":"46.5 MiB"} 2020-05-14T16:21:47-04:00 ERR (3): Indexing failed while processing "2513" (type trackeritem) with the error "Could not perform index modification: Too many columns" 2020-05-14T16:21:47-04:00 ERR (3): WARNING: PDO::query(): SQLSTATE[HY000]: General error: 1117 Too many columns {"code":null,"file":"/path/to/tikiroot/lib/core/TikiDb/Pdo.php","line":104} 2020-05-14T16:21:47-04:00 INFO (6): addDocument trackeritem 2514 {"memoryUsage":"46.5 MiB"}

Technical explanation (thanks Victor)
[+]
Solutions


The following options in Control panels → Search help for this situation:

  • MySQL use short field names
    Due to frm file constraints, number of search fields that one index can hold is usually limited to about 1500. This can be exceeded if you have numerous tracker fields. Enabling this option will try to shorten the field names internally that should allow you to use 300-500 more fields. Switching this option requires full index rebuild.
    Summary: This makes no difference in how you use Tiki.
  • Don't index non searchable fields
    Indexing will skip adding all tracker fields that are not marked as "searchable". This will free index space but also make it impossible to use those fields in search index queries.
    Summary: You need to review the Searchable property of you tracker fields. A full index rebuild will be necessary after changes in the tracker fields Searchable properties.


Hint: If the first option is enough for your site, you may ignore the second one.

Indexing log common error troubleshooting


When rebuilding your index use the following command to create a log file.
Note you need to point to the PHP for the PHP version your Tiki uses.

Copy to clipboard
php console.php i:r -p --log

In the log you may see some error, this is an attempt to help and diagnose the possible issues.

Error Possible fix
addDocument article 275
ERR (3): NOTICE: Trying to access array offset on value of type bool.../...lib/core/Search/ContentSource/ArticleSource.php","line":53
Edit article275, Advanced tab check the URL validity in the "Source" field.


Unified Index storage in the database (MySQL full-text search)

By design the unified-index store data in the MySQL database using MyISAM no matter if your Tiki uses innoDB or MyISAM (not all versions of InnoDB offer FULLTEXT).

Duplicate unified-index table

For some reasons you may found that your database has several index (tables like index_...alphanumeric...).
This happen when they were issues rebuilding the index or when moving and upgrading the database (Tiki upgrade).

You can delete those tables (always consider saving a backup of your data prior any work) and rebuild the unified-index. Tiki will recreate and relink everything properly. You can find information about the unified-index in use at the Search control panel, General settings under Unified search index.

Developer Notes

See https://dev.tiki.org/Unified-Index#Developer_Notes.

alias names for this page

Unified Search | UnifiedSearch | Enterprise search | Search Index | SearchIndex | UnifiedIndex | IndexRebuild | Index Rebuild

History

Advanced
Information Version
King David KAMBALE 133
Jonny Bradley remove mouseover from inside the fancy table heading param which seems to now break in 26.x (and replaced badly worded * note) 132
Bruno Kambere 131
Bruno Kambere 130
Marc Laporte 129
Marc Laporte 128
Marc Laporte Put in PluginMouseover 127
Marc Laporte 126
Marc Laporte 125
Marc Laporte Moved to dedicated page 124
Marc Laporte Move to own page 123
Marc Laporte Thank you Victor 122
Marc Laporte 121
Marc Laporte Index is being rebuilt at the moment and cannot start another rebuild process. 120
Bernard Sfez / Tiki Specialist 119
Bernard Sfez / Tiki Specialist Adding information about searchable and not searchable fields 118
Bernard Sfez / Tiki Specialist Fixing typos and links 117
Bernard Sfez / Tiki Specialist Improved the information especially related to rebuilding the index 116
Bernard Sfez / Tiki Specialist Removing fancy table and table sorter... not working (due to quotes in a cell ?) 115
Bernard Sfez / Tiki Specialist 114
Bernard Sfez / Tiki Specialist 113
Bernard Sfez / Tiki Specialist Link to Tiki main page (SEO) and adding an alias of a "lost" page (see gardeners blog) 112
Bernard Sfez / Tiki Specialist 111
Bernard Sfez / Tiki Specialist Adding troubleshooting from log output 110
Bernard Sfez / Tiki Specialist 109
Bernard Sfez / Tiki Specialist Adding information about clear_mysql_search_indexes.bash 108
Bernard Sfez / Tiki Specialist Added information about storage of the unified index in the database 107
Marc Laporte 106
luciash d' being 🧙 Page renamed from Unified Index to Search and List from Unified Index. Semantic alias redirect created 105
Bernard Sfez / Tiki Specialist adding title_initial and title_firstword 104
luciash d' being 🧙 103
Jean-Marc Libs 102
Jean-Marc Libs 101
Jean-Marc Libs 100
luciash d' being 🧙 formatting 99
luciash d' being 🧙 formatting 98
luciash d' being 🧙 moved to dev 97
luciash d' being 🧙 96
luciash d' being 🧙 95
Jonny Bradley autotoc off 94
Marc Laporte AutoTOC 93
drsassafras removed tiki 9 & 10 specific info 92
Marc Laporte 91
Philippe Cloutier typos 90
Xavier de Pedro 89
Xavier de Pedro 88
Xavier de Pedro 87
Xavier de Pedro 86
Xavier de Pedro 85
Xavier de Pedro 84