Estimating Query Timings in Elasticsearch

Abstract

In a shared Elasticsearch environment it can be useful to know how long a particular query will take to execute. This information can be used to enforce rate limiting or distribute requests equitably among multiple clusters. Elasticsearch uses multiple Lucene instances on multiple hosts as an underlying search engine implementation, but this abstraction makes it difficult to predict execution with previously known predictors such as the number of postings. This research investigates the ability of different pre-retrieval statistics, available through Elasticsearch, to accurately predict the execution time of queries on a typical Elasticsearch cluster. The number of terms in a query and the Total Term Frequency (TTF) from Elasticsearch’s API are found to significantly predict execution time. Regression models are then built and compared to find the most accurate method for predicting query time.