elasticsearch学习笔记高级篇(十二)——掌握phrase matching搜索技术

0

1、什么是近似搜索

假设有两个句子

java is my favourite programming langurage, and I also think spark is a very good big data system.

java spark are very related, because scala is spark's programming langurage and scala is also based on jvm like java. 

适用match query 搜索java spark

{
    {
        "match": {
            "content": "java spark"
        }
    }
}

match query 只能搜索到包含java和spark的document,但是不知道java和spark是不是离得很近。
假设我们想要java和spark离得很近的document优先返回,就要给它一个更高的relevance score,这就涉及到了proximity match近似匹配。
下面给出要实现的两个需求:
(1)搜索java spark,就靠在一起,中间不能插入任何其它字符
(2)搜索java spark,要求java和spark两个单词靠的越近,doc的分数越高,排名越靠前

2、match phrase

准备数据:

PUT /test_index/_create/1
{
  "content": "java is my favourite programming language, and I also think spark is a very good big data system."
}
PUT /test_index/_create/2
{
  "content": "java spark are very related, because scala is spark's programming language and scala is also based on jvm like java."
}

对于需求1 搜索java spark,就靠在一起,中间不能插入任何其它字符:
使用match query搜索无法实现

GET /test_index/_search
{
  "query": {
    "match": {
      "content": "java spark"
    }
  }
}

结果:

{
  "took" : 16,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 0.4255141,
    "hits" : [
      {
        "_index" : "test_index",
        "_type" : "_doc",
        "_id" : "2",
        "_score" : 0.4255141,
        "_source" : {
          "content" : "java spark are very related, because scala is spark's programming language and scala is also based on jvm like java."
        }
      },
      {
        "_index" : "test_index",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 0.37266707,
        "_source" : {
          "content" : "java is my favourite programming language, and I also think spark is a very good big data system."
        }
      }
    ]
  }
}

使用match phrase搜索就可以实现

GET /test_index/_search
{
  "query": {
    "match_phrase": {
      "content": "java spark"
    }
  }
}

结果:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 0.35695744,
    "hits" : [
      {
        "_index" : "test_index",
        "_type" : "_doc",
        "_id" : "2",
        "_score" : 0.35695744,
        "_source" : {
          "content" : "java spark are very related, because scala is spark's programming language and scala is also based on jvm like java."
        }
      }
    ]
  }
}

3、term position

假设我们有两个document

doc1: hello world, java spark
doc2: hi, spark java

hello doc1(0)
world doc1(1)
java  doc1(2) doc2(2)
spark doc1(3) doc2(1)

position详情如下:

GET /_analyze
{
  "text": ["hello world, java spark"],
  "analyzer": "standard"
}
{
  "tokens" : [
    {
      "token" : "hello",
      "start_offset" : 0,
      "end_offset" : 5,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "world",
      "start_offset" : 6,
      "end_offset" : 11,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "java",
      "start_offset" : 13,
      "end_offset" : 17,
      "type" : "<ALPHANUM>",
      "position" : 2
    },
    {
      "token" : "spark",
      "start_offset" : 18,
      "end_offset" : 23,
      "type" : "<ALPHANUM>",
      "position" : 3
    }
  ]
}
GET /_analyze
{
  "text": ["hi, spark java"],
  "analyzer": "standard"
}
{
  "tokens" : [
    {
      "token" : "hi",
      "start_offset" : 0,
      "end_offset" : 2,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "spark",
      "start_offset" : 4,
      "end_offset" : 9,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "java",
      "start_offset" : 10,
      "end_offset" : 14,
      "type" : "<ALPHANUM>",
      "position" : 2
    }
  ]
}

4、match phrase基本原理


索引中的position,match_phrase

hello world, java spark        doc1
hi, spark java                doc2

hello         doc1(0)        
wolrd        doc1(1)
java        doc1(2) doc2(2)
spark        doc1(3) doc2(1)

使用match_phrase查询要求找到每个term都在一个共有的那些doc,就是要求一个doc,必须要包含查询的每个term,并且满足位置运算。

doc1 --> java和spark --> spark position恰巧比java大1 --> java的position是2,spark的position是3,恰好满足条件
doc1符合条件
doc2 --> java和spark --> java position是2,spark position是1,spark position比java position小1,而不是大1 --> 光是position就不满足,那么doc2不匹配
doc2不符合条件

5、slop参数

含义:query string搜索文本中的几个term,要经过几次移动才能与一个document匹配,这个移动的次数就是slop。
实际举一个例子:
对于hello world, java is very good, spark is also very good. 假设我们要用match phrase 匹配到java spark。可以发现直接进行查询会查不到

PUT /test_index/_create/1
{
  "content": "hello world, java is very good, spark is also very good."
}

GET /test_index/_search
{
  "query": {
    "match_phrase": {
      "content": "java spark"
    }
  }
}
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

此时使用

GET /_analyze
{
  "text": ["hello world, java is very good, spark is also very good."],
  "analyzer": "standard"
}

结果:

{
  "tokens" : [
    {
      "token" : "hello",
      "start_offset" : 0,
      "end_offset" : 5,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "world",
      "start_offset" : 6,
      "end_offset" : 11,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "java",
      "start_offset" : 13,
      "end_offset" : 17,
      "type" : "<ALPHANUM>",
      "position" : 2
    },
    {
      "token" : "is",
      "start_offset" : 18,
      "end_offset" : 20,
      "type" : "<ALPHANUM>",
      "position" : 3
    },
    {
      "token" : "very",
      "start_offset" : 21,
      "end_offset" : 25,
      "type" : "<ALPHANUM>",
      "position" : 4
    },
    {
      "token" : "good",
      "start_offset" : 26,
      "end_offset" : 30,
      "type" : "<ALPHANUM>",
      "position" : 5
    },
    {
      "token" : "spark",
      "start_offset" : 32,
      "end_offset" : 37,
      "type" : "<ALPHANUM>",
      "position" : 6
    },
    {
      "token" : "is",
      "start_offset" : 38,
      "end_offset" : 40,
      "type" : "<ALPHANUM>",
      "position" : 7
    },
    {
      "token" : "also",
      "start_offset" : 41,
      "end_offset" : 45,
      "type" : "<ALPHANUM>",
      "position" : 8
    },
    {
      "token" : "very",
      "start_offset" : 46,
      "end_offset" : 50,
      "type" : "<ALPHANUM>",
      "position" : 9
    },
    {
      "token" : "good",
      "start_offset" : 51,
      "end_offset" : 55,
      "type" : "<ALPHANUM>",
      "position" : 10
    }
  ]
}
java        is        very        good        spark        is

java        spark
java        -->        spark
java                -->            spark
java                            -->            spark

可以发现java的position是2,spark的position是6,那么我们只需要设置slop大于等于3(也就是移动3词就可以了)就可以搜到了

GET /test_index/_search
{
  "query": {
    "match_phrase": {
      "content": {
        "query": "java spark",
        "slop": 3
      }
    }
  }
}

结果:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 0.21824157,
    "hits" : [
      {
        "_index" : "test_index",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 0.21824157,
        "_source" : {
          "content" : "hello world, java is very good, spark is also very good."
        }
      }
    ]
  }
}

此时加上slop的match phrase就是proximity match近似匹配了。加上slop之后虽然是近似匹配可以搜索到很多结果,但是距离越近的会优先返回,也就是相关度分数就会越高。


如果觉得我的文章对你有用,请随意赞赏

你可能感兴趣的

载入中...