Lucene4.3进阶开发之纯阳无极(十九)

2024-05-15 04:08

本文主要是介绍Lucene4.3进阶开发之纯阳无极(十九),希望对大家解决编程问题提供一定的参考价值,需要的开发者们随着小编来一起学习吧!

[b][color=red][size=x-large]原创不易,转载请务必注明,原创地址,谢谢配合!
[url]http://qindongliang.iteye.com/blog/2164583[/url]
[/size][/color][/b]
[b][color=green][size=large]Lucene内置很多的分词器工具包,几乎涵盖了全球所有的国家和地区,最近散仙,在搞多语言分词的一个处理,主要国家有西班牙,葡萄牙,德语,法语,意大利,其实这些语系都与英语非常类似,都是以空格为分割的语种。


那么首先,探讨下分词器的词形还原和词干提取的对搜索的意义?在这之前,先看下两者的概念:
词形还原(lemmatization),是把一个任何形式的语言词汇还原为一般形式(能表达完整语义),而词干提取

(stemming)是抽取词的词干或词根形式(不一定能够表达完整语义)。词形还原和词干提取是词形规范化的两类
重要方式,都能够达到有效归并词形的目的,二者既有联系也有区别

详细介绍,请参考[url=http://blog.csdn.net/march_on/article/details/8935462]这篇文章[/url]


在电商搜索里,词干的抽取,和单复数的还原比较重要(这里主要针对名词来讲),因为这有关搜索的查准率,和查全率的命中,如果我们的分词器没有对这些词做过处理,会造成什么影响呢?那么请看如下的一个例子?

句子: i have two cats

分词器如果什么都没有做:

这时候我们搜cat,就会无命中结果,而必须搜cats才能命中到一条数据,而事实上cat和cats是同一个东西,只不过单词的形式不一样,这样以来,如果不做处理,我们的查全率和查全率都会下降,会涉及影响到我们的搜索体验,所以stemming这一步,在某些场合的分词中至关重要。
[/size][/color][/b]
[b][color=olive][size=large]本篇,散仙,会参考源码分析一下,关于德语分词中中如何做的词干提取,先看下德语的分词声明:
[/size][/color][/b]
	 List<String> list=new ArrayList<String>();
list.add("player");//这里面的词,不会被做词干抽取,词形还原
CharArraySet ar=new CharArraySet(Version.LUCENE_43,list , true);
//分词器的第二个参数是禁用词参数,第三个参数是排除不做词形转换,或单复数的词
GermanAnalyzer sa=new GermanAnalyzer(Version.LUCENE_43,null,ar);


[b][color=olive][size=large]接着,我们具体看下,在德语的分词器中,都经过了哪几部分的过滤处理:[/size][/color][/b]
  protected TokenStreamComponents createComponents(String fieldName,
Reader reader) {
//标准分词器过滤
final Tokenizer source = new StandardTokenizer(matchVersion, reader);
TokenStream result = new StandardFilter(matchVersion, source);
//转小写过滤
result = new LowerCaseFilter(matchVersion, result);
//禁用词过滤
result = new StopFilter( matchVersion, result, stopwords);
//排除词过滤
result = new SetKeywordMarkerFilter(result, exclusionSet);
if (matchVersion.onOrAfter(Version.LUCENE_36)) {
//在lucene3.6以后的版本,采用如下filter过滤
//规格化,将德语中的特殊字符,映射成英语
result = new GermanNormalizationFilter(result);
//stem词干抽取,词性还原
result = new GermanLightStemFilter(result);
} else if (matchVersion.onOrAfter(Version.LUCENE_31)) {
//在lucene3.1至3.6的版本中,采用SnowballFilter处理
result = new SnowballFilter(result, new German2Stemmer());
} else {
//在lucene3.1之前的采用兼容的GermanStemFilter处理
result = new GermanStemFilter(result);
}
return new TokenStreamComponents(source, result);
}


[b][color=olive][size=large]OK,我们从源码中得知,在Lucene4.x中对德语的分词也做了向前和向后兼容,现在我们主要关注在lucene4.x之后的版本如何的词形转换,下面分别看下
result = new GermanNormalizationFilter(result);
result = new GermanLightStemFilter(result);
这两个类的功能:
[/size][/color][/b]
package org.apache.lucene.analysis.de;

/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

import java.io.IOException;

import org.apache.lucene.analysis.TokenFilter;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
import org.apache.lucene.analysis.util.StemmerUtil;

/**
* Normalizes German characters according to the heuristics
* of the <a href="http://snowball.tartarus.org/algorithms/german2/stemmer.html">
* German2 snowball algorithm</a>.
* It allows for the fact that ä, ö and ü are sometimes written as ae, oe and ue.
*
* [list]
* <li> 'ß' is replaced by 'ss'
* <li> 'ä', 'ö', 'ü' are replaced by 'a', 'o', 'u', respectively.
* <li> 'ae' and 'oe' are replaced by 'a', and 'o', respectively.
* <li> 'ue' is replaced by 'u', when not following a vowel or q.
* [/list]
* <p>
* This is useful if you want this normalization without using
* the German2 stemmer, or perhaps no stemming at all.
*上面的解释说得很清楚,主要是对德文的一些特殊字母,转换成对应的英文处理
*
*/

public final class GermanNormalizationFilter extends TokenFilter {
// FSM with 3 states:
private static final int N = 0; /* ordinary state */
private static final int V = 1; /* stops 'u' from entering umlaut state */
private static final int U = 2; /* umlaut state, allows e-deletion */

private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);

public GermanNormalizationFilter(TokenStream input) {
super(input);
}

@Override
public boolean incrementToken() throws IOException {
if (input.incrementToken()) {
int state = N;
char buffer[] = termAtt.buffer();
int length = termAtt.length();
for (int i = 0; i < length; i++) {
final char c = buffer[i];
switch(c) {
case 'a':
case 'o':
state = U;
break;
case 'u':
state = (state == N) ? U : V;
break;
case 'e':
if (state == U)
length = StemmerUtil.delete(buffer, i--, length);
state = V;
break;
case 'i':
case 'q':
case 'y':
state = V;
break;
case 'ä':
buffer[i] = 'a';
state = V;
break;
case 'ö':
buffer[i] = 'o';
state = V;
break;
case 'ü':
buffer[i] = 'u';
state = V;
break;
case 'ß':
buffer[i++] = 's';
buffer = termAtt.resizeBuffer(1+length);
if (i < length)
System.arraycopy(buffer, i, buffer, i+1, (length-i));
buffer[i] = 's';
length++;
state = N;
break;
default:
state = N;
}
}
termAtt.setLength(length);
return true;
} else {
return false;
}
}
}

package org.apache.lucene.analysis.de;

/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

import java.io.IOException;

import org.apache.lucene.analysis.TokenFilter;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.miscellaneous.SetKeywordMarkerFilter;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
import org.apache.lucene.analysis.tokenattributes.KeywordAttribute;

/**
* A {@link TokenFilter} that applies {@link GermanLightStemmer} to stem German
* words.
* <p>
* To prevent terms from being stemmed use an instance of
* {@link SetKeywordMarkerFilter} or a custom {@link TokenFilter} that sets
* the {@link KeywordAttribute} before this {@link TokenStream}.
*

*
*
*这个类,主要做Stemmer(词干提取),而我们主要关注
*GermanLightStemmer这个类的作用
*
*
*/
public final class GermanLightStemFilter extends TokenFilter {
private final GermanLightStemmer stemmer = new GermanLightStemmer();
private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);
private final KeywordAttribute keywordAttr = addAttribute(KeywordAttribute.class);

public GermanLightStemFilter(TokenStream input) {
super(input);
}

@Override
public boolean incrementToken() throws IOException {
if (input.incrementToken()) {
if (!keywordAttr.isKeyword()) {
final int newlen = stemmer.stem(termAtt.buffer(), termAtt.length());
termAtt.setLength(newlen);
}
return true;
} else {
return false;
}
}
}

[b][color=olive][size=large]下面看下,在GermanLightStemmer中,如何做的词干提取:源码如下:[/size][/color][/b]
 package org.apache.lucene.analysis.de;

/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

/*
* This algorithm is updated based on code located at:
* http://members.unine.ch/jacques.savoy/clef/
*
* Full copyright for that code follows:
*/

/*
* Copyright (c) 2005, Jacques Savoy
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* Redistributions of source code must retain the above copyright notice, this
* list of conditions and the following disclaimer. Redistributions in binary
* form must reproduce the above copyright notice, this list of conditions and
* the following disclaimer in the documentation and/or other materials
* provided with the distribution. Neither the name of the author nor the names
* of its contributors may be used to endorse or promote products derived from
* this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/

/**
* Light Stemmer for German.
* <p>
* This stemmer implements the "UniNE" algorithm in:
* <i>Light Stemming Approaches for the French, Portuguese, German and Hungarian Languages</i>
* Jacques Savoy
*/
public class GermanLightStemmer {

//处理特殊字符映射
public int stem(char s[], int len) {
for (int i = 0; i < len; i++)
switch(s[i]) {
case 'ä':
case 'à':
case 'á':
case 'â': s[i] = 'a'; break;
case 'ö':
case 'ò':
case 'ó':
case 'ô': s[i] = 'o'; break;
case 'ï':
case 'ì':
case 'í':
case 'î': s[i] = 'i'; break;
case 'ü':
case 'ù':
case 'ú':
case 'û': s[i] = 'u'; break;
}

len = step1(s, len);
return step2(s, len);
}


private boolean stEnding(char ch) {
switch(ch) {
case 'b':
case 'd':
case 'f':
case 'g':
case 'h':
case 'k':
case 'l':
case 'm':
case 'n':
case 't': return true;
default: return false;
}
}
//处理基于以下规则的词干抽取和缩减
private int step1(char s[], int len) {
if (len > 5 && s[len-3] == 'e' && s[len-2] == 'r' && s[len-1] == 'n')
return len - 3;

if (len > 4 && s[len-2] == 'e')
switch(s[len-1]) {
case 'm':
case 'n':
case 'r':
case 's': return len - 2;
}

if (len > 3 && s[len-1] == 'e')
return len - 1;

if (len > 3 && s[len-1] == 's' && stEnding(s[len-2]))
return len - 1;

return len;
}
//处理基于以下规则est,er,en等的词干抽取和缩减
private int step2(char s[], int len) {
if (len > 5 && s[len-3] == 'e' && s[len-2] == 's' && s[len-1] == 't')
return len - 3;

if (len > 4 && s[len-2] == 'e' && (s[len-1] == 'r' || s[len-1] == 'n'))
return len - 2;

if (len > 4 && s[len-2] == 's' && s[len-1] == 't' && stEnding(s[len-3]))
return len - 2;

return len;
}
}

[b][color=olive][size=large]具体的分析结果如下:[/size][/color][/b]

搜索技术交流群:324714439
大数据hadoop交流群:376932160

0,将一些德语特殊字符,替换成对应的英文表示
1,将所有词干元音还原 a ,o,i,u
ste(2)(按先后顺序,符合以下任意一项,就完成一次校验(return))
2,单词长度大于5的词,以ern结尾的,直接去掉
3,单词长度大于4的词,以em,en,es,er结尾的,直接去掉
4,单词长度大于3的词,以e结尾的直接去掉
5,单词长度大于3的词,以bs,ds,fs,gs,hs,ks,ls,ms,ns,ts结尾的,直接去掉s
step(3)(按先后顺序,符合以下任意一项,就完成一次校验(return))
6,单词长度大于5的词,以est结尾的,直接去掉
7,单词长度大于4的词,以er或en结尾的直接去掉
8,单词长度大于4的词,bst,dst,fst,gst,hst,kst,lst,mst,nst,tst,直接去掉后两位字母st

[b][color=olive][size=large]最后,结合网上资料分析,基于er,en,e,s结尾的是做单复数转换的,其他的几条规则主要是对非名词的单词,做词干抽取。

[/size][/color][/b]
[b][color=red][size=x-large]原创不易,转载请务必注明,原创地址,谢谢配合!
[url]http://qindongliang.iteye.com/blog/2164583[/url]
[/size][/color][/b]

这篇关于Lucene4.3进阶开发之纯阳无极(十九)的文章就介绍到这儿,希望我们推荐的文章对编程师们有所帮助!



http://www.chinasem.cn/article/990764

相关文章

Spring Security 从入门到进阶系列教程

Spring Security 入门系列 《保护 Web 应用的安全》 《Spring-Security-入门(一):登录与退出》 《Spring-Security-入门(二):基于数据库验证》 《Spring-Security-入门(三):密码加密》 《Spring-Security-入门(四):自定义-Filter》 《Spring-Security-入门(五):在 Sprin

这15个Vue指令,让你的项目开发爽到爆

1. V-Hotkey 仓库地址: github.com/Dafrok/v-ho… Demo: 戳这里 https://dafrok.github.io/v-hotkey 安装: npm install --save v-hotkey 这个指令可以给组件绑定一个或多个快捷键。你想要通过按下 Escape 键后隐藏某个组件,按住 Control 和回车键再显示它吗?小菜一碟: <template

Hadoop企业开发案例调优场景

需求 (1)需求:从1G数据中,统计每个单词出现次数。服务器3台,每台配置4G内存,4核CPU,4线程。 (2)需求分析: 1G / 128m = 8个MapTask;1个ReduceTask;1个mrAppMaster 平均每个节点运行10个 / 3台 ≈ 3个任务(4    3    3) HDFS参数调优 (1)修改:hadoop-env.sh export HDFS_NAMENOD

Java进阶13讲__第12讲_1/2

多线程、线程池 1.  线程概念 1.1  什么是线程 1.2  线程的好处 2.   创建线程的三种方式 注意事项 2.1  继承Thread类 2.1.1 认识  2.1.2  编码实现  package cn.hdc.oop10.Thread;import org.slf4j.Logger;import org.slf4j.LoggerFactory

嵌入式QT开发:构建高效智能的嵌入式系统

摘要: 本文深入探讨了嵌入式 QT 相关的各个方面。从 QT 框架的基础架构和核心概念出发,详细阐述了其在嵌入式环境中的优势与特点。文中分析了嵌入式 QT 的开发环境搭建过程,包括交叉编译工具链的配置等关键步骤。进一步探讨了嵌入式 QT 的界面设计与开发,涵盖了从基本控件的使用到复杂界面布局的构建。同时也深入研究了信号与槽机制在嵌入式系统中的应用,以及嵌入式 QT 与硬件设备的交互,包括输入输出设

OpenHarmony鸿蒙开发( Beta5.0)无感配网详解

1、简介 无感配网是指在设备联网过程中无需输入热点相关账号信息,即可快速实现设备配网,是一种兼顾高效性、可靠性和安全性的配网方式。 2、配网原理 2.1 通信原理 手机和智能设备之间的信息传递,利用特有的NAN协议实现。利用手机和智能设备之间的WiFi 感知订阅、发布能力,实现了数字管家应用和设备之间的发现。在完成设备间的认证和响应后,即可发送相关配网数据。同时还支持与常规Sof

活用c4d官方开发文档查询代码

当你问AI助手比如豆包,如何用python禁止掉xpresso标签时候,它会提示到 这时候要用到两个东西。https://developers.maxon.net/论坛搜索和开发文档 比如这里我就在官方找到正确的id描述 然后我就把参数标签换过来

[MySQL表的增删改查-进阶]

🌈个人主页:努力学编程’ ⛅个人推荐: c语言从初阶到进阶 JavaEE详解 数据结构 ⚡学好数据结构,刷题刻不容缓:点击一起刷题 🌙心灵鸡汤:总有人要赢,为什么不能是我呢 💻💻💻数据库约束 🔭🔭🔭约束类型 not null: 指示某列不能存储 NULL 值unique: 保证某列的每行必须有唯一的值default: 规定没有给列赋值时的默认值.primary key:

Linux_kernel驱动开发11

一、改回nfs方式挂载根文件系统         在产品将要上线之前,需要制作不同类型格式的根文件系统         在产品研发阶段,我们还是需要使用nfs的方式挂载根文件系统         优点:可以直接在上位机中修改文件系统内容,延长EMMC的寿命         【1】重启上位机nfs服务         sudo service nfs-kernel-server resta

【区块链 + 人才服务】区块链集成开发平台 | FISCO BCOS应用案例

随着区块链技术的快速发展,越来越多的企业开始将其应用于实际业务中。然而,区块链技术的专业性使得其集成开发成为一项挑战。针对此,广东中创智慧科技有限公司基于国产开源联盟链 FISCO BCOS 推出了区块链集成开发平台。该平台基于区块链技术,提供一套全面的区块链开发工具和开发环境,支持开发者快速开发和部署区块链应用。此外,该平台还可以提供一套全面的区块链开发教程和文档,帮助开发者快速上手区块链开发。