被標注的命名實體被放在《START》《END》范圍中,并標出了實體的類別。接下來是對命名實體識別模型的訓練,先上代碼:
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.StringReader;
import java.util.Collections;
import opennlp.tools.namefind.NameFinderME;
import opennlp.tools.namefind.NameSample;
import opennlp.tools.namefind.NameSampleDataStream;
import opennlp.tools.namefind.TokenNameFinderModel;
import opennlp.tools.util.ObjectStream;
import opennlp.tools.util.PlainTextByLineStream;
import opennlp.tools.util.featuregen.AggregatedFeatureGenerator;
import opennlp.tools.util.featuregen.PreviousMapFeatureGenerator;
import opennlp.tools.util.featuregen.TokenClassFeatureGenerator;
import opennlp.tools.util.featuregen.TokenFeatureGenerator;
import opennlp.tools.util.featuregen.WindowFeatureGenerator;
/**
* 中文命名實體識別模型訓練組件
*
* @author ddlovehy
*
*/
public class NamedEntityMultiFindTrainer {
// 默認參數
private int iterations = 80;
private int cutoff = 5;
private String langCode = “general”;
private String type = “default”;
// 待設定的參數
private String nameWordsPath; // 命名實體詞庫路徑
private String dataPath; // 訓練集已分詞語料路徑
private String modelPath; // 模型存儲路徑
public NamedEntityMultiFindTrainer() {
super();
// TODO Auto-generated constructor stub
}
public NamedEntityMultiFindTrainer(String nameWordsPath, String dataPath,
String modelPath) {
super();
this.nameWordsPath = nameWordsPath;
this.dataPath = dataPath;
this.modelPath = modelPath;
}
public NamedEntityMultiFindTrainer(int iterations, int cutoff,
String langCode, String type, String nameWordsPath,
String dataPath, String modelPath) {
super();
this.iterations = iterations;
this.cutoff = cutoff;
this.langCode = langCode;
this.type = type;
this.nameWordsPath = nameWordsPath;
this.dataPath = dataPath;
this.modelPath = modelPath;
}
/**
* 生成定制特征
*
* @return
*/
public AggregatedFeatureGenerator prodFeatureGenerators() {
AggregatedFeatureGenerator featureGenerators = new AggregatedFeatureGenerator(
new WindowFeatureGenerator(new TokenFeatureGenerator(), 2, 2),
new WindowFeatureGenerator(new TokenClassFeatureGenerator(), 2,
2), new PreviousMapFeatureGenerator());
return featureGenerators;
}
/**
* 將模型寫入磁盤
*
* @param model
* @throws Exception
*/
public void writeModelIntoDisk(TokenNameFinderModel model) throws Exception {
File outModelFile = new File(this.getModelPath());
FileOutputStream outModelStream = new FileOutputStream(outModelFile);
model.serialize(outModelStream);
}
/**
* 讀出標注的訓練語料
*
* @return
* @throws Exception
*/
public String getTrainCorpusDataStr() throws Exception {
// TODO 考慮入持久化判斷直接載入標注數據的情況 以及增量式訓練
String trainDataStr = null;
trainDataStr = NameEntityTextFactory.prodNameFindTrainText(
this.getNameWordsPath(), this.getDataPath(), null);
return trainDataStr;
}
/**
* 訓練模型
*
* @param trainDataStr
* 已標注的訓練數據整體字符串
* @return
* @throws Exception
*/
public TokenNameFinderModel trainNameEntitySamples(String trainDataStr)
throws Exception {
ObjectStream《NameSample》 nameEntitySample = new NameSampleDataStream(
new PlainTextByLineStream(new StringReader(trainDataStr)));
System.out.println(“**************************************”);
System.out.println(trainDataStr);
TokenNameFinderModel nameFinderModel = NameFinderME.train(
this.getLangCode(), this.getType(), nameEntitySample,
this.prodFeatureGenerators(),
Collections.《String, Object》 emptyMap(), this.getIterations(),
this.getCutoff());
return nameFinderModel;
}
/**
* 訓練組件總調用方法
*
* @return
*/
public boolean execNameFindTrainer() {
try {
String trainDataStr = this.getTrainCorpusDataStr();
TokenNameFinderModel nameFinderModel = this
.trainNameEntitySamples(trainDataStr);
// System.out.println(nameFinderModel);
this.writeModelIntoDisk(nameFinderModel);
return true;
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
return false;
}
}
?。?/p>
注:
參數:iterations是訓練算法迭代的次數,太少了起不到訓練的效果,太大了會造成過擬合,所以各位可以自己試試效果;
cutoff:語言模型掃描窗口的大小,一般設成5就可以了,當然越大效果越好,時間可能會受不了;
langCode:語種代碼和type實體類別,因為沒有專門針對中文的代碼,設成“普通”的即可,實體的類別因為我們想訓練成能識別多種實體的模型,于是設置為“默認”。
說明:
prodFeatureGenerators()方法用于生成個人訂制的特征生成器,其意義在于選擇什么樣的n-gram語義模型,代碼當中顯示的是選擇窗口大小為5,待測命名實體詞前后各掃描兩個詞的范圍計算特征(加上自己就是5個),或許有更深更準確的意義,請大家指正;
trainNameEntitySamples()方法,訓練模型的核心,首先是將如上標注的訓練語料字符串傳入生成字符流,再通過NameFinderME的train()方法傳入上面設定的各個參數,訂制特征生成器等等,關于源實體映射對,就按默認傳入空Map就好了。
源代碼開源在:https://github.com/Ailab403/ailab-mltk4j,test包里面對應有完整的調用demo,以及file文件夾里面的測試語料和已經訓練好的模型。
評論
查看更多