1.ES安装+ik插件
mac安装和使用:
a.由于ES为java开发 所以首先安装,由于最新版本对jdk要求>1.8.0,所以可以通过brew 升级jdk
brew cask install caskroom/versions/java8
b.brew 安装ES
$ brew update$ brew search elasticsearchelasticsearch elasticsearch@2.4 elasticsearch@5.6 $ brew install elasticsearch@2.4 #版本不需要安装太高,不然有的插件用不了....$ brew info elasticsearch@2.4 #安装成功后信息elasticsearch@2.4: stable 2.4.6 [keg-only]Distributed search & analytics enginehttps://www.elastic.co/products/elasticsearch/usr/local/Cellar/elasticsearch@2.4/2.4.6 (78 files, 38.5MB) Built from source on 2017-12-15 at 11:49:26From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/elasticsearch@2.4.rb==> RequirementsRequired: java >= 1.7 ✔==> CaveatsData: /usr/local/var/elasticsearch/elasticsearch_lph/Logs: /usr/local/var/log/elasticsearch/elasticsearch_lph.logPlugins: /usr/local/opt/elasticsearch@2.4/libexec/plugins/Config: /usr/local/etc/elasticsearch/plugin script: /usr/local/opt/elasticsearch@2.4/libexec/bin/pluginThis formula is keg-only, which means it was not symlinked into /usr/local,because this is an alternate version of another formula.If you need to have this software first in your PATH run: echo 'export PATH="/usr/local/opt/elasticsearch@2.4/bin:$PATH"' >> ~/.bash_profileTo have launchd start elasticsearch@2.4 now and restart at login: brew services start elasticsearch@2.4Or, if you don't want/need a background service you can just run: elasticsearch$ elasticsearch #启动elasticsearch
启动 elasticsearch 后,在浏览器地址:127.0.0.1:9200,浏览器会显示一段json数据
这里出现一个错误: 再次用brew 启动 elasticsearch时候
$ brew services start elasticsearch@2.4 或者 $ elasticsearch #报错 ... failed to obtain node locks, tried [[/usr/local/var/lib/elasticsearch/elasticsearch_lph]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])? ...
127.0.0.1:9200不能进行访问,这是因为已经有个节点在线程中跑了,可以查看线程
$ ps aux | grep "elastics"lph 86699 0.0 0.0 2442020 816 s000 S+ 2:02下午 0:00.00 grep elasticslph 86662 0.0 11.0 5079720 920492 ?? S 2:01下午 0:19.68 /Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home....$ kill -9 86662 #杀死jdk跑的elasticsearch 线程
之后 elasticsearch就能正常启动了
$ brew services start elasticsearch@2.4
修改elasticsearch配置
在安装完成后 brew info elasticsearch,显示了对于配置文件目录位置 /usr/local/etc/elasticsearch 打开elasticsearch.yml
$ vim /usr/local/etc/elasticsearch/elasticsearch.yml......cluster.name : test #集群名称...node.name: master-1 #当前节点...network.host: 127.0.0.1
保存修改后,启动elasticsearch
$ brew services start elasticsearch@2.4
外网访问elasticsearch配置
1.elasticsearch.yml 中network.host: 127.0.0.1 改为 network.host: 0.0.0.0
2.开启unbuntu 防火墙对9200的允许
$ sudo ufw status #查看防火墙状态22 ALLOW Anywhere80 ALLOW Anywhere$ sudo ufw allow from youallowip to any port 9200 #youallowip为你允许操作的ip,如果没有ip操作限制直接写成:sudo ufw allow 9200
3.用云平台 的朋友需要考虑云端是否设置了9200端口是否被允许访问
安装中文分词插件
Versions
| IK version | ES version |
|---|---|
| master | 6.x -> master |
| 6.0.0 | 6.0.0 |
| 5.6.4 | 5.6.4 |
| 5.5.3 | 5.5.3 |
| 5.4.3 | 5.4.3 |
| 5.3.3 | 5.3.3 |
| 5.2.2 | 5.2.2 |
| 5.1.2 | 5.1.2 |
| 1.10.6 | 2.4.6 |
| 1.9.5 | 2.3.5 |
| 1.8.1 | 2.2.1 |
| 1.7.0 | 2.1.1 |
| 1.5.0 | 2.0.0 |
| 1.2.6 | 1.0.0 |
| 1.2.5 | 0.90.x |
| 1.1.3 | 0.20.x |
| 1.0.0 | 0.16.2 -> 0.19.0 |
本例子安装ES 版本为2.4.6,所以对应下载1.10.6插件, 点击releases 查看v1.10.6版本复制下载地址
$ cd /usr/local/opt/elasticsearch@2.4/libexec/plugins/ #切换到elasticsearch插件位置$ cd mkdir ik #创建ik目录存放插件$ cd ik$ wget https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v1.10.6/elasticsearch-analysis-ik-1.10.6.zip #下载安装包$ unzip elasticsearch-analysis-ik-1.10.6.zip #解压安装包$ rm elasticsearch-analysis-ik-1.10.6.zip #删除压缩包$ brew services restart elasticsearch@2.4 #重启elasticsearch@2.4
重启之后查看log 表示插件加载成功 plugins [analysis-ik]
[2017-12-15 13:28:50,371][INFO ][plugins ] [master1] modules [reindex, lang-expression, lang-groovy], plugins [analysis-ik], sites []
对比 ik 和自带 对比测试
$ curl -XPOST "http://127.0.0.1:9200/_analyze?analyzer=ik&pretty" -d "你好世界"{ "tokens" : [ { "token" : "你好", "start_offset" : 0, "end_offset" : 2, "type" : "CN_WORD", "position" : 0 }, { "token" : "世界", "start_offset" : 2, "end_offset" : 4, "type" : "CN_WORD", "position" : 1 } ]} $ curl -XPOST "http://127.0.0.1:9200/_analyze?analyzer=standard&pretty" -d "你好世界"{ "tokens" : [ { "token" : "你", "start_offset" : 0, "end_offset" : 1, "type" : " ", "position" : 0 }, { "token" : "好", "start_offset" : 1, "end_offset" : 2, "type" : " ", "position" : 1 }, { "token" : "世", "start_offset" : 2, "end_offset" : 3, "type" : " ", "position" : 2 }, { "token" : "界", "start_offset" : 3, "end_offset" : 4, "type" : " ", "position" : 3 } ]}
ubuntu安装:
首先安装jdk
sudo add-apt-repository ppa:webupd8team/java##报错:The program 'add-apt-repository' is currently not installed. You can install it by typing:apt-get install software-properties-common需要先执行:apt-get install software-properties-common 再执行sudo add-apt-repository ppa:webupd8team/javasudo apt-get updatesudo apt-get install oracle-java8-installer #都选yessudo apt-get install oracle-java8-set-default #都选yesjava -versionwhereis javawhich java （java執行路徑）echo $JAVA_HOME
导入 Elasticsearch 公共 GPG 密钥到 apt：
$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
创建 Elasticsearch 资源列表：
$ echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
安装Elasticsearch
$ sudo apt-get update$ sudo apt-get -y install elasticsearch$ service elasticsearch start #启动elasticsearch
运行以下命令将在系统启动时启动 Elasticsearch：
$ sudo update-rc.d elasticsearch defaults 95 10
查看版本
root@iZwz92p0vf7mrdbn3qwl40Z:~# curl -XGET localhost:9200{ "name" : "masterr-1", "cluster_name" : "deraceur", "cluster_uuid" : "VV0D289YTbamI939oKzkOA", "version" : { "number" : "2.4.6",#为es版本号 ik 对应版本1.10.6 "build_hash" : "5376dca9f70f3abef96a77f4bb22720ace8240fd", "build_timestamp" : "2017-07-18T12:17:44Z", "build_snapshot" : false, "lucene_version" : "5.5.4" }, "tagline" : "You Know, for Search"}
安装maven (ik需要打包 需要先按照maven)
1 下载安装
$ wget http://mirror.bit.edu.cn/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz $ tar -zxvf apache-maven-3.3.9-bin.tar.gz $ sudo mv apache-maven-3.3.9 /usr/local/
2 为maven配置环境变量
将如下内容添加到/etc/profile和/root/.bashrc中
export M2_HOME=/usr/local/apache-maven-3.3.9 export PATH=$PATH:$M2_HOME/bin
执行如下指令使其上述配置生效或者重启系统：
source /etc/profile
检查安装是否生效
$ mvn -v Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00) Maven home: /usr/local/apache-maven-3.3.9 Java version: 1.8.0_65, vendor: Oracle Corporation Java home: /usr/local/jdk/jdk1.8.0_65/jre Default locale: en_US, platform encoding: UTF-8 OS name: “linux”, version: “3.19.0-25-generic”, arch: “amd64”, family: “unix”
安装ik插件
git clone https://github.com/medcl/elasticsearch-analysis-ik.gitcd elasticsearch-analysis-ik/git checkout -b v1.10.6/usr/local/apache-maven-3.3.9/bin/mvn package #mvn执行mkdir /usr/share/elasticsearch/plugins/ik #创建ikcp target/releases/elasticsearch-analysis-ik-1.10.6.zip /usr/share/elasticsearch/plugins/ikcd /usr/share/elasticsearch/plugins/ikunzip elasticsearch-analysis-ik-1.10.6.zip$ service elasticsearch restart #重新启动elasticsearch
修改elasticsearch配置
vim /etc/elasticsearch/elasticsearch.yml# 添加以下内容index.analysis.analyzer.default.type: ik
测试
root@dsfsd:~# curl -XGET 'http://127.0.0.1:9200/_analyze?pretty&analyzer=ik_max_word' -d '电动车不贴反光膜是违法行为'{ "tokens" : [ { "token" : "电动车", "start_offset" : 0, "end_offset" : 3, "type" : "CN_WORD", "position" : 0 }, { "token" : "电动", "start_offset" : 0, "end_offset" : 2, "type" : "CN_WORD", "position" : 1 }, { "token" : "车", "start_offset" : 2, "end_offset" : 3, "type" : "CN_CHAR", "position" : 2 }, { "token" : "不", "start_offset" : 3, "end_offset" : 4, "type" : "CN_CHAR", "position" : 3 }, { "token" : "贴反", "start_offset" : 4, "end_offset" : 6, "type" : "CN_WORD", "position" : 4 }, { "token" : "贴", "start_offset" : 4, "end_offset" : 5, "type" : "CN_WORD", "position" : 5 }, { "token" : "反光膜", "start_offset" : 5, "end_offset" : 8, "type" : "CN_WORD", "position" : 6 }, { "token" : "反光", "start_offset" : 5, "end_offset" : 7, "type" : "CN_WORD", "position" : 7 }, { "token" : "膜", "start_offset" : 7, "end_offset" : 8, "type" : "CN_WORD", "position" : 8 }, { "token" : "违法行为", "start_offset" : 9, "end_offset" : 13, "type" : "CN_WORD", "position" : 9 }, { "token" : "违法", "start_offset" : 9, "end_offset" : 11, "type" : "CN_WORD", "position" : 10 }, { "token" : "违", "start_offset" : 9, "end_offset" : 10, "type" : "CN_WORD", "position" : 11 }, { "token" : "法", "start_offset" : 10, "end_offset" : 11, "type" : "CN_CHAR", "position" : 12 }, { "token" : "行为", "start_offset" : 11, "end_offset" : 13, "type" : "CN_WORD", "position" : 13 } ]}
4.通过elasticsearch-jdbc操作数据(不推荐使用,这边我只是记录下)
使用elasticsearch-jdbc将现有MySQL数据批量导入至ElasticSearch(ubuntu环境运行)
下载并生成 mysql-import-product.sh
# wget http://xbib.org/repository/org/xbib/elasticsearch/importer/elasticsearch-jdbc/2.3.4.0/elasticsearch-jdbc-2.3.4.0-dist.zip #如果下载速度很慢 可以尝试迅雷下载# unzip elasticsearch-jdbc-2.3.4.0-dist.zip #解压# cd elasticsearch-jdbc-2.3.4.0-dist# cd bin# cp mysql-blog.sh mysql-import-product.sh# vim mysql-import-product.sh
修改 mysql-import-product.sh
#!/bin/shDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"bin=${DIR}/../binlib=${DIR}/../libecho '{ "type" : "jdbc", "jdbc" : { "url" : "jdbc:mysql://localhost:3306/test", #目标数据库为test "schedule" : "0 0-59 0-23 ? * *", #定时任务 "user" : "root", #数据库user "password" : "12345678", "sql" : "select productid,title from product" #导入数据语句 "index" : "test", "type" : "products", "elasticsearch" : { "cluster" : "test",#为 elasticsearch config 中的 cluster.name "host" : "localhost", "port" : 9300 } }}' | java \ -cp "${lib}/*" \ -Dlog4j.configurationFile=${bin}/log4j2.xml \ org.xbib.tools.Runner \ org.xbib.tools.JDBCImporter 保存后 执行该脚本
# cd ../..# bin/mysql-import-product.shbin/mysql-import-product.sh: 3: bin/mysql-import-product.sh: Bad substitutionbin/mysql-import-product.sh: 4: bin/mysql-import-product.sh: num: not found#报错 为路劲错误,google很多说是jdk path问题,这边没有时间研究,直接修改路劲# pwd #显示当前路劲/root/elasticsearch-jdbc-2.3.4.0# vim bin/mysql-import-product.sh #修改如下... bin=/root/elasticsearch-jdbc-2.3.4.0/binlib=/root/elasticsearch-jdbc-2.3.4.0/lib....# 退出保存# bin/mysql-import-product.sh #继续报错 google后 为jdk版本>1.8要求,本机为1.7Exception in thread "main" java.lang.UnsupportedClassVersionError: org/xbib/tools/Runner : Unsupported major.minor version 52.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:803) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:442) at java.net.URLClassLoader.access$100(URLClassLoader.java:64) at java.net.URLClassLoader$1.run(URLClassLoader.java:354) at java.net.URLClassLoader$1.run(URLClassLoader.java:348) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:347) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:312) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)# .... #papapa~ 升级jdk 升级方式请参考 http://blog.csdn.net/cuiaamay/article/details/51822308# chmod 777 bin/mysql-import-product.sh #添加执行权限# bin/mysql-import-product.sh # curl -XGET "http://127.0.0.1:9200/test/_search?pretty" #验证导入时候成功{ "took" : 2, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 378, ....... #有数据表示成功 指令记录:
$ curl -XPOST "http://127.0.0.1:9200/test/_search?pretty" #获取所有$ curl -XPOST "http://127.0.0.1:9200/test/_search?pretty" -d”@search.json” #通过search.json检索数据$ curl -XDELETE "http://127.0.0.1:9200/test" #删除数据
elasticsearch-jdbc自动导入增量数据
# cp mysql-import-product.sh mysql-delta-import-product.sh# vim mysql-delta-import-product.sh#!/bin/shDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"bin=${DIR}/../binlib=${DIR}/../libecho '{ "type" : "jdbc", "jdbc" : { "url" : "jdbc:mysql://localhost:3306/test", #目标数据库为test "schedule" : "0 0-59 0-23 ? * *", #定时任务 "user" : "root", #数据库user "password" : "12345678", "sql" :[{ "statement": "select productid,title from product.updated_at > unix_timestamp(?) group by bshop_product_lang.productid",#当updated_at时间大于当前时间 执行数据导入 "parameter" : ["$metrics.lastexecutionstart"] #当前时间戳 }], "index" : "test", "type" : "products", "elasticsearch" : { "cluster" : "test",#为 elasticsearch config 中的 cluster.name "host" : "localhost", "port" : 9300 } }}' | java \ -cp "${lib}/*" \ -Dlog4j.configurationFile=${bin}/log4j2.xml \ org.xbib.tools.Runner \ org.xbib.tools.JDBCImporter 其中updated_at为product更新或者创建时候的记录时间戳.
添加mysql-delta-import-product.sh 脚本权限和写进开机执行脚本
# chmod -X mysql-delta-import-product.sh# vim /etc/rc.local....sh /youpath/bin/mysql-delta-import-product.sh #ES数据增量 youpath为文件绝对位置exit 0....
这样开机将自动执行该脚本
5.通过 yii2-elasticsearch操作数据(推荐方法)
打开 yii2 根目录下 composer.json
... "require": { .... "yiisoft/yii2-elasticsearch": "^2.0" },... 保存后
$ composer update
完成后在yii2 配置文件中写入
'components' => [ 'elasticsearch' => [ 'class' => 'yii\elasticsearch\Connection', 'nodes' => [ ['http_address' => '127.0.0.1:9200'], // configure more hosts if you have a cluster ], ],
创建common\models\ESProduct类 继承 yii\elasticsearch\ActiveRecord
[ 'productid' => ['type' => 'double'], 'title' => ['type' => 'string','analyzer'=>'ik', 'searchAnalyzer' => 'ik'] ] ]; } public static function mapping() { return [ static::type() => self::mapConfig(), ]; } //索引名称 public static function index() { return "test"; } //索引类型 public static function type() { return "products"; } /** * Set (update) mappings for this model */ public static function updateMapping() { $db = static::getDb(); $command = $db->createCommand(); $command->setMapping(static::index(), static::type(), static::mapping()); } /** * Create this model's index */ public static function createIndex() { $db = static::getDb(); $command = $db->createCommand(); $command->createIndex(static::index(), [ 'settings' => [ 'index' => ['refresh_interval' => '1s'] ], 'mappings' => static::mapping(), //'warmers' => [ /* ... */ ], //'aliases' => [ /* ... */ ], //'creation_date' => '...' ]); } /** * Delete this model's index */ public static function deleteIndex() { $db = static::getDb(); $command = $db->createCommand(); $command->deleteIndex(static::index(), static::type()); } public static function updateRecord($productid, $columns){ try { $record = self::get($productid); foreach($columns as $key => $value){ $record->$key = $value; } return $record->update(); }catch (\Exception $e){ return false; } } //删除记录 public static function deleteRecord($productid){ try { $record = self::get($productid); $record->delete(); return true; }catch (\Exception $e){ return false; } } //保存数据(添加 || 更新) public function saveRecord($data){ //加载数据 if ($this->load($data) && $this->validate()){ $isExist = false; try { $record = self::get($this->productid); //不存在 为更新 if (!$record){ $record = new self(); $record->setPrimaryKey($this->productid); }else{ $isExist = true; } }catch (\Exception $e){ $record = new self(); $record->setPrimaryKey($this->productid); } //遍历数据到对象 foreach ($data['ESProductSearch'] as $key => $value) { $record->$key = $value; } try { if (!$isExist){ //插入 $result = $record->insert(); }else{ //更新 $result = $record->update(); } }catch (\Exception $e){ return false; } return true; } } } 创建测试TestControlle.php
query([ "multi_match" => [ "query" => $keyword, "fields" => ["title"] //检索属性 ], ])->all(); foreach ($res as $k){ var_dump($k->productid);//输出id } exit; } //创建 或更新 public function actionCreateES(){ $ESArray = [ 'ESProductSearch' => [ 'productid' => '1', 'title' => 'hello world' ] ]; $eSProductSearch = new ESProductSearch(); if(!$eSProductSearch -> saveRecord($ESArray)){ throw new \Exception(Common::getLayoutMessageHtml("操作失败")); } } //删除 public function actionDeteleES($id){ $eSProductSearch = new ESProductSearch(); if(!$eSProductSearch->deleteByProduct($id)){ throw new \Exception(Common::getLayoutMessageHtml("删除失败")); } }}