如何将mysql数据实时迁移到ElasticSearch

how to migrate mysql data to ElasticSearch realtime(如何将mysql数据实时迁移到ElasticSearch)

问题描述

我有一个包含几个表的 mysql 数据库，我想将 mysql 数据迁移到 ElasticSearch.通过批处理作业将整个数据库迁移到 ES 很容易.但是我应该如何从 mysql 实时更新 ES.即如果 mysql 中有更新操作，那么我应该在 ES 中执行相同的操作.我研究了 mysql binLog，它可以反映 mysql 的任何更改.但是我要把binLog解析成ES语法，我觉得真的很痛苦.谢谢！(与 Solr 相同的情况)

I have a mysql database with couple tables, I wanna migrate the mysql data to ElasticSearch. It's easy to migrate the whole database to ES via a batch job. But how should I update ES from mysql realtime. i.e if there was a update operation in mysql then I should do the same operation in ES. I researched mysql binLog which can reflect any changes from mysql. But I have to parse binLog to ES syntax, I think it's really painful. Thanks! (the same case with Solr)

推荐答案

有一个现有项目可以获取您的 binlog，将其转换并将其发送到 Elasticsearch，您可以在以下位置查看:https://github.com/siddontang/go-mysql-elasticsearch

There is an existing project which takes your binlog, transforms it and ships it to Elasticsearch, You can check it out at: https://github.com/siddontang/go-mysql-elasticsearch

另一个是这个:https://github.com/noplay/python-mysql-复制.

但是请注意，无论您选择哪种方式，在索引 binlog 之前预先创建索引和映射都是一个好习惯.这让您可以更好地控制自己的数据.

Note, however, that whichever you pick, it's a good practice to pre-create your index and mappings before indexing your binlog. That gives you more control over your data.

更新:

这是关于该主题的另一篇有趣的博客文章:如何使用 Logstash 保持 Elasticsearch 与关系数据库同步

Here is another interesting blog article on the subject: How to keep Elasticsearch synchronized with a relational database using Logstash

这篇关于如何将mysql数据实时迁移到ElasticSearch的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持编程学习网！