create account

Hivesearcher vs Hivemind Search by good-karma

View this thread on: hive.blogpeakd.comecency.com
· @good-karma · (edited)
$87.99
Hivesearcher vs Hivemind Search
In this post, I would like to talk about Elasticsearch and Postgresql search and their differences. 

As you know, we have Elasticsearch implemented on Hive which is being [rebranded as Hivesearcher](https://esteem.app/hive-139531/@good-karma/from-esteem-search-to-hivesearcher) (previously Esteem Search). System has been running for almost 2 years now without a glitch. As of this writing, there is no [Hivemind](https://gitlab.syncad.com/hive/hivemind) or Postgresql based search engine on Hive yet, so this comparison is theoretical based on our limited tests/knowledge.

![Hivesearcher_vs_Hivemind_search](https://images.ecency.com/DQmPeWLsj6qQwCUhQf251Y7wHpnC77u3pu6FPx6b59AhYPE/path50.png)


**Elasticsearch** is based on Apache Lucene as a core, adds REST API and few other improvements. Comparison should really be Lucene vs Postgresql.

Lucene is built by Apache just for one purpose, to make scalable, high-performance indexing and full-featured text search engine.

**Postgresql** is robust, relational database with SQL compliance and focuses on extensibility.

You can also [check this page to see their differences](https://db-engines.com/en/system/Elasticsearch%3BPostgreSQL). As you can see, they both built for different reasons and for different goals.

# When choose Postgresql 

We have been using Postgresql on our backend way before Hivemind came to live. And were glad to see Hivemind being written in Python and Postgres which is exactly what we use on our backend stuff for Esteem/Ecency. So we use Postgres for almost everything but not for search. We considered Postgres when building search engine because our backend was in Postgres and Hivemind was also using Postgres, so you would think it would make sense to just use same technology. After careful consideration and long term goals, we choose to go with the best search engine tech which is Elasticsearch. However, if you are web or mobile developer, consider using Postgres, it is one of the best database engine out there for relational data. You won't regret it because of opensource, community size and tools, extensions, it is growing.

# When choose Elasticsearch

Elasticsearch can also be used as datasource/database but it is not designed for that purpose and you will fail miserably if you use it as datasource. What Elasticsearch does best is building search engine and index large dataset, analyze queries. So perfect for large and long living content Hive produces.
It is #1 search engine for a reason.

Postgres Full-Text Search bring quite good improvements and can offer good alternative search. But it is very hard to scale and also I don't think, one can perform complex queries like `"esteem surfer" -"monthly digest" -giveaway author:good-karma tag:esteem,wallet type:post`. On the other hand, elasticsearch is easy to scale and you can perform any type of complex queries. 

Query above searches for posts from @good-karma with two tags together #esteem and #wallet having exact match of "Esteem Surfer" phrase but excluding posts that include "monthly digests" or "giveaways". You cannot simply do this with Postgresql, you will end up creating indexes for very different use cases and end up with large and clunky performant data.

Creating search based on Postgres would require separated tables and indexes, clean body field because it includes html + markdown combine cleaned post body + post title to search a term in multiple fields at the same time. 

Adding search into [Hivemind](https://gitlab.syncad.com/hive/hivemind) is another story altogether because then we might be affecting performance of regular API queries. Postgres uses shared_buffers which means it tries to complete each task in shared buffers. We all know hivemind servers are quite busy with reorganizing active blockchain data and if search is added into it, those search results will take some place in shared buffers. In result, it will reduce hivemind's performance and rpc nodes.

So Postgresql doesn't offer a magic wand nor does Elasticsearch. But tools are used best for their designed purpose.


#### Support Hivesearcher proposal to help us continue to improve and provide open API to Elasticsearch instances which is used by one of your favorite tools on blockchain hive.blog, peakd and esteem/ecency.


## Support proposal

- [Read details of Hivesearcher proposal](https://esteem.app/hive/@good-karma/open-search-engine-development-and-maintenance)
- [Vote for proposal using Hivesigner](https://hivesigner.com/sign/update-proposal-votes?proposal_ids=%5B114%5D&approve=true)
- HiveBlog Wallet: https://wallet.hive.blog/proposals
- PeakD: https://peakd.com/proposals/114
- HiveDao: https://hivedao.com/proposal/114
- Sourcecode: https://github.com/esteemapp/hive2elastic
πŸ‘  , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , and 552 others
πŸ‘Ž  ,
properties (23)
authorgood-karma
permlinkhivesearcher-vs-hivemind-search
categoryhive
json_metadata{"links":["https://esteem.app/hive-139531/@good-karma/from-esteem-search-to-hivesearcher","https://gitlab.syncad.com/hive/hivemind","https://db-engines.com/en/system/Elasticsearch%3BPostgreSQL","https://gitlab.syncad.com/hive/hivemind","https://esteem.app/hive/@good-karma/open-search-engine-development-and-maintenance","https://hivesigner.com/sign/update-proposal-votes?proposal_ids=%5B114%5D&approve=true","https://wallet.hive.blog/proposals","https://peakd.com/proposals/114","https://hivedao.com/proposal/114","https://github.com/esteemapp/hive2elastic"],"image":["https://images.ecency.com/DQmPeWLsj6qQwCUhQf251Y7wHpnC77u3pu6FPx6b59AhYPE/path50.png"],"users":["good-karma"],"tags":["hive","hivesearcher","hivemind","development","search","engine","elasticsearch","postgresql"],"app":"esteem/2.2.7-surfer","format":"markdown+html","community":"esteem.app"}
created2020-06-24 19:37:06
last_update2020-06-24 19:51:39
depth0
children21
last_payout2020-07-01 19:37:06
cashout_time1969-12-31 23:59:59
total_payout_value48.150 HBD
curator_payout_value39.843 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length4,766
author_reputation656,175,265,183,351
root_title"Hivesearcher vs Hivemind Search"
beneficiaries
0.
accountesteemapp
weight300
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id98,157,238
net_rshares228,811,559,698,540
author_curate_reward""
vote details (618)
@ammonite ·
I'm sure it is difficult to do search on the ever changing blockchain Database. For my personal use searching for old posts etc would be my main use I am sure it could be forgiven if search results had some sort of probability attached as the query refers to time sensitive info. It is really an interesting challenge. If I have not already voted proposal I will. 
properties (22)
authorammonite
permlinkre-good-karma-2020714t11502258z
categoryhive
json_metadata{"tags":["hive","hivesearcher","hivemind","development","search","engine","elasticsearch","postgresql"],"app":"esteem/2.2.5-mobile","format":"markdown+html","community":"hive-125125"}
created2020-07-14 10:50:27
last_update2020-07-14 10:50:27
depth1
children0
last_payout2020-07-21 10:50:27
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length364
author_reputation206,656,249,677,183
root_title"Hivesearcher vs Hivemind Search"
beneficiaries
0.
accountesteemapp
weight300
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id98,513,416
net_rshares0
@askmihai ·
Congrats for improving the platform!
What’s your opinion about the measures of our new normal?
properties (22)
authoraskmihai
permlinkqcjnko
categoryhive
json_metadata{"app":"hiveblog/0.1"}
created2020-06-26 17:36:24
last_update2020-06-26 17:36:24
depth1
children9
last_payout2020-07-03 17:36:24
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length94
author_reputation3,465,260,472,360
root_title"Hivesearcher vs Hivemind Search"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id98,192,538
net_rshares0
@good-karma ·
Thanks! If you could be more specific
πŸ‘  
properties (23)
authorgood-karma
permlinkre-askmihai-2020626t225318161z
categoryhive
json_metadata{"tags":["esteem"],"app":"esteem/2.2.7-surfer","format":"markdown+html","community":"esteem.app"}
created2020-06-26 19:53:18
last_update2020-06-26 19:53:18
depth2
children8
last_payout2020-07-03 19:53:18
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length37
author_reputation656,175,265,183,351
root_title"Hivesearcher vs Hivemind Search"
beneficiaries
0.
accountesteemapp
weight300
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id98,194,272
net_rshares0
author_curate_reward""
vote details (1)
@askmihai ·
Last year we didn't wear masks. 
- What's different this year?
- How many people ask questions?

Our leaders imposed some measures that reduce our liberties so 
- they have the burden of proof. 
- they have the responsibility to offer us the proof for their measures.

Without proof 1 theory is just 1 speculation.
# What's the proof?
By the way: so far i didn't receive 1 answer to this question. 
Everybody ignores it.
People are more or less creative in ignoring our reality.
i have a hunch: ignorance is not a virtue.
i ask this question in the idea that eventually somebody might enlighten me.
properties (22)
authoraskmihai
permlinkqcl0iq
categoryhive
json_metadata{"app":"hiveblog/0.1"}
created2020-06-27 11:13:39
last_update2020-06-27 11:13:39
depth3
children7
last_payout2020-07-04 11:13:39
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length598
author_reputation3,465,260,472,360
root_title"Hivesearcher vs Hivemind Search"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id98,204,362
net_rshares0
@attajuttjj ·
Hey why don't mirgethem
properties (22)
authorattajuttjj
permlinkre-good-karma-2020722t0418520z
categoryhive
json_metadata{"tags":["hive","hivesearcher","hivemind","development","search","engine","elasticsearch","postgresql"],"app":"esteem/2.2.5-mobile","format":"markdown+html","community":"hive-125125"}
created2020-07-21 19:41:15
last_update2020-07-21 19:41:15
depth1
children0
last_payout2020-07-28 19:41:15
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length23
author_reputation1,450,650,803,206
root_title"Hivesearcher vs Hivemind Search"
beneficiaries
0.
accountesteemapp
weight300
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id98,645,952
net_rshares0
@complaints-6 · (edited)
hivesearcher-vs-hivemind-search-complaints-6
**Dear member of HIVE

**Supplement 23 June - The witness @pfunk can't help but flag the account. So all spam messages are now automatically sent from a server. Now there is also no possibility not to unify. The text will be reformulated and every new member on HIE will be warned to continue publishing here. Also I offer every user a free account on Stemm including a delegation. Thank you @pfunk that it has come so far**  

**Supplement June 19th - Some whales had asked me to stop with all the messages as they had written to @pfunk to tell him to support me and stop destroying the account. I have stuck to it and stopped. He has now sent more downvotes and from now on the goal is to comment on every post that is published about HIVE. We have about 300 HIVE accounts, which can be used alternately and will create new accounts daily. We will program different scripts, because every hour a post will be published in the upper 100 days. These scripts will be available for every user of HIVE if they want to use them.**

I apologize for the automated comments. But we will not get another opportunity to point out our problem. 

** For the time being this message will only be published for one hour, if the proceedings against us are stopped and the donations are replaced, we will stop this. Otherwise, this message will be placed below every post that is published about HIVE through hundreds of accounts.**

Are you sure you really want to post here?

For example, we have a small account here that belongs to a social club in Germany. Donations were collected here, and every month a few dollars were collected.

But since ONE person - the witness at HIVE called @Pfunk (a witness is actually someone who is supposed to represent the members) has problems with the founder of the social organisation, he makes sure that all donations are deducted from the small account. Others who also support the account are also punished, and all income is also deducted.

If you publish here, make sure that you only publish what is of benefit to the owners of the assets on HIVE. You must never express your own opinion or criticize someone who is a witness or owner of power.

You will soon find out that you only earn a few cents here. Others, however, sometimes enrich themselves extremely. Look around, you'll soon find that some people get 2 or even 3-digit $ amounts for small contributions, but you don't even get 50 cents.

Now that the account has basically become useless to us, we have made it our mission to inform all new members about what to expect here in the block chain.

There is no way to fight against the rulers of HIVE. You can only inform others and especially new members.

If you want to see how they take away donations for the association, you can see this here https://beeme.icu/?account=nesomi

If you ever want to find out what this is all about, you can do that:
https://peakd.com/deutsch/@nesomi/ve-the-blockchain-of-coldhearts-and-egomaniacs-ve-the-blockchain-of-coldhearts-and-egomaniacs-ve-the-blockchain-of-coldhearts-and-egomaniacs

and

https://peakd.com/deutsch/@nesomi/abuse of power and love-of-the-ship-witnesses radio or abuse of power and love-of-the-ship-witnesses radio

As soon as the devaluations on our account are stopped and the destroyed donations are replaced, these comments will be stopped.

Should this action not work, we will comment less than 1000 newly published articles with this text every day.

Unfortunately we have no other possibility to take action against such behaviour. Should you really stay here, you will soon find that there is no other way to fight injustice than with many messages.

To anyone who has problems with a witness about HIVE, I am happy to give the program, which automatically distributes hundreds or even thousands of comments.

**We will delete this comment if you wish. However, this may take a few hours.
Add a comment "Delete comment "**

# In the last 10 hours, the downswings have been over $100 Think if love, not hate, were to decide for some how much good you could do.

![](https://i.imgur.com/qbMBLwL.png)


![](https://i.imgur.com/sleOd2c.png)


πŸ‘Ž  , , ,
properties (23)
authorcomplaints-6
permlinkhivesearcher-vs-hivemind-search-complaints-6
categoryhive
json_metadata{}
created2020-06-24 19:37:21
last_update2020-06-24 19:51:48
depth1
children1
last_payout2020-07-01 19:37:21
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length4,203
author_reputation-845,425,465,245
root_title"Hivesearcher vs Hivemind Search"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id98,157,245
net_rshares-26,271,630,117,115
author_curate_reward""
vote details (4)
@atma.love · (edited)
@complaints-6 I think your actions with this way if commenting will be destructive. I think it will work better if you are constructive. I in-courage you to focus your energy on improving governance. With a real time Democrity as governance the Hive population can come to an equitable concencus (not stake based) about many issues such as distribution of pool for up and downvotes not being (or only partly being) dependant on HP. Same for witness voting. There is a model, The Matrix-8 Solution, for such a system which could enable this. Please investigate and support. Namaste
https://peakd.com/hive-153630/@atma.love/matrix-8-questions-and-answers
properties (22)
authoratma.love
permlinkre-complaints-6-2020716t132852718z
categoryhive
json_metadata{"tags":[],"app":"esteem/2.2.5-mobile","format":"markdown+html","community":"hive-125125","links":["https://peakd.com/hive-153630/@atma.love/matrix-8-questions-and-answers"],"users":["complaints-6"]}
created2020-07-16 10:28:54
last_update2020-07-16 10:29:36
depth2
children0
last_payout2020-07-23 10:28:54
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length652
author_reputation144,994,480,690,778
root_title"Hivesearcher vs Hivemind Search"
beneficiaries
0.
accountesteemapp
weight300
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id98,549,715
net_rshares0
@hivebuzz ·
Congratulations @good-karma! You received a personal badge!

<table><tr><td>https://images.hive.blog/70x70/http://hivebuzz.me/badges/birthday-4.png</td><td>Happy Hive Birthday! You are on the Hive blockchain for 4 years!</td></tr></table>

<sub>_You can view [your badges on your board](https://hivebuzz.me/@good-karma) And compare to others on the [Ranking](https://hivebuzz.me/ranking)_</sub>


###### Support the HiveBuzz project. [Vote](https://hivesigner.com/sign/update_proposal_votes?proposal_ids=%5B%22109%22%5D&approve=true) for [our proposal](https://peakd.com/me/proposals/109)!
properties (22)
authorhivebuzz
permlinkhivebuzz-notify-good-karma-20200707t084644000z
categoryhive
json_metadata{"image":["http://hivebuzz.me/notify.t6.png"]}
created2020-07-07 08:46:45
last_update2020-07-07 08:46:45
depth1
children0
last_payout2020-07-14 08:46:45
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length589
author_reputation368,110,447,222,610
root_title"Hivesearcher vs Hivemind Search"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id98,386,379
net_rshares0
@iliyan90 ·
@good-karma , @esteemapp

I just noticed that the app is now showing some of the notifications like recent comments and replies.
I’ve try to change the saver and still the same.


---
---
![106503807_3081346178567603_7135964424781606109_n.jpg](https://files.peakd.com/file/peakd-hive/iliyan90/DFhAk2Xh-106503807_3081346178567603_7135964424781606109_n.jpg)
properties (22)
authoriliyan90
permlinkre-good-karma-2020628t104014397z
categoryhive
json_metadata{"tags":["hive","hivesearcher","hivemind","development","search","engine","elasticsearch","postgresql"],"app":"esteem/2.2.5-mobile","format":"markdown+html","community":"hive-125125"}
created2020-06-28 07:40:15
last_update2020-06-28 07:40:15
depth1
children0
last_payout2020-07-05 07:40:15
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length355
author_reputation139,403,337,868,520
root_title"Hivesearcher vs Hivemind Search"
beneficiaries
0.
accountesteemapp
weight300
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id98,218,817
net_rshares0
@ptaku ·
@tipu curate
properties (22)
authorptaku
permlinkqcg40p
categoryhive
json_metadata{"users":["tipu"],"app":"hiveblog/0.1"}
created2020-06-24 19:41:12
last_update2020-06-24 19:41:12
depth1
children2
last_payout2020-07-01 19:41:12
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length12
author_reputation176,884,771,885,848
root_title"Hivesearcher vs Hivemind Search"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id98,157,298
net_rshares0
@sergio4563 ·
I have made $16497 in one month by working from home. When I lost my office job 3 month ago, I was very upset and an unsuccessful try for a job hunt I was found this online job. and now I am able to earn thousands from home. Everybody can do this job and earn more dollars online by follow this link.
http://www.jobnews3.com
properties (22)
authorsergio4563
permlinkqchrw2
categoryhive
json_metadata{"links":["http://www.jobnews3.com"],"app":"hiveblog/0.1"}
created2020-06-25 17:13:39
last_update2020-06-25 17:13:39
depth2
children0
last_payout2020-07-02 17:13:39
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length324
author_reputation-41,327,018,836
root_title"Hivesearcher vs Hivemind Search"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id98,173,606
net_rshares0
@tipu ·
<a href="https://tipu.online/hive_curator?ptaku" target="_blank">Upvoted  &#128076;</a> (Mana: 0/18)
properties (22)
authortipu
permlinkre-qcg40p-20200624t194124
categoryhive
json_metadata""
created2020-06-24 19:41:24
last_update2020-06-24 19:41:24
depth2
children0
last_payout2020-07-01 19:41:24
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length100
author_reputation55,208,603,027,120
root_title"Hivesearcher vs Hivemind Search"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id98,157,306
net_rshares0
@shorna26 ·
Oh yea that's great improve.
properties (22)
authorshorna26
permlinkre-good-karma-2020715t23360253z
categoryhive
json_metadata{"tags":["hive","hivesearcher","hivemind","development","search","engine","elasticsearch","postgresql"],"app":"esteem/2.2.5-mobile","format":"markdown+html","community":"hive-125125"}
created2020-07-15 17:36:00
last_update2020-07-15 17:36:00
depth1
children0
last_payout2020-07-22 17:36:00
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length28
author_reputation1,097,701,016,431
root_title"Hivesearcher vs Hivemind Search"
beneficiaries
0.
accountesteemapp
weight300
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id98,537,470
net_rshares0
@stefano.massari ·
Thanks for sharing
properties (22)
authorstefano.massari
permlinkre-good-karma-2020628t12241281z
categoryhive
json_metadata{"tags":["hive","hivesearcher","hivemind","development","search","engine","elasticsearch","postgresql"],"app":"esteem/2.2.5-mobile","format":"markdown+html","community":"hive-125125"}
created2020-06-28 10:02:42
last_update2020-06-28 10:02:42
depth1
children0
last_payout2020-07-05 10:02:42
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length18
author_reputation175,392,049,166,413
root_title"Hivesearcher vs Hivemind Search"
beneficiaries
0.
accountesteemapp
weight300
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id98,220,500
net_rshares0