create account

Daily Hive Engine Snapshots by primersion

View this thread on: hive.blogpeakd.comecency.com
@primersion
$24.49
Daily Hive Engine Snapshots
![wallp12.png](https://images.ecency.com/DQmRT6nLW4pqLqeR1NhGVy68HUhRMDEYCNUhFQ6R1SLT7sm/wallp12.png)

## The Why

Hive engine nodes read all transactions from the HIVE blockchain and when they discover a custom json operation with a special id (ssc-mainnet-hive), they process the transaction and add it to the hive engine sidechain. The hive engine sidechain itself is basically a big database storing all transactions and processed information, such as token balances and nft instances, inside a mongodb.
To create the hive engine sidechain completely from scratch we would have to start from the genesis block and process all transactions after that. This is a very lengthy process, because it includes millions of transactions.

That's where snapshots come into play. You simply create a snapshot of the database after block X was processed and the next time you only have to replay transactions starting from block X. The same principle applies to the HIVE blockchain itself.

So whenever something goes wrong or you want to setup a fresh node you can use the snapshot to set up your node faster.

## The What

I am running a hive engine witness node myself, which is why I need to recreate my node regularly. A few other witnesses are already providing snapshots of the hive engine sidechain regularly. I think more is better, so I decided to set up my own daily hive engine snapshots. I have integrated the snapshots into my HIVE monitor tool over here: https://primersion.com/he-snapshots

Every day at 11pm UTC a snapshot is created automatically, uploaded to https://transfer.sh and provided through my page afterwards:

![image.png](https://images.ecency.com/DQmbdCZh87hkktsYM1Sot7qBg8rDXqoWG4Do8HqDBLPxm3J/image.png)

You will find snapshots of the last 7 days (in a few days, because I only started 2 days ago) and you can download them directly by clicking on the link. After downloading the snapshot you can use it to restore the database of your hive engine node using `mongorestore --gzip --archive=<path to archive>`. Additionally you will find a checksum below the link, which you can compare with the checksum created when running `md5sum <path to archive>` on the downloaded file.

## The How

I have written a small script, which does all the work for me. All I need for it to function is a running and synced hive engine node. First I am stopping my node using `pm2 stop all`.
In a next step I am dumping the mongodb to a file including the current date and the last processed block:
```
now=`date +"%m-%d-%Y"`
block=`cat config.json | grep startHiveBlock | grep -Eo '[0-9]*'`
mongodump --db hsc --gzip --archive="hsc_${now}_b${block}.archive"
```
When the dump is finished I start the node back up using `pm2 start app.js --no-treekill --kill-timeout 10000 --no-autorestart`.
And transfer the dump to https://transfer.sh using `curl --upload-file "hsc_${now}_b${block}.archive" "https://transfer.sh/hsc_${now}_b${block}.archive"`.

I am running the script daily at 11pm UTC using a cronjob.

That's all the magic.

---

Please support me by voting for my HIVE / Hive-Engine witness:

[HIVE](https://vote.hive.uno/?witness=primersion) (using Hivesigner)
[Hive-Engine](https://votify.now.sh/primersion) (using Votify)
馃憤  , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , and 166 others
properties (23)
authorprimersion
permlinkdaily-hive-engine-snapshots
categoryhive-169321
json_metadata{"links":["https://primersion.com/he-snapshots","https://transfer.sh","https://transfer.sh","https://transfer.sh/hsc_","https://vote.hive.uno/?witness=primersion","https://votify.now.sh/primersion"],"image":["https://images.ecency.com/DQmRT6nLW4pqLqeR1NhGVy68HUhRMDEYCNUhFQ6R1SLT7sm/wallp12.png","https://images.ecency.com/DQmbdCZh87hkktsYM1Sot7qBg8rDXqoWG4Do8HqDBLPxm3J/image.png"],"tags":["hive-169321","community","dev","witness","witness-category","hive","tools","hive-engine"],"app":"ecency/3.0.17-vision","format":"markdown+html"}
created2021-06-09 14:36:06
last_update2021-06-09 14:36:06
depth0
children17
last_payout2021-06-16 14:36:06
cashout_time1969-12-31 23:59:59
total_payout_value12.294 HBD
curator_payout_value12.198 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length3,240
author_reputation91,109,041,081,432
root_title"Daily Hive Engine Snapshots"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id104,231,124
net_rshares38,864,515,311,964
author_curate_reward""
vote details (230)
@denisdenis
$0.02
Interesting ;))
馃憤  ,
properties (23)
authordenisdenis
permlinkre-primersion-2021614t85742680z
categoryhive-169321
json_metadata{"tags":["hive-169321","community","dev","witness","witness-category","hive","tools","hive-engine"],"app":"ecency/3.0.18-mobile","format":"markdown+html"}
created2021-06-14 06:57:45
last_update2021-06-14 06:57:45
depth1
children0
last_payout2021-06-21 06:57:45
cashout_time1969-12-31 23:59:59
total_payout_value0.012 HBD
curator_payout_value0.012 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length15
author_reputation75,539,149,214,246
root_title"Daily Hive Engine Snapshots"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id104,321,005
net_rshares88,131,768,116
author_curate_reward""
vote details (2)
@dragokazo
$0.08
I m not to good in nodes. But for hive I have to try some time.
馃憤  
properties (23)
authordragokazo
permlinkre-primersion-2021612t131252429z
categoryhive-169321
json_metadata{"tags":["hive-169321","community","dev","witness","witness-category","hive","tools","hive-engine"],"app":"ecency/3.0.18-mobile","format":"markdown+html"}
created2021-06-12 11:12:51
last_update2021-06-12 11:12:51
depth1
children10
last_payout2021-06-19 11:12:51
cashout_time1969-12-31 23:59:59
total_payout_value0.040 HBD
curator_payout_value0.040 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length63
author_reputation85,896,569,375,880
root_title"Daily Hive Engine Snapshots"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id104,285,479
net_rshares239,575,994,490
author_curate_reward""
vote details (1)
@primersion
Running a hive-engine witness node is not too difficult and there are plenty of tutorials :)
properties (22)
authorprimersion
permlinkre-dragokazo-2021612t13550289z
categoryhive-169321
json_metadata{"tags":["hive-169321","community","dev","witness","witness-category","hive","tools","hive-engine"],"app":"ecency/3.0.17-vision","format":"markdown+html"}
created2021-06-12 11:55:03
last_update2021-06-12 11:55:03
depth2
children9
last_payout2021-06-19 11:55:03
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length92
author_reputation91,109,041,081,432
root_title"Daily Hive Engine Snapshots"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id104,286,022
net_rshares0
@dragokazo
Do you have some 
properties (22)
authordragokazo
permlinkre-primersion-2021612t143112328z
categoryhive-169321
json_metadata{"tags":["hive-169321","community","dev","witness","witness-category","hive","tools","hive-engine"],"app":"ecency/3.0.18-mobile","format":"markdown+html"}
created2021-06-12 12:31:12
last_update2021-06-12 12:31:12
depth3
children8
last_payout2021-06-19 12:31:12
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length17
author_reputation85,896,569,375,880
root_title"Daily Hive Engine Snapshots"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id104,286,461
net_rshares0
@ecency
Points Boosting refund to @primersion! 馃 <br>Due to one of these reasons: <br>1. Post is not published via Ecency. <br>2. Post is already curated by our team. <br>3. Post is not curated by our curators within 24 hours. <br>4. Post might be too old post, try more recent content.<br>5. Author already received vote in last few hours, try again later.<br>Install Android: https://android.ecency.com, iOS: https://ios.ecency.com mobile app or desktop app for Windows, Mac, Linux: https://desktop.ecency.com<br>Learn more: https://ecency.com <br>Join our discord: https://discord.me/ecency
properties (22)
authorecency
permlinkre-2021614t184823303z
categoryhive-169321
json_metadata{"tags":["ecency"],"app":"ecency/3.0.16-welcome","format":"markdown+html"}
created2021-06-14 18:48:24
last_update2021-06-14 18:48:24
depth1
children0
last_payout2021-06-21 18:48:24
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length585
author_reputation628,552,533,129,734
root_title"Daily Hive Engine Snapshots"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id104,331,649
net_rshares0
@ecency
**Yay!** 馃<br>Your content has been **boosted with Ecency Points**, by @primersion. <br>Use Ecency daily to boost your growth on platform! <br><br><b>Support Ecency</b><br>[Vote for Proposal](https://hivesigner.com/sign/update-proposal-votes?proposal_ids=%5B141%5D&approve=true)<br>[Delegate HP and earn more](https://ecency.com/hive-125125/@ecency/daily-100-curation-rewards)
馃憤  
properties (23)
authorecency
permlinkre-202169t144117722z
categoryhive-169321
json_metadata{"tags":["ecency"],"app":"ecency/3.0.16-welcome","format":"markdown+html"}
created2021-06-09 14:41:18
last_update2021-06-09 14:41:18
depth1
children0
last_payout2021-06-16 14:41:18
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length376
author_reputation628,552,533,129,734
root_title"Daily Hive Engine Snapshots"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id104,231,205
net_rshares10,247,002,248
author_curate_reward""
vote details (1)
@hivebuzz
Congratulations @primersion! You have completed the following achievement on the Hive blockchain and have been rewarded with new badge(s) :

<table><tr><td><img src="https://images.hive.blog/60x70/http://hivebuzz.me/@primersion/upvotes.png?202106192301"></td><td>You distributed more than 38000 upvotes.<br>Your next target is to reach 39000 upvotes.</td></tr>
</table>

<sub>_You can view your badges on [your board](https://hivebuzz.me/@primersion) and compare yourself to others in the [Ranking](https://hivebuzz.me/ranking)_</sub>
<sub>_If you no longer want to receive notifications, reply to this comment with the word_ `STOP`</sub>

properties (22)
authorhivebuzz
permlinkhivebuzz-notify-primersion-20210619t232747000z
categoryhive-169321
json_metadata{"image":["http://hivebuzz.me/notify.t6.png"]}
created2021-06-19 23:27:45
last_update2021-06-19 23:27:45
depth1
children0
last_payout2021-06-26 23:27:45
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length640
author_reputation370,737,118,022,828
root_title"Daily Hive Engine Snapshots"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id104,436,159
net_rshares0
@rishi556
$0.15
Awesome job on this, more snapshots copies is always a good thing. Never knew transfer.sh was a thing, its extremely cool. 10 GB does seem to be the limit though, so going to need to change this quick enough.
馃憤  
properties (23)
authorrishi556
permlinkre-primersion-qug8cb
categoryhive-169321
json_metadata{"tags":["hive-169321"],"app":"peakd/2021.06.2"}
created2021-06-09 19:19:24
last_update2021-06-09 19:19:24
depth1
children1
last_payout2021-06-16 19:19:24
cashout_time1969-12-31 23:59:59
total_payout_value0.072 HBD
curator_payout_value0.073 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length208
author_reputation134,107,100,365,867
root_title"Daily Hive Engine Snapshots"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id104,235,500
net_rshares407,983,665,155
author_curate_reward""
vote details (1)
@primersion
Yeah it's a great alternative for pages like wetransfer. Let's see how long it takes until we reach the limit ;) I am already thinking about alternatives.
properties (22)
authorprimersion
permlinkre-rishi556-202169t231622538z
categoryhive-169321
json_metadata{"tags":["hive-169321"],"app":"ecency/3.0.17-vision","format":"markdown+html"}
created2021-06-09 21:16:21
last_update2021-06-09 21:16:21
depth2
children0
last_payout2021-06-16 21:16:21
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length154
author_reputation91,109,041,081,432
root_title"Daily Hive Engine Snapshots"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id104,237,370
net_rshares0