create account

HAF application development - progress report by mahdiyari

View this thread on: hive.blog |  peakd.com |  ecency.com
hive-139531 · @mahdiyari ·
$203.22
HAF application development - progress report
Progress on HafSQL:

I have been developing views for all the operations including the virtual operations. That is done now and the result is 93 views for all the operations.

I did play with index creation for a while and after some testing on a fully synced haf node, ended up with 36 total indexes (partial indexes).

It takes around 3.5 hours for indexes to be created on i9-13900 with 12 parallel workers. This test is with ZFS+LZ4 on raid 0 NVMe setup.

Total storage usage is only by indexes and it is 185GB (non-compressed). 128GB of this space is used by indexes of `custom` and `custom_json` operations. This enables retrieving `custom_json` operations by `id` and by the username from `required_auths` or `required_posting_auths` parameters. I figured this can be a very useful feature for our apps when this gets implemented on the public RPC nodes.

The rest of the development will probably be making the live sync part of the application to track things like delegations.

The current code works as it is and is under the tag v0.0.1. Readme should be good enough to set everything up.

I did try my best to cover all the operations with indexes. Feedback is very welcome.

Code: https://gitlab.com/mahdiyari/hafsql
Proposal: [DHF proposal - Hive related development](https://peakd.com/hive-139531/@mahdiyari/dhf-proposal-hive-related-development)

<center>![cat-gf897f4638_640.jpg](https://files.peakd.com/file/peakd-hive/mahdiyari/23xp6YsMDLqUrrDKCuwHMXiWT8BCbn38G6YfPANYHBPu8mGuwFquQijHAX9XD9cbsHxzZ.jpg)</center>
πŸ‘  , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , and 595 others
πŸ‘Ž  , , ,
properties (23)
authormahdiyari
permlinkhaf-application-development-progress-report
categoryhive-139531
json_metadata{"app":"peakd/2023.4.2","format":"markdown","tags":["hivedev","dev","hive","haf","hafsql"],"users":["mahdiyari"],"image":["https://files.peakd.com/file/peakd-hive/mahdiyari/23xp6YsMDLqUrrDKCuwHMXiWT8BCbn38G6YfPANYHBPu8mGuwFquQijHAX9XD9cbsHxzZ.jpg"]}
created2023-05-07 21:05:36
last_update2023-05-07 21:05:36
depth0
children13
last_payout2023-05-14 21:05:36
cashout_time1969-12-31 23:59:59
total_payout_value101.668 HBD
curator_payout_value101.554 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length1,530
author_reputation199,858,416,089,067
root_title"HAF application development - progress report"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id123,253,946
net_rshares416,157,107,154,904
author_curate_reward""
vote details (663)
@alex500 · 
The cat looks adorable.  There is a lot of magical charm in his two eyes.  Thank you for such a beautiful post about cats.❀️❀️
properties (22)
authoralex500
permlinkre-mahdiyari-2023512t18282937z
categoryhive-139531
json_metadata{"tags":["hivedev","dev","hive","haf","hafsql"],"app":"ecency/3.0.31-vision","format":"markdown+html"}
created2023-05-12 12:28:30
last_update2023-05-12 12:28:30
depth1
children0
last_payout2023-05-19 12:28:30
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length126
author_reputation2,547,891,388
root_title"HAF application development - progress report"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id123,391,483
net_rshares0
@bittujos · 
Kitty
properties (22)
authorbittujos
permlinkre-mahdiyari-202358t233055794z
categoryhive-139531
json_metadata{"tags":["hivedev","dev","hive","haf","hafsql"],"app":"ecency/3.0.31-vision","format":"markdown+html"}
created2023-05-08 18:00:57
last_update2023-05-08 18:00:57
depth1
children0
last_payout2023-05-15 18:00:57
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length5
author_reputation1,432,513,176,124
root_title"HAF application development - progress report"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id123,281,717
net_rshares0
@ibbtammy ·
$0.36
While some of us don't understand anything you pointed out in your post we are genuinely charmed by your cat and good job with your Development thingyπŸ˜„
πŸ‘  
properties (23)
authoribbtammy
permlinkre-mahdiyari-rucl2r
categoryhive-139531
json_metadata{"tags":["hive-139531"],"app":"peakd/2023.4.2"}
created2023-05-08 16:12:39
last_update2023-05-08 16:12:39
depth1
children0
last_payout2023-05-15 16:12:39
cashout_time1969-12-31 23:59:59
total_payout_value0.179 HBD
curator_payout_value0.179 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length151
author_reputation224,799,883,985,383
root_title"HAF application development - progress report"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id123,279,067
net_rshares738,530,195,198
author_curate_reward""
vote details (1)
@itwithsm · 
Keep doing the positive work, by the way the cat is so cute @mahdiyari
properties (22)
authoritwithsm
permlinkrucyrd
categoryhive-139531
json_metadata{"users":["mahdiyari"],"app":"hiveblog/0.1"}
created2023-05-08 21:09:15
last_update2023-05-08 21:09:15
depth1
children0
last_payout2023-05-15 21:09:15
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length70
author_reputation200,131,032,351,103
root_title"HAF application development - progress report"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id123,286,994
net_rshares0
@joyce152218 · 
so cute kitty cat❀️😘
properties (22)
authorjoyce152218
permlinkre-mahdiyari-202385t3147359z
categoryhive-139531
json_metadata{"tags":["hivedev","dev","hive","haf","hafsql"],"app":"ecency/3.0.42-mobile","format":"markdown+html"}
created2023-08-04 19:14:06
last_update2023-08-04 19:14:06
depth1
children0
last_payout2023-08-11 19:14:06
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length20
author_reputation4,909,762,641,832
root_title"HAF application development - progress report"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id125,941,935
net_rshares0
@liasunshine ·
$0.40
The cat is very cute. I love cats
thanks for sharing
πŸ‘  
properties (23)
authorliasunshine
permlinkre-mahdiyari-202358t141726527z
categoryhive-139531
json_metadata{"tags":["hivedev","dev","hive","haf","hafsql"],"app":"ecency/3.0.31-vision","format":"markdown+html"}
created2023-05-08 07:17:27
last_update2023-05-08 07:17:27
depth1
children0
last_payout2023-05-15 07:17:27
cashout_time1969-12-31 23:59:59
total_payout_value0.202 HBD
curator_payout_value0.202 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length52
author_reputation2,178,023,042,705
root_title"HAF application development - progress report"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id123,266,658
net_rshares843,730,423,742
author_curate_reward""
vote details (1)
@manniman ·
$0.42
Be gentle with the kitty
πŸ‘  
properties (23)
authormanniman
permlinkre-mahdiyari-ruba91
categoryhive-139531
json_metadata{"tags":["hive-139531"],"app":"peakd/2023.4.2"}
created2023-05-07 23:22:15
last_update2023-05-07 23:22:15
depth1
children0
last_payout2023-05-14 23:22:15
cashout_time1969-12-31 23:59:59
total_payout_value0.208 HBD
curator_payout_value0.208 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length24
author_reputation77,790,724,868,389
root_title"HAF application development - progress report"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id123,256,935
net_rshares850,528,521,222
author_curate_reward""
vote details (1)
@mickiewicz · (edited)
$0.58
The HAF documentation says (https://gitlab.syncad.com/hive/haf/-/blob/develop/src/hive_fork_manager/Readme.md#important-notice-about-irreversible-data):
```
Important notice about irreversible data
⚠ Although reversible and irreversible block tables are directly visible to apps, these tables should not be queried directly.
```
As I see in the code all indexes are created on irreversible blocks data tables. Looking at the views they also are created on the irreversible data. Are You sure that new indexes are required additionally to those that are already present in HAF ? Without using views incorporated in the HAF You won't be able to present together irreversible and reversible data (live data). Please let us know if HAF is too slow for Your application, You may add a new issue. Maybe we can incorporate some changes directly inside HAF.
πŸ‘  , ,
properties (23)
authormickiewicz
permlinkruccek
categoryhive-139531
json_metadata{"links":["https://gitlab.syncad.com/hive/haf/-/blob/develop/src/hive_fork_manager/Readme.md#important-notice-about-irreversible-data"],"app":"hiveblog/0.1"}
created2023-05-08 13:06:21
last_update2023-05-08 13:07:30
depth1
children1
last_payout2023-05-15 13:06:21
cashout_time1969-12-31 23:59:59
total_payout_value0.291 HBD
curator_payout_value0.291 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length849
author_reputation20,569,771,333,459
root_title"HAF application development - progress report"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id123,274,306
net_rshares1,205,064,048,793
author_curate_reward""
vote details (3)
@mahdiyari · 
Reversible data won't matter in this case. If it was necessary we can add them to the views and it's not a big deal. Indexes are not necessary on the reversible data though as the data is too small to be worth creating an index for it.

The created indexes are necessary for this app but I don't think they are necessary for the haf itself since they won't be necessary outside this use case.
IMHO haf should stay as generalized as possible and don't put the burden of every single app on the HAF node operators unless they choose to run that app.
properties (22)
authormahdiyari
permlinkrucfam
categoryhive-139531
json_metadata{"app":"hiveblog/0.1"}
created2023-05-08 14:08:48
last_update2023-05-08 14:08:48
depth2
children0
last_payout2023-05-15 14:08:48
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length547
author_reputation199,858,416,089,067
root_title"HAF application development - progress report"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id123,275,918
net_rshares0
@nonameslefttouse · 
I should probably say something about the cat.
properties (22)
authornonameslefttouse
permlinkre-mahdiyari-rucp7h
categoryhive-139531
json_metadata{"tags":["hive-139531"],"app":"peakd/2023.4.2"}
created2023-05-08 17:42:54
last_update2023-05-08 17:42:54
depth1
children0
last_payout2023-05-15 17:42:54
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length46
author_reputation593,361,688,458,528
root_title"HAF application development - progress report"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id123,281,182
net_rshares0
@zaibkang ·
$0.42
It's a development related post but you used a picture of cat as thumbnail it's interesting anyway thanks for sharing this technical information great work Sr. Keep developing hive
πŸ‘  
properties (23)
authorzaibkang
permlinkre-mahdiyari-202358t122515451z
categoryhive-139531
json_metadata{"tags":["hive-139531","hivedev","dev","hive","haf","hafsql"],"app":"ecency/3.0.39-mobile","format":"markdown+html"}
created2023-05-08 07:25:18
last_update2023-05-08 07:25:18
depth1
children2
last_payout2023-05-15 07:25:18
cashout_time1969-12-31 23:59:59
total_payout_value0.210 HBD
curator_payout_value0.210 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length181
author_reputation55,291,837,326,050
root_title"HAF application development - progress report"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id123,266,814
net_rshares878,852,261,041
author_curate_reward""
vote details (1)
@mahdiyari · 
Cat is the clickbait
properties (22)
authormahdiyari
permlinkre-zaibkang-rubza8
categoryhive-139531
json_metadata{"tags":["hive-139531"],"app":"peakd/2023.4.2"}
created2023-05-08 08:22:57
last_update2023-05-08 08:22:57
depth2
children1
last_payout2023-05-15 08:22:57
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length20
author_reputation199,858,416,089,067
root_title"HAF application development - progress report"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id123,267,804
net_rshares0
@zaibkang · 
It's totally fine Sr.
properties (22)
authorzaibkang
permlinkre-mahdiyari-202358t155615648z
categoryhive-139531
json_metadata{"tags":["hive-139531"],"app":"ecency/3.0.39-mobile","format":"markdown+html"}
created2023-05-08 10:56:18
last_update2023-05-08 10:56:18
depth3
children0
last_payout2023-05-15 10:56:18
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length22
author_reputation55,291,837,326,050
root_title"HAF application development - progress report"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id123,270,858
net_rshares0
Help/Feedback