Microservice idea by imwatsi

View this thread on: hive.blog | peakd.com | ecency.com

hive-139531 · @imwatsi · Oct 9 '20

$9.17

Microservice idea

During development of Native Ads, I realized that a lot of dApps might need to have access to block data, but might not necessarily be in a position to write code for block streaming from scratch.

So I have decided to share the block streaming aspect of Native Ads, as a standalone microservice in two forms:

- a **module** that dApps can install as part of their backend to retrieve customizable data sets from blocks (e.g. custom_json ops for their apps only)
- a **public node**, that dApps can interact with and set parameters on the kind of data they want to keep track of

As a microservice, it will have more features and customizations to cater to a wide range of use cases.

One use case for a **public node** would be a simple app for example, that uses a few custom_json ops to provide data input for its users. It can simply subscribe to those few custom_json ops without having to write code for block streaming and parsing and without running a node connected to the blockchain.

One advantage of using the **module** implementation is that if your're making a dApp, you just host the module as part of your backend and have access to it via a local API, much like you would the PostgreSQL service. So you will have streams of block data available to you, while the module handles all the streaming and parsing logic.

*I think it will lower the barrier to entry for new dApps.*

What are your thoughts on this kind of microservice?

👍 therealwolf, tipu, ausbitbank, smartsteem, roomservice, holger80, fw206, bryan-imhoff, ezzy, exyle, tombstone, emrebeyler, dcityrewards, someguy123, gerber, netaterra, borislavzlatanov, engrave, louisthomas, alphacore, smartvote, themightyvolcano, markkujantunen, epicdice, likwid, sam99, dlike, steem.leo, nealmcspadden, payroll, dune69, followjohngalt, freddio, daan, steem.services, accelerator, hiveonboard, kryptogames, mfblack, mice-k, felander, citizensmith, roleerob, hivecur, investegg, polish.hive, maxuvd, thelogicaldude, dalz, caladan, alenox, bartheek, elikast, purefood, promobot, smolalit, fullnodeupdate, asteroids, iansart, miketr, racibo, happy-soul, steemitboard, shimozurdo, and 56 others

`author`	imwatsi
`permlink`	microservice-idea
`category`	hive-139531
`json_metadata`	{"tags":["hive","idea","development","microservice"],"app":"hiveblog/0.1","format":"markdown"}
`created`	2020-10-09 19:45:00
`last_update`	2020-10-09 19:45:00
`depth`	0
`children`	10
`last_payout`	2020-10-16 19:45:00
`cashout_time`	1969-12-31 23:59:59
`total_payout_value`	4.742 HBD
`curator_payout_value`	4.430 HBD
`pending_payout_value`	0.000 HBD
`promoted`	0.000 HBD
`body_length`	1,448
`author_reputation`	131,386,362,730,495
`root_title`	"Microservice idea"
`beneficiaries`	`[]`
`max_accepted_payout`	1,000,000.000 HBD
`percent_hbd`	10,000
`post_id`	100,037,439
`net_rshares`	34,502,283,077,501
`author_curate_reward`	""

properties (23)vote details (120)

voter	rshares	pct
tombstone	325,175,531,573	15%
gerber	176,772,399,881	8.6%
daan	53,389,544,967	8%
ezzy	574,659,348,294	8.6%
ausbitbank	3,214,228,718,357	100%
smolalit	10,328,108,288	27%
exyle	361,847,302,575	8.6%
miketr	7,359,115,549	5%
bryan-imhoff	604,676,143,854	50%
netaterra	151,537,270,209	7.74%
someguy123	196,477,369,238	8.6%
marina007	3,721,514,998	36%
ebargains	555,589,277	2.15%
steemitboard	6,821,278,617	3%
markkujantunen	116,936,099,398	100%
borislavzlatanov	144,505,023,302	100%
louisthomas	132,530,457,054	100%
dune69	64,207,223,294	8.6%
iansart	7,541,948,069	5.33%
roomservice	1,854,329,905,194	37.5%
rycharde	1,199,708,360	10%
alphacore	132,082,273,130	6.89%
sam99	103,620,268,900	25.29%
shitsignals	3,739,306,085	8.6%
cifer	1,004,108,288	20%
igrinov	5,127,113,346	25%
felander	41,817,827,661	8.6%
tipu	3,676,434,508,124	9%
accelerator	51,250,856,937	5%
therealwolf	14,844,689,677,666	50%
roleerob	34,918,357,393	4.3%
cryptogainer	372,350,250	100%
investegg	32,039,974,084	1.41%
caladan	14,637,137,923	8.17%
emrebeyler	257,023,579,590	8.6%
smartsteem	2,990,775,156,382	50%
citizensmith	40,827,658,027	8.6%
fourfourfun	814,304,703	2.88%
bartheek	13,052,878,566	4.5%
nealmcspadden	75,900,300,206	8.6%
purefood	10,688,813,116	8.6%
investyourvote	1,973,774,393	6%
chronocrypto	2,715,008,427	8.6%
holger80	1,431,389,534,029	33%
unconditionalove	826,238,264	4.3%
happy-soul	6,959,785,515	4.5%
florenceboens	1,515,106,555	10%
pkocjan	3,869,581,322	6.88%
ladysalsa	1,632,346,977	8.6%
lordjames	532,791,058	25%
bestboom	4,548,471,102	8.6%
racibo	7,084,845,296	37.5%
freddio	56,191,740,578	15%
themightyvolcano	130,021,174,206	8.6%
payroll	68,702,625,032	2%
promobot	10,428,340,944	11.55%
fw206	996,208,558,506	100%
steem.services	52,039,898,720	2.15%
swisswitness	5,359,778,642	8.6%
longer	1,087,101,233	2.25%
suonghuynh	3,765,998,833	6%
donald.porter	2,510,674,588	18%
dalz	14,710,836,357	4.3%
smartvote	130,098,730,608	6%
dlike	103,089,792,563	8.6%
voxmortis	995,283,271	0.5%
fullnodeupdate	9,977,337,891	33%
engrave	134,019,715,662	6.88%
bobby.madagascar	2,833,595,056	2.15%
laissez-faire	68,542,856	100%
ldp	1,148,818,372	8.6%
berthold	2,511,899,933	6%
followjohngalt	62,213,522,285	8.6%
marki99	5,110,770,816	100%
jacuzzi	1,846,356,356	4.5%
permaculturedude	3,360,590,808	8.6%
elikast	12,307,537,479	100%
yff	19,840,311	10%
mia-cc	1,118,495,093	9%
abbenay	4,093,697,327	18.75%
thelogicaldude	22,707,700,691	10%
hungryharish	734,835,435	2.15%
kryptogames	45,953,179,143	9%
alenox	14,068,089,998	100%
mfblack	42,790,584,405	8.17%
epicdice	113,176,590,853	15%
threejay	2,523,420,077	4.3%
sm-silva	962,613,079	4.3%
likwid	112,223,989,534	11.55%
tinyhousecryptos	527,366,995	5%
firefuture	1,093,976,243	4.3%
steemindian	805,930,990	4.3%
cryptogambit	1,443,270,458	7.5%
shimozurdo	6,563,738,878	8.6%
milu-the-dog	4,065,503,018	8.6%
triplea.bot	3,072,118,731	8.6%
steem.leo	86,777,842,732	8.6%
asteroids	9,431,959,230	7.74%
botante	6,006,925,208	2.15%
maxuvd	22,787,550,216	6%
bcm	831,103,320	6.75%
bilpcoinrecords	1,334,075,308	25%
dappstats	3,626,229,395	15%
therealyme	585,731,463	7.74%
gerbo	57,769,734	8.6%
ribary	2,840,818,128	4.3%
bilpcoinbpc	1,499,304,578	16.5%
mice-k	42,686,077,748	8.6%
curamax	5,412,817,891	8.6%
masterlamps	0	100%
hivebuzz	5,512,729,251	2%
folklure	702,266,918	4.3%
polish.hive	24,080,446,754	8.6%
dcityrewards	205,876,260,943	8.6%
hiveonboard	50,717,994,760	37.5%
hivelist	741,171,228	0.86%
hivecur	33,356,503,491	8.6%
tinycurator	2,583,170,804	8.6%
discohedge	2,153,593,286	4.3%
bitcome	1,463,038,598	25%

@elikast · Oct 9 '20

@tipu curate

properties (22)

`author`	elikast
`permlink`	re-imwatsi-2020109t214844295z
`category`	hive-139531
`json_metadata`	{"tags":["hive","idea","development","microservice"],"app":"ecency/3.0.4-mobile","format":"markdown+html"}
`created`	2020-10-09 19:48:42
`last_update`	2020-10-09 19:48:42
`depth`	1
`children`	1
`last_payout`	2020-10-16 19:48:42
`cashout_time`	1969-12-31 23:59:59
`total_payout_value`	0.000 HBD
`curator_payout_value`	0.000 HBD
`pending_payout_value`	0.000 HBD
`promoted`	0.000 HBD
`body_length`	13
`author_reputation`	94,178,068,217,451
`root_title`	"Microservice idea"
`beneficiaries`	`[]`
`max_accepted_payout`	1,000,000.000 HBD
`percent_hbd`	10,000
`post_id`	100,037,481
`net_rshares`	0

@tipu · Oct 9 '20

<a href="https://tipu.online/hive_curator?elikast" target="_blank">Upvoted  &#128076;</a> (Mana: 18/27) <a href="https://peakd.com/hive/@reward.app/reward-app-quick-guide-updated" target="_blank">Liquid rewards</a>.

properties (22)

`author`	tipu
`permlink`	re-re-imwatsi-2020109t214844295z-20201009t194901
`category`	hive-139531
`json_metadata`	""
`created`	2020-10-09 19:49:03
`last_update`	2020-10-09 19:49:03
`depth`	2
`children`	0
`last_payout`	2020-10-16 19:49:03
`cashout_time`	1969-12-31 23:59:59
`total_payout_value`	0.000 HBD
`curator_payout_value`	0.000 HBD
`pending_payout_value`	0.000 HBD
`promoted`	0.000 HBD
`body_length`	215
`author_reputation`	55,949,689,035,458
`root_title`	"Microservice idea"
`beneficiaries`	`[]`
`max_accepted_payout`	1,000,000.000 HBD
`percent_hbd`	10,000
`post_id`	100,037,488
`net_rshares`	0

@emrebeyler · Oct 10 '20

might be helpful

https://github.com/inertia186/meeseeker

properties (22)

`author`	emrebeyler
`permlink`	re-imwatsi-qhzcqp
`category`	hive-139531
`json_metadata`	{"tags":["hive-139531"],"app":"peakd/2020.09.5"}
`created`	2020-10-10 09:52:03
`last_update`	2020-10-10 09:52:03
`depth`	1
`children`	1
`last_payout`	2020-10-17 09:52:03
`cashout_time`	1969-12-31 23:59:59
`total_payout_value`	0.000 HBD
`curator_payout_value`	0.000 HBD
`pending_payout_value`	0.000 HBD
`promoted`	0.000 HBD
`body_length`	58
`author_reputation`	448,535,049,068,622
`root_title`	"Microservice idea"
`beneficiaries`	`[]`
`max_accepted_payout`	1,000,000.000 HBD
`percent_hbd`	10,000
`post_id`	100,044,757
`net_rshares`	0

@imwatsi · Oct 10 '20

Thanks, I'll check it out

properties (22)

`author`	imwatsi
`permlink`	qhzncr
`category`	hive-139531
`json_metadata`	{"app":"hiveblog/0.1"}
`created`	2020-10-10 13:41:15
`last_update`	2020-10-10 13:41:15
`depth`	2
`children`	0
`last_payout`	2020-10-17 13:41:15
`cashout_time`	1969-12-31 23:59:59
`total_payout_value`	0.000 HBD
`curator_payout_value`	0.000 HBD
`pending_payout_value`	0.000 HBD
`promoted`	0.000 HBD
`body_length`	25
`author_reputation`	131,386,362,730,495
`root_title`	"Microservice idea"
`beneficiaries`	`[]`
`max_accepted_payout`	1,000,000.000 HBD
`percent_hbd`	10,000
`post_id`	100,047,229
`net_rshares`	0

@hivebuzz · Oct 11 '20

Congratulations @imwatsi! You received a personal badge!

<table><tr><td>https://images.hive.blog/70x70/http://hivebuzz.me/badges/birthday-4.png</td><td>Happy Hive Birthday! You are on the Hive blockchain for 4 years!</td></tr></table>

<sub>_You can view your badges on [your board](https://hivebuzz.me/@imwatsi) and compare yourself to others in the [Ranking](https://hivebuzz.me/ranking)_</sub>

👍 cifer

`author`	hivebuzz
`permlink`	hivebuzz-notify-imwatsi-20201011t222307000z
`category`	hive-139531
`json_metadata`	{"image":["http://hivebuzz.me/notify.t6.png"]}
`created`	2020-10-11 22:23:06
`last_update`	2020-10-11 22:23:06
`depth`	1
`children`	0
`last_payout`	2020-10-18 22:23:06
`cashout_time`	1969-12-31 23:59:59
`total_payout_value`	0.000 HBD
`curator_payout_value`	0.000 HBD
`pending_payout_value`	0.000 HBD
`promoted`	0.000 HBD
`body_length`	398
`author_reputation`	369,663,591,473,293
`root_title`	"Microservice idea"
`beneficiaries`	`[]`
`max_accepted_payout`	1,000,000.000 HBD
`percent_hbd`	10,000
`post_id`	100,068,388
`net_rshares`	1,034,548,933
`author_curate_reward`	""

properties (23)vote details (1)

voter	weight	wgt%	rshares	pct	time
cifer	0 B		1,034,548,933	20%

@marki99 · Oct 10 '20

> a public node, that dApps can interact with and set parameters on the kind of data they want to keep track of

How can parameters be set to do that without actual programming required? Also, if an app is broadcasting custom_jsons, would it be too hard to filter them out?

There are many things I still don't understand so maybe I am missing something.

properties (22)

`author`	marki99
`permlink`	re-imwatsi-qhz4sc
`category`	hive-139531
`json_metadata`	{"tags":["hive-139531"],"app":"peakd/2020.09.5"}
`created`	2020-10-10 07:00:12
`last_update`	2020-10-10 07:00:12
`depth`	1
`children`	1
`last_payout`	2020-10-17 07:00:12
`cashout_time`	1969-12-31 23:59:59
`total_payout_value`	0.000 HBD
`curator_payout_value`	0.000 HBD
`pending_payout_value`	0.000 HBD
`promoted`	0.000 HBD
`body_length`	355
`author_reputation`	11,400,723,818,181
`root_title`	"Microservice idea"
`beneficiaries`	`[]`
`max_accepted_payout`	1,000,000.000 HBD
`percent_hbd`	10,000
`post_id`	100,043,313
`net_rshares`	0

@imwatsi · Oct 10 '20

Currently, to get custom_json ops, you have to query the blockchain directly, by querying the hived nodes, on a per-block basis.

So to get a livestream of custom_json ops as they are being created you'd have to listen to every block that's being produced and filter out the custom_json ops you're interested, from all the other post and voting data.

So what I'm proposing is a public node, that one can get these ops from without querying for complete blocks. And an alternative module to install, that you can run locally and configure to store/retrieve only the custom_json ops you're interested in.

> How can parameters be set to do that without actual programming required?

The programming required would be less, because you won't have to write code to retrieve all the data sets you want, but just a few lines and API calls. The node will do most of the work behind-the-scenes.

👍 marki99

`author`	imwatsi
`permlink`	qhznst
`category`	hive-139531
`json_metadata`	{"app":"hiveblog/0.1"}
`created`	2020-10-10 13:50:57
`last_update`	2020-10-10 13:50:57
`depth`	2
`children`	0
`last_payout`	2020-10-17 13:50:57
`cashout_time`	1969-12-31 23:59:59
`total_payout_value`	0.000 HBD
`curator_payout_value`	0.000 HBD
`pending_payout_value`	0.000 HBD
`promoted`	0.000 HBD
`body_length`	887
`author_reputation`	131,386,362,730,495
`root_title`	"Microservice idea"
`beneficiaries`	`[]`
`max_accepted_payout`	1,000,000.000 HBD
`percent_hbd`	10,000
`post_id`	100,047,354
`net_rshares`	6,865,403,898
`author_curate_reward`	""

properties (23)vote details (1)

voter	weight	wgt%	rshares	pct	time
marki99	0 B		6,865,403,898	100%

@simplegame · Oct 9 '20

$0.02

I think this will be very helpful.

A lot of devs are scared off on trying to stream and parse the blockchain.

Making this a local API will speed up development.

👍 imwatsi

`author`	simplegame
`permlink`	re-imwatsi-qhyh6t
`category`	hive-139531
`json_metadata`	{"tags":["hive-139531"],"app":"peakd/2020.09.5"}
`created`	2020-10-09 22:30:30
`last_update`	2020-10-09 22:30:30
`depth`	1
`children`	0
`last_payout`	2020-10-16 22:30:30
`cashout_time`	1969-12-31 23:59:59
`total_payout_value`	0.012 HBD
`curator_payout_value`	0.012 HBD
`pending_payout_value`	0.000 HBD
`promoted`	0.000 HBD
`body_length`	164
`author_reputation`	134,726,886,324,158
`root_title`	"Microservice idea"
`beneficiaries`	`[]`
`max_accepted_payout`	1,000,000.000 HBD
`percent_hbd`	10,000
`post_id`	100,039,350
`net_rshares`	164,687,279,944
`author_curate_reward`	""

properties (23)vote details (1)

voter	weight	wgt%	rshares	pct	time
imwatsi	0 B		164,687,279,944	100%

@therealwolf · Oct 10 '20 (edited)

$0.65

One thing that Hive needs better support for is custom_json specific queries. The only way to get a custom_json is by the transaction_id, but what if I want to access them based on a field inside the custom_json or based on the id? You could index them all yourself (i.e. https://github.com/emre/hotdog), but IMO this creates the problem of centralizing this data (if your storage gets hacked, the data is invalid)

For example, even if I host a hived and hivemind node, I still can access other nodes for the same data and in case of important data, I could even access multiple at the same time and choose the data which is consistent across the majority.

So, if we could create a public node specifically for custom_jsons querying, that people could run similarly to hived and hivemind node, that would be amazing.

Since @embrebeyler worked on this problem sort of, maybe you two can work together on it. Would be much appreciated, and even something I would def. vote for via DHF.

---

Edit: This could also be useful https://github.com/inertia186/meeseeker

👍 ausbitbank, imwatsi, sultan-indo

`author`	therealwolf
`permlink`	re-imwatsi-qhz92y
`category`	hive-139531
`json_metadata`	{"tags":["hive-139531"],"app":"peakd/2020.09.5"}
`created`	2020-10-10 08:33:00
`last_update`	2020-10-10 10:22:24
`depth`	1
`children`	1
`last_payout`	2020-10-17 08:33:00
`cashout_time`	1969-12-31 23:59:59
`total_payout_value`	0.326 HBD
`curator_payout_value`	0.326 HBD
`pending_payout_value`	0.000 HBD
`promoted`	0.000 HBD
`body_length`	1,064
`author_reputation`	582,208,885,469,814
`root_title`	"Microservice idea"
`beneficiaries`	`[]`
`max_accepted_payout`	1,000,000.000 HBD
`percent_hbd`	10,000
`post_id`	100,044,126
`net_rshares`	3,323,079,978,546
`author_curate_reward`	""

properties (23)vote details (3)

voter	rshares	pct
ausbitbank	3,150,054,750,890	100%
imwatsi	161,400,542,407	100%
sultan-indo	11,624,685,249	100%

@imwatsi · Oct 10 '20

You point out some very good issues, that I hope a public node like this will solve. We could have the node index historical ops and since it's JSON, we could leverage PostgreSQL's in-built support for JSON data types to do queries key-value. I'll investigate the options.

I'll also get in touch with @emrebeyler.

properties (22)