create account

Ingestron 0.0.2 released for every platform imaginable by faddat

View this thread on: hive.blogpeakd.comecency.com
· @faddat · (edited)
$37.81
Ingestron 0.0.2 released for every platform imaginable
![Screenshot from 2017-03-27 09-14-13.png](https://steemitimages.com/DQmcFo4qadVockxnUWv4rTwZJHvfQUu22RBPvgYvbBRz3fi/Screenshot%20from%202017-03-27%2009-14-13.png)

Ingestron is a tool for reading the steem blockchain really, really fast into a storm-formatted boltdb database.  The next step is to schlep on an API directly, which will of course have native support for swagger, and therefore libs for god knows how many languages and frameworks.  

[ingestron](github.com/faddat/ingestron)

Right now, it just reads and writes into the boltdb database.  

I'm hungry for feedback on ingestron.  It should make many a steem-project go much, much faster because it'll provide for the first time a fully native (eg: it does not depend on the one fully-developed API library, steem-js) Swagger API for steem.  At that point, building front ends should become the trivial task that the good lord intended it to be, eg: "just a front end."  (Today you cannot make "just a front end" for steem because it follows no known standards, even the ones it says it does)

[I'm banned from steem's github](https://steemit.com/steemit/@faddat/i-m-still-banned-from-steemit-s-github)
šŸ‘  , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , and 196 others
šŸ‘Ž  , , ,
properties (23)
authorfaddat
permlinkingestron-0-0-2-released-for-every-platform-imaginable
categorysteemit
json_metadata{"tags":["steemit","steem","opensource","fasterthanyourapi","betterthanyourapi"],"image":["https://steemitimages.com/DQmcFo4qadVockxnUWv4rTwZJHvfQUu22RBPvgYvbBRz3fi/Screenshot%20from%202017-03-27%2009-14-13.png"],"app":"steemit/0.1","format":"markdown","links":["github.com/faddat/ingestron","https://steemit.com/steemit/@faddat/i-m-still-banned-from-steemit-s-github"]}
created2017-03-27 02:17:30
last_update2017-03-27 04:10:42
depth0
children23
last_payout2017-04-27 11:50:36
cashout_time1969-12-31 23:59:59
total_payout_value31.431 HBD
curator_payout_value6.379 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length1,168
author_reputation36,581,868,473,026
root_title"Ingestron 0.0.2 released for every platform imaginable"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,858,077
net_rshares16,537,848,209,222
author_curate_reward""
vote details (264)
@alexpmorris ·
will it be streaming too so we don't have to keep polling the price feeds every few seconds for updates? lol
šŸ‘  ,
properties (23)
authoralexpmorris
permlinkre-faddat-ingestron-0-0-2-released-for-every-platform-imaginable-20170327t024827094z
categorysteemit
json_metadata{"tags":["steemit"],"app":"steemit/0.1"}
created2017-03-27 02:48:27
last_update2017-03-27 02:48:27
depth1
children14
last_payout2017-04-27 11:50:36
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length108
author_reputation32,063,874,290,523
root_title"Ingestron 0.0.2 released for every platform imaginable"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,858,245
net_rshares37,647,593,769
author_curate_reward""
vote details (2)
@faddat · (edited)
Maybe?  

Tell me more of what you'd like?

I haven't written a streaming API yet in go.  Do you know if libs exist for this?  

Put it in user-story format?
šŸ‘  
properties (23)
authorfaddat
permlinkre-alexpmorris-re-faddat-ingestron-0-0-2-released-for-every-platform-imaginable-20170327t024921390z
categorysteemit
json_metadata{"tags":["steemit"],"app":"steemit/0.1"}
created2017-03-27 02:49:21
last_update2017-03-27 02:49:51
depth2
children13
last_payout2017-04-27 11:50:36
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length157
author_reputation36,581,868,473,026
root_title"Ingestron 0.0.2 released for every platform imaginable"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,858,253
net_rshares3,934,374,094
author_curate_reward""
vote details (1)
@alexpmorris · (edited)
this is the way the current steem api works, which is highly inefficient.  Just "inspect" the market in Chrome and you'll see what I mean:

>SteemApi exec 5 get_following ( Array[4] )
app.56d5f0d….js:5 SteemApi exec 6 get_ticker ( Array[0] )
app.56d5f0d….js:5 SteemApi exec 5 get_following ( Array[4] )
app.56d5f0d….js:5 SteemApi exec 0 get_order_book ( Array[1] )
app.56d5f0d….js:5 SteemApi exec 6 get_trade_history ( Array[3] )
app.56d5f0d….js:5 SteemApi exec 6 get_ticker ( Array[0] )
app.56d5f0d….js:5 SteemApi exec 0 get_order_book ( Array[1] )
app.56d5f0d….js:5 SteemApi exec 6 get_trade_history ( Array[3] )
app.56d5f0d….js:5 SteemApi exec 6 get_ticker ( Array[0] )
app.56d5f0d….js:5 SteemApi exec 0 get_order_book ( Array[1] )
app.56d5f0d….js:5 SteemApi exec 6 get_trade_history ( Array[3] )
app.56d5f0d….js:5 SteemApi exec 6 get_ticker ( Array[0] )
app.56d5f0d….js:5 SteemApi exec 0 get_order_book ( Array[1] )
app.56d5f0d….js:5 SteemApi exec 6 get_trade_history ( Array[3] )
app.56d5f0d….js:5 SteemApi exec 6 get_ticker ( Array[0] )

The way it should be is more like a pub/sub, ie.  sub_order_book(10);   // subscribe to receive and update 10 top levels

Instead of repeatedly requesting updates, it feeds them to you automatically as they come in.  If it helps, here are a bunch of golang examples:

https://www.google.com/search?q=golang+pubsub+websocket

I guess the real question is can the current websocket implementations already do any of this natively, as opposed to building a new API that polls, which then in turn converts *that* to a pub/sub style API. (*ie. caveat: it's possible I don't fully understand all the features of the current API*)
šŸ‘  ,
properties (23)
authoralexpmorris
permlinkre-faddat-re-alexpmorris-re-faddat-ingestron-0-0-2-released-for-every-platform-imaginable-20170327t030029669z
categorysteemit
json_metadata{"tags":["steemit"],"links":["https://www.google.com/search?q=golang+pubsub+websocket"],"app":"steemit/0.1"}
created2017-03-27 03:00:30
last_update2017-03-27 03:05:54
depth3
children12
last_payout2017-04-27 11:50:36
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length1,667
author_reputation32,063,874,290,523
root_title"Ingestron 0.0.2 released for every platform imaginable"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,858,315
net_rshares37,647,593,769
author_curate_reward""
vote details (2)
@alexpmorris · (edited)
$0.02
Maybe I was a bit tired last night, but after looking at this again, I think I may see part of the problem, starting from when you wrote:

>Does it need to use a websocket or can pubsub be achieved by other means?

For example, you talk about "saving from latency" and a "fully native" API.  From the added context, respectfully, I think those things may mean something very different for me (and for the steemit devs) than they do for you.  For example, if you're concerned about latency, clean code, etc, why would you be thinking of adding extra layers of external dependencies and complexity such as grpc?  

And if you're already doing pull requests on their code while thinking on a completely different level than they are, well, put it this way.  I wouldn't be comfortable touching their code without being pretty darn sure I know it inside and out (short of some very obvious bug fixes). 

From your own code, you see that you are already talking to steemd with a websocket!  From my perspective, the steemd is already easily accessible from **ANY** language (and over the net via wss://).  It uses json-rpc.  It doesn't get much simpler than that!  It would be nice, though, to be able to do something like:

>'{"jsonrpc": "2.0", "method": "call", "params": ["database_api", "sub_order_book", [10,protobuf]], "id": 3}'

to open a stream of order book updates, in json or even better, maybe as a protobuf (to save bandwidth).

In fact, this describes what I was saying perfectly, though I'm not sure why the decision was made to remove *asynchronous notifications* (which to me means **subscriptions**):

>Websockets are provided by steemd because they're what we use to implement steemit.com. The steemit.com website is essentially an in-browser JavaScript implementation of most of the cli_wallet functionality (with lots of additional functionality and a pretty GUI as well, of course). Websockets are much saner to use for modern interactive web applications than HTTP connections. ***In BitShares, the previous Graphene-powered blockchain, websockets were also used for the server to send asynchronous notifications to the client, but Steem currently has little to no functionality that uses websockets in that way.***

https://steemit.com/theoretical/@theoretical/how-to-use-the-steem-api

And regarding @officialfuzzy's "promises", I thought he meant this (but perhaps I missed an intended "double entendre" lol):
https://scotch.io/tutorials/javascript-promises-for-dummies

As for latency, you should also read @dantheman's recent post on how he's thinking about "latency":
https://steemit.com/blockchain/@dantheman/blockchains-should-be-designed-like-massively-multiplayer-games
šŸ‘  ,
properties (23)
authoralexpmorris
permlinkre-faddat-ingestron-0-0-2-released-for-every-platform-imaginable-20170327t133503640z
categorysteemit
json_metadata{"tags":["steemit"],"users":["officialfuzzy","dantheman"],"links":["https://steemit.com/theoretical/@theoretical/how-to-use-the-steem-api","https://scotch.io/tutorials/javascript-promises-for-dummies","https://steemit.com/blockchain/@dantheman/blockchains-should-be-designed-like-massively-multiplayer-games"],"app":"steemit/0.1"}
created2017-03-27 13:35:03
last_update2017-03-27 13:37:30
depth1
children4
last_payout2017-04-27 11:50:36
cashout_time1969-12-31 23:59:59
total_payout_value0.017 HBD
curator_payout_value0.003 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length2,696
author_reputation32,063,874,290,523
root_title"Ingestron 0.0.2 released for every platform imaginable"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,862,909
net_rshares48,054,341,680
author_curate_reward""
vote details (2)
@faddat ·
Ingestron isn't here to save you from latency anymore.  Ingestron is here to save you from API design these days.  Originally ingestron used RethinkDB and was designed to enable real-time updates of page elements that had to be fetched from steemd.  

No, fuzzy is talking about an application that i am shamefully overdue in delivering to him, for reasons that are my fault and reasons that aren't my fault, both.  

Ingestron is a part of getting that done properly.  Tell you what alex, I will make another post that explains how ingestron would change how people build apps using steem.  So, stay tuned.  It'll be complete with a diagram.  

Dan's recent post was awesome.  He's got a lot of brain in that brain pan of his, for sure!
šŸ‘  
properties (23)
authorfaddat
permlinkre-alexpmorris-re-faddat-ingestron-0-0-2-released-for-every-platform-imaginable-20170327t152016115z
categorysteemit
json_metadata{"tags":["steemit"],"app":"steemit/0.1"}
created2017-03-27 15:20:15
last_update2017-03-27 15:20:15
depth2
children0
last_payout2017-04-27 11:50:36
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length737
author_reputation36,581,868,473,026
root_title"Ingestron 0.0.2 released for every platform imaginable"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,863,902
net_rshares114,073,401
author_curate_reward""
vote details (1)
@faddat · (edited)
also, AFAIK, steemd doesn't follow JSON-RPC 2.0, though it claims to.  In fact, one can just look at the dependencies for ingestron to see this:  Need to go into gorilla/rpc (a completley fine implementation of standard json-rpc 2.0.... and break it.  

![](https://steemitimages.com/DQmV4rTYVDDx6WS9ZGKGFu3sruPhQE4Ev3b8g7mqxPWB7cY/image.png)

EG:  Steem doesn't follow even the standards that it claims to follow.  This sucks for devs, so an API that carefully implements a Swagger API will make the situation much better.
properties (22)
authorfaddat
permlinkre-alexpmorris-re-faddat-ingestron-0-0-2-released-for-every-platform-imaginable-20170327t160115003z
categorysteemit
json_metadata{"tags":["steemit"],"image":["https://steemitimages.com/DQmV4rTYVDDx6WS9ZGKGFu3sruPhQE4Ev3b8g7mqxPWB7cY/image.png"],"app":"steemit/0.1"}
created2017-03-27 16:01:15
last_update2017-03-27 16:02:12
depth2
children2
last_payout2017-04-27 11:50:36
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length523
author_reputation36,581,868,473,026
root_title"Ingestron 0.0.2 released for every platform imaginable"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,864,326
net_rshares0
@alexpmorris ·
you really need to understand what's going on at a deeper level.  if you're having problems with a missing "2.0" in the API, well, that goes to what I said about strict frameworks wanting everything a certain way.  The real world's a lot more messy.  There should be exception handling for that, and it could just make an assumption as to the version.  If you've seen as many different APIs as I have, you'd better understand what I mean.  

It may also help to spend some more time in chrome inspect and better understand how both ends talk when you perform an action.  Steemit posts feedback in the console for every action it takes, which makes it much easier for people to understand how an actual implementation works.
properties (22)
authoralexpmorris
permlinkre-faddat-re-alexpmorris-re-faddat-ingestron-0-0-2-released-for-every-platform-imaginable-20170327t172425129z
categorysteemit
json_metadata{"tags":["steemit"],"app":"steemit/0.1"}
created2017-03-27 17:24:24
last_update2017-03-27 17:24:24
depth3
children1
last_payout2017-04-27 11:50:36
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length723
author_reputation32,063,874,290,523
root_title"Ingestron 0.0.2 released for every platform imaginable"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,865,309
net_rshares0
@screenname ·
Re: Ingestron 0.0.2 released for every platform imaginable
<p>This post has been ranked within the top 25 most undervalued posts in the first half of Mar 27. We estimate that this post is undervalued by $9.63 as compared to a scenario in which every voter had an equal say.</p> 
<p>See the full rankings and details in <a href="https://steemit.com/curation/@screenname/the-daily-tribune-most-undervalued-posts-of-mar-27---part-i">The Daily Tribune: Mar 27 - Part I</a>. You can also read about some of our methodology, data analysis and technical details in <a href="https://steemit.com/curation/@screenname/introducing-the-daily-tribune-most-undervalued-posts-of-nov-04---part-i">our initial post</a>.</p>
<p>If you are the author and would prefer not to receive these comments, simply reply "Stop" to this comment.</p>
properties (22)
authorscreenname
permlinkre-ingestron-0-0-2-released-for-every-platform-imaginable-20170327t130942
categorysteemit
json_metadata"{"replyto": "@faddat/ingestron-0-0-2-released-for-every-platform-imaginable"}"
created2017-03-27 13:09:42
last_update2017-03-27 13:09:42
depth1
children0
last_payout2017-04-27 11:50:36
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length762
author_reputation46,276,338,038,330
root_title"Ingestron 0.0.2 released for every platform imaginable"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,862,553
net_rshares0
@transisto ·
#steemdev a good tag for dev stuff.
https://steemit.com/active/steemdev
properties (22)
authortransisto
permlinkre-faddat-ingestron-0-0-2-released-for-every-platform-imaginable-20170327t143249688z
categorysteemit
json_metadata{"tags":["steemdev","steemit"],"links":["https://steemit.com/active/steemdev"],"app":"steemit/0.1"}
created2017-03-27 14:32:51
last_update2017-03-27 14:32:51
depth1
children1
last_payout2017-04-27 11:50:36
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length71
author_reputation330,357,940,720,833
root_title"Ingestron 0.0.2 released for every platform imaginable"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,863,470
net_rshares0
@faddat ·
Thank you kindly!  I will tag my next post accordingly.
šŸ‘  
properties (23)
authorfaddat
permlinkre-transisto-re-faddat-ingestron-0-0-2-released-for-every-platform-imaginable-20170327t153557045z
categorysteemit
json_metadata{"tags":["steemit"],"app":"steemit/0.1"}
created2017-03-27 15:35:57
last_update2017-03-27 15:35:57
depth2
children0
last_payout2017-04-27 11:50:36
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length55
author_reputation36,581,868,473,026
root_title"Ingestron 0.0.2 released for every platform imaginable"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,864,065
net_rshares0
author_curate_reward""
vote details (1)