create account

Ingestron 2.0 Meaner, More multithreaded; Why STEEM is ideal internet infrastructure and what that's got to do with the STEEM-conomy by faddat

View this thread on: hive.blog |  peakd.com |  ecency.com
beyondbitcoin · @faddat · (edited)
$6.03
Ingestron 2.0 Meaner, More multithreaded; Why STEEM is ideal internet infrastructure and what that's got to do with the STEEM-conomy
# <center> Feast your Eyes</center>
<center>[![flyingblocks60fps6e183.gif](https://www.steemimg.com/images/2016/10/05/flyingblocks60fps6e183.gif)](https://www.steemimg.com/image/vDG2Z)</center>

# <center> not sure what you're seeing?</center>

Okay no problem, I'll explain.  [Ingestron](github.com/faddat/ingestron) is getting some retooling.  Ingestron's big brother steemjson is just a bit grandioise for one guy at the moment.  We'll get there of course, but right now I'm more comfortable with a single main.go and less [cobra](github.com/spf13/cobra) around in the garden.  As someone who stands to gain a great deal from the chain having a new valuation of >$1B, I'm doing what I can to make that happen.  

At the moment, "what I can" can be described as "making components that allow for web developers to develop sites against the content in the STEEM chain that do privileged transactions via the browser, and non-priveliged transactions via the back end, which is cached for speed." 

Let's bust out everyone's pal, mermaid for a little diagram of which common dev problems  I think this pattern can solve by leveraging STEEM's content-blockchain:

### <center>Today's Intarwebs</center>
<center>
[![Selection_0390efeb.jpg](https://www.steemimg.com/images/2016/10/05/Selection_0390efeb.jpg)](https://www.steemimg.com/image/vpchL)</center>

This is a familiar, if somewhat inefficent pattern that can be made extremely fast by putting one caching
 proxy server in fromt of the web server like this:

[![Selection_040f436d.png](https://www.steemimg.com/images/2016/10/05/Selection_040f436d.png)](https://www.steemimg.com/image/vpg1J)
 That's still very fast, even if it's inefficent.  

But here's what we're doing with steemit.com:

[![Selection_04274cfe.png](https://www.steemimg.com/images/2016/10/05/Selection_04274cfe.png)](https://www.steemimg.com/image/vpuVa)

There are some great reasons for this design, like security.  But there are other ways to get security, which brings me right around back to my point:  

# <center> Feast your Eyes</center>
<center>[![flyingblocks60fps6e183.gif](https://www.steemimg.com/images/2016/10/05/flyingblocks60fps6e183.gif)](https://www.steemimg.com/image/vDG2Z)</center>

Ingestron is now running smooth as butter.  The next step is to define local contexts for running, and make a handbook showing how operators can deploy this customized, "state stashed but near" infrastructure.  That is to say, in the next article I'll be telling you how to use STEEM to permanently and safely store your content and Ingestron to quickly retrieve your content.  Oh, and another thing about ingestron:  The databases are in RAM.

----
I think 4 graphs and 2 hours writing is enough for now.  This post is not done.  I'll pick up tomorrow, here's what we've got to cover:

* Bitshares Compatibility
* How Ingestron improves webdev with Steem.

----
Like this post?  

Follow @faddat!
Follow @officialfuzzy!

Learn some golang and help @faddat!
----
* Over 200 followers?  I pay a buck for reblogs, because hell yeah, eyeballs!
* Over 100 followers?  Yep, you'll get fifty cents!
* New follower of mine with >200 followers & >50rep & I can see you're not a twitter-bot-of-doom?  That's a buck!
* Insightful Comment?  Twenty five cents!
* Retarded comment, probably automated?  FLAG!

Yep, @faddat's posts pay, even if your "minnow math" doesn't.
👍  , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , and 56 others
properties (23)
authorfaddat
permlinkingestron-2-0-meaner-more-multithreaded-why-steem-is-ideal-internet-infrastructure-and-what-that-s-got-to-do-with-the-steem
categorybeyondbitcoin
json_metadata{"tags":["beyondbitcoin","steem","steveballmer","linux","developersdevelopers"],"image":["https://www.steemimg.com/images/2016/10/05/flyingblocks60fps6e183.gif","https://www.steemimg.com/images/2016/10/05/Selection_0390efeb.jpg","https://www.steemimg.com/images/2016/10/05/Selection_040f436d.png","https://www.steemimg.com/images/2016/10/05/Selection_04274cfe.png"],"links":["https://www.steemimg.com/image/vDG2Z","github.com/faddat/ingestron","github.com/spf13/cobra","https://www.steemimg.com/image/vpchL","https://www.steemimg.com/image/vpg1J","https://www.steemimg.com/image/vpuVa"],"users":["faddat","officialfuzzy"]}
created2016-10-05 20:16:03
last_update2016-10-05 20:57:03
depth0
children2
last_payout2016-11-06 14:00:33
cashout_time1969-12-31 23:59:59
total_payout_value5.110 HBD
curator_payout_value0.924 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length3,386
author_reputation36,581,868,473,026
root_title"Ingestron 2.0 Meaner, More multithreaded; Why STEEM is ideal internet infrastructure and what that's got to do with the STEEM-conomy"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id1,453,675
net_rshares11,394,669,346,783
author_curate_reward""
vote details (120)
@furion ·
$2.59
Did you manage to fill the `tasks` chan buffer yet? In other words, does gorethink ever bottleneck?
👍  , , ,
properties (23)
authorfurion
permlinkre-faddat-ingestron-2-0-meaner-more-multithreaded-why-steem-is-ideal-internet-infrastructure-and-what-that-s-got-to-do-with-the-steem-20161005t212344690z
categorybeyondbitcoin
json_metadata{"tags":["beyondbitcoin"]}
created2016-10-05 21:23:45
last_update2016-10-05 21:23:45
depth1
children1
last_payout2016-11-06 14:00:33
cashout_time1969-12-31 23:59:59
total_payout_value1.940 HBD
curator_payout_value0.645 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length99
author_reputation116,503,940,714,958
root_title"Ingestron 2.0 Meaner, More multithreaded; Why STEEM is ideal internet infrastructure and what that's got to do with the STEEM-conomy"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id1,454,258
net_rshares6,895,044,637,981
author_curate_reward""
vote details (4)
@faddat ·
$2.51
For sure!  

And that's why I'm moving to an in-memory DB.  

But yes, absolutely, it got all constipated, and went down to approximately the same speed as the original because of gorethink which left me with two options:  

* Refactor to pass slices of strings of json to gorethink
* Refactor to an in-memory DB.  I'm playing with: 
    * github.com/hashicorp/go-memdb (don't like the API much)
    * github.com/cayleygraph/cayley (leaning this way: it's embedded and can be totally in-memory.)
👍  ,
properties (23)
authorfaddat
permlinkre-furion-re-faddat-ingestron-2-0-meaner-more-multithreaded-why-steem-is-ideal-internet-infrastructure-and-what-that-s-got-to-do-with-the-steem-20161005t214001897z
categorybeyondbitcoin
json_metadata{"tags":["beyondbitcoin"]}
created2016-10-05 21:40:03
last_update2016-10-05 21:40:03
depth2
children0
last_payout2016-11-06 14:00:33
cashout_time1969-12-31 23:59:59
total_payout_value1.883 HBD
curator_payout_value0.627 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length495
author_reputation36,581,868,473,026
root_title"Ingestron 2.0 Meaner, More multithreaded; Why STEEM is ideal internet infrastructure and what that's got to do with the STEEM-conomy"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id1,454,397
net_rshares6,771,972,251,930
author_curate_reward""
vote details (2)
Help/Feedback