Viewing a response to: @krnel/subchains-and-multi-chain-matrices-for-massive-blockchain-data-propagation
It is very critical to separate read demand from writers. Steem can easily scale to unlimited read requests. It is the business logic, not storage that must be scaled. You are right about needing a large number interconnected chains. We can divide steem into about 10 chains such that exchanges don't need to load all content and posts. That alone only gives us at most 10x and that assumes equal load balance, but of course that isn't the case. We need to enable chains to apply all transactions in a block in parallel. Next we need to pipeline evaluation like CPUs do.
author | dantheman |
---|---|
permlink | re-krnel-subchains-and-multi-chain-matrices-for-massive-blockchain-data-propagation-20161201t165337732z |
category | blockchain |
json_metadata | {"tags":["blockchain"]} |
created | 2016-12-01 16:53:36 |
last_update | 2016-12-01 16:53:36 |
depth | 1 |
children | 4 |
last_payout | 2017-01-01 16:38:27 |
cashout_time | 1969-12-31 23:59:59 |
total_payout_value | 6.275 HBD |
curator_payout_value | 0.010 HBD |
pending_payout_value | 0.000 HBD |
promoted | 0.000 HBD |
body_length | 584 |
author_reputation | 240,292,002,602,347 |
root_title | "Subchains and Multi-Chain Matrices for Massive Blockchain Data Propagation" |
beneficiaries | [] |
max_accepted_payout | 1,000,000.000 HBD |
percent_hbd | 10,000 |
post_id | 1,890,607 |
net_rshares | 25,926,096,342,245 |
author_curate_reward | "" |
voter | weight | wgt% | rshares | pct | time |
---|---|---|---|---|---|
dantheman | 0 | 25,921,995,494,276 | 100% | ||
ppitonak | 0 | 3,407,739,482 | 100% | ||
southbaybits | 0 | 63,929,415 | 100% | ||
cnmtz | 0 | 58,169,756 | 100% | ||
johnathanhenry | 0 | 50,125,330 | 76% | ||
ivanvan | 0 | 520,883,986 | 100% |
Thanks for the clarification. Steem may be able to handle unlimited read requests, but the nodes that host the data and have to deliver those requests, is the issue when you are talking about huge data files like movies, or the size of the blockchain itself. I still see that as an issue, when talking about nodes hosting all the data on one chain. A million terrabyte ledger is not viable. That proves how size is an issue, and why subchains are required. You break it up into different databases, which is how the whole internet is structures, networks of networks. I was just making the case that a blockchain can't stay one blockchain for all data, as it's not possible in the long term.
author | krnel |
---|---|
permlink | re-dantheman-re-krnel-subchains-and-multi-chain-matrices-for-massive-blockchain-data-propagation-20161201t172551903z |
category | blockchain |
json_metadata | {"tags":["blockchain"]} |
created | 2016-12-01 17:25:51 |
last_update | 2016-12-01 17:25:51 |
depth | 2 |
children | 2 |
last_payout | 2017-01-01 16:38:27 |
cashout_time | 1969-12-31 23:59:59 |
total_payout_value | 0.000 HBD |
curator_payout_value | 0.000 HBD |
pending_payout_value | 0.000 HBD |
promoted | 0.000 HBD |
body_length | 692 |
author_reputation | 1,343,547,270,297,082 |
root_title | "Subchains and Multi-Chain Matrices for Massive Blockchain Data Propagation" |
beneficiaries | [] |
max_accepted_payout | 1,000,000.000 HBD |
percent_hbd | 10,000 |
post_id | 1,890,808 |
net_rshares | 520,883,986 |
author_curate_reward | "" |
voter | weight | wgt% | rshares | pct | time |
---|---|---|---|---|---|
ivanvan | 0 | 520,883,986 | 100% |
Large datasets are best stored in a DHT. The business logic is hard to compress or divide once you get down to sequence dependent operations. A large ledger can be distributed via DHT, but the interpretation of the ledger requires the active state. The structure you describe for a block chain database records is different than how we organize it.
author | dantheman |
---|---|
permlink | re-krnel-re-dantheman-re-krnel-subchains-and-multi-chain-matrices-for-massive-blockchain-data-propagation-20161201t173624045z |
category | blockchain |
json_metadata | {"tags":["blockchain"]} |
created | 2016-12-01 17:36:24 |
last_update | 2016-12-01 17:36:24 |
depth | 3 |
children | 1 |
last_payout | 2017-01-01 16:38:27 |
cashout_time | 1969-12-31 23:59:59 |
total_payout_value | 0.000 HBD |
curator_payout_value | 0.000 HBD |
pending_payout_value | 0.000 HBD |
promoted | 0.000 HBD |
body_length | 354 |
author_reputation | 240,292,002,602,347 |
root_title | "Subchains and Multi-Chain Matrices for Massive Blockchain Data Propagation" |
beneficiaries | [] |
max_accepted_payout | 1,000,000.000 HBD |
percent_hbd | 10,000 |
post_id | 1,890,883 |
net_rshares | 571,009,316 |
author_curate_reward | "" |
voter | weight | wgt% | rshares | pct | time |
---|---|---|---|---|---|
johnathanhenry | 0 | 50,125,330 | 76% | ||
ivanvan | 0 | 520,883,986 | 100% |
Alright, I'm more talking about the node itself that has to download the blockchain ledger. It's not manageable as one single blockchain ledger when it gets too big. That was my main issue. Thanks for the feedback and clarity.
author | krnel |
---|---|
permlink | re-dantheman-re-krnel-re-dantheman-re-krnel-subchains-and-multi-chain-matrices-for-massive-blockchain-data-propagation-20161201t175925358z |
category | blockchain |
json_metadata | {"tags":["blockchain"]} |
created | 2016-12-01 17:59:24 |
last_update | 2016-12-01 17:59:24 |
depth | 4 |
children | 0 |
last_payout | 2017-01-01 16:38:27 |
cashout_time | 1969-12-31 23:59:59 |
total_payout_value | 0.000 HBD |
curator_payout_value | 0.000 HBD |
pending_payout_value | 0.000 HBD |
promoted | 0.000 HBD |
body_length | 226 |
author_reputation | 1,343,547,270,297,082 |
root_title | "Subchains and Multi-Chain Matrices for Massive Blockchain Data Propagation" |
beneficiaries | [] |
max_accepted_payout | 1,000,000.000 HBD |
percent_hbd | 10,000 |
post_id | 1,891,033 |
net_rshares | 0 |
Better than manual splitting would be automatic. The demand driven model he refers to is from me, and it is about using frequent associations to find data that should be pooled so it can be added to without waiting for the data to come from other nodes, in the case of nodes not caching all the data for the purpose of increasing the number of domains where transaction authorisations can be immediately certified to their provenance. To implement it I suggest something like a hierarchy of nodes like we now have Witnesses, and runners up. Another hierarchy related to capacity, both storage and processing. These nodes don't keep the whole blockchain, but a related subset, and clustered according to transaction history and frequency that leaves from them are added on, such as user accounts, tokens, and contracts. Some have to be more broadly spread, but if these subnodes are sufficient in number, they in effect break the blockchain in a more temporary and specific way, driven by use. Tracking client request frequency, correlating other entities that associate with them, is not for increasing read speed, but decreasing latency by having the necessary data already in cache. I am pretty sure to some degree Graphene already does some of this within the node's live buffer caches, I am mainly talking about expanding the caching to various kinds of caching nodes who specialise in aggregating associated data on disk, instead of having to store it all, only a little, and other nodes know to propagate transactions to them. I think we are talking much the same thing with breaking into subchains, but My idea is derived from the very solutiin Witnesses and Masternodes enable - reducing the cost of convergence by delaying the replication so the data is canonical within a cluster of nodes overlapping in their focal areas, and thus confirmed quickly. In the background the node propagates first to near neighbours, and much later than currently, the network converges. But where it is used, it is nearly instant. Well, I am just trying to help here anyway. Parallelisation is key here so knowledge from routing and caching systems is very pertinent, as is graph analysis to find divisible regions. From what I understand, Graphene is very adapted to graph manipulation. Alsi his reminds me about how 3D graphics systems extensively work with graphs and GPUs have special processing systems for dealing with them (matrix transforms).
author | l0k1 |
---|---|
permlink | re-dantheman-re-krnel-subchains-and-multi-chain-matrices-for-massive-blockchain-data-propagation-20161201t165337732z-2016122t223033355z |
category | blockchain |
json_metadata | {"tags":"blockchain","app":"esteem/1.3.2","format":"markdown+html"} |
created | 2016-12-02 21:30:36 |
last_update | 2016-12-02 21:30:36 |
depth | 2 |
children | 0 |
last_payout | 2017-01-01 16:38:27 |
cashout_time | 1969-12-31 23:59:59 |
total_payout_value | 0.000 HBD |
curator_payout_value | 0.000 HBD |
pending_payout_value | 0.000 HBD |
promoted | 0.000 HBD |
body_length | 2,448 |
author_reputation | 94,800,257,230,993 |
root_title | "Subchains and Multi-Chain Matrices for Massive Blockchain Data Propagation" |
beneficiaries | [] |
max_accepted_payout | 1,000,000.000 HBD |
percent_hbd | 10,000 |
post_id | 1,901,466 |
net_rshares | 0 |