create account

Beem: Threaded blockchain.stream() fetched blocks without ops twice by stmdev

View this thread on: hive.blogpeakd.comecency.com
· @stmdev ·
$20.78
Beem: Threaded blockchain.stream() fetched blocks without ops twice
#### Project Information
* Repository: https://github.com/holgern/beem
* Project Name: Beem
* Publisher: @holger80

A [GitHub issue](https://github.com/holgern/beem/issues/69) was created and the bug was confirmed and fixed by the PO.

#### Expected behavior
`Blockchain.stream(threading=True)` should use threads to fetch several blocks in parallel. Each block should be fetched once from the RPC node.

#### Actual behavior
`Blockchain.stream(threading=True)` requests blocks twice from the RPC node, if those block don't contain operations or only virtual operations. This limits the benefits from the parallel threads.

#### How to reproduce
```python
from beem.blockchain import Blockchain
b = Blockchain()
for op in b.stream(start=2500000, stop=2500010, threading=True):
    continue
```

Plus printf-debugging:
```patch
index 5cb196c..2e19545 100644
--- a/beemapi/steemnoderpc.py
+++ b/beemapi/steemnoderpc.py
@@ -53,6 +53,7 @@ class SteemNodeRPC(GrapheneRPC):
             :raises ValueError: if the server does not respond in proper JSON format
             :raises RPCError: if the server returns an error
         """
+        print(payload)
         if self.url is None:
             raise except
```

Output:
```
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': ['database_api', 'get_dynamic_global_properties', []], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': ['database_api', 'get_feed_history', []], 'jsonrpc': '2.0', 'id': 3}
{'method': 'call', 'params': ['database_api', 'get_feed_history', []], 'jsonrpc': '2.0', 'id': 4}
{'method': 'call', 'params': ['database_api', 'get_next_scheduled_hardfork', []], 'jsonrpc': '2.0', 'id': 5}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 6}
{'method': 'call', 'params': ['database_api', 'get_witness_schedule', []], 'jsonrpc': '2.0', 'id': 7}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 8}
{'method': 'call', 'params': ['database_api', 'get_reward_fund', ['post']], 'jsonrpc': '2.0', 'id': 9}
{'method': 'call', 'params': ['database_api', 'get_dynamic_global_properties', []], 'jsonrpc': '2.0', 'id': 10}
{'method': 'call', 'params': [None, 'get_block', [25031061]], 'jsonrpc': '2.0', 'id': 11}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': [None, 'get_block', [2500000]], 'jsonrpc': '2.0', 'id': 12}
{'method': 'call', 'params': [None, 'get_block', [2500001]], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': [None, 'get_block', [2500002]], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': [None, 'get_block', [2500003]], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': [None, 'get_block', [2500004]], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': [None, 'get_block', [2500005]], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': [None, 'get_block', [2500006]], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': [None, 'get_block', [2500007]], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': [None, 'get_block', [2500000]], 'jsonrpc': '2.0', 'id': 13}
{'method': 'call', 'params': [None, 'get_block', [2500003]], 'jsonrpc': '2.0', 'id': 14}
{'method': 'call', 'params': [None, 'get_block', [2500004]], 'jsonrpc': '2.0', 'id': 15}
{'method': 'call', 'params': [None, 'get_block', [2500005]], 'jsonrpc': '2.0', 'id': 16}
{'method': 'call', 'params': [None, 'get_block', [2500007]], 'jsonrpc': '2.0', 'id': 17}
{'method': 'call', 'params': [None, 'get_block', [2500008]], 'jsonrpc': '2.0', 'id': 18}
{'method': 'call', 'params': [None, 'get_block', [2500009]], 'jsonrpc': '2.0', 'id': 3}
{'method': 'call', 'params': [None, 'get_block', [2500010]], 'jsonrpc': '2.0', 'id': 3}
{'method': 'call', 'params': [None, 'get_block', [2500008]], 'jsonrpc': '2.0', 'id': 19}
{'method': 'call', 'params': [None, 'get_block', [2500009]], 'jsonrpc': '2.0', 'id': 20}
{'method': 'call', 'params': [None, 'get_block', [2500010]], 'jsonrpc': '2.0', 'id': 21}
```

8 of the 11 blocks are fetched twice: 2500000, 2500003, 2500004, 2500005, 2500007, 2500008, 2500009, 2500010. Those blocks contain no operations.

---

The reason was confirmed to be here:
https://github.com/holgern/beem/blob/78281e724248b44724f985f8b57f936558d0d769/beem/blockchain.py#L463
with `len(b.operations) == 0`, the block is treated as if it was never fetched from the chain.

The check for `len(b.operations) == 0` was removed by the PO.

#### Environment
```
# beempy --version
beempy, version 0.19.53
# python --version
Python 3.6.6
```

#### GitHub Account
https://github.com/crokkon
πŸ‘  , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , and 6 others
properties (23)
authorstmdev
permlinkbeem-threaded-blockchain-stream-fetched-blocks-without-ops-twice-1535100683291
categoryutopian-io
json_metadata{"app":"steeditor/0.1.0","format":"markdown","image":[],"tags":["utopian-io","bug-hunting","beem","python"],"users":["holger80"],"links":["https://github.com/holgern/beem/issues/69"],"community":"utopian"}
created2018-08-24 08:51:24
last_update2018-08-24 08:51:24
depth0
children4
last_payout2018-08-31 08:51:24
cashout_time1969-12-31 23:59:59
total_payout_value15.620 HBD
curator_payout_value5.162 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length5,253
author_reputation10,427,398,400,460
root_title"Beem: Threaded blockchain.stream() fetched blocks without ops twice"
beneficiaries
0.
accountutopian.pay
weight500
max_accepted_payout100,000.000 HBD
percent_hbd10,000
post_id69,207,572
net_rshares14,686,130,162,490
author_curate_reward""
vote details (70)
@fego ·
$4.20
Hi @stmdev, thanks for making this contribution.

I see the issue has been fixed, thanks to you pointing what exactly to look for.

Feedback:
1. Though the title contains sufficient information, it is missing one tiny bit of detail, the release version. The format ought to be ([Product Name] [Product Version] – Succinct description of the bug).
2. The expected and actual behavior were properly explained. The discrepancy between the two behaviors could be spotted with ease. 
3. The *"how to reproduce"* is what is exactly expected of a bug report. 
4. It was helpful of you to have provided the output, and also pointing the PO to where the error possibly originated.

This adds significant value to the open source community, I look forward to your next contribution.


Your contribution has been evaluated according to [Utopian policies and guidelines](https://join.utopian.io/guidelines), as well as a predefined set of questions pertaining to the category.

To view those questions and the relevant answers related to your post, [click here](https://review.utopian.io/result/5/311112).

---- 
Need help? Write a ticket on https://support.utopian.io/. 
Chat with us on [Discord](https://discord.gg/uTyJkNm). 
[[utopian-moderator]](https://join.utopian.io/)
πŸ‘  , , , , , ,
properties (23)
authorfego
permlinkre-stmdev-beem-threaded-blockchain-stream-fetched-blocks-without-ops-twice-1535100683291-20180824t094545103z
categoryutopian-io
json_metadata{"community":"busy","app":"busy/2.5.6","format":"markdown","tags":["utopian-io"],"users":["stmdev"],"links":["/@stmdev","https://join.utopian.io/guidelines","https://review.utopian.io/result/5/311112","https://support.utopian.io/","https://discord.gg/uTyJkNm","https://join.utopian.io/"],"image":[]}
created2018-08-24 09:45:48
last_update2018-08-24 09:45:48
depth1
children1
last_payout2018-08-31 09:45:48
cashout_time1969-12-31 23:59:59
total_payout_value3.170 HBD
curator_payout_value1.027 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length1,263
author_reputation168,836,422,039,217
root_title"Beem: Threaded blockchain.stream() fetched blocks without ops twice"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id69,211,515
net_rshares2,848,621,578,895
author_curate_reward""
vote details (7)
@utopian-io ·
Thank you for your review, @fego!

So far this week you've reviewed 1 contributions. Keep up the good work!
properties (22)
authorutopian-io
permlinkre-re-stmdev-beem-threaded-blockchain-stream-fetched-blocks-without-ops-twice-1535100683291-20180824t094545103z-20180825t030509z
categoryutopian-io
json_metadata"{"app": "beem/0.19.42"}"
created2018-08-25 03:05:09
last_update2018-08-25 03:05:09
depth2
children0
last_payout2018-09-01 03:05:09
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length107
author_reputation152,955,367,999,756
root_title"Beem: Threaded blockchain.stream() fetched blocks without ops twice"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id69,284,673
net_rshares0
@steem-ua ·
Hi @stmdev! We are @steem-ua, a new Steem dApp, computing UserAuthority for all accounts on Steem. We are currently in test mode upvoting quality Utopian-io contributions! Nice work!
properties (22)
authorsteem-ua
permlinkre-beem-threaded-blockchain-stream-fetched-blocks-without-ops-twice-1535100683291-20180824t100834z
categoryutopian-io
json_metadata"{"app": "beem/0.19.54"}"
created2018-08-24 10:08:33
last_update2018-08-24 10:08:33
depth1
children0
last_payout2018-08-31 10:08:33
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length182
author_reputation23,214,230,978,060
root_title"Beem: Threaded blockchain.stream() fetched blocks without ops twice"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id69,213,149
net_rshares0
@utopian-io ·
Hey @stmdev
 **Thanks for contributing on Utopian**.
Congratulations! Your contribution was Staff Picked to receive a maximum vote for the bug-hunting category on Utopian for being of significant value to the project and the open source community.

We’re already looking forward to your next contribution!

**Want to chat? Join us on Discord https://discord.gg/h52nFrV.**

<a href='https://steemconnect.com/sign/account-witness-vote?witness=utopian-io&approve=1'>Vote for Utopian Witness!</a>
properties (22)
authorutopian-io
permlinkre-beem-threaded-blockchain-stream-fetched-blocks-without-ops-twice-1535100683291-20180825t085508z
categoryutopian-io
json_metadata"{"app": "beem/0.19.42"}"
created2018-08-25 08:55:09
last_update2018-08-25 08:55:09
depth1
children0
last_payout2018-09-01 08:55:09
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length492
author_reputation152,955,367,999,756
root_title"Beem: Threaded blockchain.stream() fetched blocks without ops twice"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id69,304,259
net_rshares0