create account

steemd produces incorrect JSON. No, really, no foolin, you can verify this for yourself. UPDATE: This is a piston-lib error by l0k1

View this thread on: hive.blogpeakd.comecency.com
· @l0k1 · (edited)
$1.32
steemd produces incorrect JSON. No, really, no foolin, you can verify this for yourself. UPDATE: This is a piston-lib error
<div class="pull-left"><h1>In</h1></div> <br />my previous post I posted the code for a python script that streams the json output from <code>steemd</code>, and being a relative newbie to JSON encoding, I didn't realise that JSON does not use or accept single quotes. Here is the output of it (unmodified) from block one:

### This has turned out to be an error in piston-lib, which I now am not using anymore, because it's broken and useless. @xeroc

```
{'extensions': [],
 'previous': '0000000000000000000000000000000000000000',
 'timestamp': '2016-03-24T16:05:00',
 'transaction_merkle_root': '0000000000000000000000000000000000000000',
 'transactions': [],
 'witness': 'initminer',
 'witness_signature': '204f8ad56a8f5cf722a02b035a61b500aa59b9519b2c33c77a80c0a714680a5a5a7a340d909d19996613c5e4ae92146b9add8a7a663eef37d837ef881477313043'}
```

This is the raw output that I pushed into a file you can see on <a href="ftp://projectinception.lt/block_log.json.xz">ftp://projectinception.lt/block_log.json.xz</a> (I am not making the uncompressed form because it is 31Gb (this is basically the whole `block_log`up until sometime yesterday). 

Anyone who knows JSON can tell you this is the wrong format. But guess what, this is what comes out of an RPC node. I had to write this little gem to fix it:

```
        out_block = json.loads ( str ( block ).replace("'", '%temp%').replace('"', '\"').replace('%temp%', '"') )
```

and to be clear, this is how that data was acquired, in the same code:

```
props = rpc.get_dynamic_global_properties ( )
current_block = args.startblock
last_confirmed_block = props [ 'last_irreversible_block_num' ]
if args.startblock == -1:
  current_block = last_confirmed_block

while True:
    os.write ( 2, str.encode( "Block #" + str ( current_block ) + '\r' ) )
    #try:
    block = rpc.get_block(current_block)
```

I don't think this is @xeroc's error, this is a raw string coming out of `steemd`

This is the kind of inane nonsense that us developers are dealing with. I am making it known exactly what goes on because it is precisely this kind of repeated experience of incorrect programming that is probably an unspoken reason why so few developers are working with the chain.

I am doing this because I am determined to resolve this for the community and if Steemit isn't going to do the right thing, then we are going to give you all an exit route.

If you know a bit of how to work with python, you can find this code in the 'misc' repository on the `dawn-network` here: https://github.com/dawn-network/misc

## Update

I have realised that the already long string substitution is going to break the single quotes inside the double quotes. I am importing the whole chain snapshot I made two days ago, so I have to write the correct substitution. I will write it in sed script because it will be a lot faster. Stay tuned for an update.

It gets worse too:

```json
'Without any further ado, here is '
'the python script, it depends on '
"`piston-lib`'s module `pistonapi` "
'from @xeroc.\n'
```

For absolutely no good reason, now I have to write a very complex string processing algorithm that recognises correct sections, and leaves them alone, but fixes all the ones that are wrong. This is so nonstandard and wrong, my mind is blowing at the amount of work that this is going to cause me to have to do.

In fact I'd say that it would probably be less work after everything I have had to do so far, to just write a program that reads the binary data out of the `block_log` itself.

# UPDATE

It is an error in piston-lib. So now I am quitting with working with python. Here is the request performed by `curl`:

<code>
curl --data '{"jsonrpc": "2.0", "method": "get_block", "params": [1], "id": 1 }' http://127.0.0.1:8090/rpc

{"id":1,"result":{"previous":"0000000000000000000000000000000000000000","timestamp":"2016-03-24T16:05:00","witness":"initminer","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"204f8ad56a8f5cf722a02b035a61b500aa59b9519b2c33c77a80c0a714680a5a5a7a340d909d19996613c5e4ae92146b9add8a7a663eef37d837ef881477313043","transactions":[]}}
</code>

<div class="pull-right">😎</div><br /><hr />

<center><code>We can't code here! This is Whale country!</code></center>
<div class="pull-left"><a href="https://steemit.com/@l0k1"><img src="http://s20.postimg.org/igf27v79p/signature_new_small.png" /></a></div>

<h2>Vote #1 <code>l0k1</code></h2><sub>Go to <a href="https://steemit.com/~witnesses">steemit.com/~witnesses</a> to cast your vote by copying and pasting <code>l0k1</code> into the text entry at the bottom of the leaderboard.</sub>

<sub>(note, my username is spelled <code>El Zero Kay One</code> or <code>Lima Zero Kilo One</code>, all lower case)</sub>

</div>
👍  , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , and 255 others
👎  ,
properties (23)
authorl0k1
permlinksteemd-produces-incorrect-json-no-really-no-foolin-you-can-verify-this-for-yourself
categorydawn-network
json_metadata{"tags":["dawn-network","steemdev","development","python"],"users":["xeroc"],"image":["http://s20.postimg.org/igf27v79p/signature_new_small.png"],"links":["ftp://projectinception.lt/block_log.json.xz","https://github.com/dawn-network/misc","https://steemit.com/@l0k1","https://steemit.com/~witnesses"],"app":"steemit/0.1","format":"markdown"}
created2017-04-02 20:19:09
last_update2017-04-03 10:59:21
depth0
children11
last_payout2017-04-09 20:19:09
cashout_time1969-12-31 23:59:59
total_payout_value1.110 HBD
curator_payout_value0.205 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length4,787
author_reputation94,800,257,230,993
root_title"steemd produces incorrect JSON. No, really, no foolin, you can verify this for yourself. UPDATE: This is a piston-lib error"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,926,547
net_rshares10,834,840,318,574
author_curate_reward""
vote details (321)
@faddat · (edited)
I just want to say that I've been saying this for more than eight months now.  Corp-chains lie, and those lies have consequences for folks simply trying to do their job.  

Thanks to @l0k1 and his diligence in this regard.  Please know that lies on such a low level of a system, destroy it.  To see my past posts coming to the exact same conclusion (there were many, many of which earned me more than my fair share of scorn and being called a liar, google faddat json steem.  That should do more than well enough.

Regarding that exit route, you can participate in the creation of an exit, here:


https://discord.gg/8dWYbFS
👍  
properties (23)
authorfaddat
permlinkre-l0k1-steemd-produces-incorrect-json-no-really-no-foolin-you-can-verify-this-for-yourself-20170402t215516819z
categorydawn-network
json_metadata{"tags":["dawn-network"],"users":["l0k1"],"app":"steemit/0.1","links":["https://discord.gg/8dWYbFS"]}
created2017-04-02 21:55:18
last_update2017-04-02 21:56:39
depth1
children0
last_payout2017-04-09 21:55:18
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length624
author_reputation36,581,868,473,026
root_title"steemd produces incorrect JSON. No, really, no foolin, you can verify this for yourself. UPDATE: This is a piston-lib error"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,926,990
net_rshares145,440,421,836
author_curate_reward""
vote details (1)
@kenny-crane ·
I know JSON.  But anyone who does not, can validate JSON over at http://jsonlint.com/

Pasting in the alleged JSON object above gives this error:
Expecting 'STRING', got 'undefined'

Turn the single quotes into double quotes and then you will have the strings in the proper format, and the object validates.

Good catch, @l0k1
👍  , , , , , ,
properties (23)
authorkenny-crane
permlinkre-l0k1-steemd-produces-incorrect-json-no-really-no-foolin-you-can-verify-this-for-yourself-20170402t203710924z
categorydawn-network
json_metadata{"tags":["dawn-network"],"users":["l0k1"],"links":["http://jsonlint.com/"],"app":"steemit/0.1"}
created2017-04-02 20:37:03
last_update2017-04-02 20:37:03
depth1
children2
last_payout2017-04-09 20:37:03
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length326
author_reputation234,044,041,897,835
root_title"steemd produces incorrect JSON. No, really, no foolin, you can verify this for yourself. UPDATE: This is a piston-lib error"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,926,634
net_rshares239,219,623,131
author_curate_reward""
vote details (7)
@l0k1 ·
yeah, how does this get past everyone for a whole year, is my question...
👍  
properties (23)
authorl0k1
permlinkre-kenny-crane-re-l0k1-steemd-produces-incorrect-json-no-really-no-foolin-you-can-verify-this-for-yourself-20170402t204537025z
categorydawn-network
json_metadata{"tags":["dawn-network"],"app":"steemit/0.1"}
created2017-04-02 20:45:39
last_update2017-04-02 20:45:39
depth2
children1
last_payout2017-04-09 20:45:39
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length73
author_reputation94,800,257,230,993
root_title"steemd produces incorrect JSON. No, really, no foolin, you can verify this for yourself. UPDATE: This is a piston-lib error"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,926,670
net_rshares94,355,925,604
author_curate_reward""
vote details (1)
@faddat ·
It did not get past me.  

The way it "got past" everyone at steemit, inc is: LIES.
👍  
properties (23)
authorfaddat
permlinkre-l0k1-re-kenny-crane-re-l0k1-steemd-produces-incorrect-json-no-really-no-foolin-you-can-verify-this-for-yourself-20170402t215814600z
categorydawn-network
json_metadata{"tags":["dawn-network"],"app":"steemit/0.1"}
created2017-04-02 21:58:15
last_update2017-04-02 21:58:15
depth3
children0
last_payout2017-04-09 21:58:15
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length83
author_reputation36,581,868,473,026
root_title"steemd produces incorrect JSON. No, really, no foolin, you can verify this for yourself. UPDATE: This is a piston-lib error"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,927,010
net_rshares145,440,421,836
author_curate_reward""
vote details (1)
@markush ·
One day I will understand this better..
👍  
properties (23)
authormarkush
permlinkre-l0k1-steemd-produces-incorrect-json-no-really-no-foolin-you-can-verify-this-for-yourself-20170402t202105654z
categorydawn-network
json_metadata{"tags":["dawn-network"],"app":"steemit/0.1"}
created2017-04-02 20:21:03
last_update2017-04-02 20:21:03
depth1
children0
last_payout2017-04-09 20:21:03
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length39
author_reputation13,440,209,801,477
root_title"steemd produces incorrect JSON. No, really, no foolin, you can verify this for yourself. UPDATE: This is a piston-lib error"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,926,558
net_rshares148,602,170,137
author_curate_reward""
vote details (1)
@renzoarg ·
Can this kind of in-congruence be abused by injection... ?
👍  
properties (23)
authorrenzoarg
permlinkre-l0k1-steemd-produces-incorrect-json-no-really-no-foolin-you-can-verify-this-for-yourself-20170402t221918314z
categorydawn-network
json_metadata{"tags":["dawn-network"],"app":"steemit/0.1"}
created2017-04-02 22:19:18
last_update2017-04-02 22:19:18
depth1
children1
last_payout2017-04-09 22:19:18
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length58
author_reputation62,960,081,117,872
root_title"steemd produces incorrect JSON. No, really, no foolin, you can verify this for yourself. UPDATE: This is a piston-lib error"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,927,099
net_rshares145,440,421,836
author_curate_reward""
vote details (1)
@l0k1 ·
I doubt it could be abused but it certainly will waste a lot of time for developers.
properties (22)
authorl0k1
permlinkre-renzoarg-re-l0k1-steemd-produces-incorrect-json-no-really-no-foolin-you-can-verify-this-for-yourself-20170402t224510280z
categorydawn-network
json_metadata{"tags":["dawn-network"],"app":"steemit/0.1"}
created2017-04-02 22:45:12
last_update2017-04-02 22:45:12
depth2
children0
last_payout2017-04-09 22:45:12
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length84
author_reputation94,800,257,230,993
root_title"steemd produces incorrect JSON. No, really, no foolin, you can verify this for yourself. UPDATE: This is a piston-lib error"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,927,219
net_rshares0
@xeroc · (edited)
$2.91
> This has turned out to be an error in piston-lib, which I now am not using anymore, because it's broken and useless. @xeroc

How about you stop using `pprint` and instead use `print(json.dumps(x))`

All the data that is obtained through the API is python `dict`. If you don't convert it to JSON properly, it's not the error of the library, but an error in your code.
👍  , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
properties (23)
authorxeroc
permlinkre-l0k1-steemd-produces-incorrect-json-no-really-no-foolin-you-can-verify-this-for-yourself-20170403t140548103z
categorydawn-network
json_metadata{"tags":["dawn-network"],"users":["xeroc"],"app":"steemit/0.1"}
created2017-04-03 14:05:48
last_update2017-04-03 14:07:36
depth1
children3
last_payout2017-04-10 14:05:48
cashout_time1969-12-31 23:59:59
total_payout_value2.793 HBD
curator_payout_value0.114 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length368
author_reputation118,819,064,085,695
root_title"steemd produces incorrect JSON. No, really, no foolin, you can verify this for yourself. UPDATE: This is a piston-lib error"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,931,488
net_rshares16,385,110,918,132
author_curate_reward""
vote details (36)
@l0k1 · (edited)
You are right, but other factors are bigger in my decision :) Like the ongoing issue of the unjust amplification of whale power, of the impossibility to get any of the issues that are leading to user attrition, the idiotic inflation policy, which I understand is thought to be necessary because of the forum system, which I think is the cart before the horse because reduced supply allows a more stable price. I mean, it would work, **if** the economy was growing at 100% or 9.5% per year, but the reality is the economy is shrinking so even 2% is too high. And because they invested so much in building a monolithic chain, within 5 years the `block_log` file will be over 200Gb in size. It's time to start again with a different model, and with people who don't secretly value more their bank balance than a growing community and internal economy. I am not covering the whole topic at all but there's too much and I have better things to do than explain what should be obvious.

Oh, btw, I used this:

> variable = json.loads (x)
> print (variable)

I may have used simplejson, in fact, but neither explains why the output came out in python string convention instead of json.

Good luck with your work, anyway.  I just need to move to a more supportive environment where I can learn instead of spending so much time with such fractious people. (not you, of course).
properties (22)
authorl0k1
permlinkre-xeroc-re-l0k1-steemd-produces-incorrect-json-no-really-no-foolin-you-can-verify-this-for-yourself-20170403t203619071z
categorydawn-network
json_metadata{"tags":["dawn-network"],"app":"steemit/0.1"}
created2017-04-03 20:36:24
last_update2017-04-03 20:41:42
depth2
children1
last_payout2017-04-10 20:36:24
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length1,367
author_reputation94,800,257,230,993
root_title"steemd produces incorrect JSON. No, really, no foolin, you can verify this for yourself. UPDATE: This is a piston-lib error"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,934,058
net_rshares0
@abit ·
$0.31
Then please change the title to avoid misleading others.
👍  
properties (23)
authorabit
permlinkre-l0k1-re-xeroc-re-l0k1-steemd-produces-incorrect-json-no-really-no-foolin-you-can-verify-this-for-yourself-20170404t203249497z
categorydawn-network
json_metadata{"tags":["dawn-network"],"app":"steemit/0.1"}
created2017-04-04 20:34:24
last_update2017-04-04 20:34:24
depth3
children0
last_payout2017-04-11 20:34:24
cashout_time1969-12-31 23:59:59
total_payout_value0.232 HBD
curator_payout_value0.077 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length56
author_reputation141,171,499,037,785
root_title"steemd produces incorrect JSON. No, really, no foolin, you can verify this for yourself. UPDATE: This is a piston-lib error"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,941,387
net_rshares3,963,974,369,082
author_curate_reward""
vote details (1)
@pharesim ·
$0.27
omg lol.
STEEM ERROR, steem is shit!
Oh no, piston error. Fucking dirt!
Erm...my own error? Can happen!
👍  
properties (23)
authorpharesim
permlinkre-xeroc-re-l0k1-steemd-produces-incorrect-json-no-really-no-foolin-you-can-verify-this-for-yourself-20170403t141853981z
categorydawn-network
json_metadata{"tags":["dawn-network"],"app":"steemit/0.1"}
created2017-04-03 14:18:54
last_update2017-04-03 14:18:54
depth2
children0
last_payout2017-04-10 14:18:54
cashout_time1969-12-31 23:59:59
total_payout_value0.199 HBD
curator_payout_value0.066 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length103
author_reputation239,450,405,799,183
root_title"steemd produces incorrect JSON. No, really, no foolin, you can verify this for yourself. UPDATE: This is a piston-lib error"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,931,569
net_rshares3,884,694,881,700
author_curate_reward""
vote details (1)