create account

Filtering of operations returned by the get_account_history API call. by miniature-tiger

View this thread on: hive.blogpeakd.comecency.com
· @miniature-tiger · (edited)
$5.37
Filtering of operations returned by the get_account_history API call.
![gears_blockops_pink.png](https://ipfs.busy.org/ipfs/QmRHkXakScTvN5xorda64DLbMsDiHC6moUxpbPRhJnKyhP)


##### OVERVIEW
A very useful Hive non-consensus change was added by the @blocktrades core coding team last week; namely the filtering of operations returned by the "get_account_history" API call.	

For those not intimately acquainted with Hive API calls, the "get_account_history" request allows you to obtain the Hive transactions performed by a single Hive account. These are returned in batches of up to 1000 transactions, in reverse chronological order, starting from a transaction number that you define. 

Prior to last week a history request would include all transactions for the account with no possibility to filter by type. So if you only wanted to look at the account's power-up transactions, of which there are, say, 3, you would still need to pull though their 10,000 comments, 100,000 votes and 300,000 splinterlands custom json transactions. This made the "get_account_history" API call particularly data-heavy and time consuming. 

Following the change it's now possible to filter the results by transaction type, allowing you to grab just the 3 power-up transactions you were looking for.

I've spent the last few months updating my HeyStack project (more on this in a couple of weeks) for which the above change is hugely useful. As such, when I saw it mentioned in @blocktrades' post (https://hive.blog/hive-139531/@blocktrades/update-on-blocktrades-hive-core-coding) I immediately tried to put it into action.

Easier said than done! But after a fair amount of digging around, a little trial and error, and the breaking-out of the BigInt, I got there in the end. Here's what I learned! 

##### NEW PARAMETERS
Prior to the change there were three main parameters:
* account (string): the account name, e.g. "miniature-tiger";
* start (int): the starting transaction number of the batch (which can be set to -1 for the most recent transaction); and
* limit (int): the number of transactions in the batch (a max of 1000). 

So if I have made 125,499 transactions (true at the time of writing) then parameters ["miniature-tiger", -1, 0] would have returned my most recent single transaction and ["miniature-tiger", 999, 999] would have returned my first 1000 transactions (you get 1 more transaction than you ask for because Hive is very inclusive, and you count from zero, because arrays).

After the change there are two new parameters:
* operation_filter_low (int64)
* operation_filter_high (int64) 

They are defined here:
https://gitlab.syncad.com/hive/hive/-/blob/master/libraries/plugins/apis/account_history_api/include/hive/plugins/account_history_api/account_history_api.hpp

These parameters operate a matching bitwise filter based on the transaction list shown here (once the SMT transactions are stripped out):
https://gitlab.syncad.com/hive/hive/-/blob/master/libraries/protocol/include/hive/protocol/operations.hpp

There are two 64 bit parameters: the first one covers the first 64 transaction types, the second one covers the remainder. There are currently 73 transaction types listed.

##### HOW DOES IT WORK - IN SIMPLE TERMS!
Take the transaction list, strip out the SMT transactions, and use 2^(n-1) of the transaction number in the list for the low parameter.

For example (using transactions from the start of the list):
1: vote_operation: 1 (i.e. 000001)
2: comment_operation: 2 (i.e. 000010) 
3: transfer_operation: 4 (i.e. 000100)
4: transfer_to_vesting_operation: 8 (i.e. 001000) 

So if you just want transfer transactions and no others then the low parameter would be 4 (and the high parameter would be 0). 

Parameters of ["miniature-tiger", 999, 999, 4, 0] would return all the transfer transactions out of my first 1000 transactions (i.e. an array of, say, 2 transactions, or possibly no transactions at all if I hadn't got around to making transfers early in my Hive career). 

You will still need to loop through numerous calls if you want all my transfers (e.g. ["miniature-tiger", 1999, 999, 4, 0] for the next 1000 etc etc) but most users will probably only be interested in the latest transactions rather than the whole account history.

For transaction types listed above 64 use the high parameter, deducting 64 from the transaction number. Again from the transactions list:  
65: proposal_pay_operation: 2^(65 - 64 -1) = 1
66: sps_fund_operation: 2^(66 - 64 - 1) = 2
i.e. basically the same as for the low parameter but starting the list over from transaction 65.

So if you just want my proposal_pay transactions (I don't got none before you ask) and no others then the low parameter would be 0 and the high parameter would be 1 (i.e. 2^(65-64-1) = 1) and so forth. 

Why this complex approach? Because with the bitwise filter if you want both comments and transfers but no other transactions you can set the low parameter to 6 (2 + 4 = 6 i.e. 000110), based on:
2: comment_operation: 2 (i.e. 000010) 
3: transfer_operation: 4 (i.e. 000100)

And if you want comments, transfers, and proposal payments then you set the low parameter to 6 and the high parameter to 1.
  
Super-useful!!!

##### BREAKING OUT THE BIGINT()

One issue I found coding in js for HeyStack is that Int cannot handle numbers over 2^53-1. So trying to calculate a low parameter number for transactions above 53 in the list (and below 65) won't fly.

There's undoubtedly a sexier way to do this using hexes and other bitwise-shift witchery but I decided just to break out the BigInt().

The code below works to calculate the low and high parameters. Just add your own desired transaction numbers in "paramNumbers".

let paramNumbers = [3, 4, 10, 23, 51, 55, 56, 62, 63, 65];
let lowParam = paramNumbers.filter(x => x <= 64).reduce((total, amount) => total + (2n ** BigInt(amount-1)), 0n);
let highParam = paramNumbers.filter(x => x > 64).reduce((total, amount) => total + (2n ** BigInt(amount-64-1)), 0n);

That's about it! As I said, a very useful change and hopefully the above explanation is useful for anyone trying to use it!
👍  , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , and 124 others
properties (23)
authorminiature-tiger
permlinkfiltering-of-operations-returned-by-the-getaccounthistory-api-call
categoryhive-139531
json_metadata"{"app":"hiveblog/0.1","format":"markdown","description":"A simple explanation for a very useful Hive code change!","tags":["devs","stemgeeks","leofinance","palnet","neoxian","hive"],"users":["blocktrades"],"links":["https://hive.blog/hive-139531/@blocktrades/update-on-blocktrades-hive-core-coding","https://gitlab.syncad.com/hive/hive/-/blob/master/libraries/plugins/apis/account_history_api/include/hive/plugins/account_history_api/account_history_api.hpp","https://gitlab.syncad.com/hive/hive/-/blob/master/libraries/protocol/include/hive/protocol/operations.hpp"],"image":["https://ipfs.busy.org/ipfs/QmRHkXakScTvN5xorda64DLbMsDiHC6moUxpbPRhJnKyhP"]}"
created2020-10-31 22:27:24
last_update2020-10-31 22:35:33
depth0
children4
last_payout2020-11-07 22:27:24
cashout_time1969-12-31 23:59:59
total_payout_value3.100 HBD
curator_payout_value2.267 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length6,062
author_reputation110,039,466,555,434
root_title"Filtering of operations returned by the get_account_history API call."
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id100,336,286
net_rshares29,539,966,675,456
author_curate_reward""
vote details (188)
@hivebuzz ·
<center>[![](https://images.hive.blog/175x175/http://hivebuzz.me/@miniature-tiger/level.png?202010312322)](https://hivebuzz.me/@miniature-tiger)
**Congratulations @miniature-tiger**!
You raised your level and are now a **Dolphin**!</center>

**Do not miss the last post from @hivebuzz:**
<table><tr><td><a href="/hivebuzz/@hivebuzz/pud-202011"><img src="https://images.hive.blog/64x128/https://i.imgur.com/805FIIt.jpg"></a></td><td><a href="/hivebuzz/@hivebuzz/pud-202011">Next Hive Power Up Day is November 1st 2020</a></td></tr><tr><td><a href="/halloween/@hivebuzz/trick-or-treat-share-your-scariest-story-and-get-your-halloween-badge"><img src="https://images.hive.blog/64x128/https://i.imgur.com/vvqCEPy.png"></a></td><td><a href="/halloween/@hivebuzz/trick-or-treat-share-your-scariest-story-and-get-your-halloween-badge">Trick or Treat - Share your scariest story and get your Halloween badge</a></td></tr></table>
properties (22)
authorhivebuzz
permlinkhivebuzz-notify-miniature-tiger-20201031t232504000z
categoryhive-139531
json_metadata{"image":["http://hivebuzz.me/notify.t6.png"]}
created2020-10-31 23:25:03
last_update2020-10-31 23:25:03
depth1
children0
last_payout2020-11-07 23:25:03
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length923
author_reputation367,967,710,900,820
root_title"Filtering of operations returned by the get_account_history API call."
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id100,336,866
net_rshares0
@inertia · (edited)
I've noticed that not all full nodes are running the plugin that is required for this feature.  But one that is: `api.hive.blog`

Here's an example of how to invoke this from the command-line, using the `condenser_api` namespace, filtering on the `vote_operation` bit:

```bash
curl -s --data '{
  "jsonrpc": "2.0",
  "method": "condenser_api.get_account_history",
  "params": ["hiveio", 1000, 1000, 1, null],
  "id":1
}' https://api.hive.blog
```

And here's the same example, using the `account_history_api` namespace, again filtering on the `vote_operation` bit:

```bash
curl -s --data '{
  "jsonrpc":"2.0",
  "method": "account_history_api.get_account_history",
  "params": {
    "account": "hiveio",
    "start": 1000,
    "limit": 1000,
    "include_reversible": false,
    "operation_filter_low": 1,
    "operation_filter_high": null
  },
  "id":1
}' https://api.hive.blog
```
👍  
properties (23)
authorinertia
permlinkre-miniature-tiger-qj59lg
categoryhive-139531
json_metadata{"app":"hiveblog/0.1"}
created2020-11-02 01:03:15
last_update2020-11-02 15:32:45
depth1
children2
last_payout2020-11-09 01:03:15
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length884
author_reputation344,124,293,856,703
root_title"Filtering of operations returned by the get_account_history API call."
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id100,352,211
net_rshares19,274,033,308
author_curate_reward""
vote details (1)
@miniature-tiger ·
> I've noticed that not all full nodes are running the plugin that is required for this feature. But one that is: api.hive.blog

Yes. I think that as the code change was made only very recently it could take a little while to be rolled out by all nodes.

> curl -s --data '{
  "jsonrpc": "2.0",
  "method": "condenser_api.get_account_history",
  "params": ["hiveio", 1000, 1000, 1, null],
  "id":1
}' https://api.hive.blog

If you wanted both high and low parameters (both set to 1) would it be as follows?
"params": ["hiveio", 1000, 1000, 1, 1, null],
properties (22)
authorminiature-tiger
permlinkqj6031
categoryhive-139531
json_metadata{"links":["https://api.hive.blog"],"app":"hiveblog/0.1"}
created2020-11-02 10:35:27
last_update2020-11-02 10:35:27
depth2
children1
last_payout2020-11-09 10:35:27
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length552
author_reputation110,039,466,555,434
root_title"Filtering of operations returned by the get_account_history API call."
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id100,357,146
net_rshares0
@inertia ·
No.  The final `null` would become the high bits:

```bash
curl -s --data '{
  "jsonrpc": "2.0",
  "method": "condenser_api.get_account_history",
  "params": ["hiveio", 1000, 1000, 1, 1],
  "id":1
}' https://api.hive.blog
```
properties (22)
authorinertia
permlinkqj6duw
categoryhive-139531
json_metadata{"app":"hiveblog/0.1"}
created2020-11-02 15:32:57
last_update2020-11-02 15:32:57
depth3
children0
last_payout2020-11-09 15:32:57
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length225
author_reputation344,124,293,856,703
root_title"Filtering of operations returned by the get_account_history API call."
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id100,360,627
net_rshares0