create account

Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*() by jfollas

View this thread on: hive.blogpeakd.comecency.com
· @jfollas · (edited)
$113.59
Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()
_(Previous Post: [Part 8](/@jfollas/write-a-steemit-web-app-part-8-retrieving-content-with-getstate))_

Previously, I introduced the `getState()` API function that is used by the [Condenser](https://github.com/steemit/condenser) app (a.k.a., Steemit.com) to retrieve content for a given URL path, such as `'trending/steemdev'`. That function is great in order to get a page-worth of posts and other information, such as Account objects and the current feed price, all in one call. But, in order to retrieve more than just the first 20 posts, you must use a different API function.

Today, we'll look into the various `getDiscussionsBy*` functions.

## The `get_discussions_by` API Functions
A lot of the documentation for the Steem API only exists in the form of source code - that's part of the reason why I started this series of posts (to document what I find as I explored the source code while trying to figure out how the API works). The [database_api header file](https://github.com/steemit/steem/blob/master/libraries/app/include/steemit/app/database_api.hpp) is a good place to start, since header files tend to be better documented than the rest of the source.

In that file, you'll find a group of functions with the same signature:

```c
vector<discussion> get_discussions_by_payout(const discussion_query& query )const;
vector<discussion> get_discussions_by_trending( const discussion_query& query )const;
vector<discussion> get_discussions_by_created( const discussion_query& query )const;
vector<discussion> get_discussions_by_active( const discussion_query& query )const;
vector<discussion> get_discussions_by_cashout( const discussion_query& query )const;
vector<discussion> get_discussions_by_votes( const discussion_query& query )const;
vector<discussion> get_discussions_by_children( const discussion_query& query )const;
vector<discussion> get_discussions_by_hot( const discussion_query& query )const;
vector<discussion> get_discussions_by_feed( const discussion_query& query )const;
vector<discussion> get_discussions_by_blog( const discussion_query& query )const;
vector<discussion> get_discussions_by_comments( const discussion_query& query )const;
vector<discussion> get_discussions_by_promoted( const discussion_query& query )const;
```

<br/>_Note: In the C API, snake_case is used, while Steem.js uses camelCase. Simply remove the underscores and capitalize each subsequent word to figure out what the function name needs to be for JavaScript._

So, let's say that you used `getState('trending/steemdev')` to retrieve the first 20 posts, and you want to fetch the next 20. How would you do that with one of these API functions?

First, you need to know which database index you were really using for the `getState()` call. From a pure data point of view, you can check the data returned by `getState()` to see which array under `discussion_idx` is populated. 

In this case, it was `'trending'`, so it's pretty easy to figure out that you probably want the `get_discussions_by_trending(query)` function (or, rather, `getDiscussionsByTrending(query)` in JavaScript).  But, what's that `query` argument all about?

## `discussion_query` Struct

Again, referring to [the header file](https://github.com/steemit/steem/blob/master/libraries/app/include/steemit/app/database_api.hpp#L83), we see that `discussion_query` is defined as:

```c
/**
 *  Defines the arguments to a query as a struct so it can be easily extended
 */
struct discussion_query {
   void validate()const{
      FC_ASSERT( filter_tags.find(tag) == filter_tags.end() );
      FC_ASSERT( limit <= 100 );
   }

   string           tag;
   uint32_t         limit = 0;
   set<string>      filter_tags;
   set<string>      select_authors;     ///< list of authors to include, posts not by this author are filtered
   set<string>      select_tags;        ///< list of tags to include, posts without these tags are filtered
   uint32_t         truncate_body = 0;  ///< the number of bytes of the post body to return, 0 for all
   optional<string> start_author;
   optional<string> start_permlink;
   optional<string> parent_author;
   optional<string> parent_permlink;
};
```

<br/>Essentially, this is a structure that is used by a lot of the API functions, but functions only take what they need, so some properties will be ignored even if you provided data. 

# A JavaScript Example
Here's an example of how a call into `getState()` can be followed by a call into `getDiscussionsByTrending()`:

```javascript
let index = 'trending'
let tag = 'steemdev'

steem.api.getStateAsync(`${index}/${tag}`)
  .then(function (o) {
    let posts = o.discussion_idx[tag][index]
    let last = o.content[_.last(posts)]

    let query = {
      tag: tag,
      limit: 20, 
      start_author: last.author,
      start_permlink: last.permlink
    }

    steem.api.getDiscussionsByTrendingAsync(query) 
      .then(r => console.log(JSON.stringify(r,null,2)))
        .catch(console.log)          
      })
  })
  .catch(console.log)
```

<br/>_Note: As @pilcrow points out in his [Angular tutorial](/@pilcrow/tutorial-show-steemit-posts-in-your-angular-app) that also introduces the `getDiscussionsBy*` functions, the function itself returns a promise. I'm a creature of habit, so that's why my code shows `*Async()` in the function name because it explicitly tells me that the function was promisified by Bluebird.js under the covers (and that's a Bluebird convention). You'll get the same result with or without `Async` in the function name if you follow the function call with a `.then()`._

So, what's happening here? Well, first we get the top 20 posts of `trending/steemdev` using `getState()`. The sorted list of posts will be in the `.discussion_idx.steemdev.trending` property (a string array of permlinks):

```javascript
"discussion_idx": {
  "steemdev": {
    "category": "",
    "trending": [
      "good-karma/esteem-filters-community-input-om0vmqj9sw",
      "ausbitbank/steemvids-alpha-update",
      "steemreports/steemreports-outgoing-votes-analysis-tool",
      "ontofractal/glasnost-v0-12-released-now-with-postgresql-realtime-and-7-day-lookback-comments-data-sync-open-source-app-server-for-steem",
      "morning/just-another-wordpress-steem-plugin-also-introducing-steemeasy-com",
      "good-karma/good-karma-witness-update-22nd-july-2017-ulf0cx9y6o",
      "almost-digital/dsteem-playground",
      "rycharde/proposal-for-new-rules-regarding-self-votes-and-voting-rings",
      "adept/steemh-com-a-hacker-news-styled-interface-for-steemit",
      "steepshot/the-practice-of-programming-using-graphene-based-blockchains",
      "davidk/steemphp-v0-2-released",
      "djvidov/osteem-chrome-extension-it-s-alive-and-need-20-40-alpha-testers",
      "good-karma/esteem-calling-for-volunteer-translators-16-get-reward-lqpst47n77",
      "calamus056/extensive-curation-stats-overview-since-hf19-june-20th-2017",
      "dez1337/steemj-v0-3-1-has-been-released-update-13",
      "recrypto/wordpress-steem-1-0-2",
      "klye/klye-witness-update-07-22-2017",
      "freyman/esteem-steem-no-mobil-preguntas-frecuentes-faq",
      "djvidov/osteem-first-alpha-version-its-almost-ready-its-time-to-create-an-chrome-developer-account",
      "jfollas/write-a-steemit-web-app-part-8-retrieving-content-with-getstate"
    ],
    "payout": [],
    "payout_comments": [],
    "trending30": [],
    "updated": [],
    "created": [],
    "responses": [],
    "active": [],
    "votes": [],
    "maturing": [],
    "best": [],
    "hot": [],
    "promoted": [],
    "cashout": []
  }
}
```

<br/>From this array, the code picks the last one to use as the `start_author` and `start_permlink` for the continuation list of the next 20. 

Note that the maximum limit is 100, so there's no real reason to only retrieve 20 posts at a time if you need more. Also note that the results of the `getDiscussionsByTrending()` call will start with the last post that you already have - so be prepared to handle duplicate data if you are going to merge the results of the two calls.

Unlike `getState()`, the `getDiscussionsByTrending()` function will return just an array of posts - if you need the Author's Account metadata, etc, then you will need to make subsequent API calls to fetch that data. 

Also, while `getState()` truncates the post body at 1024 characters, `getDiscussionsBy*` will not truncate the body unless you provide a `truncate_body` value in the query struct.

## What about Comments?
Stay tuned - I'll cover how to retrieve (and post) Comments next time.

![javascriptlogo.png](https://steemitimages.com/DQmZBbDGvKqM8V46NHaXoT7nZvrXhHLtTysrhixRu8eS87g/javascriptlogo.png)

_(Next Post: [Part 10](/@jfollas/write-a-steemit-web-app-part-10-retrieving-comments-with-getcontentreplies))_
👍  , , , , , , , , , , , , , , , , ,
properties (23)
authorjfollas
permlinkwrite-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby
categorysteemdev
json_metadata{"tags":["steemdev","developer","javascript","steemjs","steem-dev"],"users":["pilcrow"],"image":["https://steemitimages.com/DQmZBbDGvKqM8V46NHaXoT7nZvrXhHLtTysrhixRu8eS87g/javascriptlogo.png"],"links":["/@jfollas/write-a-steemit-web-app-part-8-retrieving-content-with-getstate","https://github.com/steemit/condenser","https://github.com/steemit/steem/blob/master/libraries/app/include/steemit/app/database_api.hpp","https://github.com/steemit/steem/blob/master/libraries/app/include/steemit/app/database_api.hpp#L83","/@pilcrow/tutorial-show-steemit-posts-in-your-angular-app","/@jfollas/write-a-steemit-web-app-part-10-retrieving-comments-with-getcontentreplies"],"app":"steemit/0.1","format":"markdown"}
created2017-07-28 03:30:36
last_update2017-08-02 02:25:24
depth0
children10
last_payout2017-08-04 03:30:36
cashout_time1969-12-31 23:59:59
total_payout_value85.921 HBD
curator_payout_value27.672 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length8,769
author_reputation4,351,701,088,490
root_title"Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd0
post_id9,977,977
net_rshares31,032,418,543,893
author_curate_reward""
vote details (18)
@bikash-tutor ·
Nice post. Upvoted and following. Please visit my blog @bikash-tutor and upvote and follow. Thank you.
👎  
properties (23)
authorbikash-tutor
permlinkre-jfollas-write-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby-20170728t033311808z
categorysteemdev
json_metadata{"tags":["steemdev"],"users":["bikash-tutor"],"app":"steemit/0.1"}
created2017-07-28 03:33:09
last_update2017-07-28 03:33:09
depth1
children0
last_payout2017-08-04 03:33:09
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length102
author_reputation4,579,993,214,824
root_title"Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id9,978,147
net_rshares-261,210,713
author_curate_reward""
vote details (1)
@mkt ·
$0.05
Your posts are very helpful, since the documentation is not very comprehensive yet. Thank you! I'm having a try to built a simple [jqurey plugin](https://github.com/mktcode/steemit-posts) to display posts and profile information on websites. News version is already on it's way. Maybe you want to have a look at it. :)
👍  
properties (23)
authormkt
permlinkre-jfollas-write-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby-20170728t085844274z
categorysteemdev
json_metadata{"tags":["steemdev"],"links":["https://github.com/mktcode/steemit-posts"],"app":"steemit/0.1"}
created2017-07-28 08:58:45
last_update2017-07-28 08:58:45
depth1
children1
last_payout2017-08-04 08:58:45
cashout_time1969-12-31 23:59:59
total_payout_value0.034 HBD
curator_payout_value0.011 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length318
author_reputation45,513,283,519,678
root_title"Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id9,999,428
net_rshares12,655,452,777
author_curate_reward""
vote details (1)
@sdavignon ·
Your link is broken. 

![giphy.gif](https://steemitimages.com/DQmZ4zmjNSdPzKEAqdGQfWPj5A98NSr57w4wdTxzDtYvpUf/giphy.gif)
properties (22)
authorsdavignon
permlinkre-mkt-re-jfollas-write-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby-20180214t102334247z
categorysteemdev
json_metadata{"tags":["steemdev"],"image":["https://steemitimages.com/DQmZ4zmjNSdPzKEAqdGQfWPj5A98NSr57w4wdTxzDtYvpUf/giphy.gif"],"app":"steemit/0.1"}
created2018-02-14 10:23:33
last_update2018-02-14 10:23:33
depth2
children0
last_payout2018-02-21 10:23:33
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length120
author_reputation202,327,117,878
root_title"Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id37,446,355
net_rshares0
@money-dreamer ·
Great documentation! I can't wait to read all the posts in this series. I'm currently working on some NodeJS Steemit bots & articles.
properties (22)
authormoney-dreamer
permlinkre-jfollas-write-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby-20171217t225407610z
categorysteemdev
json_metadata{"tags":["steemdev"],"app":"steemit/0.1"}
created2017-12-17 22:54:06
last_update2017-12-17 22:54:06
depth1
children0
last_payout2017-12-24 22:54:06
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length133
author_reputation4,495,208,779,741
root_title"Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id23,940,032
net_rshares0
@neuromancer ·
I wish I had the time to experiment in interacting with steemit via API calls and (attempting) scripted routines, I'll happily settle for reading your articles with keen interest.

Some things I'd really like to see would be, for one, a tag explorer that was aware of votes/views for various tags and able to suggest the best tags to apply to an article based on a selection of keywords.

I'm really looking forward to your next post about retrieving and posting comments, it's giving me hope that there will someday be a means of checking comments at submission time to see if it's:
* very short/generic XOR
* clone of another comment AND
* Nth clone of X comment in Y period

Then warning the submitter of potential reputation consequences should they choose to proceed, the second 2 conditions being by far more important. That's the other one I'd *really* like to see.
properties (22)
authorneuromancer
permlinkre-jfollas-write-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby-20170728t074055777z
categorysteemdev
json_metadata{"tags":["steemdev"],"app":"steemit/0.1"}
created2017-07-28 07:41:06
last_update2017-07-28 07:41:06
depth1
children2
last_payout2017-08-04 07:41:06
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length872
author_reputation865,641,939,516
root_title"Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id9,994,451
net_rshares0
@jfollas ·
Not sure that out of the box API would support your type of query. You'd probably have to use one of the database implementations (blockchain/transactions stored in a searchable database) because you're looking to find matches across the comments from a multitude of posts at a time.
properties (22)
authorjfollas
permlinkre-neuromancer-re-jfollas-write-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby-20170730t145341449z
categorysteemdev
json_metadata{"tags":["steemdev"],"app":"steemit/0.1"}
created2017-07-30 14:53:39
last_update2017-07-30 14:53:39
depth2
children1
last_payout2017-08-06 14:53:39
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length283
author_reputation4,351,701,088,490
root_title"Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id10,226,146
net_rshares0
@neuromancer ·
An API call would go to steemit from an external source, but what I'm describing probably cannot be achieved by anything other than steemit server side, essentially something along these lines (and please excuse the faux verbs/functions):

FOR `user` submitting a comment, return an array of `comments`,`datetimestamp` WHERE `datediff(now,12H)` AND `string(exact(comment))`,  count all perfect matches of current comment, return `count, time, string`

`user`, this is the `count` time you have posted the exact phrase "`string`" in `time`, please consider a less generic response, steemians thrive on original content & meaningful contributions, proceeding with this comment may negatively impact your reputation.
[POST]() &nbsp;&nbsp;&nbsp; [EDIT]()

EG:
[copypastamasta](), this is the [11th]() time you have posted the exact phrase "[good post, I upvote please upvote me]()" in 37 minutes, please consider a less generic response, steemians thrive on original content & meaningful contributions, proceeding with this comment may negatively impact your reputation.
[POST]() &nbsp;&nbsp;&nbsp; [EDIT]()

Not something within your purview, I know, more just a hypothetical item for discussion.
properties (22)
authorneuromancer
permlinkre-jfollas-re-neuromancer-re-jfollas-write-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby-20170731t071946486z
categorysteemdev
json_metadata{"tags":["steemdev"],"app":"steemit/0.1"}
created2017-07-31 07:19:48
last_update2017-07-31 07:19:48
depth3
children0
last_payout2017-08-07 07:19:48
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length1,193
author_reputation865,641,939,516
root_title"Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id10,293,636
net_rshares0
@pilcrow ·
Ah, I never realized *Async is just a convention for Bluebird promises. I might as well start using that too then. It makes sense that it's nice to see at a glance whether a method returns a promise or not, just like the $ suffix for Observables that many people use.
properties (22)
authorpilcrow
permlinkre-jfollas-write-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby-20170808t205225153z
categorysteemdev
json_metadata{"tags":["steemdev"],"app":"steemit/0.1"}
created2017-08-08 20:52:24
last_update2017-08-08 20:52:24
depth1
children1
last_payout2017-08-15 20:52:24
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length267
author_reputation2,531,070,549,481
root_title"Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id11,214,284
net_rshares0
@jfollas ·
$0.08
It's at least a convention for Bluebird's `Promise.promisifyAll()` feature to turn functions with Node-style callbacks into promises:

http://bluebirdjs.com/docs/api/promise.promisifyall.html
👍  
properties (23)
authorjfollas
permlinkre-pilcrow-re-jfollas-write-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby-20170809t011346806z
categorysteemdev
json_metadata{"tags":["steemdev"],"links":["http://bluebirdjs.com/docs/api/promise.promisifyall.html"],"app":"steemit/0.1"}
created2017-08-09 01:13:48
last_update2017-08-09 01:13:48
depth2
children0
last_payout2017-08-16 01:13:48
cashout_time1969-12-31 23:59:59
total_payout_value0.063 HBD
curator_payout_value0.020 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length191
author_reputation4,351,701,088,490
root_title"Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id11,233,634
net_rshares23,181,777,527
author_curate_reward""
vote details (1)
@thescubageek ·
Thanks for your work digging into this. One thing that I'm hitting my head against is how to retrieve all posts for a given day... right now I'm iterating through the results of `get_discussions_by_created` until I'm in the correct time range, but this takes an extremely long time for any dates longer than a few days ago since it has to retrieve a large amount of unnecessary discussions until we hit the correct timeframe. Do you have any ideas on how to make this more efficient?
properties (22)
authorthescubageek
permlinkre-jfollas-write-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby-20180113t080230608z
categorysteemdev
json_metadata{"tags":["steemdev"],"app":"steemit/0.1"}
created2018-01-13 08:02:39
last_update2018-01-13 08:02:39
depth1
children0
last_payout2018-01-20 08:02:39
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length483
author_reputation2,455,841,396,924
root_title"Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id29,178,472
net_rshares0