_(Previous Post: [Part 8](/@jfollas/write-a-steemit-web-app-part-8-retrieving-content-with-getstate))_ Previously, I introduced the `getState()` API function that is used by the [Condenser](https://github.com/steemit/condenser) app (a.k.a., Steemit.com) to retrieve content for a given URL path, such as `'trending/steemdev'`. That function is great in order to get a page-worth of posts and other information, such as Account objects and the current feed price, all in one call. But, in order to retrieve more than just the first 20 posts, you must use a different API function. Today, we'll look into the various `getDiscussionsBy*` functions. ## The `get_discussions_by` API Functions A lot of the documentation for the Steem API only exists in the form of source code - that's part of the reason why I started this series of posts (to document what I find as I explored the source code while trying to figure out how the API works). The [database_api header file](https://github.com/steemit/steem/blob/master/libraries/app/include/steemit/app/database_api.hpp) is a good place to start, since header files tend to be better documented than the rest of the source. In that file, you'll find a group of functions with the same signature: ```c vector<discussion> get_discussions_by_payout(const discussion_query& query )const; vector<discussion> get_discussions_by_trending( const discussion_query& query )const; vector<discussion> get_discussions_by_created( const discussion_query& query )const; vector<discussion> get_discussions_by_active( const discussion_query& query )const; vector<discussion> get_discussions_by_cashout( const discussion_query& query )const; vector<discussion> get_discussions_by_votes( const discussion_query& query )const; vector<discussion> get_discussions_by_children( const discussion_query& query )const; vector<discussion> get_discussions_by_hot( const discussion_query& query )const; vector<discussion> get_discussions_by_feed( const discussion_query& query )const; vector<discussion> get_discussions_by_blog( const discussion_query& query )const; vector<discussion> get_discussions_by_comments( const discussion_query& query )const; vector<discussion> get_discussions_by_promoted( const discussion_query& query )const; ``` <br/>_Note: In the C API, snake_case is used, while Steem.js uses camelCase. Simply remove the underscores and capitalize each subsequent word to figure out what the function name needs to be for JavaScript._ So, let's say that you used `getState('trending/steemdev')` to retrieve the first 20 posts, and you want to fetch the next 20. How would you do that with one of these API functions? First, you need to know which database index you were really using for the `getState()` call. From a pure data point of view, you can check the data returned by `getState()` to see which array under `discussion_idx` is populated. In this case, it was `'trending'`, so it's pretty easy to figure out that you probably want the `get_discussions_by_trending(query)` function (or, rather, `getDiscussionsByTrending(query)` in JavaScript). But, what's that `query` argument all about? ## `discussion_query` Struct Again, referring to [the header file](https://github.com/steemit/steem/blob/master/libraries/app/include/steemit/app/database_api.hpp#L83), we see that `discussion_query` is defined as: ```c /** * Defines the arguments to a query as a struct so it can be easily extended */ struct discussion_query { void validate()const{ FC_ASSERT( filter_tags.find(tag) == filter_tags.end() ); FC_ASSERT( limit <= 100 ); } string tag; uint32_t limit = 0; set<string> filter_tags; set<string> select_authors; ///< list of authors to include, posts not by this author are filtered set<string> select_tags; ///< list of tags to include, posts without these tags are filtered uint32_t truncate_body = 0; ///< the number of bytes of the post body to return, 0 for all optional<string> start_author; optional<string> start_permlink; optional<string> parent_author; optional<string> parent_permlink; }; ``` <br/>Essentially, this is a structure that is used by a lot of the API functions, but functions only take what they need, so some properties will be ignored even if you provided data. # A JavaScript Example Here's an example of how a call into `getState()` can be followed by a call into `getDiscussionsByTrending()`: ```javascript let index = 'trending' let tag = 'steemdev' steem.api.getStateAsync(`${index}/${tag}`) .then(function (o) { let posts = o.discussion_idx[tag][index] let last = o.content[_.last(posts)] let query = { tag: tag, limit: 20, start_author: last.author, start_permlink: last.permlink } steem.api.getDiscussionsByTrendingAsync(query) .then(r => console.log(JSON.stringify(r,null,2))) .catch(console.log) }) }) .catch(console.log) ``` <br/>_Note: As @pilcrow points out in his [Angular tutorial](/@pilcrow/tutorial-show-steemit-posts-in-your-angular-app) that also introduces the `getDiscussionsBy*` functions, the function itself returns a promise. I'm a creature of habit, so that's why my code shows `*Async()` in the function name because it explicitly tells me that the function was promisified by Bluebird.js under the covers (and that's a Bluebird convention). You'll get the same result with or without `Async` in the function name if you follow the function call with a `.then()`._ So, what's happening here? Well, first we get the top 20 posts of `trending/steemdev` using `getState()`. The sorted list of posts will be in the `.discussion_idx.steemdev.trending` property (a string array of permlinks): ```javascript "discussion_idx": { "steemdev": { "category": "", "trending": [ "good-karma/esteem-filters-community-input-om0vmqj9sw", "ausbitbank/steemvids-alpha-update", "steemreports/steemreports-outgoing-votes-analysis-tool", "ontofractal/glasnost-v0-12-released-now-with-postgresql-realtime-and-7-day-lookback-comments-data-sync-open-source-app-server-for-steem", "morning/just-another-wordpress-steem-plugin-also-introducing-steemeasy-com", "good-karma/good-karma-witness-update-22nd-july-2017-ulf0cx9y6o", "almost-digital/dsteem-playground", "rycharde/proposal-for-new-rules-regarding-self-votes-and-voting-rings", "adept/steemh-com-a-hacker-news-styled-interface-for-steemit", "steepshot/the-practice-of-programming-using-graphene-based-blockchains", "davidk/steemphp-v0-2-released", "djvidov/osteem-chrome-extension-it-s-alive-and-need-20-40-alpha-testers", "good-karma/esteem-calling-for-volunteer-translators-16-get-reward-lqpst47n77", "calamus056/extensive-curation-stats-overview-since-hf19-june-20th-2017", "dez1337/steemj-v0-3-1-has-been-released-update-13", "recrypto/wordpress-steem-1-0-2", "klye/klye-witness-update-07-22-2017", "freyman/esteem-steem-no-mobil-preguntas-frecuentes-faq", "djvidov/osteem-first-alpha-version-its-almost-ready-its-time-to-create-an-chrome-developer-account", "jfollas/write-a-steemit-web-app-part-8-retrieving-content-with-getstate" ], "payout": [], "payout_comments": [], "trending30": [], "updated": [], "created": [], "responses": [], "active": [], "votes": [], "maturing": [], "best": [], "hot": [], "promoted": [], "cashout": [] } } ``` <br/>From this array, the code picks the last one to use as the `start_author` and `start_permlink` for the continuation list of the next 20. Note that the maximum limit is 100, so there's no real reason to only retrieve 20 posts at a time if you need more. Also note that the results of the `getDiscussionsByTrending()` call will start with the last post that you already have - so be prepared to handle duplicate data if you are going to merge the results of the two calls. Unlike `getState()`, the `getDiscussionsByTrending()` function will return just an array of posts - if you need the Author's Account metadata, etc, then you will need to make subsequent API calls to fetch that data. Also, while `getState()` truncates the post body at 1024 characters, `getDiscussionsBy*` will not truncate the body unless you provide a `truncate_body` value in the query struct. ## What about Comments? Stay tuned - I'll cover how to retrieve (and post) Comments next time.  _(Next Post: [Part 10](/@jfollas/write-a-steemit-web-app-part-10-retrieving-comments-with-getcontentreplies))_
author | jfollas |
---|---|
permlink | write-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby |
category | steemdev |
json_metadata | {"tags":["steemdev","developer","javascript","steemjs","steem-dev"],"users":["pilcrow"],"image":["https://steemitimages.com/DQmZBbDGvKqM8V46NHaXoT7nZvrXhHLtTysrhixRu8eS87g/javascriptlogo.png"],"links":["/@jfollas/write-a-steemit-web-app-part-8-retrieving-content-with-getstate","https://github.com/steemit/condenser","https://github.com/steemit/steem/blob/master/libraries/app/include/steemit/app/database_api.hpp","https://github.com/steemit/steem/blob/master/libraries/app/include/steemit/app/database_api.hpp#L83","/@pilcrow/tutorial-show-steemit-posts-in-your-angular-app","/@jfollas/write-a-steemit-web-app-part-10-retrieving-comments-with-getcontentreplies"],"app":"steemit/0.1","format":"markdown"} |
created | 2017-07-28 03:30:36 |
last_update | 2017-08-02 02:25:24 |
depth | 0 |
children | 10 |
last_payout | 2017-08-04 03:30:36 |
cashout_time | 1969-12-31 23:59:59 |
total_payout_value | 85.921 HBD |
curator_payout_value | 27.672 HBD |
pending_payout_value | 0.000 HBD |
promoted | 0.000 HBD |
body_length | 8,769 |
author_reputation | 4,351,701,088,490 |
root_title | "Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()" |
beneficiaries | [] |
max_accepted_payout | 1,000,000.000 HBD |
percent_hbd | 0 |
post_id | 9,977,977 |
net_rshares | 31,032,418,543,893 |
author_curate_reward | "" |
voter | weight | wgt% | rshares | pct | time |
---|---|---|---|---|---|
transisto | 0 | 11,582,146,546,625 | 100% | ||
julianita | 0 | 4,467,337,199,034 | 100% | ||
dragosroua | 0 | 31,177,775,934 | 10% | ||
sdavignon | 0 | 0 | 100% | ||
comfortgenius | 0 | 1,231,324,936 | 100% | ||
soon8013 | 0 | 6,826,158,480 | 100% | ||
newsflash | 0 | 14,327,325,331,766 | 100% | ||
mkt | 0 | 19,389,306,169 | 100% | ||
staticinstance | 0 | 560,605,730,559 | 100% | ||
davidk | 0 | 4,156,248,462 | 100% | ||
jfollas | 0 | 12,838,301,011 | 100% | ||
pilcrow | 0 | 9,970,006,738 | 100% | ||
alexandruionescu | 0 | 343,954,091 | 20% | ||
bikash-tutor | 0 | 9,070,660,088 | 60% | ||
scorer | 0 | 0 | 100% | ||
kaudrius | 0 | 0 | 100% | ||
zaya | 0 | 0 | 100% | ||
pymike | 0 | 0 | 100% |
Nice post. Upvoted and following. Please visit my blog @bikash-tutor and upvote and follow. Thank you.
author | bikash-tutor |
---|---|
permlink | re-jfollas-write-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby-20170728t033311808z |
category | steemdev |
json_metadata | {"tags":["steemdev"],"users":["bikash-tutor"],"app":"steemit/0.1"} |
created | 2017-07-28 03:33:09 |
last_update | 2017-07-28 03:33:09 |
depth | 1 |
children | 0 |
last_payout | 2017-08-04 03:33:09 |
cashout_time | 1969-12-31 23:59:59 |
total_payout_value | 0.000 HBD |
curator_payout_value | 0.000 HBD |
pending_payout_value | 0.000 HBD |
promoted | 0.000 HBD |
body_length | 102 |
author_reputation | 4,579,993,214,824 |
root_title | "Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()" |
beneficiaries | [] |
max_accepted_payout | 1,000,000.000 HBD |
percent_hbd | 10,000 |
post_id | 9,978,147 |
net_rshares | -261,210,713 |
author_curate_reward | "" |
voter | weight | wgt% | rshares | pct | time |
---|---|---|---|---|---|
shalinoth | 0 | -261,210,713 | -100% |
Your posts are very helpful, since the documentation is not very comprehensive yet. Thank you! I'm having a try to built a simple [jqurey plugin](https://github.com/mktcode/steemit-posts) to display posts and profile information on websites. News version is already on it's way. Maybe you want to have a look at it. :)
author | mkt |
---|---|
permlink | re-jfollas-write-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby-20170728t085844274z |
category | steemdev |
json_metadata | {"tags":["steemdev"],"links":["https://github.com/mktcode/steemit-posts"],"app":"steemit/0.1"} |
created | 2017-07-28 08:58:45 |
last_update | 2017-07-28 08:58:45 |
depth | 1 |
children | 1 |
last_payout | 2017-08-04 08:58:45 |
cashout_time | 1969-12-31 23:59:59 |
total_payout_value | 0.034 HBD |
curator_payout_value | 0.011 HBD |
pending_payout_value | 0.000 HBD |
promoted | 0.000 HBD |
body_length | 318 |
author_reputation | 45,513,283,519,678 |
root_title | "Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()" |
beneficiaries | [] |
max_accepted_payout | 1,000,000.000 HBD |
percent_hbd | 10,000 |
post_id | 9,999,428 |
net_rshares | 12,655,452,777 |
author_curate_reward | "" |
voter | weight | wgt% | rshares | pct | time |
---|---|---|---|---|---|
jfollas | 0 | 12,655,452,777 | 100% |
Your link is broken. 
author | sdavignon |
---|---|
permlink | re-mkt-re-jfollas-write-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby-20180214t102334247z |
category | steemdev |
json_metadata | {"tags":["steemdev"],"image":["https://steemitimages.com/DQmZ4zmjNSdPzKEAqdGQfWPj5A98NSr57w4wdTxzDtYvpUf/giphy.gif"],"app":"steemit/0.1"} |
created | 2018-02-14 10:23:33 |
last_update | 2018-02-14 10:23:33 |
depth | 2 |
children | 0 |
last_payout | 2018-02-21 10:23:33 |
cashout_time | 1969-12-31 23:59:59 |
total_payout_value | 0.000 HBD |
curator_payout_value | 0.000 HBD |
pending_payout_value | 0.000 HBD |
promoted | 0.000 HBD |
body_length | 120 |
author_reputation | 202,327,117,878 |
root_title | "Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()" |
beneficiaries | [] |
max_accepted_payout | 1,000,000.000 HBD |
percent_hbd | 10,000 |
post_id | 37,446,355 |
net_rshares | 0 |
Great documentation! I can't wait to read all the posts in this series. I'm currently working on some NodeJS Steemit bots & articles.
author | money-dreamer |
---|---|
permlink | re-jfollas-write-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby-20171217t225407610z |
category | steemdev |
json_metadata | {"tags":["steemdev"],"app":"steemit/0.1"} |
created | 2017-12-17 22:54:06 |
last_update | 2017-12-17 22:54:06 |
depth | 1 |
children | 0 |
last_payout | 2017-12-24 22:54:06 |
cashout_time | 1969-12-31 23:59:59 |
total_payout_value | 0.000 HBD |
curator_payout_value | 0.000 HBD |
pending_payout_value | 0.000 HBD |
promoted | 0.000 HBD |
body_length | 133 |
author_reputation | 4,495,208,779,741 |
root_title | "Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()" |
beneficiaries | [] |
max_accepted_payout | 1,000,000.000 HBD |
percent_hbd | 10,000 |
post_id | 23,940,032 |
net_rshares | 0 |
I wish I had the time to experiment in interacting with steemit via API calls and (attempting) scripted routines, I'll happily settle for reading your articles with keen interest. Some things I'd really like to see would be, for one, a tag explorer that was aware of votes/views for various tags and able to suggest the best tags to apply to an article based on a selection of keywords. I'm really looking forward to your next post about retrieving and posting comments, it's giving me hope that there will someday be a means of checking comments at submission time to see if it's: * very short/generic XOR * clone of another comment AND * Nth clone of X comment in Y period Then warning the submitter of potential reputation consequences should they choose to proceed, the second 2 conditions being by far more important. That's the other one I'd *really* like to see.
author | neuromancer |
---|---|
permlink | re-jfollas-write-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby-20170728t074055777z |
category | steemdev |
json_metadata | {"tags":["steemdev"],"app":"steemit/0.1"} |
created | 2017-07-28 07:41:06 |
last_update | 2017-07-28 07:41:06 |
depth | 1 |
children | 2 |
last_payout | 2017-08-04 07:41:06 |
cashout_time | 1969-12-31 23:59:59 |
total_payout_value | 0.000 HBD |
curator_payout_value | 0.000 HBD |
pending_payout_value | 0.000 HBD |
promoted | 0.000 HBD |
body_length | 872 |
author_reputation | 865,641,939,516 |
root_title | "Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()" |
beneficiaries | [] |
max_accepted_payout | 1,000,000.000 HBD |
percent_hbd | 10,000 |
post_id | 9,994,451 |
net_rshares | 0 |
Not sure that out of the box API would support your type of query. You'd probably have to use one of the database implementations (blockchain/transactions stored in a searchable database) because you're looking to find matches across the comments from a multitude of posts at a time.
author | jfollas |
---|---|
permlink | re-neuromancer-re-jfollas-write-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby-20170730t145341449z |
category | steemdev |
json_metadata | {"tags":["steemdev"],"app":"steemit/0.1"} |
created | 2017-07-30 14:53:39 |
last_update | 2017-07-30 14:53:39 |
depth | 2 |
children | 1 |
last_payout | 2017-08-06 14:53:39 |
cashout_time | 1969-12-31 23:59:59 |
total_payout_value | 0.000 HBD |
curator_payout_value | 0.000 HBD |
pending_payout_value | 0.000 HBD |
promoted | 0.000 HBD |
body_length | 283 |
author_reputation | 4,351,701,088,490 |
root_title | "Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()" |
beneficiaries | [] |
max_accepted_payout | 1,000,000.000 HBD |
percent_hbd | 10,000 |
post_id | 10,226,146 |
net_rshares | 0 |
An API call would go to steemit from an external source, but what I'm describing probably cannot be achieved by anything other than steemit server side, essentially something along these lines (and please excuse the faux verbs/functions): FOR `user` submitting a comment, return an array of `comments`,`datetimestamp` WHERE `datediff(now,12H)` AND `string(exact(comment))`, count all perfect matches of current comment, return `count, time, string` `user`, this is the `count` time you have posted the exact phrase "`string`" in `time`, please consider a less generic response, steemians thrive on original content & meaningful contributions, proceeding with this comment may negatively impact your reputation. [POST]() [EDIT]() EG: [copypastamasta](), this is the [11th]() time you have posted the exact phrase "[good post, I upvote please upvote me]()" in 37 minutes, please consider a less generic response, steemians thrive on original content & meaningful contributions, proceeding with this comment may negatively impact your reputation. [POST]() [EDIT]() Not something within your purview, I know, more just a hypothetical item for discussion.
author | neuromancer |
---|---|
permlink | re-jfollas-re-neuromancer-re-jfollas-write-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby-20170731t071946486z |
category | steemdev |
json_metadata | {"tags":["steemdev"],"app":"steemit/0.1"} |
created | 2017-07-31 07:19:48 |
last_update | 2017-07-31 07:19:48 |
depth | 3 |
children | 0 |
last_payout | 2017-08-07 07:19:48 |
cashout_time | 1969-12-31 23:59:59 |
total_payout_value | 0.000 HBD |
curator_payout_value | 0.000 HBD |
pending_payout_value | 0.000 HBD |
promoted | 0.000 HBD |
body_length | 1,193 |
author_reputation | 865,641,939,516 |
root_title | "Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()" |
beneficiaries | [] |
max_accepted_payout | 1,000,000.000 HBD |
percent_hbd | 10,000 |
post_id | 10,293,636 |
net_rshares | 0 |
Ah, I never realized *Async is just a convention for Bluebird promises. I might as well start using that too then. It makes sense that it's nice to see at a glance whether a method returns a promise or not, just like the $ suffix for Observables that many people use.
author | pilcrow |
---|---|
permlink | re-jfollas-write-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby-20170808t205225153z |
category | steemdev |
json_metadata | {"tags":["steemdev"],"app":"steemit/0.1"} |
created | 2017-08-08 20:52:24 |
last_update | 2017-08-08 20:52:24 |
depth | 1 |
children | 1 |
last_payout | 2017-08-15 20:52:24 |
cashout_time | 1969-12-31 23:59:59 |
total_payout_value | 0.000 HBD |
curator_payout_value | 0.000 HBD |
pending_payout_value | 0.000 HBD |
promoted | 0.000 HBD |
body_length | 267 |
author_reputation | 2,531,070,549,481 |
root_title | "Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()" |
beneficiaries | [] |
max_accepted_payout | 1,000,000.000 HBD |
percent_hbd | 10,000 |
post_id | 11,214,284 |
net_rshares | 0 |
It's at least a convention for Bluebird's `Promise.promisifyAll()` feature to turn functions with Node-style callbacks into promises: http://bluebirdjs.com/docs/api/promise.promisifyall.html
author | jfollas |
---|---|
permlink | re-pilcrow-re-jfollas-write-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby-20170809t011346806z |
category | steemdev |
json_metadata | {"tags":["steemdev"],"links":["http://bluebirdjs.com/docs/api/promise.promisifyall.html"],"app":"steemit/0.1"} |
created | 2017-08-09 01:13:48 |
last_update | 2017-08-09 01:13:48 |
depth | 2 |
children | 0 |
last_payout | 2017-08-16 01:13:48 |
cashout_time | 1969-12-31 23:59:59 |
total_payout_value | 0.063 HBD |
curator_payout_value | 0.020 HBD |
pending_payout_value | 0.000 HBD |
promoted | 0.000 HBD |
body_length | 191 |
author_reputation | 4,351,701,088,490 |
root_title | "Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()" |
beneficiaries | [] |
max_accepted_payout | 1,000,000.000 HBD |
percent_hbd | 10,000 |
post_id | 11,233,634 |
net_rshares | 23,181,777,527 |
author_curate_reward | "" |
voter | weight | wgt% | rshares | pct | time |
---|---|---|---|---|---|
pilcrow | 0 | 23,181,777,527 | 50.51% |
Thanks for your work digging into this. One thing that I'm hitting my head against is how to retrieve all posts for a given day... right now I'm iterating through the results of `get_discussions_by_created` until I'm in the correct time range, but this takes an extremely long time for any dates longer than a few days ago since it has to retrieve a large amount of unnecessary discussions until we hit the correct timeframe. Do you have any ideas on how to make this more efficient?
author | thescubageek |
---|---|
permlink | re-jfollas-write-a-steemit-web-app-part-9-retrieving-content-with-getdiscussionsby-20180113t080230608z |
category | steemdev |
json_metadata | {"tags":["steemdev"],"app":"steemit/0.1"} |
created | 2018-01-13 08:02:39 |
last_update | 2018-01-13 08:02:39 |
depth | 1 |
children | 0 |
last_payout | 2018-01-20 08:02:39 |
cashout_time | 1969-12-31 23:59:59 |
total_payout_value | 0.000 HBD |
curator_payout_value | 0.000 HBD |
pending_payout_value | 0.000 HBD |
promoted | 0.000 HBD |
body_length | 483 |
author_reputation | 2,455,841,396,924 |
root_title | "Write a Steemit Web App: Part 9 - Retrieving Content with getDiscussionsBy*()" |
beneficiaries | [] |
max_accepted_payout | 1,000,000.000 HBD |
percent_hbd | 10,000 |
post_id | 29,178,472 |
net_rshares | 0 |