create account

AskSteem v1.2 "Angelfish" Update by thekyle

View this thread on: hive.blogpeakd.comecency.com
· @thekyle · (edited)
$50.83
AskSteem v1.2 "Angelfish" Update
Hello everyone, today Iโ€™m releasing v1.2 of both the AskSteem API and AskSteem Web UI which consists of many backend improvements to the AskSteem Core.

# What is AskSteem?
AskSteem is the steem search engine, you can find more info about it [here](https://steemit.com/steemit/@thekyle/introducing-asksteem-a-steem-search-engine).

# API Updates
## Adjustable Ranking Weights
There are two new parameters available for the AskSteem Search API which are `quality_weight` and `relevancy_weight`. These new parameters allow API users to customize the AskSteem ranking algorithm to better suit their needs.

To understand what these new options do I will briefly summarize how our ranking algorithm works. When users search on AskSteem we try to find the results that are most relevant but still high quality. To do this we utilize many different ranking factors that fall into two categories: quality or relevancy. By default we give each of these categories a weight of 1, however, with these new API options, you can choose to weigh one higher than the other. For example, if you wanted to ensure the highest quality results make it to the top you could set `relevancy_weight=1&quality_weight=3`, or if you wanted more relevant results you could set `relevancy_weight=3&quality_weight=1`.

While most use cases will not require you to change these values we are making them available for special cases. Additionally, please note that relying too heavily on relevancy will make your search results susceptible to keyword stuffing. 

## Related Posts API
There is also a new API endpoint available that allows developers to easily find posts related to an existing post by providing its author and permlink.

`https://api.asksteem.com/related?author=thekyle&permlink=introducing-asksteem-a-steem-search-engine`

It also works for users

`https://api.asksteem.com/related?user=thekyle` 

# Blockchain Crawler Updates
In an update to AskSteem Core (the core technology that is utilized by all AskSteem services), we have updated our blockchain crawler. This is the software that syncs our search index with that actual data on the blockchain. 

The previous version of the AskSteem crawler worked by periodically scanning old posts to check if the content had changed in any way such as new upvotes or comments. While this worked well when AskSteem was launched the blockchain has grown significantly and this is no longer a viable strategy. 

The new version of the AskSteem crawler uses a technique called block linking. Where it scans the head block for activities that link back to previous blocks in the blockchain in order to update the data, such as posts, created in those previous blocks. This is a much more efficient method for keeping our index in sync with the blockchain and will result in more accurate information in search results. We are also working on applying this crawling technique to our user crawler. 

# Introducing AuthorRank
AuthorRank is a blockchain implementation of Googleโ€™s PageRank algorithm. Currently, we are testing it only on user search results but in the future, we will likely use this as a quality factor in post rankings too. 

Right now, AuthorRank works by counting each follower of any steem user as a vote recommending that user. However, not all followers are given the same voting weight. Although our actual algorithm differs a good way to estimate the AuthorRank voting power of any given follower is to divide their user reputation by the number of accounts they are following. This means that followers with low reputations or who are following lots of accounts will have a smaller individual vote compared to accounts with high reputation or who are following few accounts.

# AskSteem Web UI Updates
Over the past few days, I have completely rewritten the AskSteem Web UI from the ground up. Originally when AskSteem launched there was no API available so the Web UI needed to house all of the components required to run AskSteem Search including the ranking algorithm, logic, formatting, suggestions, etc. 

Later I decided to fork large portions of the AskSteem Web UI codebase to create the AskSteem Search API, however, this meant I had a lot of duplicate code between the two services.  Iโ€™ve had plans to migrate the AskSteem Web UI to using the AskSteem Search API for a few months but needed the API to reach feature parody before that could be done. In the AskSteem Search API v1.1 feature parody was achieved between the two services and I began purging the AskSteem Web UIโ€™s codebase of duplicate code and replacing it with calls to the AskSteem Search API. In AskSteem Web UI v1.2 there is none of the original code used for searching and everything gets passed to the AskSteem API.

# Version Codenames
I've also decided to make referencing AskSteem versions easier I will be adding a codename for each major version release which will be a type of fish. For this release, the codename is "Angelfish".
๐Ÿ‘  , , , , , , , , , , , , , , , , , , , , , , , , , ,
properties (23)
authorthekyle
permlinkasksteem-v1-2-angelfish-update
categoryasksteem
json_metadata{"tags":["asksteem","steemit","steem","search","steemtools"],"app":"steemit/0.1","format":"markdown","links":["https://steemit.com/steemit/@thekyle/introducing-asksteem-a-steem-search-engine"]}
created2017-09-21 17:09:33
last_update2017-09-21 17:14:15
depth0
children11
last_payout2017-09-28 17:09:33
cashout_time1969-12-31 23:59:59
total_payout_value38.346 HBD
curator_payout_value12.486 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length4,963
author_reputation4,705,240,024,157
root_title"AskSteem v1.2 "Angelfish" Update"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id15,541,444
net_rshares19,374,983,397,561
author_curate_reward""
vote details (27)
@codewithcheese · (edited)
Thanks for building this tool. Earlier today I tried a search 'monero' I was expecting to find some interesting analysis of Monero. Instead the results seem to heavily favor the single word Monero titles.

https://www.asksteem.com/search?q=monero

I have developed search before and understand how hard it is, I will hang in there while you perfect it. You might benefit from more user testing.
properties (22)
authorcodewithcheese
permlinkre-thekyle-asksteem-v1-2-angelfish-update-20170921t174312073z
categoryasksteem
json_metadata{"tags":["asksteem"],"links":["https://www.asksteem.com/search?q=monero"],"app":"steemit/0.1"}
created2017-09-21 17:43:12
last_update2017-09-21 17:45:18
depth1
children4
last_payout2017-09-28 17:43:12
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length394
author_reputation2,076,698,718,573
root_title"AskSteem v1.2 "Angelfish" Update"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id15,544,198
net_rshares0
@thekyle · (edited)
Thanks for the feedback, in order to implement some of the new API features we had to change our ranking algorithm which is likely why those results are such poor quality, even compared to previous versions of AskSteem. For the next update, I'll focus more on improving that.
๐Ÿ‘  
properties (23)
authorthekyle
permlinkre-codewithcheese-re-thekyle-asksteem-v1-2-angelfish-update-20170921t175313256z
categoryasksteem
json_metadata{"tags":["asksteem"],"app":"steemit/0.1"}
created2017-09-21 17:53:12
last_update2017-09-21 17:54:09
depth2
children3
last_payout2017-09-28 17:53:12
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length275
author_reputation4,705,240,024,157
root_title"AskSteem v1.2 "Angelfish" Update"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id15,544,910
net_rshares5,710,905,499
author_curate_reward""
vote details (1)
@codewithcheese ·
Ok thanks, Ill keep it in mind when I think of something to search.
properties (22)
authorcodewithcheese
permlinkre-thekyle-re-codewithcheese-re-thekyle-asksteem-v1-2-angelfish-update-20170921t175607967z
categoryasksteem
json_metadata{"tags":["asksteem"],"app":"steemit/0.1"}
created2017-09-21 17:56:09
last_update2017-09-21 17:56:09
depth3
children2
last_payout2017-09-28 17:56:09
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length67
author_reputation2,076,698,718,573
root_title"AskSteem v1.2 "Angelfish" Update"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id15,545,172
net_rshares0
@defreca ·
https://s26.postimg.org/3kassc0ft/U5dr_TZb_CJRia_MNVmeqoja_LLS4_Acdxx_Y.gif
properties (22)
authordefreca
permlinkre-thekyle-asksteem-v1-2-angelfish-update-20170921t172146799z
categoryasksteem
json_metadata{"tags":["asksteem"],"image":["https://s26.postimg.org/3kassc0ft/U5dr_TZb_CJRia_MNVmeqoja_LLS4_Acdxx_Y.gif"],"app":"steemit/0.1"}
created2017-09-21 17:21:48
last_update2017-09-21 17:21:48
depth1
children0
last_payout2017-09-28 17:21:48
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length75
author_reputation-435,672,531,499
root_title"AskSteem v1.2 "Angelfish" Update"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id15,542,456
net_rshares0
@hodorhodor ·
Thanks to provide Great New and Useful Information
properties (22)
authorhodorhodor
permlinkre-thekyle-asksteem-v1-2-angelfish-update-20170921t172052553z
categoryasksteem
json_metadata{"tags":["asksteem"],"app":"steemit/0.1"}
created2017-09-21 17:20:57
last_update2017-09-21 17:20:57
depth1
children0
last_payout2017-09-28 17:20:57
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length50
author_reputation-18,132,307,114
root_title"AskSteem v1.2 "Angelfish" Update"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id15,542,367
net_rshares0
@mkt · (edited)
Great work you're doing there! Thank you very much! I just wonder how up-to-date the data is. How often does the crawler run and can I use asksteem to implement mention notifications in my project?
๐Ÿ‘  
properties (23)
authormkt
permlinkre-thekyle-asksteem-v1-2-angelfish-update-20170926t155017249z
categoryasksteem
json_metadata{"tags":["asksteem"],"app":"steemit/0.1"}
created2017-09-26 15:50:18
last_update2017-09-26 15:51:24
depth1
children1
last_payout2017-10-03 15:50:18
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length197
author_reputation45,513,283,519,678
root_title"AskSteem v1.2 "Angelfish" Update"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id15,996,086
net_rshares1,799,706,307
author_curate_reward""
vote details (1)
@thekyle · (edited)
As was mentioned in the post we've just deployed a brand new crawler for updating posts. New posts are indexed within 5 minutes of being posted, old posts are updated whenever new activity such as comments, upvotes, etc. occur. Feel free to use our API in your service you can contact me on steemit.chat if you have any questions.
properties (22)
authorthekyle
permlinkre-mkt-re-thekyle-asksteem-v1-2-angelfish-update-20170928t185225383z
categoryasksteem
json_metadata{"tags":["asksteem"],"app":"steemit/0.1"}
created2017-09-28 18:52:24
last_update2017-09-28 18:54:12
depth2
children0
last_payout2017-10-05 18:52:24
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length330
author_reputation4,705,240,024,157
root_title"AskSteem v1.2 "Angelfish" Update"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id16,208,697
net_rshares0
@sadekj ·
$1.71
Hey, @thekyle. Upvoted! and resteemed!

- Thank you very much for such a great tool. @biophil introduced me to asksteem.com a week ago, and since then, I use it a lot. 

- Steemit lacks a good search engine. Google search is not good enough. It has many shortcomings. AskSteem is helping a lot to fill this gap. Moreover, AskSteem can serve as an archive for steem. Steemit [does not have history](https://steemit.com/steemit/@sadekj/we-need-history-in-steemit). So, you can use AskSteem for that.

- However, **I think it is not getting the support it deserves from Steemit and the community**. I am sure that running such a service is costing a lot. So, instead of being rewarded for your efforts, you are probably spending some money.

- AskSteem is great. Yet, it is still not perfect. I have some **suggestions** based on my experience with it:

1- We need a GUI for advanced search. most steemians are not technical, and they would not be able to use many of the advanced features. So, a GUI would be very much appreciated. 

2- Until that is ready, please put a cheat sheet on the site.

3- We need to specify if the search is in a post or in a comment. Sometimes, I want to search only in comments.

4- We need to be able to use (>=  and <=) with dates. currently you can specify a fixed day or a range. 

5- We need to be able to search by reward amount.

6- We need sorting options. Sort by: relevance, date, votes, author_reputation,  ... etc.

Again, thanks a lot for such a great tool.
๐Ÿ‘  , , , ,
properties (23)
authorsadekj
permlinkre-thekyle-asksteem-v1-2-angelfish-update-20170922t052130217z
categoryasksteem
json_metadata{"tags":["asksteem"],"users":["thekyle","biophil"],"links":["https://steemit.com/steemit/@sadekj/we-need-history-in-steemit"],"app":"steemit/0.1"}
created2017-09-22 05:21:30
last_update2017-09-22 05:21:30
depth1
children1
last_payout2017-09-29 05:21:30
cashout_time1969-12-31 23:59:59
total_payout_value1.282 HBD
curator_payout_value0.423 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length1,498
author_reputation5,340,306,134,692
root_title"AskSteem v1.2 "Angelfish" Update"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id15,585,489
net_rshares621,886,892,367
author_curate_reward""
vote details (5)
@thekyle ·
Thanks for the feedback. An advanced search page has been a commonly requested feature so its definitely on the to-do list. The AskSteem API already supports [sorting by specific fields](https://steemit.com/asksteem/@thekyle/asksteem-api-v1-1-update-user-search-includes-and-sorting) so it should not be too difficult to add that functionality to the Web UI.
properties (22)
authorthekyle
permlinkre-sadekj-re-thekyle-asksteem-v1-2-angelfish-update-20170923t162451144z
categoryasksteem
json_metadata{"tags":["asksteem"],"links":["https://steemit.com/asksteem/@thekyle/asksteem-api-v1-1-update-user-search-includes-and-sorting"],"app":"steemit/0.1"}
created2017-09-23 16:24:51
last_update2017-09-23 16:24:51
depth2
children0
last_payout2017-09-30 16:24:51
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length358
author_reputation4,705,240,024,157
root_title"AskSteem v1.2 "Angelfish" Update"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id15,721,278
net_rshares0