[COMPLETE] Maintenance of chainBB - indexes missing by jesta

View this thread on: hive.blog | peakd.com | ecency.com

chainbb · @jesta · May 17 '17 (edited)

$148.61

[COMPLETE] Maintenance of chainBB - indexes missing

**UPDATE**: This maintenance is now complete and the forums are fully populated again. 

----

Just as a heads up - the chainBB forums are looking pretty empty because the DB got wiped within the last 8 hours. Not sure how/why, could be that the DB was accessible somehow and someone dropped them, or that the beta environment running within docker just took a dump. I haven't spent a lot of time with the infrastructure yet because this is still one huge test, but this highlights the need for a little of my time towards it.

Luckily, nothing of value is actually stored in these indexes or the database! 

I have a huge EC2 instance on standby that needs to sync the blockchain and then rerun the indexer, then just import it over into the beta server, and we'll be right back to where we started. The beta forums should be back and 100% running again within the next few hours. 

On a separate topic, I'm hoping to add "Sticky" posts at some point this week, and hopefully add a "Decline Payouts" button to the post creation process. Both of these options would be very well suited for posts like this.

👍 arama, engagement, cryptoctopus, freeyourmind, ausbitbank, furion, sponge-bob, thecyclist, rok-sivante, teamsteem, twinner, jrcornel, lukestokes, knircky, steemservices, ajvest, inchonbitcoin, robrigo, inertia, aizensou, cryptomancer, anotherjoe, joele, steemrollin, richman, gregm, corn113, chitty, good-karma, myvoter, eric-boucher, anca3drandom, penguinpablo, barrydutton, yougotflagged, grandpere, themonetaryfew, tomino, p0o, twinner01, sebastianjago, blueorgy, ace108, theyeti, beers, kachinhenry, asmolokalo, thebotkiller, altccy, fortinbuff, nspart, judasp, jonathanyoung, giantbear, tmendieta, albertogm, piedpiper, ssekulji, bitshares101, youngkim, berniesanders, alexpmorris, steemprentice, dez1337, and 98 others

properties (23)vote details (162)

voter	rshares	pct
berniesanders	15,959,973,706	20%
ajvest	334,843,873,425	100%
steemservices	374,297,432,635	20%
aizensou	270,925,349,024	100%
masteryoda	1,021,778,770	1%
pheonike	10,027,407,414	6%
stoner19	0	100%
steemrollin	168,636,057,591	50%
chitty	159,364,503,487	100%
piedpiper	20,266,823,678	100%
carlos-cabeza	401,690,297	23%
teamsteem	641,881,223,618	15%
cryptoctopus	1,131,385,238,283	100%
richman	164,707,341,401	100%
tmendieta	24,902,934,002	100%
eric-boucher	114,594,179,966	100%
applecrisp	234,485,162	100%
rok-sivante	695,773,637,059	77%
samether	533,825,934	15%
dahaz159	415,910,640	15%
schro	1,005,179,785	100%
grandpere	76,451,288,893	100%
raphaelom	685,290,106	100%
albertogm	23,099,700,305	100%
lukestokes	505,020,197,609	100%
christoph3	386,508,357	25%
jakevanderark	116,784,514	100%
grey580	11,033,928,446	100%
michaellamden68	631,954,696	22%
asmolokalo	39,757,877,566	25%
good-karma	125,659,707,965	23%
robrigo	301,403,348,671	100%
stranger27	7,144,296,873	100%
furion	773,155,089,379	100%
bitshares101	17,793,388,906	100%
knircky	482,399,987,828	100%
ausbitbank	919,003,814,573	25%
freeyourmind	1,099,435,825,534	100%
asim	6,198,000,106	100%
bitland	1,738,985,743	100%
edrivegom	5,243,322,743	31%
gregm	163,975,436,548	100%
krystle	9,499,919,374	25%
inertia	276,955,981,395	100%
fiveboringgames	2,980,686,253	4.5%
blueorgy	49,800,932,612	100%
mysteem	353,441,044	23%
anca3drandom	101,086,211,165	100%
ace108	46,574,322,611	15%
trisnawati	5,802,057,194	100%
youngkim	16,743,507,389	100%
alexpmorris	12,838,124,313	100%
twinner	637,195,745,576	50%
dez1337	11,819,710,249	100%
timcliff	4,695,329,182	1%
transhuman	2,413,984,545	33%
letc	1,487,691,048	100%
ullikume	4,154,574,211	100%
ipumba	4,011,315,571	100%
dolov	451,068,307	6%
joele	177,725,493,097	100%
hms818	1,520,642,906	100%
thebotkiller	37,718,497,352	20%
jrcornel	632,820,066,183	100%
fooblic	4,674,799,996	99%
virtualgrowth	9,275,626,642	30%
anotherjoe	224,503,182,598	100%
gammagooblin	6,316,255,729	100%
lamech-m	698,468,282	15%
ripplerm	5,858,573,265	50%
cryptomancer	230,218,044,142	100%
leahmchenry	549,433,574	100%
inchonbitcoin	331,398,694,674	100%
themonetaryfew	71,891,907,119	100%
escapeamericanow	1,423,092,854	100%
runridefly	9,681,051,251	13%
barrydutton	85,150,607,136	100%
penguinpablo	94,767,680,501	35%
sponge-bob	736,081,279,449	100%
thecyclist	724,220,147,732	20%
jonathanyoung	28,792,327,676	100%
jeanviete	688,276,070	100%
arama	1,639,100,314,634	100%
bestofreddit	393,014,533	4.05%
thegame	1,567,095,977	30%
lloyddavis	4,636,708,346	100%
fortinbuff	30,395,869,256	100%
ssekulji	19,884,683,309	100%
steembets	1,900,057,042	30%
siniceku	541,518,245	100%
p0o	62,591,242,946	100%
thomasp	149,716,854	100%
beers	42,249,545,510	50%
tomino	62,882,500,472	100%
aurorax	1,106,353,840	100%
kachinhenry	40,395,606,514	100%
giantbear	26,174,367,207	100%
promx	141,720,571	100%
mokluc	744,482,326	100%
judasp	28,953,377,712	100%
unipsycho	7,238,550,791	50%
steemland.com	1,593,614,326	30%
selwi	677,207,217	100%
trans-juanmi	4,451,593,503	60%
steemgold	4,874,316,863	100%
steem-myanmar	495,581,671	100%
steemprentice	11,926,536,335	30%
wagnertamanaha	2,659,689,004	100%
reisman	665,626,170	23%
engagement	1,583,795,065,611	20%
irawandedy	805,291,605	23%
seablue	2,755,208,460	4%
steemquestions	8,746,030,113	100%
juliosalas	602,955,594	60%
nspart	30,274,594,876	100%
tamersameeh	235,585,088	100%
barvon	756,474,426	100%
sebastianjago	52,380,829,344	50%
steemcenterwiki	992,133,501	100%
davidcurwain	242,123,137	100%
firesteem	298,619,639	100%
kochmaster	632,178,535	50%
ogochukwu	1,178,623,951	15%
rizkiavonna	1,696,045,958	100%
tonylondon	116,740,152	15%
eusebio	567,997,140	100%
theyeti	45,658,374,358	20%
yougotflagged	79,002,806,020	20%
justinashby	394,164,100	15%
torkot	2,516,707,676	100%
evildeathcore	396,220,810	15%
bedroomshaman	258,474,664	11.11%
bookman	4,255,098,757	100%
vadbars	344,988,774	100%
corn113	162,502,842,131	100%
motivator	343,052,802	23%
thomasgutierrez	87,966,999	100%
sassal	9,794,613,039	100%
torem-di-torem	1,876,764,149	100%
myvoter	115,341,801,214	70%
syarrf	714,356,842	100%
twinner01	52,443,745,288	50%
xarchangelx	1,206,502,919	100%
altccy	35,739,946,595	100%
saifulbahri	1,170,214,982	100%
jackchalat	735,927,627	23%
jeff001	917,068,428	100%
rdickey	119,384,854	100%
froghollow	1,053,413,438	100%
reig3790	879,012,475	100%
mic1211	1,193,357,452	100%
nitinchugh	143,202,667	100%
ecuadorianin	501,207,157	100%
cryptoadvocator	525,073,601	100%
ba531	214,802,400	100%
crepa	596,671,013	100%
ayahme	1,515,370,208	100%
toodeep	1,363,832,379	100%
lautenglye	60,614,765	100%
svinsent	1,242,602,564	100%
taeya	1,485,058,643	100%
milad70	0	100%

@barvon · May 16 '17

Good job @jesta
I like it.
I like your way!!!

properties (22)

`author`	barvon
`permlink`	re-jesta-maintenance-of-chainbb-indexes-missing-20170516t182249138z
`category`	chainbb
`json_metadata`	{"tags":["chainbb"],"users":["jesta"],"app":"steemit/0.1"}
`created`	2017-05-16 18:22:57
`last_update`	2017-05-16 18:22:57
`depth`	1
`children`	0
`last_payout`	2017-05-23 18:22:57
`cashout_time`	1969-12-31 23:59:59
`total_payout_value`	0.000 HBD
`curator_payout_value`	0.000 HBD
`pending_payout_value`	0.000 HBD
`promoted`	0.000 HBD
`body_length`	45
`author_reputation`	46,236,405,475,497
`root_title`	"[COMPLETE] Maintenance of chainBB - indexes missing"
`beneficiaries`	`[]`
`max_accepted_payout`	1,000,000.000 HBD
`percent_hbd`	10,000
`post_id`	3,467,814
`net_rshares`	0

@eric-boucher · May 18 '17

$0.19

What a troopper! Great work, thanks for the major update on this one.  Namaste :)

👍 eric-boucher

`author`	eric-boucher
`permlink`	re-jesta-maintenance-of-chainbb-indexes-missing-20170518t000231339z
`category`	chainbb
`json_metadata`	{"tags":["chainbb"],"app":"steemit/0.1"}
`created`	2017-05-18 00:02:30
`last_update`	2017-05-18 00:02:30
`depth`	1
`children`	0
`last_payout`	2017-05-25 00:02:30
`cashout_time`	1969-12-31 23:59:59
`total_payout_value`	0.188 HBD
`curator_payout_value`	0.000 HBD
`pending_payout_value`	0.000 HBD
`promoted`	0.000 HBD
`body_length`	81
`author_reputation`	68,503,601,066,539
`root_title`	"[COMPLETE] Maintenance of chainBB - indexes missing"
`beneficiaries`	`[]`
`max_accepted_payout`	1,000,000.000 HBD
`percent_hbd`	10,000
`post_id`	3,510,775
`net_rshares`	114,594,179,966
`author_curate_reward`	""

properties (23)vote details (1)

voter	weight	wgt%	rshares	pct	time
eric-boucher	0 B		114,594,179,966	100%

@jesta · May 17 '17

This maintenance is now complete, forum indexes restored, snapshot saved (with todays date for backups), and the big EC2 used to regenerate the indexes is on standby for another occurrence of this.

properties (22)

@omrusman · May 20 '17

I really like all of your blogs 
and i am also following you 
can you please follow me back :D
And i am new here so can you please
upvote some of my post please 

Thanks!

properties (22)

`author`	omrusman
`permlink`	re-jesta-maintenance-of-chainbb-indexes-missing-20170520t173422970z
`category`	chainbb
`json_metadata`	{"tags":["chainbb"],"app":"steemit/0.1"}
`created`	2017-05-20 17:34:12
`last_update`	2017-05-20 17:34:12
`depth`	1
`children`	0
`last_payout`	2017-05-27 17:34:12
`cashout_time`	1969-12-31 23:59:59
`total_payout_value`	0.000 HBD
`curator_payout_value`	0.000 HBD
`pending_payout_value`	0.000 HBD
`promoted`	0.000 HBD
`body_length`	170
`author_reputation`	2,021,244,563,068
`root_title`	"[COMPLETE] Maintenance of chainBB - indexes missing"
`beneficiaries`	`[]`
`max_accepted_payout`	1,000,000.000 HBD
`percent_hbd`	10,000
`post_id`	3,604,515
`net_rshares`	0

@p0o · May 16 '17

$0.05

Why you are not using Steem to store all the information in chainBB? I thought it's already like that in your previous post. There are so many flexibility in using specific tags combined with custom meta data to format and sort content differently.

👍 jesta

`author`	p0o
`permlink`	re-jesta-maintenance-of-chainbb-indexes-missing-20170516t190816818z
`category`	chainbb
`json_metadata`	{"tags":["chainbb"],"app":"steemit/0.1"}
`created`	2017-05-16 19:08:18
`last_update`	2017-05-16 19:08:18
`depth`	1
`children`	3
`last_payout`	2017-05-23 19:08:18
`cashout_time`	1969-12-31 23:59:59
`total_payout_value`	0.036 HBD
`curator_payout_value`	0.012 HBD
`pending_payout_value`	0.000 HBD
`promoted`	0.000 HBD
`body_length`	248
`author_reputation`	8,373,104,781,065
`root_title`	"[COMPLETE] Maintenance of chainBB - indexes missing"
`beneficiaries`	`[]`
`max_accepted_payout`	1,000,000.000 HBD
`percent_hbd`	10,000
`post_id`	3,469,159
`net_rshares`	31,250,318,435
`author_curate_reward`	""

properties (23)vote details (1)

voter	weight	wgt%	rshares	pct	time
jesta	0 B		31,250,318,435	10%

@jesta · May 16 '17 (edited)

$1.79

The user data, posts and replies are all stored in Steem, but I maintain the forum indexes in an  external database for raw speed and scalability. 

Steem also doesn't offer API's for retrieving content as the forums would need it either. The forums themselves pull based on category (steemd pulls on tags) and then sorts them by when the last response occurred. It would be possible to build a custom plugin for steemd that achieved this, but that's a lot harder than using more traditional web technologies.

There's also the small matter of a steemd full node requiring upwards of 40GB of RAM to run smoothly, where as this forum index I'm running requires around 2GB. Adding a custom plugin to steemd could reduce the 40GB, but probably not a lot, as most of the normal plugins would still be required to maintain the proper data.

It's just not possible to efficiently pull the data needed with steemd at this point, so the indexes are in a database :)

Hope that helps explain why I went the route I did!

👍 furion, good-karma, alexpmorris, irawandedy, jackchalat, reisman, dahaz159, carlos-cabeza, evildeathcore, motivator

`author`	jesta
`permlink`	re-p0o-re-jesta-maintenance-of-chainbb-indexes-missing-20170516t195915398z
`category`	chainbb
`json_metadata`	{"tags":["chainbb"],"app":"steemit/0.1"}
`created`	2017-05-16 19:59:15
`last_update`	2017-05-16 20:07:39
`depth`	2
`children`	2
`last_payout`	2017-05-23 19:59:15
`cashout_time`	1969-12-31 23:59:59
`total_payout_value`	1.342 HBD
`curator_payout_value`	0.443 HBD
`pending_payout_value`	0.000 HBD
`promoted`	0.000 HBD
`body_length`	1,010
`author_reputation`	140,605,453,893,072
`root_title`	"[COMPLETE] Maintenance of chainBB - indexes missing"
`beneficiaries`	`[]`
`max_accepted_payout`	1,000,000.000 HBD
`percent_hbd`	10,000
`post_id`	3,470,719
`net_rshares`	915,417,184,530
`author_curate_reward`	""

properties (23)vote details (10)

voter	rshares	pct
carlos-cabeza	401,690,297	23%
dahaz159	415,910,640	23%
good-karma	125,660,250,887	23%
furion	773,155,089,379	100%
alexpmorris	12,838,124,313	100%
reisman	665,626,170	23%
irawandedy	805,291,605	23%
evildeathcore	396,220,810	23%
motivator	343,052,802	23%
jackchalat	735,927,627	23%

@p0o · May 17 '17 (edited)

I understand the problem and I think keeping the indexes in a database was a good choice. There are ways to implement it over blockchain with a nodeJS app as a middleware to provide a different API using steem-js but it also involve some hacks and caching which is not necessarily easier.

However, since you are looking for a better database solution I recommend you to take a look at Gundb as it could provide a decentralized solution with an API as good as firebase.

properties (22)

`author`	p0o
`permlink`	re-jesta-re-p0o-re-jesta-maintenance-of-chainbb-indexes-missing-20170517t045057043z
`category`	chainbb
`json_metadata`	{"tags":["chainbb"],"app":"steemit/0.1"}
`created`	2017-05-17 04:51:00
`last_update`	2017-05-17 04:51:30
`depth`	3
`children`	1
`last_payout`	2017-05-24 04:51:00
`cashout_time`	1969-12-31 23:59:59
`total_payout_value`	0.000 HBD
`curator_payout_value`	0.000 HBD
`pending_payout_value`	0.000 HBD
`promoted`	0.000 HBD
`body_length`	469
`author_reputation`	8,373,104,781,065
`root_title`	"[COMPLETE] Maintenance of chainBB - indexes missing"
`beneficiaries`	`[]`
`max_accepted_payout`	1,000,000.000 HBD
`percent_hbd`	10,000
`post_id`	3,482,458
`net_rshares`	0

1 reply

@steemitboard · May 17 '17

Congratulations @jesta! You have completed some achievement on Steemit and have been rewarded with new badge(s) :

[![](https://steemitimages.com/70x80/http://steemitboard.com/notifications/comments.png)](http://steemitboard.com/@jesta) Award for the number of comments

Click on any badge to view your own Board of Honnor on SteemitBoard.
For more information about SteemitBoard, click [here](https://steemit.com/steemitboard/@arcange/steemitboard-com-introducing-the-first-steemit-game-experience)

If you no longer want to receive notifications, reply to this comment with the word `STOP`

If you want to support the SteemitBoard project, your upvote for this notification is welcome!

properties (22)

`author`	steemitboard
`permlink`	steemitboard-notify-jesta-20170517t081826000z
`category`	chainbb
`json_metadata`	{"image":["https://steemitboard.com/img/notifications.png"]}
`created`	2017-05-17 06:18:24
`last_update`	2017-05-17 06:18:24
`depth`	1
`children`	0
`last_payout`	2017-05-24 06:18:24
`cashout_time`	1969-12-31 23:59:59
`total_payout_value`	0.000 HBD
`curator_payout_value`	0.000 HBD
`pending_payout_value`	0.000 HBD
`promoted`	0.000 HBD
`body_length`	687
`author_reputation`	38,975,615,169,260
`root_title`	"[COMPLETE] Maintenance of chainBB - indexes missing"
`beneficiaries`	`[]`
`max_accepted_payout`	1,000,000.000 HBD
`percent_hbd`	10,000
`post_id`	3,484,043
`net_rshares`	0