create account

STEEMSQL - A public SQL database with all blockchain data – Update 5 by arcange

View this thread on: hive.blogpeakd.comecency.com
· @arcange · (edited)
$5.73
STEEMSQL - A public SQL database with all blockchain data – Update 5
<center>https://i.imgsafe.org/77e5fc4a12.png</center>

**SteemSQL is a public MS-SQL database with all the blockchain data in it.**

#### Previous posts:
* [Introduction post](https://steemit.com/steemit/@arcange/steemsql-com-a-public-sql-server-database-with-all-steemit-blockchain-data)
* [SteemSQL - Update 1](https://steemit.com/steemit/@arcange/steemsql-a-public-sql-database-with-all-blockchain-data-update-1)
* [SteemSQL - Update 2 - Language  detection](https://steemit.com/steemit/@arcange/steemsql-a-public-sql-database-with-all-blockchain-data-new-language-detection)
* [SteemSQL - Update 3](https://steemit.com/steemit/@arcange/steemsql-a-public-sql-database-with-all-blockchain-data-update3)
* [SteemSQL - Update 4](https://steemit.com/steemsql/@arcange/steemsql-a-public-sql-database-with-all-blockchain-data-update-2)

# What’s new?

Two days ago, I announced a maintenance of my servers infrastructure. 

Indeed, more and more people are using SteemSQL and rely on its availability and performance to provide services and information to all Steemit users

### A never-ending growing database

One of the problem we face is that the amount of data managed by the server grows every day.

To give you an idea, here is a graph that shows the daily amount of data added to the blockchain:

![](http://i.imgsafe.org/fd665698c5.png)

You can see that each day, approximately 75000 KB of data are added to the blockchain. 
As more active users are joining Steem, we can expect this amount to continue to increase even faster.

### Performance matter

When you have to deal with such an amount of data, the main bottleneck are:

1. The storage performances

The speed of the storage to deliver the data for processing is the key factor. The faster the server can read data, the faster it can use it to make computation.
As user on Voice are more and more active, they server has also to constantly write the new data generated. This can also affect the performances because when you write data, the server put a lock on its database, preventing others to read uncommitted data to avoid computation errors.

2. The computation power

When you have to aggregate several millions of values to compute a sum or an average trend, you need a lot of computation power and preferably, you do not want to be interrupted by others while you are doing the math.

### A new high-performance dedicated infrastructure

SteemSQL was previously hosted on an existing shared infrastructure. Even if it is a high-end server, it still uses some legacy technology like SAS hard drives and shared its environment with some others process (like my witness and seed nodes).

Given the success of SteemSQL and the growing and growing load on the server, I have decided to move it to a brand new high-performance fully dedicated infrastructure.

The new server hosting Steem SQL now has the following spec:
* 4 CPUs running at 3.5Ghz with latest Intel Xeon v5 technology 
This is far enough to handle the most demanding computation requests. The actual average load on the CPU is now down to 6.7%

* 2x500GB RAID SSD drives with the latest NVMe technology<br>
This is the very latest storage technology and it is BLAZING FAST !!!
To give you an idea, here a performance comparison chart:<br>
![](http://i.imgsafe.org/e92a44c3b2.png)<br>
The previous server was using SAS HDD with high-end cache (similar to the first red column)
The new server use NVMe RAM (the green column)

I used to wait a lot when doing some stats. I now get the results in a few seconds. I have to admit i was really impressed by its performance the first time I ran some heavy queries.

On top of this, the server has a guaranteed dedicated 500Mb/s internet bandwidth.

### No limits ... no tea

This major infrastructure migration will allow me and my fellow Steem statisticians to provide you with more crazy statistics and fancy graphs.

The only drawback is that we all have no more time to take a break and drink a cup of tea.

That’s life ...
<br>
Thanks for reading!

---
All payout from this post will be dedicated  keep SteemSQL running. Thanks for your support.

<center>![](https://steemitimages.com/50x61/http://steemitboard.com/@arcange/commented.png)![](https://steemitimages.com/60x70/http://steemitboard.com/@arcange/votes.png) ![](https://steemitimages.com/70x80/http://steemitboard.com/@arcange/posts.png) [![](https://steemitimages.com/100x80/http://steemitboard.com/@arcange/level.png)](http://steemitboard.com/@arcange) ![](https://steemitimages.com/70x80/http://steemitboard.com/@arcange/comments.png) ![](https://steemitimages.com/60x70/http://steemitboard.com/@arcange/voted.png)![](https://steemitimages.com/50x60/http://steemitboard.com/@arcange/payout.png)</center>

### <center>Support me and my work as a [witness](https://steemit.com/steemit/@arcange/arcange-witness-thread) by voting for me [here!](https://steemit.com/~witnesses)</center>
<center>https://i.imgsafe.org/ce6527e1a7.png</center>

You like this post, do not forget to https://s22.postimg.org/n01rpphmp/pointing_fingers.gif **upvote** or **[follow](https://steemit.com/@arcange) me** or **resteem**
👍  , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , and 87 others
properties (23)
authorarcange
permlinksteemsql-a-public-sql-database-with-all-blockchain-data-update-5
categorysteemsql
json_metadata{"tags":["steemsql","steemit","database","sql","programming"],"image":["https://i.imgsafe.org/77e5fc4a12.png","http://i.imgsafe.org/fd665698c5.png","http://i.imgsafe.org/e92a44c3b2.png","https://steemitimages.com/50x61/http://steemitboard.com/@arcange/commented.png","https://steemitimages.com/60x70/http://steemitboard.com/@arcange/votes.png","https://steemitimages.com/70x80/http://steemitboard.com/@arcange/posts.png","https://steemitimages.com/100x80/http://steemitboard.com/@arcange/level.png","https://steemitimages.com/70x80/http://steemitboard.com/@arcange/comments.png","https://steemitimages.com/60x70/http://steemitboard.com/@arcange/voted.png","https://steemitimages.com/50x60/http://steemitboard.com/@arcange/payout.png","https://i.imgsafe.org/ce6527e1a7.png","https://s22.postimg.org/n01rpphmp/pointing_fingers.gif"],"links":["https://steemit.com/steemit/@arcange/steemsql-com-a-public-sql-server-database-with-all-steemit-blockchain-data","https://steemit.com/steemit/@arcange/steemsql-a-public-sql-database-with-all-blockchain-data-update-1","https://steemit.com/steemit/@arcange/steemsql-a-public-sql-database-with-all-blockchain-data-new-language-detection","https://steemit.com/steemit/@arcange/steemsql-a-public-sql-database-with-all-blockchain-data-update3","https://steemit.com/steemsql/@arcange/steemsql-a-public-sql-database-with-all-blockchain-data-update-2","http://steemitboard.com/@arcange","https://steemit.com/steemit/@arcange/arcange-witness-thread","https://steemit.com/~witnesses","https://steemit.com/@arcange"],"app":"steemit/0.1","format":"markdown"}
created2017-03-08 12:36:36
last_update2017-03-26 13:11:45
depth0
children2
last_payout2017-04-08 14:25:18
cashout_time1969-12-31 23:59:59
total_payout_value4.786 HBD
curator_payout_value0.944 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length5,149
author_reputation1,146,621,535,585,811
root_title"STEEMSQL - A public SQL database with all blockchain data – Update 5"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,672,296
net_rshares29,352,613,777,832
author_curate_reward""
vote details (151)
@oflyhigh ·
Thank you very much for providing this service, upvoted!
properties (22)
authoroflyhigh
permlinkre-arcange-steemsql-a-public-sql-database-with-all-blockchain-data-update-5-20170308t231909861z
categorysteemsql
json_metadata{"tags":["steemsql"],"app":"steemit/0.1"}
created2017-03-08 23:19:12
last_update2017-03-08 23:19:12
depth1
children1
last_payout2017-04-08 14:25:18
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length56
author_reputation6,315,854,928,524,282
root_title"STEEMSQL - A public SQL database with all blockchain data – Update 5"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,676,320
net_rshares0
@arcange ·
Thanks =)
properties (22)
authorarcange
permlinkre-oflyhigh-re-arcange-steemsql-a-public-sql-database-with-all-blockchain-data-update-5-20170308t232525736z
categorysteemsql
json_metadata{"tags":["steemsql"],"app":"steemit/0.1"}
created2017-03-08 23:25:27
last_update2017-03-08 23:25:27
depth2
children0
last_payout2017-04-08 14:25:18
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length9
author_reputation1,146,621,535,585,811
root_title"STEEMSQL - A public SQL database with all blockchain data – Update 5"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id2,676,356
net_rshares0