This post will be a go-to reference for myself, and a resource for anyone interested in setting up an api node. It's very technical, so if you don't belong to one of those two groups you can skip it.

<center>https://upload.wikimedia.org/wikipedia/commons/thumb/b/b8/Compiling.jpg/640px-Compiling.jpg</center>

---------

# Introduction

My current setup is close to the default hivemind setup provided in the example jussi config. After talks with @gtg one of the nodes offers a few more apis for performance reasons, as the fat node is very slow. There surely are not-optimal or even misconfigurations, optimization will be an ongoing process. See the end of the post for more info, make sure to read and understand everything before you start. I will update this post with any important changes.

---------

# Hardware

First, we need hardware. The setup consists of 3 nodes, for which I selected the following specs:

- _hivemind_
32GB RAM
2x240GB SSD RAID0
- _fat_
64GB RAM
2x480GB SSD RAID0
- _accounthistory_
64GB RAM
2x512GB NVMe RAID0 (64GB SWAP)

All are set up with a clean install of Ubuntu 18.04

-------------------

# Setup

## Common steps

Set up each server to log in securely and with a dedicated user. The user on all machines will be called "hive" during this process. Individual needs may differ so I don't go into details here and only provide the steps necessary to proceed.

```sudo apt-get update && sudo apt-get upgrade```
```sudo apt-get install -y screen git nginx  certbot python-certbot-nginx```

## Step 1: make everything sync

### hivemind node

install software
```
cd ~
sudo apt-get install -y python3 python3-pip postgresql postgresql-contrib docker.io
git clone https://gitlab.syncad.com/hive/hivemind.git
```

```
cd hivemind
sudo pip3 install -e .[test]
```

setup database
```
sudo su postgres
createdb hive
```

Create db user hive and grant access to database
```createuser --interactive```
```psql```
```GRANT ALL PRIVILEGES ON DATABASE hive TO hive;```
```\q```
```exit```

optimize postgres
```sudo nano /etc/postgresql/10/main/postgresql.conf```
use https://pgtune.leopard.in.ua/ to find the optimal settings for your machine. I used the following:
```
# DB Version: 10
# OS Type: linux
# DB Type: web
# Total Memory (RAM): 32 GB
# Data Storage: ssd

max_connections = 200
shared_buffers = 8GB
effective_cache_size = 24GB
maintenance_work_mem = 2GB
checkpoint_completion_target = 0.7
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 20971kB
min_wal_size = 1GB
max_wal_size = 4GB
```
```sudo service postgresql restart```

The irredeemables list is a blacklist containing mass spammers mostly. It's recommended to use it if you serve browser based interfaces, because the amount of comments by these accounts creates a lot of traffic and is a burden on browsers. It's defined in ```/home/hive/hivemind/hive/conf.py``` under ```--muted-accounts-url```. You can change it there, or add the environment variable ```MUTED_ACCOUNTS_URL``` in both scripts if you do not want to use the default. I offer an [empty version](https://raw.githubusercontent.com/pharesim/irredeemables/master/full.txt) if you don't want to filter the results.

Create sync script
```nano sync.sh```
Insert the following (the STEEMD_URL is temporary until your own fat node has synced, update and restart it in a few days to speed up the hivemind sync):
```
#!/bin/bash
export DATABASE_URL=postgresql://hive:pass@localhost:5432/hive
export STEEMD_URL='{"default": "https://fat.pharesim.me"}'
export HTTP_SERVER_PORT=28091
hive sync
```

```
chmod +x sync.sh
screen -S hivesync
```
```./sync.sh```
Use ```Ctrl-a d``` to detach screen, ```screen -r hivesync``` to reattach

The whole sync process takes about a week. Don't forget changing the STEEMD_URL when your fat node is finished. Unlike the steemd replays, you can interrupt this sync at any time and it picks up where you stopped.

The sync is finished when you see single blocks coming in. Keep it running, and set up the server:
```
cp sync.sh hivemind.sh
nano hivemind.sh
```
Change ```sync``` in the end for ```server```

```
screen -S hivemind
```
```./hivemind.sh```
Use ```Ctrl-a d``` to detach screen, ```screen -r hivemind``` to reattach

### steemd nodes

Both the fat and the accounthistory node will run an instance of steemd, these are the steps to prepare them:
```
sudo apt-get install -y autoconf automake cmake g++ git libbz2-dev libsnappy-dev libssl-dev libtool make pkg-config python3-jinja2 libboost-chrono-dev libboost-context-dev libboost-coroutine-dev libboost-date-time-dev libboost-filesystem-dev libboost-iostreams-dev libboost-locale-dev libboost-program-options-dev libboost-serialization-dev libboost-signals-dev libboost-system-dev libboost-test-dev libboost-thread-dev doxygen libncurses5-dev libreadline-dev perl ntp
```
```
cd
git clone https://github.com/openhive-network/hive
cd hive
git checkout v0.23.0
git submodule update --init --recursive
mkdir build
cd build
```
The build options differ for the two nodes

_fat_
```
cmake -DCMAKE_BUILD_TYPE=Release -DLOW_MEMORY_NODE=OFF -DCLEAR_VOTES=OFF -DSKIP_BY_TX_ID=OFF -DBUILD_STEEM_TESTNET=OFF -DENABLE_MIRA=ON -DSTEEM_STATIC_BUILD=ON ..
```

_accounthistory_
```
cmake -DCMAKE_BUILD_TYPE=Release -DLOW_MEMORY_NODE=ON -DCLEAR_VOTES=ON -DSKIP_BY_TX_ID=OFF -DBUILD_STEEM_TESTNET=OFF -DENABLE_MIRA=OFF -DSTEEM_STATIC_BUILD=ON ..
```

Again on both:
```
make -j$(nproc) steemd
cd
mkdir bin
cp /home/hive/hive/build/programs/steemd bin/v0.23.0
mkdir .steemd
nano .steemd/config.ini
```

And again, the configs differ for the two nodes

_fat_
```
log-appender = {"appender":"stderr","stream":"std_error"} {"appender":"p2p","file":"logs/p2p/p2p.log"}
log-logger = {"name":"default","level":"info","appender":"stderr"} {"name":"p2p","level":"warn","appender":"p2p"}
backtrace = yes

plugin = webserver p2p json_rpc witness account_by_key reputation market_history
plugin = database_api account_by_key_api network_broadcast_api reputation_api
plugin = market_history_api condenser_api block_api rc_api

history-disable-pruning = 0
account-history-rocksdb-path = "blockchain/account-history-rocksdb-storage"
block-data-export-file = NONE
block-log-info-print-interval-seconds = 86400
block-log-info-print-irreversible = 1
block-log-info-print-file = ILOG
sps-remove-threshold = 200

shared-file-dir = "blockchain"
shared-file-size = 360G

shared-file-full-threshold = 0
shared-file-scale-rate = 0
follow-max-feed-size = 500
follow-start-feeds = 0
market-history-bucket-size = [15,60,300,3600,86400]
market-history-buckets-per-size = 5760

p2p-seed-node = anyx.io:2001 gtg.steem.house:2001 seed.jesta.us:2001

rc-skip-reject-not-enough-rc = 0
rc-compute-historical-rc = 0
statsd-batchsize = 1
tags-start-promoted = 0
tags-skip-startup-update = 0
transaction-status-block-depth = 64000
transaction-status-track-after-block = 0

webserver-http-endpoint = 0.0.0.0:28091
webserver-ws-endpoint = 0.0.0.0:28090
webserver-thread-pool-size = 32

enable-stale-production = 0
required-participation = 33
witness-skip-enforce-bandwidth = 1
```
Not sure about the shared_file_size here, it's rocksdb? Better safe than sorry...

_accounthistory_
```
log-appender = {"appender":"stderr","stream":"std_error"} {"appender":"p2p","file":"logs/p2p/p2p.log"}
log-logger = {"name":"default","level":"info","appender":"stderr"} {"name":"p2p","level":"warn","appender":"p2p"}
backtrace = yes

plugin = webserver p2p json_rpc witness
plugin = rc market_history_account_history_rocksdb transaction_status account_by_key
plugin = database_api condenser_api market_history_api account_history_api transaction_status_api account_by_key_api
plugin = block_api network_broadcast_api rc_api

history-disable-pruning = 1
account-history-rocksdb-path = "blockchain/account-history-rocksdb-storage"
block-data-export-file = NONE
block-log-info-print-interval-seconds = 86400
block-log-info-print-irreversible = 1
block-log-info-print-file = ILOG
sps-remove-threshold = 200

shared-file-dir = "/run/hive"
shared-file-size = 120G

shared-file-full-threshold = 9500
shared-file-scale-rate = 1000
flush-state-interval = 0
follow-max-feed-size = 500
follow-start-feeds = 0
market-history-bucket-size = [15,60,300,3600,86400]
market-history-buckets-per-size = 5760

p2p-seed-node = anyx.io:2001 seed.jesta.us:2001

rc-skip-reject-not-enough-rc = 0
rc-compute-historical-rc = 0
statsd-batchsize = 1
tags-start-promoted = 0
tags-skip-startup-update = 0
transaction-status-block-depth = 64000
transaction-status-track-after-block = 42000000

webserver-http-endpoint = 0.0.0.0:28091
webserver-ws-endpoint = 0.0.0.0:28090
webserver-thread-pool-size = 32

```

The _fat_ node also needs a database.cfg
```nano .steemd/database.cfg```

These settings are for 32GB of RAM. Adapt global.shared_cache.capacity, global.write_buffer_manager.write_buffer_size and global.object_count accordingly
```
{
  "global": {
    "shared_cache": {
      "capacity": "21474836480"
    },
    "write_buffer_manager": {
      "write_buffer_size": "4294967296"
    },
    "object_count": 250000,
    "statistics": false
  },
  "base": {
    "optimize_level_style_compaction": true,
    "increase_parallelism": true,
    "block_based_table_options": {
      "block_size": 8192,
      "cache_index_and_filter_blocks": true,
      "bloom_filter_policy": {
        "bits_per_key": 10,
        "use_block_based_builder": false
      }
    }
  }
}
```
As well as increased file limits (hive is the username, adapt to your memory again)
```sudo nano /etc/security/limits.conf```
Insert near the end of the file
```
hive soft  nofile 262140
hive hard nofile 262140
```
```sudo nano /etc/sysctl.conf ```
Insert near the end of the file
```
fs.file-max = 2097152
```
You need to log in to the server again for these to take effect.

The _accounthistory_ requires a change of the size of /run 
```mount -o remount ,size=120G /run```
and a directory /run/hive
```
sudo mkdir /run/hive
sudo chown hive:hive /run/hive
```

Then continue on both servers.
Download block_log.index and block_log
```
rsync -avh --progress --append rsync://files.privex.io/hive/block_log.index .steemd/blockchain/block_log.index
rsync -avh --progress --append rsync://files.privex.io/hive/block_log .steemd/blockchain/block_log
```
Go for a walk or have dinner.

Start up steemd and replay blockchain
```screen -S hive```
```
echo    75 | sudo tee /proc/sys/vm/dirty_background_ratio
echo  1000 | sudo tee /proc/sys/vm/dirty_expire_centisecs
echo    80 | sudo tee /proc/sys/vm/dirty_ratio
echo 30000 | sudo tee /proc/sys/vm/dirty_writeback_centisecs
```
```~/bin/v0.23.0 --replay```
Use ```Ctrl-d a``` to detach from screen, and ```screen -r hive``` to reattach.

The sync process takes a bit more than 2 days on the fat node, less on accounthistory.  Do not interrupt them, or you will have to start over. If you are syncing a hivemind node, don't forget to switch to the fat node when that's finished.

## Step 2: webserver + routing

### all nodes

All requests will be proxied by nginx, so we need this on all machines. We will install SSL certificates, so all communication is encrypted and all nodes can be called individually.

```sudo nano /etc/nginx/sites-enabled/hive```
The config is the same for each node, only change the server_name
```
upstream hivesrvs {
# Dirty Hack. Causes nginx to retry node
   server 127.0.0.1:28091;
   server 127.0.0.1:28091;
   server 127.0.0.1:28091;
   server 127.0.0.1:28091;
   keepalive 10;
}

server {
    server_name hivemind/fat/acchist.you.tld;
    root /var/www/html/;

    location ~ ^(/|/ws) {
        proxy_pass http://hivesrvs;
        proxy_set_header Connection "";
        include snippets/rpc.conf;
    }
```
Add the rpc.conf to each server
```sudo nano /etc/nginx/snippets/rpc.conf```
Insert
```
access_log off;

proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
proxy_connect_timeout 10;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

keepalive_timeout 65;
keepalive_requests 100000;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
proxy_ssl_verify off;
```

Let certbot configure the domains for automatic redirect to https
```sudo certbot --nginx```

### hivemind node

We need an additional file for nginx, for the general entry point 
```
sudo cp /etc/nginx/sites-enabled/hive /etc/nginx/sites-enabled/api
sudo nano /etc/nginx/sites-enabled/api
```

Change both occurances of ```hivesrvs``` to ```jussisrv```, the ports from ```28091``` to ```9000```, and the server_name to api., or whatever you want your node to be accessible on.


#### jussi

```
cd ~
git clone https://gitlab.syncad.com/hive/jussi.git
```

Create build script
```nano build.sh```
and insert
```
#!/bin/bash
cd /home/hive/jussi
sudo docker build -t="$USER/jussi:$(git rev-parse --abbrev-ref HEAD)" .
```
```chmod +x build.sh```

Create run script
```nano jussi.sh```
and insert
```
#!/bin/bash
cd /home/hive/jussi
sudo docker run -itp 9000:8080 --log-opt max-size=50m "$USER/jussi:$(git rev-parse --abbrev-ref HEAD)"
```
```chmod +x jussi.sh```

```screen -S jussi```
```
cd ~/jussi
nano DEV_config.json
```

Currently, my config looks like this:

```
{
    "limits": { "accounts_blacklist": [ "accounttoblock" ] },
    "upstreams": [
      {
        "name": "steemd",
        "translate_to_appbase": true,
        "urls": [["steemd", "https://fat1.pharesim.me" ]],
        "ttls": [["steemd", 3]],
        "timeouts": [["steemd",3]]
      },
      {
        "name": "appbase",
        "urls": [
          ["appbase", "https://fat1.pharesim.me"],

          ["appbase.account_history_api", "https://acchist1.pharesim.me"],
          ["appbase.condenser_api.get_account_history", "https://acchist1.pharesim.me"],
          ["appbase.condenser_api.get_ops_in_block", "https://acchist1.pharesim.me"],
          ["appbase.condenser_api.get_transaction", "https://acchist1.pharesim.me"],

          ["appbase.condenser_api.get_followers", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_following", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_follow_count", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_discussions_by_trending", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_discussions_by_hot", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_discussions_by_promoted", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_discussions_by_created", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_discussions_by_blog", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_discussions_by_feed", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_discussions_by_comments", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_reblogged_by", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_replies_by_last_update", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_trending_tags", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_discussions_by_author_before_date", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_post_discussions_by_payout", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_comment_discussions_by_payout", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_blog", "http://localhost:28091"],
          ["appbase.condenser_api.get_blog_entries", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_account_votes", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_state", "https://hivemind.pharesim.me"],

          ["appbase.condenser_api.get_state.params=['witnesses']", "https://acchist1.pharesim.me"],
          ["appbase.condenser_api.get_state.params=['/witnesses']", "https://acchist1.pharesim.me"],
          ["appbase.condenser_api.get_state.params=['/~witnesses']", "https://acchist1.pharesim.me"],
          ["appbase.condenser_api.get_state.params=['~witnesses']", "https://acchist1.pharesim.me"],

          ["appbase.follow_api", "https://hivemind.pharesim.me"],
          ["appbase.tags_api", "https://hivemind.pharesim.me"],

          ["appbase.market_history_api", "https://acchist1.pharesim.me"],
          ["appbase.transaction_status_api", "https://acchist1.pharesim.me"],
          ["appbase.account_by_key_api", "https://acchist1.pharesim.me"],
          ["appbase.block_api", "https://acchist1.pharesim.me"],
          ["appbase.network_broadcast_api", "https://acchist1.pharesim.me"],
          ["appbase.rc_api", "https://acchist1.pharesim.me"]
        ],
        "ttls": [
          ["appbase", 3],
          ["appbase.login_api",-1],
          ["appbase.network_broadcast_api", -1],
          ["appbase.follow_api", 10],
          ["appbase.market_history_api", 1],
          ["appbase.condenser_api", 3],
          ["appbase.condenser_api.get_block", -2],
          ["appbase.condenser_api.get_block_header", -2],
          ["appbase.condenser_api.get_content", 1],
          ["appbase.condenser_api.get_state", 1],
          ["appbase.condenser_api.get_state.params=['/trending']", 30],
          ["appbase.condenser_api.get_state.params=['trending']", 30],
          ["appbase.condenser_api.get_state.params=['/hot']", 30],
          ["appbase.condenser_api.get_state.params=['/welcome']", 30],
          ["appbase.condenser_api.get_state.params=['/promoted']", 30],
          ["appbase.condenser_api.get_state.params=['/created']", 10],
          ["appbase.condenser_api.get_dynamic_global_properties", 3]
        ],
        "timeouts": [
          ["appbase", 3],
          ["appbase.network_broadcast_api",0],
          ["appbase.chain_api.push_block", 0],
          ["appbase.chain_api.push_transaction", 0],
          ["appbase.condenser_api.broadcast_block", 0],
          ["appbase.condenser_api.broadcast_transaction", 0],
          ["appbase.condenser_api.broadcast_transaction_synchronous", 0],
          ["appbase.condenser_api.get_account_history", 20],
          ["appbase.condenser_api.get_account_votes", 20],
          ["appbase.condenser_api.get_ops_in_block.params=[2889020,false]", 20],
          ["appbase.account_history_api.get_account_history", 20],
          ["appbase.account_history_api.get_ops_in_block.params={\"block_num\":2889020,\"only_virtual\":false}", 20]
        ]
      },
      {
        "name": "hive",
        "urls": [["hive", "http://localhost:28091"]],
        "ttls": [["hive", -1]],
        "timeouts": [["hive", 30]]
      },
    {
      "name": "bridge",
      "translate_to_appbase": false,
      "urls": [["bridge","http://localhost:28091"]],
      "ttls": [["bridge",-1]],
      "timeouts": [["bridge",30]]
    }
    ]
  }
```

```cd```
```./build.sh```
This takes a while, when finished
```./run.sh```
Use ```Ctrl-a d``` to detach screen, ```screen -r jussi``` to reattach
If you update your config, run build.sh outside of screen, then attach to screen and restart the run script. (There may be faster ways to update the docker with a new config, but I'm new to this).

### Something about steemd apis

As you might have realized, there are some duplicates in the apis _fat_ and _accounthistory_ provide. That's because of what I mentioned above, _fat_ is _slow_. I did not investigate which aren't needed on _fat_ now to still work for hivemind, so I didn't change anything in that default configuration. I also just took the list for apis on _accounthistory_ from @gtg without questioning. There is unnecessary redundancy for sure, instructions may change in the future to improve on this. Your perfect setup may differ completely, depending on which apis you need to serve (most).

## Finishing words

That's it. After everything is synced, you should have a working public node ready to serve requests! If this guide has helped you and/or you want to support my work on Hive infrastructure, education, onboarding and retention, please [help me secure my witness spot](https://peakd.com/me/witnesses)!

## Stats

Current output of ```df -h``` on the three servers (April 14):

_hivemind_
```
/dev/md2        407G  272G  115G  71% /
```

_fat_
```
/dev/md2        815G  369G  406G  48% /
```

_accounthistory_
```
tmpfs           120G   58G   63G  48% /run
/dev/md2        874G  531G  299G  65% /
```