We’ve spent the past week analyzing the performance of Eclipse hived/hivemind in our API service (api.hive.blog) and making improvements based on real-world loading patterns. 

It’s been a very productive week and we’ve resolved most of the outstanding issues reported and performance problems we observed. We’ve also identified and fixed several longstanding issues in the hived mutex locking code that prevented peak performance and led to potential data corruption issues.

# Hived work (blockchain node software)

We made a number of non-consensus changes to improve hived performance and fix old bugs:

We reviewed the locking used to protect the chainbase database and found several serious longstanding bugs that showed up under high stress API loading of a hived node. These bugs could lead to data corruption and non-optimal performance. Those fixes are incorporated into this merge request:
https://gitlab.syncad.com/hive/hive/-/merge_requests/143


After fixing these problems, we ran several high stress tests using the `siege` tool and were unable to “break” hived’s operation anymore. We were also able to get hived to fully utilize all the server’s cores (48 on our test system), demonstrating that the locking is fairly optimal for realistic loads.

As a performance optimization, we enabled filtering of operations returned by one of the most time-consuming API calls, `get_account_history`: 
https://gitlab.syncad.com/hive/hive/-/merge_requests/136
https://gitlab.syncad.com/hive/hive/-/merge_requests/137
https://gitlab.syncad.com/hive/hive/-/merge_requests/139


We made several changes to API responses returned by hived, mostly in response to reports from apps developers:
https://gitlab.syncad.com/hive/hive/-/merge_requests/125
https://gitlab.syncad.com/hive/hive/-/merge_requests/126
https://gitlab.syncad.com/hive/hive/-/merge_requests/133

We also limited the max number of operation of operations to  that can be fetched by this call to 1k operations (previously it allowed up to 10k operations) to optimize hived throughput: https://gitlab.syncad.com/hive/hive/-/commit/9a05f4d780fc98b891f51ab4b88ebec89fde467b

We corrected account-impacting for several virtual operations created recently:
https://gitlab.syncad.com/hive/hive/-/merge_requests/134

## Hived status

We started a full replay and a full sync early this morning to check all the above changes (full replay takes around 18 hours). We don’t expect any issues, but there is a very small chance for issues due to the mutex locking changes, even though no consensus changes were made.

With these changes, I think hived’s performance and reliability has been substantially improved over its  past condition and it seems quite solid in the face of very substantial API traffic loading. 

One minor issue still to be resolved is the longstanding issue with hived crashing if it receives API traffic when starting up, so we may tackle that next, since it can be an annoyance when updating several hiveds deployed inside of a API server node configuration. 

**As this version contains substantial operational improvements, we’ve tagged this version as [v1.24.5](https://gitlab.syncad.com/hive/hive/-/releases/v1.24.5), and it’s a strongly recommended upgrade for all API server nodes. However, there’s no need for exchanges or consensus nodes to upgrade to this version, since the performance improvements mostly come into play when a node is handling significant API traffic.**

# Hivemind

We made numerous optimizations and bug fixes in hivemind this past week:

https://gitlab.syncad.com/hive/hivemind/-/merge_requests/311
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/306
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/312
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/314
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/291
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/280
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/316
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/318
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/323
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/226
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/322
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/289
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/315
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/326
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/321
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/325
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/328



## Hivemind status (2nd layer social media microservice)

By late tomorrow I expect we’ll have resolved all the remaining big issues with hivemind responses and performance, other than reputation support (which we’ve intentionally left out until all other issues were resolved).

# Condenser + Condenser wallet (open-source code for **hive.blog**)

Merged in support for decentralized lists:
https://gitlab.syncad.com/hive/condenser/-/merge_requests/135
This functionality is still waiting one last issue to be resolved in hivemind, so it won’t be fully testable until tomorrow when we deploy a new version of hivemind.

We made numerous changes this week to improve condenser’s wallet (corrected some display issues and improved performance of page loads by using newer, faster API calls), including new features in the proposals page:
https://gitlab.syncad.com/hive/wallet/-/merge_requests/42
https://gitlab.syncad.com/hive/wallet/-/merge_requests/43
https://gitlab.syncad.com/hive/wallet/-/merge_requests/45
https://gitlab.syncad.com/hive/wallet/-/merge_requests/52
https://gitlab.syncad.com/hive/wallet/-/merge_requests/54
https://gitlab.syncad.com/hive/wallet/-/merge_requests/55
https://gitlab.syncad.com/hive/wallet/-/merge_requests/56
https://gitlab.syncad.com/hive/wallet/-/merge_requests/57
https://gitlab.syncad.com/hive/wallet/-/merge_requests/60
https://gitlab.syncad.com/hive/wallet/-/merge_requests/62
https://gitlab.syncad.com/hive/wallet/-/merge_requests/64
https://gitlab.syncad.com/hive/wallet/-/merge_requests/65
https://gitlab.syncad.com/hive/wallet/-/merge_requests/68
https://gitlab.syncad.com/hive/wallet/-/merge_requests/70

Proposals page changes:
https://gitlab.syncad.com/hive/wallet/-/merge_requests/47
https://gitlab.syncad.com/hive/wallet/-/merge_requests/49
https://gitlab.syncad.com/hive/wallet/-/merge_requests/51


## Condenser status

Condenser is looking pretty good at this point, but we’ll continue to test and look for any remaining problems.

# Hive-js improvements

In order to make hive.blog faster, we modified hive-js to support the new option to filter the operations fetched by the get_account_history call:
https://gitlab.syncad.com/hive/hive-js/-/merge_requests/12
https://gitlab.syncad.com/hive/hive-js/-/merge_requests/13
https://gitlab.syncad.com/hive/hive-js/-/merge_requests/14



# What’s next for the week?

Tomorrow we’ll hopefully be deploying a new hivemind with all the performance improvements and fixes we made in the last week, after some real world traffic testing on our API node.

We’ll also be testing condenser and the wallet and looking for fixes and optimizations we can make.

**Assuming deployment of latest hivemind goes well tomorrow, we’ll likely still need to make a few more fixes as issues get reported, but I expect that by the latter part of the coming week we can begin to analyze feasibility for various previously suggested features for HF25 and also intermediate changes that aren’t consensus changes that can be released earlier.**