# 🎉 Hive is five 🎂

Hello Hive Community ❗

Birthday needs a gift 🎁 and we, 🐝 thebeedevs group, prepared a few for you, to make this platform better and simplify application development.

To make quick retrospection, Hive was born in a block [41818752](https://explore.openhive.network/block/41818752), by executing the `HardFork #23`.

Here are our gifts... A few has been already mentioned (but they got a lot of new features), some (like HAF itself and a few base HAF applications powering the platform) are being continuously developed to make it faster, better and more reliable. Finally, we have also created new applications to address community needs.

[![HIVE](https://hive.io/opengraph.png)](https://hive.blog)

***𝔹*** like a ***BeekeePy*** tool
***𝕀*** like ***Improvements***
***ℝ*** like ***Reliability***
***𝕋*** like a ***Transaction Inspector*** web application
***ℍ*** like a ***hAIve*** (our Artificial Intelligence powered search engine for Hive)
***𝔻*** like a ***Denser*** web application
***𝔸*** like ***A wAx library***
***𝕐*** like ***Yet more wallets to sign Your transactions***

Below I would like to describe shortly mentioned things... 

## 🔑 BeekeePy tool
One of previously published posts introduced a tool called `beekeeper` ["for those developing a Hive"](https://hive.blog/hive-139531/@thebeedevs/for-those-developing-a-hive). The purpose of this tool is to preserve your keys from direct access and allow only performing basic operations like:

- signing,
- buffer encryption,
- buffer decryption

Beekeeper is a ***SUPERHERO*** 🦸 right now 😊, and guards your keys in **three** environments:
- 🕸 WASM code directly executed inside Javascript environment by the WebBrowser. This allows you to hold your keys in the wallet (e.g.) this version used by Denser application) 
- 💻 as a desktop service, where its functionality can be accessed through network API calls. This way, you can configure it to run the service using separate OS account and use UNIX as a mainguard of your keys stored inside encrypted beekeeper storage
- 🐍 Python based (this is why we've created a Beekee**Py** wrapper), which significantly simplifies interaction to underlying beekeeper service. The beekeepy python package allows easily operating on explicitly pointed beekeeper service instance, or own-managed process. Again, UNIX security features prevent accessing sensitive data (they are managed by separate process) directly from Python side which can be running in a potentially not safe environment (due to the 3rd party Python package dependency problems).

## 💡 Improvements
It is very hard to select most important enhancements being done by the group teammates. Our main focus is a performance of provided tools... The greatest motivation for the work put into optimization 🚅 is the constantly growing blockchain, which increments every 3s ⏰ together with local resource needs required to hold it.

To address the issue, we did several things like:
- block log splitting/pruning implementation to reduce disk space needed to run given hived node
- significant block processing optimization in the [reputation_tracker](https://gitlab.syncad.com/hive/reputation_tracker/), [balance_tracker](https://gitlab.syncad.com/hive/balance_tracker/) applications which reduced application processing times from more than **21hrs** to **12hrs 15mins** even it has to process **additional 2 million** blocks ❗❗❗
- HAF and Hivemind storage reduction (due to database index optimization) to reduce disk space usage required to hold a HAF node.

We are also planning new ones especially specific to critical parts the Hive services like [Hived](https://gitlab.syncad.com/hive/hive/-/issues/675). Hopefully some of them will be also ready soon.

## 🐛 Reliability
To achieve good software quality it must be... tested. And ideally not only by final users 😊 but during software development process. To make testing efficient, during last few years of Hive development we have created a lot of new testcases like also a Python based library simplifying their creation (called [test-tools](https://gitlab.syncad.com/hive/test-tools)).

Below you can find some information on the number of regression tests that exist in each of the most important projects included in the Hive platform:

|Project|Total number of tests|Example pipeline URL|
|--------:|--------------------:|:--------:|
|Hive|5015|[develop branch pipeline](https://gitlab.syncad.com/hive/hive/-/pipelines/118202/test_report)|
|HAF|117|[develop branch pipeline](https://gitlab.syncad.com/hive/haf/-/pipelines/118166/test_report)|
|Hivemind|1467|[develop branch pipeline](https://gitlab.syncad.com/hive/hivemind/-/pipelines/117893/test_report)|
|HAfAH|656|[develop branch pipeline](https://gitlab.syncad.com/hive/HAfAH/-/pipelines/116997/test_report)|
|WAX|923|[develop branch pipeline](https://gitlab.syncad.com/hive/wax/-/pipelines/118175/test_report)|

To make this testing possible and efficient like also reproducible, all our projects have implemented Continuous Integration processes, being useful for automatic building, testing as well as deployment of each project.

Additionally, making good software not only needs tests, but also some tools which allow performing scenarios not easy (**nor safe**) to happen in production environment. To make them a reality, during the last few years we created testing tools like:
- [suite of tools supporting a Hive mirrornet](https://gitlab.syncad.com/hive/hive/-/tree/develop/programs/blockchain_converter?ref_type=heads) (be able to transform mainnet blocks into fake mirrornet ones), where - due to the isolation of both networks - disasters can be safely performed...  💣
- dedicated unit test fixtures emulating actual execution environment during performing low level unit tests. The one of biggest challenges in this part, was creation of fixture mimicking hived start and shutdown actions, mostly by directly reusing [appbase code](https://gitlab.syncad.com/hive/hive/-/tree/develop/libraries/appbase?ref_type=heads). The source code responsible for such emulation is located inside [hived_fixture](https://gitlab.syncad.com/hive/hive/-/blob/develop/tests/unit/db_fixture/hived_fixture.hpp?ref_type=heads) 
- low level hived plugins called: [colony](https://gitlab.syncad.com/hive/hive/-/tree/develop/libraries/plugins/colony?ref_type=heads), [queen](https://gitlab.syncad.com/hive/hive/-/tree/develop/libraries/plugins/queen?ref_type=heads) and [pacemaker](https://gitlab.syncad.com/hive/hive/-/tree/develop/libraries/plugins/pacemaker?ref_type=heads) being useful in executing performance tests leading to generating thousands of transactions, producing huge blocks and next collect statistics and ideas for... other optimizations to be done in Hive software 🚀 or features to prevent problems in production environment like described here: [Flood protection](https://blog.openhive.network/hive-139531/@andablackwidow/flood-protection-in-1-27).

What is  worth noting, 5 years ago when Hive was born, its testing ability was minimal and actually the whole test suite contained chain_test and plugin_test tools being able to run multiple simple unit tests to verify basic funcions. They are both still present right now...  

## 🔬 Transaction inspector
Regression tests 🧪, additional tools 🔧 and automation of development processes 🤖 are useful for us, developers... But we also think about you: Hive users. One of the biggest problems you have to deal with is the rejection of transactions due to improper authorization. To address this issue and help solve it ***without a need to look for a Blockchain Wizard*** we created a new application: [Transaction Inspector](https://tx.openhive.network/).

Its purpose is to visualize the process of verifying the authenticity of the transaction by showing analysis steps performed starting from account signing the transaction, then coming up through (potential) intermediate account authority redirections (e.g. if you permitted some other entities to operate on behalf of your account) to finally end with the account satisfying the authority required by a given transaction.

Let's discuss briefly the main application features - we will publish soon a separate article where authority analysis steps will be discussed in details. To work with it you need to point a transaction being an analysis subject. You have various ways to do it:

- interactively, by using a `Provide transaction` button and pasting transaction contents into the dialog as:
  - a JSON form, in this case you can point transaction which was not accepted yet by the chain which let you find the reason for evaluation failure
  - a transaction binary form encoded as a hexstring (by choosing `Binary` option); similar to the previous case you can point potentially broken/not yet broadcasted transaction
  - an uploaded file which can contain transaction JSON or a binary form encoded as a hexstring or just as a binary buffer. The application will detect automatically the type of passed content.
  ![Tx Input dialog](https://images.hive.blog/DQmRMFJKgLMrFoc4RfY5R1pTDoE9qNJRvHVMevoJz1kW7qS/txInputDialog.png)
- by pointing it using URL parameters while opening a Transaction Inspector application. In this case you can point a transaction hash (to look at the details of transaction already accepted by blockchain) or its base64 serialized content. Please be advised that the application automatically generates URL params also after interactive transaction load to make simple sharing it with others. Here is a simple example:
[https://tx.openhive.network/?transaction=07e7d9ecaad2a11bc36db6d7046a55e231627574](https://tx.openhive.network/?transaction=07e7d9ecaad2a11bc36db6d7046a55e231627574)

After opening the above link, you should get a similar view:
 ![ProcessedTx.png](https://images.hive.blog/DQmZRQn743a2TodWsNbyET7by3N9V9ifmftR9AdP3BvH1mX/txProcessedTx_resized.png)
 
As you can see, application shows information divided into several sections specific to:
- overall transaction verification status,
- signatures associated with the transaction, together with authority paths evaluated (by chain algorithm) while verifying transaction authority,
- set of required authorities (it depends on operations included in transaction),
- basic transaction header data which allow preserving blockchain sanity (TAPOS, expiration time),
- body (set of operations) which can be displayed using a `Formatted`, `JSON` or `Binary` view to allow you a better analysis by choosing a preferred presentation form.

The following application documentation contains the description of mentioned things: [Transaction inspector documentation](https://gitlab.syncad.com/hive/tx-inspector/#app-specification).

The important note is that this application uses the same algorithm to process authority as Hive Protocol due to the functionality exposed by a wax library.

Please enjoy yourself! We hope you will find this application useful. We also plan to improve it more soon to simplify significantly the analysis of the negative cases.

## 🛠 hAIve - our new Arificial Intelligence powered search engine

One of our wishes for Hive's birthday was to do something exciting both for you - the users of the platform - and for us - programmers involved in its development. I think we all love to have contact with the cutting edge technologies... That's why we decided to spend some of our time on the prototype of the engine, powered by AI and offering us some practical features. During our brainstorm we had a lot of great ideas on how we can use AI technology to improve our social platform. They all lead to a common requirement: massive processing of blockchain data (e.g. posts) and teaching some AI models with it. Fortunately, we have HAF already so making a dedicated app to process posts was quite simple at least at side specific to receiving input data...

Another big challenge was to select a proper AI model and configure it to make it work efficiently. In the end, we need a solution to provide ability to serve API calls and here again we could reuse our experience related to PostgREST integration into HAF applications.
Finally, we created the application called internally `HiveSense` having following architecture:

 ![HiveSense diagram.png](https://images.hive.blog/DQmXqdYiRkf2aGALbMbGHQAS6SnhDxzy6zkt25s9gUGz3bQ/HiveSense%20diagram.png)
 
To make the entire project successful we had to achieve an acceptable level of performance. The beginnings (as always) were difficult and first versions of the application were able to process the first 5M blocks in a few days. Currently, the optimizations made both in the software layer and the system and hardware configuration allowed the `HiveSense` application to be supplied with a full set of `94M` blocks in `38 hours` and to stick at head block continuously with `live-sync` processing block times at level: `0.078173 s`.

The amazing thing is that our software development and system administration team members involved into this project, had to use just two regular PCs:
- one serving AI models having onboard: AMD Ryzen 7 3800X and GeForce RTX 3060 12GB VRAM
- second holding HAF instance (and PostgreSQL database) built using: AMD Ryzen 7 5800X, 128GB RAM

⚡ Monster(s) ⚡ - yes, They are software development monsters and magicians 🧙🏼‍♂️ doing best things ❗

The important thing to mention is also the timeframe of this task: it took ca. **one month** to start from the rough idea to implement the working prototype 🪄✨. Of course you can test it yourself: we prepared a special version of our Blog App where regular search backend has been replaced with our AI powered engine.

You can find it here: [AI search engine prototype](https://blog.dev.openhive.network/search).

Below, there are some interesting search prompts where you can see the difference (yes, really big) in the quality of received results - please compare to condenser search, available here: [Current search engine](https://hive.blog/search).

Some example prompts:

| Prompt | HiveSense results | Current search results |
| -------- | -------- | -------- |
|russian classic cars| [russian classic cars](https://blog.dev.openhive.network/search?q=russian%20classic%20cars)| [russian classic cars](https://hive.blog/search?q=russian%20classic%20cars)|
|breed frogs| [breed frogs](https://blog.dev.openhive.network/search?q=breed%20frogs)| [breed frogs](https://hive.blog/search?q=breed%20frogs&s=newest)|
|swimming competition| [swimming competition](https://blog.dev.openhive.network/search?q=swimming%20competition)| [swimming competition](https://hive.blog/search?q=swimming%20competition&s=newest)|

Another nice feature provided by this version are `related posts` shown for given one, where the list of such posts is created by looking for similar word vector values computed during AI model processing.

Please look at side panel described as `You Might Also Like` e.g. displayed for this [post](https://blog.dev.openhive.network/cars/@mattsbeamer/an-old-car-rally-driving-through-moscow-russia)

## 📡 Denser

Today we would also like to announce providing the new version of [denser](https://gitlab.syncad.com/hive/denser) application.

The idea behind the creation of this application was to provide functional equivalent of old Condenser software, but made by using nowadays technology stack. This way we could reduce maintenance effort (especially) required to preserve acceptable security level.

Our frontend team did a great job to implement a wallet (similar to the one available e.g. at [hive.blog](https://hive.blog), improve posts rendering and **finally** add support for signing transactions using [Keychain](https://hive-keychain.com/). We hope you will find this application handy and appreciate it, just by switching your blog application to [denser](https://blog.openhive.network/). 

## 🍯 A wAx library

As we have already announced in past articles:
* [wax announcement](https://blog.dev.openhive.network/hive-139531/@thebeedevs/for-those-developing-a-hive)
* [development guilde](https://blog.dev.openhive.network/hive-139531/@small.minion/hive-application-development-guide)

the purpose of `wax` library was to provide flexible and safe way to operate with Hive platform. Another important feature is sharing the Hive protocol code that allows the 3rd party programmers (e.g. developing using Typescript or Python) to use directly types and algorithms specific to Hive Protocol. Such architecture (where blockchain code can be immediately called from a Web application instead of gathering results through network API calls) opens a lot of opportunities for implementing new features, making common functional layer where things are evaluated in the same way as hived service does. The examples can be `Transaction inspector` application (mentioned above, peforming transaction authority verification in the same way as hived), as well as binary view component (embeeded in it but also available separately) able to process transactions the same way as low level blockchain code.

The last several months of development allowed us to enhance this library a lot. Implemented features contain:

- support for external signature providers (wax can be still used the same way as previously - it just additionally allows to use another signature provider (i.e. Keychain instead of Beekeeper) to sign your transactions,
- implementation of REST API wrappers (the previous version contained support for JSONRPC calls). Thanks to that, now you can call network APIs just like regular Typescript method (regardless of its kind: wax hides complexity of specific requirements to call `REST` or `JSONRPC` API)
- providing an ability to generate (above mentioned) API call wrappers from `swagger.json` specification which simplified the development process of e.g. [haf_block_explorer API calls](https://gitlab.syncad.com/hive/haf_block_explorer/-/jobs/1749168)
- a component ([Healthchecker](https://gitlab.syncad.com/hive/wax/-/wikis/classes/HealthChecker)) useful for analysing network call statistics of REST/JSONRPC call endpoints. Such a tool class can be used by 3rd applications to measure (at webbrowser client side) the quality of Hive service providers and allow the final user to select the best one
- new meta operation classes, e.g. [AccountAuthorityUpdateOperation](https://gitlab.syncad.com/hive/wax/-/wikis/classes/AccountAuthorityUpdateOperation) which simplifies a way to create transactions to update account authority. It also implements many additional checks that prevent damaging account authority by misusing direct blockchain operation
- preventing private key leakage while scanning prepared transactions before broadcast or explicitly by calling a [dedicated method](https://gitlab.syncad.com/hive/wax/-/wikis/interfaces/IOnlineTransaction#performonchainverification)

The library test suite has been also extended a lot to allow us to provide you with a much better and stable product. 

We also did a lot of optimisations, including great [reducing of package size](https://gitlab.syncad.com/hive/wax/-/merge_requests/260).

The big step forward is also **Python** support which has been done lately [Python support](https://gitlab.syncad.com/hive/wax/-/merge_requests/230).

The whole list of done features you can see by visiting a [project releases page](https://gitlab.syncad.com/hive/wax/-/releases)

The library is used by a few applications offered by Hive platform like:
- denser (frontend, Typescript)
- transaction-inspector (frontend, Typescript)
- block_explorer_ui (frontend, Typescript)
- clive (desktop, python)
- test_tools (desktop, python)

We would like to encourage you to try it as it offers great integration to IDEs intellisense (like Visual Studio Code) which speeds up the development process a lot !

**Happy** Coding ❗

## 🔒 Yet more wallets to operate with Hive

The last important thing to mention - especially for non-developer part of Hive Community - is extending the support of alternative wallets to be used for Hive transaction signing.
As you probably know, our team also offers a desktop tool [clive](https://gitlab.syncad.com/hive/clive) which provides safe capabilities to sign transactions (e.g. using its offline signing features).

But that's not all - the Hive platform already contains several tools (like Keachain, PeakVault, Beekeeper) which allow signing your transactions inside a Webbrowser we lacked a tool which allow to enclose management of your keys in single place. Especially, if such keys are not only specific to Hive blockchain. To address it, we need to have a tool which by using modern cryptography features will be able to provide signing services to different blockchains. 

Creating such a tool would be very difficult, so we decided to use a great feature of wax instead: [external signature providers](https://gitlab.syncad.com/hive/wax/-/merge_requests/282). This way we are able to implement integration to... [MetaMask wallet](https://metamask.io/) and provide them a [Hive dedicated snap](https://gitlab.syncad.com/hive/metamask-snap/).

Besides, to make Hive onboarding process much easier (for MetaMask users already having a Hive account and those who - hopefully - would like to join our platform) we implemented dedicated service and wizard application to simplify configuration steps.

We hope that this service will become a *bridge* between Hive and the external world... You can find it [here](https://auth.openhive.network).

 ![HiveBridge.png](https://images.hive.blog/DQmbxXhUHpXXFCaQL4Yx4oruMbNsmQrXUqmzCRAARqZ1CB2/HiveBridge.png) 

Offered functions cover:

- preparing a request to create a new Hive account. Yes we know, creating a Hive account can be troublesome. Most users would like to avoid all such terrible things like saving private keys, then choosing their **PUBLIC** part to send them to a *Hive Friend* be able to create an account for you. We hope we simplified this process maximally and our application let you focus on selecting beloved name of your account, then allow copying a link to be shared with your *Hive Friend* which will create the account for you. You don't need to save private keys nor extract public keys from them - the MetaMask wallet involved by us in this process will do it for you 
- processing an account creation request (e.g. received from your friend which would like to join Hive)
- updating your Hive account authority, e.g. to associate it with just configured MetaMask instance (this case you can sign your transaction using another wallet e.g. Keychain). Such a process can be also initiated by our wizard simplifying MetaMask and Hive account configuration
- encrypt/decrypt a buffer of arbitrary text, using keys associated with your Hive account. This feature can be useful for interacting with other services (like some bots) that may require you to prove your identity by encrypting the content they provide, then attempting to decrypt it using your public key, thus verifying your identity
- signinig (and optional broadcasting) of any transaction prepared outside of Hive Bridge service

Following animation shows steps to be performed to setup Meta Mask wallet to Hive account:

https://youtu.be/zKT1GXO6G-0


***WARNING*** At the moment our [snap](https://gitlab.syncad.com/hive/metamask-snap/) implementation needs to be accepted by Meta Mask as trustworthy. See [how MetaMask accepts snaps](https://docs.metamask.io/snaps/how-to/get-allowlisted/). Because of it, we need to operate with MetaMask Flask browser extension instead of the regular one.

---

🎂🎈🎉🥳 After party 😊

We hope you enjoyed reading this article. We would like to see you  using successfully our applications and libraries described above.

See you on Hive ❗
Thanks for reading and we are waiting for your feedback - which is invaluable...

Best regards,
    @thebeedevs team