Hive hosts lots of different languages. If you want to categorize the content, one of the categorization points is the language used for the posts/comments.

Detecting the post's language was one of the requirements in my current project. It turns out it was pretty easy to do it, dropping here some experience for future reference.

#### What do we need?
***

- A function fetches the post body and removes the noise from the post body
- A function returns the language code detected in the post body

#### Clearing the noise
****
To feed the language detector, we need to filter out the HTML and Markdown tags. Passing clean data as much as possible will increase the language detection's correctness.

```
from bs4 import BeatifulSoup
from markdown import markdown

def clear_noise(post_body):
    body_as_html = markdown(post_body)
    soup = BeautifulSoup(body_as_html, "html.parser")
    post_body = ''.join(soup.findAll(text=True))
    post_body = re.sub('<.*?>', '', post_body)
    post_body = re.sub(
    '```.*?```', '', post_body, flags=re.MULTILINE | re.DOTALL)
    return post_body
```
***
This simple snippet removes Markdown, HTML tags, and try to return only sentences/words. 

#### Detecting the language
***
*"There is a library for that"* is the answer to most questions in Python world. I've used [langdetect](https://pypi.org/project/langdetect/) in the past and it was working considerably well.

```
from lighthive.client import Client
from langdetect import detect

def detect_language(author, permlink):
    c = Client()
    post_body = c.get_content(author, permlink)["body"]
    post_body = clear_noise(post_body)

    return detect(post_body)
```
***
We've used lighthive's Client to fetch the post body, cleared the post text with the `clear_noise` and passed it to the language detector library.

Let's test it with some example posts; I've picked four different posts with four different languages:

```
posts = [
    ('themarkymark', 'stemgeek-s-first-hackathon'),
    ('emrebeyler', 'yeni-baslayanlar-icin-hive'),
    ('clayop', '4v8phu-9'),
    ('satren', 'virtuelles-dach-meetup-am-sonntag-dem-17-05-2020-18-00-q9r1qv'),
]

for author, permlink in posts:
    language = detect_language(author, permlink)
    print("Language of @%s/%s is %s" % (author, permlink, language))
```
***

Output:
```
Language of @themarkymark/stemgeek-s-first-hackathon is en
Language of @emrebeyler/yeni-baslayanlar-icin-hive is tr
Language of @clayop/4v8phu-9 is ko
Language of @satren/virtuelles-dach-meetup-am-sonntag-dem-17-05-2020-18-00-q9r1qv is de
```
***
Pretty accurate. You can try the script without installing anything at [repl.it](https://repl.it/@emre2/HIVE-language-detection).


#### Bonus data: Language saturation on HIVE
***

This was for a small period of time, but probably gives a good idea on the actively used languages at HIVE.

![Screen Shot 2020-05-03 at 18.59.24.png](https://images.hive.blog/DQmZ9U8ujS5zy1CcsTvTVBoGcRWNdeS7yPaTZm1SKsueKxP/Screen%20Shot%202020-05-03%20at%2018.59.24.png)


#### Notes
***
- langdetect supports 55 languages and the returned code is [ISO 639-1 ](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) language code.

- There is also a [langid](https://github.com/saffsd/langid.py) library with a little bit less precision, but with faster detection times. Check it out if you have tight time constraints.