# Back Office Release 4 statistics module V 1.1.0

## Introduction

The purpose of this document is to provide a simple guideline in order to be able to installing the statistic module belonging to the Release 4 for Backoffice application.

### Prerequisites

* Linux based OS
* Docker and Docker Compose installed
* Basic understanding of Docker and Docker Compose
* Knowledge of Logstash, Elasticsearch and Kibana
* Experience with Linux command line

If you are using some other operating system, you can follow the instructions specific to that OS but the process remains more or less the same.

### Docker application

The *SPBO Release 4 statistic module* is provided as a set of [Docker](https://www.docker.com/) images. Currently, there are three differents images:

* **elastic**: Elasticsearch server.
* **kibana**: Kibana server.
* **logstash**: Logstash server.

The elastic, kibana and logstash images will work together for the data processing and display the statistic dashboards.

![Elastic Stack](https://cdn-images-1.medium.com/max/800/1*vZJLwJ2XrENyo9sYnI-Ehg.png)

### Files Structure.

Short explanation how the application have been structured:

* **docker-compose.yml:** File where is defined all services needed to run the system.
    - **elasticsearch**
        * **dockerfile.conf:** File where is defined all information needed to create elasticsearch image
        * **elasticsearch.yml:** File where is defined specific configuration for elasticsearch.
        * **cert:** Folder containing certificate to Basic authentication.
    - **kibana**
        * **dockerfile.conf:** File where is defined all information needed to create kibana image
        * **kibana.yml:** File where is defined specific configuration for kibana.
        * **objects:** File where is placed all dashboard, index-patterns, visualizations and searches.
    - **logstash**
        * **dockerfile.conf:** File where is defined all information needed to create logstash image
        * **mysql-connector-java-5.1.36-bin.jar/mssql-jdbc-9.4.1.jre8.jar:** Connector needed to connect to backoffice database
        * **pipeline:** Folder where are defined all pipelines needed to feed up elasticsearch.
    - **Wipo-Converting**
        * **wipo-converting.war:** Application developed to transforms csv files from kibana into wipo templates.

## Installation

Back Office Release 4 statistics module package comes setting up by dockers images, therefore just a straitforward steps are need to install the system.

### Step 0 - Configuration Pipelines Parameters

Before executing docker-compose command is going to be needed to review some environment parameters placed within dockerfile.conf under logstash folder, inside of this file you will the next parameters:

```
  "${jdbc_connection_string}" : url database connection
  "${jdbc_user}" : user used for database
  "${jdbc_password}" : password used for database
```
Also review and update the next two parameters depending on the type of database Mysql (by default) or SQL server
```
"${jdbc_library}" : JDBC library used to connect with the database
"${jdbc_driver_class}" : JDBC Class path inside the library.
```


### Step 1 - Using docker-compose to start intallation process

In the root of package intallation where docker-compose.yml file is located, run following instruction.

```
> docker-compose up -d
``` 
This command will start the proccess, downloading the images, creating the containers and running them.

If the previous procces have worked correctly, it will appear in the console the following messages, meaning servers are running.

```
Successfully built 2a7d147d1de7
Successfully tagged kibanareportingbo_logstash:latest
WARNING: Image for service logstash was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating elasticsearch ... done
Creating logstash      ... done
Creating kibana        ... done

```
### Step 2 - Testing environment have been installed correctly

let's check that containers are running, executing following command

```
> docker container ls

```
it should appear something similar to:

```
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS                              NAMES
50e0ecb1d50a        spbackofficekibana_kibana          "/usr/local/bin/kiba…"   12 minutes ago      Up 6 minutes        0.0.0.0:5701->5601/tcp             kibana
837088ce2afd        spbackofficekibana_elasticsearch   "/usr/local/bin/dock…"   12 minutes ago      Up 10 minutes       9300/tcp, 0.0.0.0:9700->9200/tcp   elasticsearch
a1b2c48af47f        spbackofficekibana_logstash        "/usr/local/bin/dock…"   12 minutes ago      Up 5 seconds        5044/tcp, 9600/tcp                 logstash
```

Go to the browser and typing >  http://ip_machine:5701/app/kibana#  you should see kibana environment working.

### Step 3 - Installing Objects

In kibana environment in the section 'managment -> saved objects' in the upper corner is the import option. Importing all json files you will see in the folder 'objects' under kibana structure.

#### Warning !

After importing all objects it will be needed select a default index-pattern in order to be able to see all dashboards. To do this action please go to index-pattern option and select one of them, after that, click on the star, placed upper corner of this screen.

## Activating Security - Basic Authentication.

We will be setting up basic authentication on Elasticsearch. Also, Kibana UI will require the username and password to login. For our setup, we will be using Docker Compose which makes our setup very easy to deploy anywhere.

### Step 1 - Modifying container elasticsearch.yml file

Let's access to the container of elasticsearch, executing following command

```
> docker-compose exec elasticsearch bash

```
You will know your inside to the container because your promt will change to something similar to:

```
[root@837088ce2afd elasticsearch]

```

once you are inside to elasticsearch container let's edit elasticsearch.yml file running vi command.

```
[root@837088ce2afd elasticsearch] vi ./config/elasticsearch.yml 
```
and it needed to add following information at the bottom.

```
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: test.p12
xpack.security.transport.ssl.truststore.path: test.p12
```

now, it's needed to go out from the container executing exit command:

```
[root@837088ce2afd elasticsearch] exit 
```

and to finish with elasticsearch configuration restart the container with

```
> docker-compose restart elasticsearch
```

### Step 2 - Creating Passwords in Elasticsearch.

let's create password needed to access to elasticsearch once security have been activated.

To create passwords it needed to access again to the elasticsearch container:

```
> docker-compose exec elasticsearch bash

```
once inside container, execute following elasticsearch command

```
[root@837088ce2afd elasticsearch] ./bin/elasticsearch-setup-password auto
``` 
it will be created several passwords similar to:

```
Changed password for user apm_system
PASSWORD apm_system = random password

Changed password for user kibana
PASSWORD kibana = random password

Changed password for user logstash_system
PASSWORD logstash_system = random password

Changed password for user beats_system
PASSWORD beats_system = random password

Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = random password

Changed password for user elastic
PASSWORD elastic = random password

```
Finally, it's needed to go out from the container executing exit command:

```
[root@837088ce2afd elasticsearch] exit 
```
#### Warning!
Please, save passwords information in a save place and do not remove it.

### Step 3 - Modifying container kibana.yml file

Let's setting up Kibana in order to be able to connect to elasticsearch.

Go inside to kibana container, executing following command

```
> docker-compose exec kibana bash

```
You will know you are inside to the container because your promt will change to something similar to:

```
[bash-4.2$]
```

once you are inside to kinbana container let's edit kibana.yml file running vi command.

```
[bash-4.2$] vi ./config/kibana.yml 
```
and it's needed to add following information at the bottom.
```
elasticsearch.username: "kibana"
elasticsearch.password: random password
```
save this information and go out to the container executing exit command:

```
[bash-4.2$] exit
```
Finally, restart kibana container.

```
> docker-compose restart kibana 
```

after this final step, once the service have been restarted, kibana credentials screen should appear.

### Step 4 - Configuring security in Logstash

Setting up logstash for connecting to elasticsearch is straightforward, and very well explained in this section from elasticsearch documentation.

```
https://www.elastic.co/guide/en/logstash/6.8/ls-security.html
```

## Troubleshooting

### Container does not appear in the list with "docker container ls"

There are mainly two reason why containers do not start.

* Configuration error.
* Fatal error, it's not possible to run the container again.

#### Checking logs in order to know the problem.

It's possible to see logs of every container executing following command

```
> docker logs elasticsearch 
``` 
or checking last numbers of lines specified.

```
> docker logs --tail 100 elasticsearch
``` 

#### Removing unreachable container.

In case a fatal error making a container impossible to running, it's possible to fix the system removing the container created and running again the command docker-compose to run the system.

Step 1 - shutdown the system running following command

```
> docker-compose stop container_name (eg. elasticsearch or kibana or logstash)
```

Step 2 - check corrupt container is removed running following command

```
> docker container ls -a
```

Step 3 - remove it with command

```
> docker rm container_name (eg. elasticsearch or kibana or logstash)
```

Step 4 - run again the system

```
> docker-compose up -d container_name (eg. elasticsearch or kibana or logstash)
```

### Cleaning all docker system to start from scratch.

When it's needed to clean all system to starting with a clean envoriment then execute following command

```
> docker system prune
```

### Wipo-Converting tool.

Wipo-converting tool is an application developed to convert csv data from kibana dashboards into wipo standards templates.

This application is delivered ready to be deployed into an applications server like for instance, wildfly.

Once have been deployed the url to access to this application is

```
http://server:port/wipo-converting/app/
```

#### Caution.

In case, it is the first time running the application, it needed to pre-install proccesing data folder, running following url.

```
http://server:port/wipo-converting/resources/ping
```
After executing this url should appear a message as follow:

```
Hey Ready to Work Dude !
```
If you see this message, application is ready to use.

#### How to install templates

Once the application have been deployed and executed previous steps you will have to install last version wipo template in the url
../procesing_templates/templates.

To get wipo templates you can use two urls.

Application resources ( wipo 2019 templates )

```
https://code.tmdn.org/sp/sp-backoffice-wipo-converting/tree/master/src/main/resources 
```
or directly from wipo website for last version

```
https://www.wipo.int/ipstats/en/data_collection/questionnaire/
```

# Kibana Statistics Versions

### V 1.1.0

Improvements and fixes

    List of representatives to dossiers, updated dashboards
    New index for correspondence:
         08. R4 Dashboard Trademarks / Letters sent
         09. R4 Dashboard Trademark IR / Letters sent    
         10. R4 Dashboard Designs / Letters sent    
         11. R4 Dashboard Design IR / Letters sent
         12. R4 Dashboard TM Recordals / Letters sent
         13. R4 Dashboard / Overall view - Dossiers and Letters sent
         13. R4 Dashboard / Overall view - Letters sent
         14. R4 Dashboard / Overall view - Dossier/Letter index
         27. R4 Dashboard TM Recordals / Sent letters monthly 
     New fields to dossier and tasks index

Microsoft SQL Server compatibility

     Library & Prepare configurtion
     All fields to lowercase

### V 1.0.4

16.1 R4 WIPO - TM Applications by Origin

     Dashboards adapted to use the new field applicant

16.2 R4 WIPO - DS Applications by Origin

     Dashboards adapted to use the new field applicant


### V 1.0.3

new wipo-application version.

    Minor issue converting text in cells to numbers 


### V 1.0.2

04 R4 Dashboard / Designs

     Designs with Withdrawn status filter added.

05 R4 Dashboard / Design IRs

     Design IR designated to Finland - removed

21 R4 WIPO - DS in force by year of registration

     Test Registration_designs - added

28 QS - Trademarks - Time from filing to registration

     Exclude letter VP filter added. 
     R4 QS  trademarks_median_time_to_registration added

new wipo-application version.

     Now is possible to update wipo reports annually.
     Number format applied to templates.

### V 1.0.1

Minor changes in visualizations:

1. Changed average time to Median time in QS dashboards.
1. Fixed range of time in dashboar 23. R4 Dashboard / Overall view - Monthly applications comparison

### V 1.0.0

First release.

## Author

* **Carlos Miguel Benitez Prieto** - carlos.benitez@iecisa.altia.es




