Build a scalable and fault tolerance application — Part 1 — Create the project

In the next posts we will develop a social web application, for travelers with the following tools and frameworks:

• Java 11 and Spring Boot for the backend
• React for the frontend
• Spring Cloud for a microservices architecture
• JHipster to generate the project
• MongoDB as database
• Redis for caching
• Apache Kafka for asynchronous messaging between microservices
• Elasticsearch for logging and analytics
• Docker for containerisation

The application will be easy to scale, fault tolerant and secure.

Fault tolerance is the property that enables a system to continue operating properly in the event of the failure of (or one or more faults within) some of its components.

Scalability is a characteristic of a system, model, or function that describes its capability to cope and perform well under an increased or expanding workload.

The application it’ll let you to create an account, create a post about a country or city where you traveled, search after other posts, to give a review to a post and of course a statistic page for the admin where he’ll find out how many users have created an account in the past days, how many posts where created and which countries or cities were most visited until now.

The architecture of the application

Above, we have the architecture of the application:

• Gateway is the the main entrance of the external users, where we have a React App for the user interface, Zuul Proxy and Ribbon for routing and load balancing and an access control service.
• Registry for service discovery and microservices configration.
• Each microservices (User, Posts, Statistics and Gateway) have a distributed Redis cache and a MongoDb database.
• Elastichsearch is use for logs analysis.
• Kafka for asynchronous communication between microservices.

Now we have all the informations regarding the application, we have to be sure that we have all the below tools installed:

• Java 11: https://www.oracle.com/java/technologies/javase-jdk11-downloads.html
• Git: https://git-scm.com/downloads
• Node.JS: https://nodejs.org
• Docker: https://www.docker.com/get-started

The GitHub repository for this tutorial:

• Gateway: https://github.com/Daniel194/travelers-gateway
• Users: https://github.com/Daniel194/travelers-users
• Posts: https://github.com/Daniel194/travelers-posts
• Statistics: https://github.com/Daniel194/travelers-statistics
• Config: https://github.com/Daniel194/travelers-config

For this post, the corresponding branch is post 1 on each repository.

JHipster is a free and open-source application generator used to quickly develop modern web applications and Microservices using Angular, React or Vue.js, for the frontend, and the Spring Framework, for the backend.

First of all, we’ll have to install JHipster to our machine, so go to a command line and install JHipster with the following command:

npm install -g generator-jhipster

After the installation of JHipster is done, create a directory and go into it:

mkdir travelers-gateway && cd travelers-gateway

From the directory, run JHipster, with the next command and follow instructions on screen:

jhipster

Create the gateway

Now that we have the gateway created, let’s go back and create the microservices for Users, Posts and Statistics. Do the following instruction for each of them:

mkdir travelers-users && cd travelers-users

From the directory, run JHipster, with the next command and follow instructions on screen:

jhipster

Create microservice

For the Statistics and Posts microservices do the same things, but change the port from 8081 to 8083 for Posts and to 8085 for Statistics

Let’s run the application with DEV profile to be sure that everything is running fine, but first we’ll have to do some modifications in the configurations files.

First open Users microservice in your favorite IDE and modify the port of MongoDB container from 27017 to 27081 in the following files:
application-dev.yml, application-prod.yml, mongodb.yml and mongodb-cluster.yml . Don’t make any change in the app.yml file.

Now we’ll change the port of the Redis container for Users microservices from 6379 to 6381 in the following files: application-prod.yml, redis.yml and RedisTestContainerExtension.java and add the next property in application-dev.yml because by default it’s set to port 6379 by JHipster.

jhipster:
  cache:
    redis:
      expiration: 3600
      server: redis://localhost:6381

JHipster, create a instance of Elasticsearch per container, but since we are using only one instance of Elasticsearch in our application for log analyse we have to change the name of the container from user-elasticsearch to elasticsearch in the following files: app.yml and elasticsearch.yml and add the next property in application-dev.yml

data:
  jest:
    uri: http://localhost:9200

From the command line, we’ll have to runt the following Docker commands:

• MongoDB: docker-compose -f src/main/docker/mongodb.yml up -d
• JHipster-registry: docker-compose -f src/main/docker/jhipster-registry.yml up -d
• Redis: docker-compose -f src/main/docker/redis.yml up -d
• Elasticsearch: docker-compose -f src/main/docker/elasticsearch.yml up -d
• Kafka: docker-compose -f src/main/docker/kafka.yml up -d

Before to start the app we have to fix some errors generated by Netty, add the next line in logback-spring.xml to enable logs for Netty

<logger name="io.netty.util" level="ERROR"/>

Both Elastichsearch and Redis use Netty, and they will try to start the same resource. For this reason we’ll have a concurency problem. To avoid this to happen we’ll add the next command at the top of the main method:

System.setProperty("es.set.netty.runtime.available.processors", "false");

Now, let’s run the application locally to see if everything run correctly. Run the next command in the console:

./gradlew

The above command will start the application with DEV profile.

We have now to do the same thing for Posts, Statistics and Gateway microservices. Just change the port for MongoDB from 27017 to 27083 for Posts, to 28085 for Statistics and to 28080 for Gateway. Change the port for Redis from 6379 to 6383 for Posts, to 6385 for Statistics and to 6380 for Gateway and change the name of Elasticsearch instance from posts-elasticsearch, statistics-elasticsearch and gateway-elasticsearch to elasticsearch. All the other steps are the same.

For Gateway microservices you have to run the next command in the terminal before to start the container:

npm install

The above command will bring all the dependencies for the user interface, which is written in React.

Run the following commands in two separate terminals to create a blissful development experience where your browser auto-refreshes when files change on your hard drive.

./gradlew -x webpack
npm start

Now that we have the application run in DEV mood, let’s dockerize our application and run it with PROD profile. We’ll start with the Gateway microservice, but first be sure that you have the related microservices for the gateway: Elasticsearch, MongoDB, Redis, Kafka and Registry in PROD profile, until now all the microservices were started in DEV profile. To do this you have to shut down and delete all the container started until at this moment. You can do this by running next command in a terminal:

docker stop $(docker ps -a -q) && docker rm $(docker ps -a -q)

To achieve dockerize for a microservices, first build a docker image of your app by running:

./gradlew bootJar -Pprod jibDockerBuild

Then run the next command to build and start the container for Gateway:

docker-compose -f src/main/docker/app.yml up -d

The above command will run seven containers: MongoDB, Redis, Elasticsearch, Registry, Kafka, Zookeeper and Gateway so be sure you have enough RAM and CPU available to run that command.

If everything is fine, do the same thing for Users, Posts and Statistics microservices. At the final of this process you should have 16 containers running. You could verify this by running the next command.

docker ps -a --format="table {{.Image}}\t{{.Names}}\t{{.Ports}}"

Open you favorite browser an go to http://localhost:8080/ and login in as admin. You should have access to all API endpoints and their descriptions by Swagger at this address: http://localhost:8080/admin/docs

You could also access Registry service to be sure that you have four application instances running at this moment by accessing this address: http://localhost:8761/registry/applications . You should see something like this

Congratulations, you have now a fully functional microservices application running on your local machine. In the next post we’ll add some business logic to our application, because at this moment you can only just login and create an account in the application.

Stay tuned !