- read

Developing Nest.js Microservices Within a Monorepo: A Step-by-Step Guide

Web Development
Trending

Moein Moeinnia 86

Welcome to a journey into the world of microservices development within a monorepo. In the world of modern software, combining microservices and monorepos has become a helpful way to develop software more efficiently, make it easy to grow, and reuse code better.

Table Of Content:
Step 1: Converting Project To Monorepo
Step 2: Transitioning to Microservices
Step 3: Configure Communication Patterns
Step 4: Configure Intra-Service Communication
Step 5: Debugging Microservices
Step 6: Running All Microservices Together

Before we dive into the code and see how we can utilize microservices inside NestJS, we need to be familiar with 2 concepts:

Microservices: Microservices is a widely used architecture that aims to decouple services serving different business purposes. Microservices are popular for various reasons, including decoupling, faster CI/CD, easy management, and scalability.

Monorepo: A monorepo is a term used for a repository that contains multiple projects within it. In our case, it’s a repository that encapsulates all of our services (also known as microservices) and has a single Git repository.

Benefits of Monorepo over Polyrepo:

  1. Unified Development: Everything in one place for easier work.
  2. Code Sharing and Reusability: Share and reuse code across projects easily.
  3. Dependency Management: Easily manage and share all your dependencies across all services.
  4. Simplified CI/CD

While there are various methods like Nx, TurboPack, and others to create monorepos, the good news is that NestJS simplifies the process of crafting and managing multiple microservices. Without relying on third-party tools, NestJS offers native support for various communication patterns. It also provides the flexibility to independently run and build distinct microservices using the Nest CLI.

Step 1 — Converting Project To Monorepo

According to NestJS documentation, there are two methods for organizing your code:

  1. Standard Mode
  2. Monorepo

By default, projects created using the Nest CLI nest new new-projectare set up in the standard mode. Initially your nest-cli.json file will appear like this:

{
 "$schema": "https://json.schemastore.org/nest-cli",
 "collection": "@nestjs/schematics",
 "sourceRoot": "src",
 "compilerOptions": {
 "deleteOutDir": true
 }
}

To set up your project in monorepo mode, you need to introduce projects within it. There are two primary project types:

1. Applications: These are standalone NestJS projects.

  • Create an application: nest generate app

2. Libraries: These encompass shared modules used across applications.

  • Create a library: nest generate library

Following the generation of these projects, your project’s structure will be structured as follows:

📦NestJS_Project
 ┣ 📂apps
 ┃ ┗ 📂app1
 ┃ ┃ ┣ 📂src
 ┃ ┃ ┃ ┣ 📜app1.controller.spec.ts
 ┃ ┃ ┃ ┣ 📜app1.controller.ts
 ┃ ┃ ┃ ┣ 📜app1.module.ts
 ┃ ┃ ┃ ┣ 📜app1.service.ts
 ┃ ┃ ┃ ┗ 📜main.ts
 ┃ ┃ ┗ 📜tsconfig.app.json
 ┣ 📂libs
 ┃ ┗ 📂lib1
 ┃ ┃ ┣ 📂src
 ┃ ┃ ┃ ┣ 📜index.ts
 ┃ ┃ ┃ ┣ 📜lib1.module.ts
 ┃ ┃ ┃ ┣ 📜lib1.service.spec.ts
 ┃ ┃ ┃ ┗ 📜lib1.service.ts
 ┃ ┃ ┗ 📜tsconfig.lib.json
 ┣ 📜nest-cli.json
 ┣ 📜package.json
 ┗ 📜tsconfig.json

And your nest-cli.json is going to look like this :

{
 "$schema": "https://json.schemastore.org/nest-cli",
 "collection": "@nestjs/schematics",
 "sourceRoot": "apps/app1/src",
 "compilerOptions": {
 "deleteOutDir": true,
 "webpack": true,
 "tsConfigPath": "apps/app1/tsconfig.app.json"
 },
 "monorepo": true,
 "root": "apps/app1",
 "projects": {
 "app1": {
 "type": "application",
 "root": "apps/app1",
 "entryFile": "main",
 "sourceRoot": "apps/app1/src",
 "compilerOptions": {
 "tsConfigPath": "apps/app1/tsconfig.app.json"
 }
 },
 "lib1": {
 "type": "library",
 "root": "libs/lib1",
 "entryFile": "index",
 "sourceRoot": "libs/lib1/src",
 "compilerOptions": {
 "tsConfigPath": "libs/lib1/tsconfig.lib.json"
 }
 }
 }
}

Step 2 — Transitioning to Microservices

Creating microservices in NestJS might seem as easy as a piece of cake, but there are certain aspects you should be mindful of. Essentially, there are two primary approaches to creating microservices in NestJS:

1. Utilizing NestFactory.createMicroservice() :

const app = await NestFactory.createMicroservice<MicroserviceOptions>(
AppModule,
 {
 transport: Transport.TCP,
 options: {
 host: 'localhost',
 port: 3001,
 },
 });
await app.listen();

While this method works well, it has two drawbacks that have given me pause in fully adopting it:

  • The instance created using the createMicroservice method is of type INestMicroservice instead of INestApp , which can introduce limitations when interacting with certain third-party packages and libraries. This is because these packages only accept inputs of INestApptype.
  • It only allows the initialization of a single communication protocol. For instance, you can solely listen on one port using one specific protocol.

2. Leveraging connectMicroservice() :

This approach is utilized to create an app instance that’s more adaptable and powerful, often referred to as a Hybrid app in the documentation. It performs better compared to the previous method, offering increased flexibility. With this approach, you can create an app of INestApp type, which allows you to listen on multiple ports and work with multiple transporters at the same time.

const app = await NestFactory.create(AppModule);
// microservice #1
const microserviceTcp = app.connectMicroservice<MicroserviceOptions>({
transport: Transport.TCP,
options: {
host: 'localhost',
port: 3001,
},
});
// microservice #2
const microserviceRedis = app.connectMicroservice<MicroserviceOptions>({
transport: Transport.REDIS,
options: {
host: 'localhost',
port: 6379,
},
});
await app.startAllMicroservices();
await app.listen(3002);

NOTE : When utilizing the connectMicroservice method and specifying the same port for both HTTP and TCP , the HTTP port configuration is ignored and we get only one TCP listener.

NOTE : It’s important to remember that if you omit the host parameter or provide a falsy value (such as null, undefined, or an empty string), the IPv6 version of localhost (::1) will be returned as the default value.

After completing the earlier steps, you can use the Nest CLI to launch your microservices individually. Use this command :

nest start [app_name]

For instance, in our example, you would run nest start app1.

Step 3 — Configure Communication Patterns

Within NestJS, two distinct types of transports are embedded:

  1. Broker-based: This category encompasses NATS, along with other options such as Redis, RabbitMQ, MQTT, and Kafka.
  2. Point-to-point: This group comprises TCP and gRPC communication mechanisms.

To configure your transporter, you need to modify the transport option in your microservice settings. Here's an example using the Redis or NATS transporter:

#REDIS trasnporter
transport: Transport.REDIS

----------------------------

#NATS transporter
transport: Transport.NATS

Step 4 — Configure Intra-Service Communication

To enable communication between different services, configuring their respective modules is essential and we can also use our services inside libraries.

Configuring Responding Server:

After we transitioned our server to microservice we have to add controllers to that server :

import { EventPattern, MessagePattern, Payload } from '@nestjs/microservices';

import { AppService } from './app.service';

@Controller()
 export class AppController {
 constructor(private readonly appService: AppService) {}
 
 //Broker-Based controller
 @EventPattern('testEvent')
 getHello(@Payload() data: any) {
 return `Hello !`
 }

 //Point-To-Point controller
 @MessagePattern('testMessage')
 getHelloTCP(name: string): string {
 return `Hello !`;
 }
}

NestJS supports two distinct message patterns, each catering to specific communication needs:
1. @MessagePattern: This pattern is tailored for point-to-point transport, offering direct communication between components.

  • Sync Responses: Controllers utilize a standard return type.
@MessagePattern({ cmd: 'sum' })
 accumulate(data: number[]): number {
 return (data || []).reduce((a, b) => a + b);
}
  • Async Responses: Controllers return a Promise.
@MessagePattern({ cmd: 'sum' })
async accumulate(data: number[]): Promise<number> {}
  • Stream Responses: Controllers return an Observable. this approach requires some basic knowledge about RxJS .
@MessagePattern({ cmd: 'sum' })
accumulate(data: number[]): Observable<number> {
 return from([1, 2, 3]);
}

2. @EventPattern: Designed for broker-based transporters, this pattern facilitates communication through a message broker. In this example, our server will listen to the user_created event :

@EventPattern('user_created')
async handleUserCreated(data: Record<string, unknown>) {
 // business logic
}

NOTE : We can assign multiple event handlers for a single event pattern,and this handlers run in parallel

Configuring Requesting Server:

When setting up the requesting server (also known as the client), you’ll need to make changes to the [app_name].module.ts file and register responding servers :

@Module({
 imports: [
 ClientsModule.register([
 {
 name: 'AUTH_REDIS',
 transport: Transport.REDIS,
 options: {
 client: {
 clientId: 'auth',
 brokers: ['localhost:6379'],
 },
 producerOnlyMode: true,
 consumer: {
 groupId: 'auth-consumer',
 },
 },
 },
 {
 name: 'AUTH_TCP',
 transport: Transport.TCP,
 options: {
 options: {
 host: "localhost",
 port: 3001
 }
 },
 },
 ]),
 ],
 providers: [AuthService],
 controllers: [AuthController],
})

NOTE : Libraries can also work as requesters, and you need to modify them in a similar way to the requesting server in the [library_name].module.tsfile.

Here’s how we can inject microservice and consume it:

import { Controller, Get , Inject} from '@nestjs/common';

@Controller()
 export class AppController {
 constructor(
 @Inject('AUTH_REDIS') private redisClient: ClientProxy;
 @Inject('AUTH_TCP') private tcpClient: ClientProxy
 ) {}

 //TCP requester
 @Get('call_tcp')
 getTCP(data: any) {
 return this.tcpClient.send('pattern_name' , data)
 }

 //Redis requester
 @Get('call_redis')
 getRedis(data: any) {
 return this.redisClient.emit('event_name' , data)
 }
}

NOTE : For injecting pattern we have to ues custom providers , its a technique commonly used in nest js , read this link for more info.

NOTE : By default, the connection is not created when the client server starts up (although we can achieve this using the onApplicationBootstrap hook). The connection will be established before the first microservice call and will be reused for later calls (the TCP channel will remain open).

Step 5 — Debugging Microservices

Since we use tools like Postman or Insomnia to debug REST APIs, we might also require tools to directly communicate with and debug our microservices. Setting up communication between Nest services is straightforward because everything is designed to work together and you can easily import your target microservices and call it.

However, what if some of your services are built using different programming languages? How do you handle situations like that?

While going into full detail about this subject would need its own article, in this blog post, I’ll cover how to identify problems with TCP Servers. However, I’m also planning to write another blog post that will explore this interesting topic further 😉.

Calling TCP Microservice:

To achieve this goal, we require a tool that can send and receive TCP packets. To fulfill this need, you can use tools like Packet Sender or Hercules.

For our testing purposes, we will be sending requests to this controller:

@MessagePattern('get_multiple'})
 accumulate(data: number): number {
 return data * 2;
}

Our packet will have the following ASCII-based structure:

37#{"pattern":"get_multiple","data":"1"}

In general, the metadata we transmit is formatted in JSON. However, there’s an additional element at the beginning of the string, indicated as (37#). Here, 37 signifies the character count within the string. You can visit this link to calculate the length of the string.

There are two methods to specify the pattern. The first way is by using a string input, 'create_user’. The second method involves providing an object, such as { cmd: 'create_user' }.
Based on the situation, our packets will have varying structures:


@MessagePattern('create_user') => "pattern":"create_user"

-------------------------------------

@MessagePattern({ cmd: 'create_user'}) => "pattern":{"cmd": "create_user"}

Step 6 — Running All Microservices Together

Typically, to run multiple microservices concurrently, you would dockerize each microservice and then use Docker Compose to bring all the images together. However, I’ve developed a simpler approach through a bash script that achieves the same outcome.

To get started, you’ll need to install concurrencyas a dev dependency:

npm install concurrency --save-dev

Now use this bash script to run all your Microservices concurrently:

#!/bin/bash
initCMD='npx concurrently'

cyanColor="\033[0;36m"

for app in $(cd ./apps && ls); do
 echo -e "${cyanColor}${app%/*} service starting ..."
 initCMD+=" \"npx nest start ${app%/*}\""
done

eval "$initCMD"

# For not closing terminal if we face an error
read -p "Press any key to continue" x

Thank you 🙏 for taking the time to read my blog post! Your comments 💬and claps👏would be greatly appreciated as they would motivate me to continue writing and improve.

Moein Moeinnia , My linkedin

Resources:

https://trilon.io/blog/using-nestjs-microservices-with-laravel

https://dev.to/nestjs/integrate-nestjs-with-external-services-using-microservice-transporters-part-1-p3

https://tkssharma.com/nestjs-microservices-of-all-types/

https://dev.to/lampewebdev/monorepo-and-microservice-setup-in-nest-js-41n4

In Plain English

Thank you for being a part of our community! Before you go: