Pitfalls in Testing NestJS Modules using HttpService
In this blog post, we will delve into the challenges developers face when testing NestJS modules utilizing HttpService
from @nestjs/axios
and explore the crucial role played by the RxJS TestScheduler
in overcoming these hurdles. Additionally, we will unravel the intricacies of injecting HttpService
into the test module, providing you with a comprehensive guide to fortify your testing strategies and ensure the robustness of your NestJS applications.
Table Of Contents
The Universities API
We will use the free Universities API as an example for our NestJS application. The response payload to the GET
request http://universities.hipolabs.com/search?country=Luxembourg looks like this:
[
{
"name": "International University Institute of Luxembourg",
"alpha_two_code": "LU",
"state-province": null,
"domains": [
"iuil.lu"
],
"country": "Luxembourg",
"web_pages": [
"http://www.iuil.lu/"
]
},
{
"name": "University of Luxemburg",
"alpha_two_code": "LU",
"state-province": null,
"domains": [
"uni.lu"
],
"country": "Luxembourg",
"web_pages": [
"http://www.uni.lu/"
]
}
]
We will now write a client for that API using NestJS. The source code for the application is available at https://github.com/gflohr/test-nestjs-http-service but you can also follow the step-by-step instructions below for creating that mini application yourself.
At the time of this writing, the application is using NestJS 10.3.0 and RxJS version 7.8.1 but the instructions should also be valid for other versions.
If you are an experienced Nest developer, you will know how to implement such a client. In that case you may want to jump directly to the section Writing Tests.
Scaffolding the Client
NestJS comes with a command-line tool for scaffolding new applications or individual components of existing applications. We will start by generating an application
$ npx nest new --strict universities
$ npx nest new universities
⚡ We will scaffold your app in a few seconds..
? Which package manager would you ❤️ to use? (Use arrow keys)
❯ npm
yarn
pnpm
CREATE universities/.eslintrc.js (663 bytes)
...
✔ Installation in progress... ☕
🚀 Successfully created project universities
...
We pass the option --strict
to nest new
to enable script mode in TypeScript.
Installing the required dependencies takes a little time. Once the installation is done, we change directory and start the application.
$ cd universities
$ npm run start
Pointing your browser to http://localhost:3000 should now show a hello world webpage. But we are targeting a client, not a server, and terminate the application with CTRL-C
.
Next we generate the heart of our application, the universities
module and inside of it the universities
service:
$ npx nest generate module universities
CREATE src/universities/universities.module.ts (89 bytes)
UPDATE src/app.module.ts (340 bytes)
$ npx nest generate service universities
CREATE src/universities/universities.service.spec.ts (502 bytes)
CREATE src/universities/universities.service.ts (96 bytes)
UPDATE src/universities/universities.module.ts (187 bytes)
Implementing the Client
Install Dependencies
We have some additional dependencies that have to be installed first:
$ npm install --save @nestjs/axios @nestjs/config
Configuration File
The next step is to create a stub configuration.
Create a directory src/config
and inside a new file src/config/configuration.ts
:
const DEFAULT_UNIVERSITIES_BASE_URL = 'http://universities.hipolabs.com';
export default () => ({
universities: {
baseUrl: process.env.UNIVERSITIES_BASE_URL || DEFAULT_UNIVERSITIES_BASE_URL,
},
});
University Interface
Create a new file src/universities/university.interface.ts
:
export interface University {
name: string;
country: string;
'state-province': string | null;
alpha_two_code: string;
web_pages: string[];
domains: string[];
}
This is the data type sent back by the university API.
Modifiying the Universities Module
The universities module has to include the new dependencies. Change src/universities/universities.module.ts
to look like this:
import { Module } from '@nestjs/common';
import { ConfigModule } from '@nestjs/config';
import { HttpModule } from '@nestjs/axios';
import { UniversitiesService } from './universities.service';
@Module({
imports: [ConfigModule, HttpModule],
providers: [UniversitiesService],
})
export class UniversitiesModule {}
Both ConfigModule
and HttpModule
are added to the imports
array.
Implementing the Universities Service
We now have to modify the universities service src/universities/universities.service.ts
:
import { Injectable, Logger } from '@nestjs/common';
import { Observable, map, of } from 'rxjs';
import { University } from './university.interface';
import { HttpService } from '@nestjs/axios';
import { ConfigService } from '@nestjs/config';
@Injectable()
export class UniversitiesService {
private readonly baseUrl: string;
private readonly logger = new Logger(UniversitiesService.name);
constructor(
private readonly configService: ConfigService,
private readonly httpService: HttpService,
) {
this.baseUrl = configService.get('universities.baseUrl') as string;
}
findByCountry(country: string): Observable<University[]> {
if (null == country) {
this.logger.error('no country specified');
return of([]);
}
this.logger.log(`getting universities for ${country}`);
const url = new URL(this.baseUrl);
url.pathname = '/search';
url.search = '?country=' + country;
const o$ = this.httpService.get<University[]>(url.toString());
return o$
.pipe(map(response => response.data));
}
}
The code should be self-explanatory. If not, please read the documenation for the NestJS Http module
. A real application should also implement some kind of error handling but this is left as an exercise to the reader.
Modify the App Module
Since we are implementing a REST client, not a server, we should delete the controller generated by nest new
:
$ rm src/app.controller.ts src/app.controller.spec.ts
Now that the controller files are deleted, the application module src/app.module.ts
no longer compiles. We fix that and also add the http module and config module. The file src/app.module.ts
should now look like this:
import { Module } from '@nestjs/common';
import { ConfigModule } from '@nestjs/config';
import { HttpModule } from '@nestjs/axios';
import { UniversitiesModule } from './universities/universities.module';
import { AppService } from './app.service';
import { UniversitiesService } from './universities/universities.service';
import configuration from './config/configuration';
@Module({
imports: [
ConfigModule.forRoot({
load: [configuration],
}),
UniversitiesModule,
HttpModule,
],
controllers: [],
providers: [AppService, UniversitiesService],
})
export class AppModule {}
Modify the App Service
We also have to change the app service src/app.service.ts
:
import { Injectable } from '@nestjs/common';
import { UniversitiesService } from './universities/universities.service';
import { first } from 'rxjs';
@Injectable()
export class AppService {
constructor(private universitiesService: UniversitiesService) {}
getUniversities(country: string): void {
this.universitiesService
.findByCountry(country)
.pipe(first())
.subscribe(console.log);
}
}
The app service has now just one method getUniversities()
that uses the university service for getting the list of universities and log them to the console.
Modify the Entry File src/main.ts
The last step is to adapt the entry file of the application src/main.ts
to reflect the changes made:
import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { AppService } from './app.service';
async function bootstrap() {
const app = await NestFactory.create(AppModule);
const appService = app.get(AppService);
appService.getUniversities(process.argv[2]);
await app.close();
}
bootstrap();
Try Out the Application
It is time to try out the new application:
$ npx ts-node src/main.ts Luxembourg
[Nest] 88309 - 01/18/2024, 3:31:57 PM LOG [NestFactory] Starting Nest application...
[Nest] 88309 - 01/18/2024, 3:31:57 PM LOG [InstanceLoader] HttpModule dependencies initialized +12ms
[Nest] 88309 - 01/18/2024, 3:31:57 PM LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms
[Nest] 88309 - 01/18/2024, 3:31:57 PM LOG [InstanceLoader] ConfigModule dependencies initialized +1ms
[Nest] 88309 - 01/18/2024, 3:31:57 PM LOG [InstanceLoader] ConfigModule dependencies initialized +0ms
[Nest] 88309 - 01/18/2024, 3:31:57 PM LOG [InstanceLoader] UniversitiesModule dependencies initialized +1ms
[Nest] 88309 - 01/18/2024, 3:31:57 PM LOG [InstanceLoader] AppModule dependencies initialized +0ms
[Nest] 88309 - 01/18/2024, 3:31:57 PM LOG [UniversitiesService] getting universities for Luxembourg
[
{
name: 'International University Institute of Luxembourg',
alpha_two_code: 'LU',
'state-province': null,
domains: [ 'iuil.lu' ],
country: 'Luxembourg',
web_pages: [ 'http://www.iuil.lu/' ]
},
{
name: 'University of Luxemburg',
alpha_two_code: 'LU',
'state-province': null,
domains: [ 'uni.lu' ],
country: 'Luxembourg',
web_pages: [ 'http://www.uni.lu/' ]
}
]
Works like a charm. By the way, Luxembourg is always taken as an example because the country has only two universities that the API knows about.
Writing Tests
Unit Tests
Now it is time to change the tests. In fact, the test file src/universities.service.spec.ts
that the Nest command-line tool has generated does not even compile at the moment:
$ npm run test
npm run test
> universities@0.0.1 test
> jest
FAIL src/universities/universities.service.spec.ts
UniversitiesService
✕ should be defined (11 ms)
● UniversitiesService › should be defined
Nest can't resolve dependencies of the UniversitiesService (?, HttpService). Please make sure that the argument ConfigService at index [0] is available in the RootTestModule context.
Potential solutions:
- Is RootTestModule a valid NestJS module?
- If ConfigService is a provider, is it part of the current RootTestModule?
- If ConfigService is exported from a separate @Module, is that module imported within RootTestModule?
@Module({
imports: [ /* the Module containing ConfigService */ ]
})
...
Test Suites: 1 failed, 1 total
Tests: 1 failed, 1 total
Snapshots: 0 total
Time: 2.839 s
Ran all test suites.
Provide the Module Dependencies
Obviously there is a problem with the ConfigModule
. In fact, the same problem exists with the HttpModule
. It becomes clear, when you look at the constructor of UniversityService
in src/universities/universities.ts
:
constructor(
private readonly configService: ConfigService,
private readonly httpService: HttpService,
) {
this.baseUrl = configService.get('universities.baseUrl') as string;
}
The constructor needs two injected dependencies, a ConfigService
and an HttpService
. But that does not happen automatically for test modules.
The reason is that it is discouraged to use "real" services in your test code. After all, the UniversitiesService
should be tested and not the ConfigService
and HttpService
that both ship with NestJS.
import { Test, TestingModule } from '@nestjs/testing';
import { HttpService } from '@nestjs/axios';
import { ConfigService } from '@nestjs/config';
import { AxiosRequestHeaders } from 'axios';
import { UniversitiesService } from './universities.service';
import { Observer, of } from 'rxjs';
import { University } from './university.interface';
const MOCK_URL = 'http://localhost/whatever';
describe('UniversitiesService', () => {
let service: UniversitiesService;
let httpService: HttpService;
beforeEach(async () => {
const module: TestingModule = await Test.createTestingModule({
providers: [
UniversitiesService,
{
provide: HttpService,
useValue: {
get: jest.fn(),
},
},
{
provide: ConfigService,
useValue: {
get: () => MOCK_URL,
}
}
],
}).compile();
service = module.get<UniversitiesService>(UniversitiesService);
httpService = module.get<HttpService>(HttpService);
});
it('should be defined', () => {
expect(service).toBeDefined();
});
});
The interesting lines are lines 19-30. We have to pass objects that are injected into our service and conventionally, plain objects are used that just provide mocks for all the methods that are used.
Of the ConfigService
only the get()
method is used and we mock it with a lambda function that always returns the same URL. This is because we know that the method is only called for getting the configured URL of the REST API. It does not matter what URL is returned here because no real requests are sent.
Of the HttpService
we also use just one method and by pure coincidence the name is again get()
. The reason is simply that we send GET
requests to the API. If we were also sending POST
, OPTIONS
, or DELETE
requests, we would have to implement stubs for the methods post()
, options()
, and delete()
respectively.
Whereas we provided a lambda function as the implementation of the get()
method of the ConfigService
we now use jest.fn()
as the stub. The reason is that we can mock the implementation in our actual test case and can also spy on the method.
While we were at it, we also set a variable httpService
that holds the instance of our mocked implementation, see line 32.
Running the tests again with npm run test
should now succeed. But we actually just test that the service can be instantiated and not that it actually works.
Create a Test Case
Let's change that now and create a test for the findByCountry()
method. That test will be on the same level as the should be defined
test (the rest of the file is unchanged):
it('should be defined', () => {
expect(service).toBeDefined();
});
it('should return universities of Lalaland', () => {
const data: University[] = [
{
name: 'University of Lalaland',
alpha_two_code: 'LU',
'state-province': null,
domains: [ 'uni.ll' ],
country: 'Lalaland',
web_pages: [ 'http://www.uni.ll/' ]
},
{
name: 'International Institute of Advanced Misanthropolgy',
alpha_two_code: 'LL',
'state-province': null,
domains: [ 'iiam.ll' ],
country: 'Lalaland',
web_pages: [ 'http://www.iiam.ll/' ]
},
];
const spy = jest
.spyOn(httpService, 'get')
.mockReturnValue(of({
data,
headers: {},
config: {
url: MOCK_URL,
headers: undefined,
},
status: 200,
statusText: 'OK',
})
);
const observer: Observer<University[]> = {
next: (universities: University[]) => {
expect(universities.length).toBe(2);
expect(universities[0].name).toBe('University of Lalaland');
},
error: (error: any) => expect(error).toBeNull,
complete: () => {
expect(spy).toHaveBeenCalledTimes(1);
},
};
service.findByCountry('Lalaland').subscribe(observer);
});
That is a lot of code but it is actually not so hard to understand.
In line 6 we define an array with some test data that our mocked implementation of the HttpService
should return.
In line 25 we create a Jest spy for the get()
method of the HttpService
and mock the return value of the method. The return value must be an AxiosResponse
object that has the mandatory properties data
, headers
, config
, status
, and statusText
.
In line 39 we then define an observer that implements our assertions and finally in line 50, we activate the observable returned by service.findByCountry()
by subscribing to it.
Stackoverflow is full of responses that contain similar code but it actually has issues.
Strict Null Check Issues
The first problem may or may not occur in your NestJS project. When you run the test with npm run test
, you may see this error:
src/universities/universities.service.spec.ts:64:21 - error TS2345: Argument of type 'Observable<{ data: University[]; headers: {}; config: { url: string; headers: undefined; }; status: number; statusText: string; }>' is not assignable to parameter of type 'Observable<AxiosResponse<unknown, any>>'.
Type '{ data: University[]; headers: {}; config: { url: string; headers: undefined; }; status: number; statusText: string; }' is not assignable to type 'AxiosResponse<unknown, any>'.
The types of 'config.headers' are incompatible between these types.
Type 'undefined' is not assignable to type 'AxiosRequestHeaders'.
Type 'undefined' is not assignable to type 'Partial<RawAxiosHeaders & { "Content-Length": AxiosHeaderValue; "Content-Encoding": AxiosHeaderValue; Accept: AxiosHeaderValue; "User-Agent": AxiosHeaderValue; Authorization: AxiosHeaderValue; } & { ...; }>'.
Apparently, there is a problem with something being undefined. When you check the code above in line 27 (actually line 64 in the file), you see that the property config.headers
of the mocked AxiosResponse
is undefined. Sometimes you see the usage of {}
instead of undefined
. The error message is very similar:
src/universities/universities.service.spec.ts:64:21 - error TS2345: Argument of type 'Observable<{ data: University[]; headers: {}; config: { url: string; headers: {}; }; status: number; statusText: string; }>' is not assignable to parameter of type 'Observable<AxiosResponse<unknown, any>>'.
Type '{ data: University[]; headers: {}; config: { url: string; headers: {}; }; status: number; statusText: string; }' is not assignable to type 'AxiosResponse<unknown, any>'.
The types of 'config.headers' are incompatible between these types.
Type '{}' is not assignable to type 'AxiosRequestHeaders'.
Type '{}' is missing the following properties from type 'AxiosHeaders': set, get, has, delete, and 23 more.
So the problem is that we pass an empty object but TypeScript wants one that has the properties set
, get
, has
, delete
, and 23 more. Given that we are totally not interested in that part of the response, that seems like a lot of work.
What is the actual source of the problem? Remember that we had created the application with the option --strict
that enables TypeScript strict mode? As a consequence you will find the compiler option strictNullChecks
set to true
in the top-level tsconfig.json
file of the application.
So you could set strictNullChecks
to false
, and the problem is gone. The effect of disabling strict null
checks is that both null
and undefined
can be used whereever a (typed) object is expected. But you probably enabled these checks for a reason.
A better solution is therefore to simply use a typecast. First, we have to add an import
statement for AxiosRequestHeaders
:
...
import { AxiosRequestHeaders } from 'axios';
...
And then change the offending line to this:
...
config: {
url: MOCK_URL,
headers: {} as AxiosResponseHeaders,
},
...
The typecast is okay here because we do not need the headers for our test at all.
The code now compiles, and the test should be green again. Try it out with npm run test
.
Asynchronous Issues
A golden rule of unit testing is that you should never trust a test case that you have not made fail. Please change the line that checks that the spy has been called exactly once to this:
expect(spy).toHaveBeenCalledTimes(42);
Run npm run test
again, and you can see a little surprise.
$ npm run test
> universities@0.0.1 test
> jest
PASS src/universities/universities.service.spec.ts
UniversitiesService
✓ should be defined (11 ms)
✓ should return universities of Lalaland (5 ms)
Test Suites: 1 passed, 1 total
Tests: 2 passed, 2 total
Snapshots: 0 total
Time: 2.856 s, estimated 3 s
Ran all test suites.
/path/to/universities/node_modules/rxjs/dist/cjs/internal/util/reportUnhandledError.js:13
throw err;
^
JestAssertionError: expect(jest.fn()).toHaveBeenCalledTimes(expected)
Expected number of calls: 42
Received number of calls: 1
...
$ echo $?
1
At first, success is reported but then the test fails and an exception is reported.
Obviously the assertion is executed after the test suite completed. How can that happen?
If you look at the error message closely again, you can see a hint what has gone wrong:
/path/to/universities/node_modules/rxjs/dist/cjs/internal/util/reportUnhandledError.js:13
throw err;
So there was an unhandled error reported. "Unhandled" probably means here that the assertion failed after the test suite had terminated.
Know the top 3 reasons why programming with asynchronous events is complicated?
1) They are asynchronous.
2) They are asynchronous.
What can be done about that? The solution to the problem is the RxJS TestScheduler
. When you read its documentation you may wonder what marble testing has to do with the current problem. And you are right. Most of the documentation is really irrelevant for the problem we are facing. The most interesting sentence is at the top of the page:
We can test our asynchronous RxJS code synchronously and deterministically by virtualizing time using the TestScheduler.
The TestScheduler
was initially written for unit testing RxJS itself and most of the features described in its documentation are probably mostly useful when you want to test your own operators or have to test that certain things happen in a specific order. All this is not really important for us.
Let's just fix our test case. We start by adding an import at the top of the file src/universities/universites.service.spec.ts
:
import { TestScheduler } from 'rxjs/testing';
Inside our top-level describe()
function, we instantiate a TestScheduler
:
let service: UniversitiesService;
let httpService: HttpService;
const testScheduler = new TestScheduler(() => {});
If you were to do marble testing, you should pass a real assertion function:
let service: UniversitiesService;
let httpService: HttpService;
const testScheduler = new TestScheduler((actual: any, expected: any) => {
expect(actual).toBe(expected);
});
But since the function will no be called in our use case you can just pass a dummy instead.
Finally, change the code that is running the observable to be invoked from the run()
method of the TestScheduler
:
testScheduler.run(() => {
service.findByCountry('Lalaland').subscribe(observer);
});
That is all. If the unit test fails, failure is now reported properly. If you fix the test case again (replace 42 with 1 again), everything is green again.
You can find the final version of the test here: https://github.com/gflohr/test-nestjs-http-service/blob/main/src/universities/universities.service.spec.ts.
End-to-end Test
The scaffolding tool of NestJS creates an end-to-end test by default. The generated one does not work because it assumes that the application exposes a REST API.
We should instead test the getUniversities()
method of the AppService
. However, it is not easy to test it because the actual logic happens asynchronously inside of the next()
method of the observer. But we can fix that by making getUniversities()
return a Promise
instead. This is the new version of src/app.service.ts
:
import { Injectable } from '@nestjs/common';
import { UniversitiesService } from './universities/universities.service';
import { first } from 'rxjs';
@Injectable()
export class AppService {
constructor(private universitiesService: UniversitiesService) {}
getUniversities(country: string): Promise<void> {
return new Promise((resolve, reject) => {
this.universitiesService
.findByCountry(country)
.pipe(first())
.subscribe({
next: console.log,
complete: () => resolve(),
error: (err) => reject(err),
});
});
}
}
The next()
method of the observer is, of course, called just once. When the observable returned by the findByCountry()
method completes, we resolve the promise. And for completeness, the promise gets rejected in case of an error. And once more for completeness and consistency, you should also change src/main.ts
to await
the call to getUniversities()
although things also work without that.
Now that the method returns a promise, we can leverage async/await
in the test code of test/app.e2e-spec.ts
and that works well as long as the tests succeed whether we are using the TestScheduler
or not. But it is playing up again, when a test fails. Then we can still see the failure but no test summary is generated. That means that the technique that works for the unit tests causes trouble for the e2e test.
I am sure that you can play with that approach yourself by just cloning the final version of the mini app from https://github.com/gflohr/test-nestjs-http-service and reverting to the Promise
based approach. I did not investigate too much into the issues because I think that mixing promises with observables is in general a recipe for trouble. I therefore modified getUniversities()
in src/app.service.ts
once more to return an Observable
instead of a Promise
:
import { Injectable } from '@nestjs/common';
import { UniversitiesService } from './universities/universities.service';
import { Observable, first, tap } from 'rxjs';
import { University } from './universities/university.interface';
@Injectable()
export class AppService {
constructor(private universitiesService: UniversitiesService) {}
getUniversities(country: string): Observable<University[]> {
return this.universitiesService
.findByCountry(country)
.pipe(
tap(console.log)
);
}
}
The console logging is now done as a side effect inside the tap()
operator.
That change required, of course, a modification of the entry file src/main.ts
:
import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { AppService } from './app.service';
async function bootstrap() {
const app = await NestFactory.create(AppModule);
const appService = app.get(AppService);
appService.getUniversities(process.argv[2])
.subscribe({
next: () => {},
error: console.error,
complete: () => app.close(),
});
}
bootstrap();
With these changes in place, the end-to-end test in src/app.e2e-spec.ts
could now invoke the test code from the run()
method of the TestScheduler
. See below or in the GitHub repository for a way to test the application end-to-end.
Feel free to leave a comment or a pull request on GitHub if you can contribute improvements. I wish you succes with your next NestJS project!
Following is the final version of the end-to-end test in src/app.e2e-spec.ts
:
import { Test, TestingModule } from '@nestjs/testing';
import { INestApplication } from '@nestjs/common';
import { Observer, of } from 'rxjs';
import { AppModule } from './../src/app.module';
import { AppService } from './../src/app.service';
import { UniversitiesService } from './../src/universities/universities.service';
import { University } from './../src/universities/university.interface';
import { TestScheduler } from 'rxjs/testing';
describe('AppController (e2e)', () => {
let app: INestApplication;
let appService: AppService;
const universitiesService = {
findByCountry: jest.fn(),
};
const testScheduler = new TestScheduler(() => {});
beforeEach(async () => {
const moduleFixture: TestingModule = await Test.createTestingModule({
imports: [AppModule],
})
.overrideProvider(UniversitiesService)
.useValue(universitiesService)
.compile();
app = moduleFixture.createNestApplication();
await app.init();
appService = moduleFixture.get<AppService>(AppService);
});
it('should print the universities of Lalaland', async () => {
const data: University[] = [
{
name: 'University of Lalaland',
alpha_two_code: 'LU',
'state-province': null,
domains: [ 'uni.ll' ],
country: 'Lalaland',
web_pages: [ 'http://www.uni.ll/' ]
},
{
name: 'International Institute of Advanced Misanthropolgy',
alpha_two_code: 'LL',
'state-province': null,
domains: [ 'iiam.ll' ],
country: 'Lalaland',
web_pages: [ 'http://www.iiam.ll/' ]
},
];
const findByCountrySpy = jest
.spyOn(universitiesService, 'findByCountry')
.mockImplementation(() => {
return of(data);
});
const logSpy = jest
.spyOn(global.console, 'log')
.mockImplementation(() => {});
const observer: Observer<University[]> = {
next: () => {},
error: (error: any) => expect(error).toBeNull,
complete: () => {
expect(findByCountrySpy).toHaveBeenCalledTimes(1);
expect(logSpy).toHaveBeenCalledTimes(1);
expect(logSpy).toHaveBeenCalledWith(data);
},
};
testScheduler.run(() => {
appService.getUniversities('Lalaland').subscribe(observer);
});
});
});
Leave a comment