In Data Engineer’s Lunch #46: API Calls on Node.js, we introduce some key architectural elements of node.js and demonstrate two ways to do API Calls within it. The live recording of the Data Engineer’s Lunch, which includes a more in-depth discussion, is also embedded below in case you were not able to attend live. If you would like to attend a Data Engineer’s Lunch live, it is hosted every Monday at noon EST. Register here now!
Node.js
Node.js (usually pronounced “node” “jay” “ess” or just “node”) is an asynchronous event-driven JavaScript runtime, and I encountered it while researching development environments.
Some of the key takeaways from my exploration of node.js include:
- Open-source
- Big support community
- Lots of tutorials
- Comes with easily extruded frameworks for web applications
- Event-driven loop helps it handle high volume requests from client and scale effectively
- Javascript (that’s the “.js”) core that integrates very easily with a lot of other languages (C++, HTML, PHP, Python, …)
Who Uses node.js?
Developers who use node.js typically use it to develop back-end or full-stack projects.
The community praises how easy it is to get started with node.js. I can confirm it didn’t take more than an hour to get to the first deliverables.
Getting Started with node.js
The framework has installation options for all major operating systems at
I developed the demo you’ll see below using WSL on VSCode, but you could have more easily used a gitpod. If, like me, you’re using node.js through the command line and Linux terminal, be sure to install Linux packages:
nodejs – the main package that includes a lot of baked-in libraries
npm – “node package manager” and will be used to install libraries/modules into your node framework.
Once you’ve installed those two, there are a few npm libraries you’ll want to make sure you have as well.
express – a web application framework module that lets you build a web application in a single line of code. Of course, you’ll have to make modifications to the files express generates for you in order for your app to do anything, but it’s a great starting place if you know you want a web app, and you can go a long way working with exclusively with express.js.
aysnc – a module that further empowers asynchronous operations within your JavaScript framework.
request – one of many options for managing API calls produced by the very active node.js community.
Node.js architecture
The event loop running at the core of node.js is the key to its success. The loop itself is stateless, making it fast and light. There are two main repositories within the main node.js architecture, the event queue, and the thread pool. If the system is going to bog down, you’ll see it in the expansion of the event queue, typically as the thread pool drowns in calls to other resources.
Requests from the customer are processed as events, which enter the Event Queue in order and are assigned threads by the Thread Pool as needed based on the event processing code defined within the framework or by designers.
Two distinct processing paths occur for “blocking” or “non-blocking” operations.
Blocking operations, such as those which require querying an external database, file-system, or other non-JavaScript resources, take place separately from those which can run within the JavaScript framework alone.
A few more thorough looks at node’s event loop are shown below:
API Calls in node.js
The index file in almost any node.js application is the app.js file (you can change this when you first run express and autogenerate your web application files). In the demo below, the app.js file creates a server that listens at port 3000. There are two endpoints defined for the server in the app.js file: the “/request” endpoint uses the “Request” module you downloaded using node package manager; the “/node” endpoint uses the “HTTPS” module included with the “nodejs” package you installed using your Linux package manager. You can reuse the code here to generate any number of endpoints.
There are a number of other modules you can use to handle the API calls: axios, Got, and Super Agent are among those developed by the community for this purpose.
I’d consider there to be two main functional parts of the API call itself.
The first is the request. The request typically needs to include some information about what information you are asking for and who you are. The first part is handled by the specific URI associated with the API that you are calling. The second is commonly handled by API keys. These can also be incorporated into the call itself, or included in the header of your request. In the case of the National Weather Service API that the demo below calls, the request header must include a “User-Agent”. This is an older authentication method.
The second part of the call is the response. In order to manage the response correctly, you’ll need to include code to handle the data the API is sending you (often referred to as the “body” or, understandably enough, the “data”) or an error message sent by the API in response to your call.
The Demo
To access the demo files and see it in all its glory, follow this Link: GitHub – Anant/example-nodejs-api-calls-with-https-and-request: Node.js demo with API call by https and Request modules
1.1. Why nodejs?
- Open Source Runtime Environment
- Event-Driven Single-Threaded Architecture
- Excellent for interacting with high volume of client request
- Less good at handling a large number of database queries
- Shallow Learning Curve
1.2. Getting Started with nodejs
- Assumes you know how to
- access a Linux terminal,
- update your Linux distribution,
- install packages from the command line,
- make and move between directories,
- create and edit files, and
- use root access if necessary.
1.2.1 Get and install packages.
apt-get install nodejs npm git
1.2.2 Use npm to install libraries and modules you’ll use in the demo
npm install request async
1.2.3 Create and change to your working directory
1.2.4 Replicate the files in your working directory
git clone https://github.com/Anant/example-nodejs-api-calls-with-https-and-request.git
1.2.5 Initialize the node
node app.js
1.2.5 Navigate in your browser to http://localhost:3000/node
You should receive a response from the National Weather Service API specified in the JavaScript file you cloned. The message should indicate that you are not an authorized user of the API. To correct that you’ll have to make some changes in the NodeJSCall.js file
1.2.6 Make sure to kill the node server running on your terminal by hitting cntrl+C
2.1 Configure the User-agent agent header
2.1.2 In the NodeJSCall.js file on line 7, change the uniqueID to something of yours, and add your email address.
- This effectively allows the API to know and track who is making calls to their interface
- Your unique ID can be anything, it doesn’t need to identify you. Your github username is a good default.
2.1.3 In the Request.js file on line 7, change the uniqueID and your email
2.1.4 Reinitialize the node from your terminal
node app.js
2.2 Navigate back to http://localhost:3000/node or http://localhost:3000/request to see the results of your API calls.
You should see weather data displayed at both!
2.3 Always remember to shut down the node with cntrl-c before letting your computer idle.
Thanks for following along with the demo. We have just scratched the surface of this topic, and I hope you got something out of it! If you have questions or comments, we are always interested in hearing them!
Thanks for reading!
References
HTTP | Node.js v17.1.0 Documentation
How to set user-agent in nodejs?
The A-Z of Node.js Architecture [Updated 2021]
Node JS Architecture – Single Threaded Event Loop – JournalDev
Overview of Blocking vs Non-Blocking | Node.js
Cassandra.Link
Cassandra.Link is a knowledge base that we created for all things Apache Cassandra. Our goal with Cassandra.Link was to not only fill the gap of Planet Cassandra but to bring the Cassandra community together. Feel free to reach out if you wish to collaborate with us on this project in any capacity.
We are a technology company that specializes in building business platforms. If you have any questions about the tools discussed in this post or about any of our services, feel free to send us an email!
