### Monitoring How To Build An Application Monitoring System With FastAPI and RabbitMQ - Python - by Carlos Armando Marcano Vargas - Medium
### Monitoring How To Build An Application Monitoring System With FastAPI and RabbitMQ - Python - by Carlos Armando Marcano Vargas - Medium
Get unlimited access to the best of Medium for less than $1/week. Become a member
In this article, we will build a service monitoring system using FastAPI and
RabbitMQ. The system will monitor the web traffic of various web services and
microservices. This is just for educational purposes, and is not a replacement for
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 1/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
Requirements
Python installed
Pip installed
Postgres installed
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 2/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
mkdir visitor_tracker
cd
#Windows users
py -m venv venv
cd venv/Scripts
./activate
#Linux
python3 -m venv venv
source venv/bin/activate
init_db.py
import os
import psycopg2
from dotenv import load_dotenv
load_dotenv()
USER = os.getenv('USER')
PASSWORD = os.getenv('PASSWORD')
def get_db_connection():
conn = psycopg2.connect(
dbname = "logs_db",
user = "postgres",
password = PASSWORD
)
return conn
conn = get_db_connection()
cur = conn.cursor()
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 3/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
conn.commit()
cur.close()
conn.close()
Here we setting up a database connection to store log data. First, we load the .env
file using dotenv to get the database username and password variables. Then, we
define a get_db_connection() function that establishes a connection to a PostgreSQL
database named logs_db. Then, the code calls that function to get a database
connection and cursor.
In this file, also the code drops the logs table if it exists and recreates it with the
given schema — with columns to store IP address, request URL, port, path, method,
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 4/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
browser, OS, time etc. It inserts sample log data into the table with its values. It
commits the changes to the database and closes the cursor and connection.
helpers.py
import collections
def to_dict(psycopg_tuple:tuple):
tracker = collections.OrderedDict()
tracker['id'] = psycopg_tuple[0]
tracker["ip_address"] = psycopg_tuple[1]
tracker["request_url"] = psycopg_tuple[2]
tracker["request_port"] = psycopg_tuple[3]
tracker["request_path"] = psycopg_tuple[4]
tracker["request_method"] = psycopg_tuple[5]
tracker["browser_type"] = psycopg_tuple[6]
tracker["request_time"] = psycopg_tuple[7].strftime("%d-%m-%Y, %H:%M:%S")
tracker["service_name"] = psycopg_tuple[8]
return tracker
def list_dict(rows:list):
row_list = []
for row in rows:
book_dict = to_dict(row)
row_list.append(book_dict)
return row_list
This file has two functions: to_dict() and list_dict() . The to_dict() function
converts a PostgreSQL tuple to a dictionary. The list_dict() function converts a list
of PostgreSQL tuples to a list of dictionaries.
The to_dict() function takes a PostgreSQL tuple as input and returns a dictionary.
The dictionary contains the values of the tuple in the same order as the tuple. The
list_dict() function takes a list of PostgreSQL tuples as input and returns a list of
dictionaries. The dictionaries are created using the to_dict() function.
controllers.py
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 5/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
def all_logs():
conn = get_db_connection()
cur = conn.cursor()
cur.execute('SELECT * FROM logs;')
logs = list_dict(cur.fetchall())
cur.close()
conn.close()
return logs
conn = get_db_connection()
cur = conn.cursor()
cur.execute('INSERT INTO logs (ip_address, request_url, request_port, reque
'VALUES (%s, %s, %s, %s, %s, %s, %s, %s) RETURNING *;',(ip_
request_url,
request_port,
request_path,
request_method,
browser_type,
request_time,
service_name))
log = cur.fetchone()[:]
log_dict = to_dict(log)
conn.commit()
cur.close()
conn.close()
return json.dumps(log_dict)
The all_logs() function gets all logs from the database and returns a list of
dictionaries. Each dictionary contains information about a single log.
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 6/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
consumer.py
import aio_pika
import ast
In the consumer.py file we write the code responsible for receiving the messages
from the RabbitMQ queue. The on_message() function will be called when a message
is received. It parses the JSON data from the message body and calls the new_log()
app.py
app = FastAPI()
@app.get("/logs")
async def receiver():
logs = all_logs()
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 7/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
return logs
@app.get("/")
async def hello():
return "hello"
On startup, we create a connection to the AMQP broker and declare a name of the
queue, in this case, “logs”. And consume the messages from that queue using the
on_message() function.
Then, we create an endpoint with the path “/logs” to show all the logs stored in the
database.
import pika
import json
tracker = {
"ip_address": '127.0.0.1',
"request_url": 'http://localhost:8000',
"request_port": 8000,
"request_path": "/",
"request_method": "GET",
"request_time": "2023-06-25T16:03:24.722256",
"browser_type": "Firefox",
"operating_system": "Windows 11",
"service_name": "Fastapi_service",
}
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
channel.queue_declare(queue='logs')
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 8/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
We run this file, python3 sender.py . It will show [x] Sent 'Logs' in our command
line. And in the command line that is showing the logs of the visitor tracker, it will
appear the data is received from the sender.py .
Publisher
The publisher will be the service we want to track its activity.
sender.py
import pika
In the sender.py file, we define the sender() function, it will receive a dictionary,
that will be the tracking information. It creates a connection to RabbitMQ, declares
a queue, and publishes the body (dictionary with the tracking information) to the
queue.
middleware.py
class Tracker:
def __init__(self, service_name: str):
self.service_name = service_name
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 9/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
request_port = request.url.port
request_path = request.url.path
request_method = request.method
browser_type = request.headers["User-Agent"]
request_time = str(datetime.now())
service_name = self.service_name
return {
"ip_address": ip_address,
"request_url": request_url,
"request_port": request_port,
"request_path": request_path,
"request_method": request_method,
"request_time": request_time,
"browser_type": browser_type,
"service_name": service_name,
}
In the middleware.py file, we create the Tracker class that takes the service_name as
an argument in its constructor. It has a visitor_tracker() method which takes a
FastAPI Request object as an argument. It extracts various information from the
request like IP address, URL, port, request path, method (GET, POST etc.), browser
type from the User-Agent header, request time, and the service name passed in the
constructor. This method returns a dictionary containing all this information which
can be logged to track visitor requests.
main.py
@app.middleware("tracker")
async def tracker(request: Request, call_next):
service_tracker = Tracker("service_one")
tracker = str(service_tracker.visitor_tracker(request))
sender(tracker)
response = await call_next(request)
return response
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 10/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
@app.get("/")
def index():
return "Hello, world"
@app.get("/json")
def some_func():
return {
"some_json": "Some Json"
}
if __name__ == "__main__":
app.run(debug=True)
We define two routes. If we make a request to these routes, its information will be
collected by the middleware and sent to the RabbitMQ queue.
We navigate through one of the endpoints with a browser or request with an HTTP
client.
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 11/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
Building a UI
We will create a React app to visualize the tracking information in a table.
Adding CORS
app.py
app = FastAPI()
origins = ["*"]
methods = ["GET", "POST", "PUT", "DELETE"]
headers = ["*"]
app.add_middleware(
CORSMiddleware,
allow_origins=origins,
allow_methods=methods,
allow_headers=headers,
)
...
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 12/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
#npm
npm create vite@latest table -- --template react-ts
#yarn
yarn create vite@latest table --template react-ts
#pnpm
pnpm create vite@latest table --template react-ts
After all the packages are installed we run Vite with the npm run dev command.
We navigate to localhost:5173 and we should see the Vite and React homepage.
app.ts
interface Table {
id: number,
ip_address: string,
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 13/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
request_url: string,
request_port: string,
request_path: string,
request_method: string,
request_time: string,
browser_type: string,
service_name: string,
}
useEffect(() => {
fetch(url)
.then(res => res.json())
.then(data => setData(data));
console.log(data);
}, []);
return (
<div>
<h1>Logs</h1>
<table>
<thead>
<tr>
<th>Id</th>
<th>IP Address</th>
<th>Request URL</th>
<th>Request Port</th>
<th>Request Path</th>
<th>Request Method</th>
<th>Request Time</th>
<th>Browser Type</th>
<th>Service Name</th>
</tr>
</thead>
<tbody>
{data.map((item, index) => (
<tr key={index}>
<td>{item.id}</td>
<td>{item.ip_address}</td>
<td>{item.request_url}</td>
<td>{item.request_port}</td>
<td>{item.request_path}</td>
<td>{item.request_method}</td>
<td>{item.request_time}</td>
<td>{item.browser_type}</td>
<td>{item.service_name}</td>
</tr>
))}
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 14/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
</tbody>
</table>
</div>
);
};
Let’s add a .css file to add style and see the data easily.
index.css
:root {
font-family: Inter, system-ui, Avenir, Helvetica, Arial, sans-serif;
line-height: 1.5;
font-weight: 400;
font-synthesis: none;
text-rendering: optimizeLegibility;
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
-webkit-text-size-adjust: 100%;
}
table {
border-collapse: collapse;
margin-bottom: 1rem;
}
th,
td {
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 15/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
padding: 0.5rem;
border: 1px solid #ccc;
}
th {
text-align: left;
}
td {
text-align: left;
}
.column-gap-10 {
column-gap: 10px;
}
Conclusion
In this article, we built a simple service monitoring system using FastAPI and
RabbitMQ. We saw how to create a FastAPI project, create exchanges and queues,
publish and consume messages from RabbitMQ queues in FastAPI, and store
monitoring data in a Postgres database.
Note: I didn’t try this app to track the activities of other services that use a different
framework than FastAPI. But if you want to, you can try using the same steps: Create
a middleware that collects the tracking information and then push them to the
RabbitMQ queue you declare to manage this data.
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 16/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
Resources
How to Send Logs to a Telegram Bot Using RabbitMQ and FastAPI
How to Build a Web Traffic Monitor with Python, Flask, SQLite, and Pusher
RabbitMQ documentation
Following
I'm self taught developer. I like to write about what I'm learning, and building. Always writting about Python,
Go and Rust.
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 17/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
37
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 18/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
11
Sometimes, when I’m writing a web application or installing one, I find that is not possible to
start it up because the port that is…
11
85
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 20/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
Darya Plotnikova
I‘d been writing for 1 year on FastAPI after 5-year experience on Django.
And here what I’ve found
For whom this article is written — for people who just started their way in backend development
and are deciding now what a framework to…
288 8
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 21/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
429 2
Lists
New_Reading_List
174 stories · 166 saves
dboost.me
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 22/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
Let’s think about dependency injection. In comparison with the java ecosystem, dependency
injection is not commonly used in the python…
Tom Jay
2K 135
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 23/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
Akshat Gadodia
133 2
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 24/25
31/10/2023, 11:26 How to Build an Application Monitoring System with FastAPI and RabbitMQ | Python | by Carlos Armando Marcano Vargas | Medium
1.3K 19
https://medium.com/@carlosmarcano2704/how-to-build-a-application-monitoring-system-with-fastapi-and-rabbitmq-python-9267227fc60d 25/25