MAJOR PROJECT 1: MCS345-ADVANCED DATABASE.
Firestore: The NoSQL Serverless Database for
the Application Developer.
Friday 8TH, September 2023
STUDENT DETAILS:
Name: Paingo Maima
Email: [email protected]
Course: MCS345-Advance Database
Coordinator: Mr. Lan Hanspeter
Submission Date: Saturday 7th October, 2023
TABLE OF CONTENT.
▪ EXECUTIVE SUMMARY
▪ HISTORY
▪ ORGANISATION
➢ Mission and goals
➢ Structure of Firestore
➢ Architecture Of Firestore
➢ Impacted/Advantages of Firestore
➢ Creating Firestore for customers
▪ PROJECT DESCRIPTION
▪ PROJECT PLAN
▪ PROPOSED BUDGET
▪ RECOMMENDATIONS
▪ REFERENCES
Executive Summary
Firestore is an alternative of Datastore that came out of Google’s App engine datastore
database which was introduced in 2008. It is one o f the kind that provide various types of
services to any business houses around the world nowadays. It was introduced in 20218, but
then came into full effect in 2014, where Google acquired Firebase that came with serverless
functionality and this led to the merger of Firebase’s serverless and datastore’s ease of usage
resulting in Firestore.
However, Firestore provide safe and secured data storage management system nowadays
and an be used any where around the globe. It helps customers or company to have access to
their Data any where in the world and can also, Delete, Changes, Adding and others
management by either during Online or Offline then update.
It can be created by using a mobile/web client library or using a server client library and it is
free at start, which means you can get registered for free then start to pay the service at later
time. This allows customers and users to see the importance of Firestore and have a strong
heart before do the billing. In addition, Firestore is designed to hold very large amount of data
that a company or business cannot hold and so is required by large extended companies and
businesses all around the globe.
Thus, Firestore is one of the fast, secured and reliable service provided by the Google Cloud
Storage that works with the goal of storing and managing data efficiently.
History
Firestore is an alternative of Datastore that came out of Google’s App engine datastore
database which was introduced in 2008. Initially it was accessible only through App engine.
Internally it is built on top of Bigtable which makes the queries to the storage eventually
consistent and in turn complicate the application development. In 2013, the datastore API
was exposed to be used outside the App engine environment. In 2014, Google acquired
Firebase that came with serverless functionality and this led to the merger of Firebase’s
serverless and datastore’s ease of usage resulting in Firestore.
There are also some thoughts going into moving the custom Firestore query planner to
Spanner for improved performance. Though this would require converting Firestore query to
Spanner query.
Organisations
In the world of application development, most of the computation is performed on the edge.
Though for certain use cases such as syncing user data across multiple platforms or to
interact with other users in the system, a remote storage is required. For an application
developer, this can soon turn out to be a challenging experience where they are spending
substantial amount of time maintaining the database instead of focussing on the business
logic of the application. Firestore is a schema-less and serverless database with scale in
millions of queries per second that makes life of application developers easier for their
storage needs. Well known applications such as NewYork Times have used Firestore for their
apps. In this paper we will take a look at the Firestore architecture and how it addresses
scalability challenges for the developers.
Key aspects that make Firestore a great storage system for developers are as follows:
• Ease of use
• Fully serverless and rapid scalability
• Efficient real-time querying ability
• Support for disconnected operations
Using Firestore
Firestore is a schema-less, serverless, document database offered by Google Cloud that can either be
initialized as a regional or multi-regional service, where the latter provides higher availability. This
section describes how the application developer interacts with the service.
Structure Of Firestore
Firestore supports a rich set of primitive and complex data types, such as maps and arrays. Cloud
Firestore is a cloud-hosted NoSQL database that is directly accessible via SDKs from iOs, Android, and
any other web application.
In Cloud Firestore, the data is stored in the form of documents which are then organized to form a
collection. The collection acts as containers for documents that users can use to organize the data
and build queries. Example as follows.
Figure 1: Sample Modelling of Firestore Design.
Bellow is an example of data model implementation in firestore designed.
Since Document is a record of information, it contains certain details such as user’s information.
//Document 1
first : "Mac"
last : "Anthony"
born : 1992
A collection is a container of documents which holds information on specific document file.
//Colection
users
//Document 1
user1
first : "Mac"
last : "Anthony"
born : 1990
//Document 2
user2
first : "Abdul"
last : "Ahamed"
born : 1987
Noticed that data can be stored hierarchically, thus can form what is known as Sub-Collection.
//Collection
Rooms
//Document 1
roomA
name: "my chat room"
messages
//Sub-collection 1
message1
from : "Shubham"
msg : "Hello Shubham"
//Sub-collection 2
message2
from : "Mac"
msg : "Hello Mac"
//Document 2
roomB
name: "my chat room two"
messages
//Sub-collection 1
message1
from : "Sam"
msg : "Hello Sam"
//Sub-collection 2
message2
from : "Mac"
msg : "Hello Mac"
Indexes
Firestore automatically indexes all the fields in a document and these secondary indexes are used to
execute queries at scale. These indexes are also defined in both ascending and descending order to
support for faster querying. This functionality is super helpful from developer’s perspective as a
developer won’t know before-hand which part of their application can end up becoming viral and
result in spike as part of the access pattern. But at the same time, having indexes on all fields can
end up making the write operations slower as each write will have to update multiple indexes.
Developers have the ability to specify fields that need to be excluded from automatic indexing.
Querying
Developers can query the storage using point-in-time queries which are strongly consistent or tied to
a close timestamp. Firestore also provides support for strongly consistent real-time queries. The
realtime query returns a series of timestamped snapshot where each snapshot contains strongly
consistent result of the query at given timestamp.
Server SDKs
Firestore provides SDKs for multiple programming languages. Server SDKs are used to run the
applications in environments such as Google cloud or App Engine. Server SDK provides mapping
between Firestore’s data model to the programming language of developer’s choice and additional
abstractions such as automatic retries.
Mobile & Web SDKs
Firestore allows third-party access to its storage which is guarded by Firebase authentication using a
range of identity providers such as Google, Apple etc. Developer can authenticate users with any of
these providers and then leverage the mobile & web SDKs to interact with the storage system.
Abstractions over operations such as real-time querying makes the application development a
breeze. Application queries the state whenever the application is opened and then continues to
update it in realtime whenever an update is made.
In case of network disconnect, the abstraction is fast enough to reconcile the application state. The
SDK provides transactional writes while connected and blind writes (Transaction that does not read
the current state before writing the new state) at all times.
Write Triggers
Google cloud functions can be invoked through triggers on specific database changes. Developers
can define these functions and tie them to specific database changes and the delta of change will be
transferred to the handlers of these functions.
Impact/Advantage of Firestore
Firestore is an alternative of firebase database developed by Google. It was designed to provide
better developer/customer experience and simplify the development process. It is a powerful tool
for storing data. Thus, it provides certain benefits to the developer such as,
1. Asynchronous Querying
Data can be retrieved from mobile and web in real-time. When a query is performed one can attach
a listener to it that receives real-time callbacks, hence query listener will be notified every time it
changes. Thus, real-time updates can be achieved from the document.
2. Offer Excellent Data Handling Capabilities
Firestore stores data in documents organized into collections and provides a high-performance
query engine with which developers can run complicated queries against the NoSQL database.
Cloud Firestore is optimized for storing large collections of small documents to provide extensive
flexibility in data structuring. Hence, With Cloud Firestore, it is easier to handle and organize data.
3. Offline Support and
Firestore provides offline synchronization which helps users to interact with the app when there is
no network connectivity. Hence, the users can store data offline and sync it to the database
immediately when connected.
Firestore libraries handle all the syncing, data merging, notifications, and other problems that allow
apps to run perfectly when they are not connected to the internet
4. Designed to Scale.
Firestore is a fast NoSQL database designed for scaling with the Google cloud platform offering
automatic horizontal scaling as per load. It offers global scalability and strong reliability.
Architecture of Firestore
Below is a very simplified version of Firestore’s architecture. Firestore service is available in multiple
geo-locations and customer chooses a location for database while creating the database. As part of
this section, we will look into the key components that are part of the Firestore architecture.
Global Routing
Consider an app that was built in USA but got viral and now users from around the world are using it.
Global writing becomes essential at this moment as user requests are coming from multiple geo-
locations. So, whenever a user request comes in, it first lands in closest Google point of presence
where the network infrastructure does a lookup for Firestore’s database location using the Firestore
metadata and routes the request to Firestore frontend task in that region. Firestore location lookup
service is a critical component in this flow and needs to be highly available. Firestore architecture
achieves this by persisting database location in a global Spanner database which consists of multiple
replicas.
Billing & Free Quota
Developers get a daily free quota of 1GB storage, 50K document reads and 20K document writes.
This is what makes Firestore an attractive option for developers to experiment with the Firestore
without incurring any major cost. Firestore performs billing by logging the count of resources
accessed by each RPC and then integrating this count with Google’s billing system.
Multi-tenancy & Isolation
As Firestore is used by multiple application developers for their storage needs, it makes sense for it
to be multi-tenant and provide an abstraction where the developer feels that they are
communicating with just one node for their storage. So multiple apps that are generating low
amount of traffic can be kept under a single node whereas an app that generates huge number of
requests can be spread into multiple physical machines. But with this multi-tenant architecture
comes a set of challenges where misbehaviour by one application can end up affecting the other
tenants i.e., the noisy neighbour problem. To avoid such problems, Firestore makes use of a fair-
scheduler. This limits the amount of CPU used by a tenant using the database id and ensures that no
single tenant overpowers the system.
Writes, Queries & Real-time Queries
Write queries update all the matching secondary indexes and also send updated data to the real-
time queries for all the respective clients. As Firestore underlying makes use of Spanner for storage,
each Firestore database maps to a specific directory in a spanner. Indexes in Firestore database are
stored as inverted index. Firestore manages its own query engine as it has to perform automatic
indexing and also query semantics of Firestore are different from that of Spanner. But building on
top of Spanner proves to be equivalent to standing on the shoulders of a giant as Firestore gets
benefits of high availability, transactional guarantees etc.
All the indexes created for a Firestore table are stored under a special table under spanner known
as IndexEntries. Adding/removing an index will require bulk-update on the IndexEntries table and is
done by a background service. Retrieving these indexes is done mostly through a cache in order to
improve the performance.
Writes are performed by leveraging the transaction abstraction exposed by Spanner. Read is
performed by either scanning over a single IndexEntries table or joining multiple secondary indexes.
As Firestore maintains index in both ascending as well as descending order, it doesn’t need to
perform any in-memory sorting when required as part of the query. So, query such as below will be
executed by joining salary asc and age desc secondary index.
SELECT * FROM employees WHERE salary > 1000 ORDER BY age desc;
If a query is performed using a field which is not indexed, Firestore returns an error along with
instructions to create a secondary index. Clients can also register for real-time queries which will
continue sending updated data through a long-lived connection. This is helpful for applications that
are presenting time-series data (Average score in last 5 innings or top 10 items sold in each
category).
Disconnected Operations
In case of Firestore, client SDKs build a local cache of documents requested by clients along with
required local indexes. This helps in providing low latency response to the client even when network
is up. Updates are acknowledged immediately after adding the update to local cache and processed
using the Firestore API in the background. User has the ability to persist the user cache and this will
help when a device is restarted as the client will be working with a warm cache. Also, API users are
not charged for operations that are performed using the cache only and hence it is economically
efficient to maintain the cache for the application.
Creating Firestore for Customers.
There are several numbers of ways to create or set up your d Database in Firestore Database
Management System. However, the common way to do that is by using a mobile/web client library
or using a server client library. For example. To create a Firestore database by using a web or mobile
client library.
This QuickStart shows you how to set up Firestore, add data, and read data by using the Android,
Apple platforms, Web, Unity, or C++ client library.
Create a Firestore database
1.) If you haven't already, create a Firebase project: In the Firebase console, click Add project,
then follow the on-screen instructions to create a Firebase project or to add Firebase
services to an existing GCP project.
2.) From the Firebase console's navigation pane, select Firestore, then click Create database for
Firestore.
3.) Select Test mode for your Firestore Security Rules:
Test mode
Good for getting started with the mobile and web client libraries, but allows anyone to read
and overwrite your data. After testing, make sure to review the Secure your data section.
4.) Select a location for your database.
▪ This location setting is your project's default Google Cloud Platform (GCP) resource
location. Note that this location will be used for GCP services in your project that
require a location setting, specifically, your default Cloud Storage bucket and your
App Engine app (which is required if you use Cloud Scheduler).
▪ If you aren't able to select a location, then your project already has a default GCP
resource location. It was set either during project creation or when setting up
another service that requires a location setting.
5.) Click Done.
Firestore and App Engine: You can't use both Firestore and Datastore in the same project, which
might affect apps using App Engine. Try using Firestore with a different project if you need to use
Datastore.
When you enable Firestore, it also enables the API in the Cloud API Manager.
Steps to set up your development environment.
For this part, it needs someone of the greater experiences because it is an important part of the
databases of the customers or the company. Also, it requires certain programming languages such as
the C++, JSON, JAVA and so, so experts to do the designing.
Set up your development environment
Remember to add the required dependencies and client libraries to your app.
1.) Follow the instructions to add Firebase to your Web app.
2.) Import both Firebase and Firestore:
import { initializeApp } from "firebase/app";
import { getFirestore } from "firebase/firestore";
NB: Looking for a compact Firestore library, and only need simple REST/CRUD capabilities? Try the
Firestore Lite SDK, available only via npm.
Initialize Firestore
Initialize an instance of Firestore:
// Initialize Firestore through Firebase
import { initializeApp } from "firebase/app"
import { getFirestore } from "firebase/firestore"
const firebaseApp = initializeApp({
apiKey: '### FIREBASE API KEY ###',
authDomain: '### FIREBASE AUTH DOMAIN ###',
projectId: '### CLOUD FIRESTORE PROJECT ID ###'
});
const db = getFirestore();
NB: The values for `initializeApp` can be found in your web app's `firebaseConfig`. To persist data
when the device loses its connection, see the Enable Offline Data documentation.
Add data
Firestore stores data in Documents, which are stored in Collections. Firestore creates collections and
documents implicitly the first time you add data to the document. You do not need to explicitly
create collections or documents.
Create a new collection and a document using the following example code.
NB: Learn more about the tree-shakeable modular Web API and upgrade from the namespaced API.
import { collection, addDoc } from "firebase/firestore";
try {
const docRef = await addDoc(collection(db, "users"), {
first: "Ada",
last: "Lovelace",
born: 1815
});
console.log("Document written with ID: ", docRef.id);
} catch (e) {
console.error("Error adding document: ", e);
}
Now add another document to the user collection. Notice that this document includes a key-value
pair (middle name) that does not appear in the first document. Documents in a collection can
contain different sets of information.
NB: Learn more about the tree-shakeable modular Web API and upgrade from the namespaced API.
// Add a second document with a generated ID.
import { addDoc, collection } from "firebase/firestore";
try {
const docRef = await addDoc(collection(db, "users"), {
first: "Alan",
middle: "Mathison",
last: "Turing",
born: 1912
});
console.log("Document written with ID: ", docRef.id);
} catch (e) {
console.error("Error adding document: ", e);
}
Read data
Use the data viewer in the Firebase console to quickly verify that you've added data to Firestore.
You can also use the get method to retrieve the entire collection.
NB: Learn more about the tree-shakeable modular Web API and upgrade from the namespaced API.
import { collection, getDocs } from "firebase/firestore";
const querySnapshot = await getDocs(collection(db, "users"));
querySnapshot.forEach((doc) => {
console.log(`${doc.id} => ${doc.data()}`);
});
Secure your data
Use Firebase Authentication and Firestore Security Rules to secure your data in Firestore.
Here are some basic rule sets you can use to get started. You can modify your security rules in the
Rules tab of the Firebase console.
// Allow read/write access on all documents to any user signed in to the application
service cloud.firestore {
match /databases/{database}/documents {
match /{document=**} {
allow read, write: if request.auth != null;
}
}
}
NB: Before you deploy your Web, Android, or iOS app to production, also take steps to ensure that
only your app clients can access your Firestore data. See the App Check documentation.
Project Descriptions
This project was elaborated on basics information about the Firestore. It provides the brief history
about the evolution of Firestore. It shows main ideal idea behind the Firestore creation, where any
customers or developers can easily understand the concept. This project also talked about the
benefits of Firestore and the implementation of customer’s filing systems.
Project Plan
This project discussed the summary details of Firestore database, how it can be used by the
developers to enhance their business managements, how to create and register to Firestore, the
importance of Firestore in-terms of services and uses of it and the advantages of it.
Proposed Budget
Firestore is a cloud storage, that manages, store, control and delete information online or offline for
some cases, since data can be edit off-line then update whenever you on-line again. However, it is
free at start. This means registration and joining the Firestore at first time is free of charge, but as
the time goes by, it will cost according to your businesses’ database stored in Firestore Database(See
sample of Firestore billing at: https://firebase.google.com/docs/firestore/billing-example#small-50k-
installs ).
However, the billing cost of Firestore is based on several factors, that is sometimes referred to as
” Cost By Usage Level”. Some are as follows;
1.) Read/Write Cost
Example for Read/Write Cost.
For 50,000 app installs (5,000 Daily Active Users): $12.14/month
400K total = 50K No-cost reads + (350K = 3.5 *
daily reads reads at $0.06/100K) $0.06
$0.21 / day * 30 = $6.30
Total monthly cost = 100K total = 20K No-cost writes + (80K = .8 *
$11.10/month daily writes writes at $0.18/100K) $0.18
$0.14 / day * 30 = $4.20
100K total = 20K No-cost deletes + (80K = .8 *
daily deletes deletes at $0.02/100K) $0.02
$0.02 / day * 30 = $0.60
2.) Storage/Networking Costs.
Example for Storage/Networking Costs.
For 50,000 app installs (5,000 Daily Active Users): $12.14/month.
20KB / DAU of = 100MB of = 3GB monthly network
daily egress * 5K daily egress egress
DAUs * 30
Total monthly 3 GB No-cost egress = No-cost1
cost =
$1.04/month
15KB daily = 45KB of = 225MB of = 6.75GB
message storage / storage / daily storage monthly
DAU + 3KB storage DAU * 5K / DAU * 30 storage
/ install 2 DAUs usage
1GB No-cost storage + (5.75 * $0.18) = $1.04 / month
1
10GB of monthly network egress are no-cost for Cloud Firestore.
2
Since our assumption is that DAUs are 10% of total app installs, this number accounts for the
total number of users that have installed your app.
Thus, a benefit of the Cloud Firestore billing model worth considering is that you only pay for what
you use. As a result, your bill may grow and shrink with your DAU count.
Recommendation.
o Firestore is a secured and save database management system that is offered and run by
Google services.
o Firestore is free at start. Thus, you can get yourself registered first and then when your
business grows or you become familiar with what you are doing then you will do some
payments, according to your businesses’ data management stored in firestore.
o Firestore Database can be managed, control and viewed any where around the world, since
it an online form of database management system.
o Firestore stored large amount of Data. Thus, if any company with a huge amount of data
needed to be stored, then Firestore is designed to manage large amount of data and can last
for very long amount of time.
o Firestore can be manage off-line and online. That is, if you are online then interactions with
your database is directly done, but if you are off-line then changes to your database can be
done when you are online again by which you updated the changes done.
References
o Google Cloud, September 20, 2023, Create a Firestore database by using a web or mobile
client library, September 27, 2023, at: https://cloud.google.com/firestore/docs/create-
database-server-client-library
o Firestore and its advantages-GeeksforGeeks, February 19, 2021,
at:https://www.geeksforgeeks.org/firestore-and-its-advantages/ , September 27, 2023.
o PDF, April 1, 2023 at:
https://www.researchgate.net/publication/372668019_Firestore_The_NoSQL_Serverless_D
atabase_for_the_Application_Developer ,September 27, 2023.
