0% found this document useful (0 votes)
10 views12 pages

26

Uploaded by

abdul
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
10 views12 pages

26

Uploaded by

abdul
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
You are on page 1/ 12

26.

Iscsi
Aug 28, 2025, 12:09 AM
26.Iscsi
Play

00:00
1:12:50
Mute

Settings
(0:02) Yeah, so from today onwards we are going to start the SAN. (0:15) So first
SAN, what is the full form of SAN? (0:20) Storage Area Network, okay, this is the
full form of SAN. (0:30) In SAN, we have two protocols, right, again, one is iSCSI
and the other one is FCP.
(0:40) The full form of iSCSI is Internet Small Computer System Interface, on
computer, okay, that (0:58) is the full form of iSCSI. (1:00) FCP means Fiber
Channel Protocol, okay. (1:12) So what is the difference between SAN and NAS?
(1:16) I'm not going to discuss now, okay.
(1:18) Let's talk about it once we complete the SAN topic because if I discuss it
now, you'll (1:25) be not able to understand that, okay. (1:29) So these two
protocols are SAN protocols, right, like normally, in NAS, we'll use a (1:37) LAN
network, okay, but SAN, again, that is protocol based. (1:44) What is the medium?
(1:45) That is, again, it is based upon the protocol.
(1:48) For example, if you are using FCP, then you will build a dedicated separate
SAN network, (1:57) okay, dedicated separated SAN network. (2:00) But for iSCSI,
you don't need to build any separate SAN network, okay. (2:05) What you can do, you
can use the existing LAN only, okay, you can use the existing LAN (2:10) only.
(2:11) But for the namesake, iSCSI will work, I mean, the background mechanism and
everything will (2:17) work like SAN only, even though, but it will use LAN only as
its own medium. (2:27) LAN means, again, you need IPs and all if you want to
implement iSCSI. (2:32) Like, why iSCSI using LAN network? (2:35) Why FCP needs a
dedicated network? (2:38) And when we implement iSCSI, okay, FCP is a very
expensive, very, very, very costly (2:45) thing, costly protocol to implement,
because you need to create a dedicated SAN network, (2:50) you need to purchase SAN
switches, which are very expensive, like 1 crore, 2 crore, 50 (2:56) lakhs,
something like that, those switches costs are, okay.
(2:59) Along with that, server side also, you need to install HBAs, storage side,
you need to (3:05) have a fiber channel adapters, which are very, very expensive,
the HBAs itself, it will cost (3:10) you around 2 to 4 lakh, okay, HBA ports.
(3:14) So, FCP is a very, very expensive thing, okay. (3:18) For example, mine is a
very startup company, but I want to implement SAN, okay.
(3:25) Every organization will have their own LAN, right, without LAN, they'll be
not able to (3:29) run an organization, because they have to communicate
internally, right. (3:33) So, if I want SAN, I don't need to go for FCP, okay.
(3:39) I can configure iSCSI, so, by using iSCSI protocol also, I can satisfy my
SAN requirements, (3:49) okay, I'll show you how to do that.
(3:54) Same thing, we'll be having LAN here, okay, and storage, simple, normally
everything is (4:06) common. (4:07) Here, we'll use network ports only, okay, for
iSCSI, and server end also, let's assume (4:15) that this is our server, okay,
server end also, they'll use network ports only, and (4:19) all these are normal
network cables only. (4:22) Everything is simple, same, just like NAS only, when it
comes to iSCSI, but whatever (4:28) the background working mechanism, and whatever
the host address we are going to use, for example, (4:35) in NFS, we'll use the IP
address, right, and when it comes to CIFS, we'll use the Active (4:42) Directory
username to provision the storage, but for iSCSI, it is little different, which
(4:47) I'm going to discuss a little later, okay, and one more thing, the
difference between (4:54) iSCSI and NAS protocols are, even though the iSCSI is
using LAN, you cannot, iSCSI protocol (5:02) is not routable, routable in the sense
of, for example, my entire, this storage setup (5:09) is in Hyderabad location,
let's assume that, okay, Hyderabad location.
(5:14) Now, I'm doing work from home, from Vijayawada, or for example, Chennai,
okay, Chennai, so, (5:23) if it is through CIFS, I can access it, I can access a
CIFS share from a remote location, (5:33) even for NFS also, you can access the
data from a remote location, but for NFS also, (5:40) that is not recommended,
actually, okay, you cannot, it is not best practice to assign (5:46) storage from
Hyderabad location to Chennai servers, because it will not fulfill your (5:53)
requirements, if it is NFS protocol, but for CIFS, you can do that, because mostly
CIFS (5:58) will use it for user data and all, right, yeah, but when it comes to
iSCSI, you cannot (6:06) do that, okay, you need to have servers in the same
location where we have our storage (6:16) and both servers and storage should be
connected to a dedicated LAN or at least a dedicated (6:26) VLAN, virtual LAN,
okay, and yeah, like that, you need to configure, okay, that is the reason (6:35)
and iSCSI will not support routing, routing in the sense of, as I told you, if
routing, (6:41) if any network is routable, you can access it from anywhere, okay,
CIFS and NFS, they (6:47) are the routable protocols, that means you can access it
from different geographical (6:52) location as well, but when it comes to iSCSI,
the servers should be in the same location (6:58) as the storage, okay, let us see
how to configure iSCSI, first, same thing, let us validate (7:10) the license
first, license show is the command to validate the license and check it on the
(7:16) both the controllers, if you have iSCSI license is enabled or not, see, I
have iSCSI license (7:21) on both of my controllers, controller 1 as well as
controller 2, okay, now let us check (7:32) if iSCSI is already configured on top
of any of your vServers or not, vServer iSCSI show, (7:40) okay, this is the
command to check iSCSI configuration is enabled on top of any vServer or not,
(7:47) okay, I do not have any iSCSI service configured on top of my existing
vServers, now what (7:53) I am going to do, I am going to, I want to configure
iSCSI on top of one of my vServers, (8:00) so before that what I need to do, I need
to create lifts, right, I need to create lifts, (8:07) so network interface show,
this is the command to check the lifts, you know that, okay, I (8:15) have already
configured so many ports like A0A, E0B, I mean these all are part of previous
(8:23) class, I created them, I do not want to delete them for now because it is a
waste of time, (8:27) so what I am going to do, I am going to use the ports which
are not under if group, okay, (8:34) which are not under if group, see, I have used
D and E, so what I am going to do, I (8:41) am going to use E0F for, E0F and E0G
for my iSCSI configuration, iSCSI lifts, as iSCSI (8:49) lifts, and one more thing
guys, iSCSI require dedicated lifts, you cannot use NAS lifts (8:58) for iSCSI,
okay, you cannot enable NAS protocols and iSCSI on the same lifts, okay, iSCSI must
(9:06) need dedicated lifts, got it right, but when it comes to NAS, you can
configure, you can (9:12) use the same lifts for CIFS and NFS traffic, right, which
is not possible with iSCSI, iSCSI (9:17) needs a dedicated lifts, now let's see how
to configure the iSCSI lifts, network, interface, (9:24) create, okay, vserver
name, I just want to do it on svm1 only, let's do that, okay, (9:33) next lift
name, I am just keeping it as iSCSI01, something like that, okay, simply, (9:41)
okay, data, not files, right, you remember we discussed about service policy, we
have (9:46) three different service policies, default management, default files,
default block, (9:51) now as I want to implement iSCSI, I have to use default data
blocks, that means I am planning (9:59) to use this lift for block data, block data
means nothing but SAN data, okay, roll data, (10:09) data protocol, I want to use
it for iSCSI, if you want to do it for FCP, yeah, you need (10:15) to select FCP,
okay, like that, but I want to do it for iSCSI, right, so I am selecting (10:21)
iSCSI here and data protocol, next, address, what is the IP address, 192.168.0.121,
yeah, (10:31) 121 is not used, so I am using that, okay, you know that, right, we
will get all these (10:37) IPs and all from networking and subnet mask
255.255.255.0, I want to create it on top (10:47) of node 1, which port on node 1,
no, not on A0A, I want to create it on top of E0F, that's (11:04) it, oh, okay, I
have not sent them, I guess, I can do that, okay, the same way, I want (12:03) to
configure another iSCSI lift on top of E0G, okay, I want to maintain 2 on node 1
(12:10) and 2 on node 2, 2 iSCSI lifts on node 1 and 2 iSCSI lifts on node 2, this
time I want to (12:16) use IP address 122, same thing, only thing I am going to
change is again, name of my (12:24) lift 02, okay, the same on second node as well,
23, 124, because this is sand, right, so I (13:07) need more performance, that is
the reason I am creating 4 lifts, it is not like that, (13:11) if you want, if you
are okay with 2 lifts, you can create only 2 lifts, 1 per node, okay, (13:17) now
let's check the network interface show, okay, now I am able to see 4 iSCSI lifts,
(13:23) okay, by seeing the name, I am able to identify that they are iSCSI lifts,
for example, I (13:29) have used the same generic name like lift 5, lift 6, lift 7,
lift 8, then how can I (13:33) identify which lifts are iSCSI and which lifts are
FCP, then what you can do, you can simply (13:39) run network interface show hyphen
data protocol iSCSI, if you run that, then it will show (13:50) you only the lifts
which are enabled with that particular protocol, okay, this is the (13:56) way you
can find the lifts which are configured with iSCSI protocol or same thing, CIFS
also (14:04) or NFS, yeah, like that, now next step, now let's configure iSCSI on
top of our vServer, (14:15) vServer iSCSI create, vServer name, nothing, I am just
creating a virtual iSCSI server, (14:26) same thing like a CIFS, I am creating a
virtual iSCSI server, that's it, very simple iSCSI (14:33) creation, vServer iSCSI
create, vServer name, iSCSI virtual server name and status admin, (14:42) now check
vServer, iSCSI show, if you run this command, see on top of svm1, iSCSI is (14:50)
configured, it is showing up and this is a address called iqn address, okay,
whenever (14:56) you configure iSCSI on top of the vServer, every vServer will get
a unique and unique (15:04) iqn address, okay, will use this iqn address only to
identify the devices in iSCSI protocol, (15:13) okay, with the help of this iqn
address only, okay, will be, one device will be able to (15:19) identify other
device in iSCSI protocol, okay, let's talk about it a little later, (15:27) okay,
so let's discuss what we have done so far, first, I validated the iSCSI license,
(15:34) okay, second, I have checked that iSCSI is enabled on top of my vServer,
any of the (15:38) vServer or not, so there is, I mean, there is no vServer that
configured with iSCSI, (15:45) so, I want to configure it now, before that, before
configuring iSCSI vServer, I have created (15:51) lifts with iSCSI protocol, okay,
once lifts are created, now, I have configured iSCSI (15:58) service on top of my
vServer, okay, that is what we have done so far, now, the next step (16:07) is, I
have to enable, okay, I have to enable iSCSI protocol on top of my iSCSI lifts, by
(16:18) default, it will be enabled, but it's better to validate that, if it is not
enabled, you (16:24) need to enable it manually, okay, let's see how to do that,
network interface, sorry, (16:30) network interface, server, iSCSI interface, okay,
show and your vServer name, that's it, (16:45) see, showing all four iSCSI lifts
here, their IPs, blah blah, so many things, okay, and (16:52) here, it is showing
enabled is true, that means, okay, iSCSI protocol is enabled on (17:00) these
ports, okay, four ports, four lifts, if it is not enabled, then, we need to do
(17:06) it manually, okay, vServer, iSCSI interface, enable, vServer name and lift
name, you need (17:17) to enter, okay, like that, this is the way to enable and the
same way, you can disable (17:25) it as well, okay, that's up to you, any questions
guys, so far, Aditya, yeah, Rabin, how many (17:41) iSCSI lifts we can create, is
there any specification like, no, you can create as many as you want, (17:52) there
is nothing like that, that's based on your iSCSI load, for example, I'm using only
(17:58) two or three servers, then two lifts are fine, right, for example, I'm
using 50 servers, then (18:04) I'll go ahead with four lifts, because, so that it
will increase the performance, right, (18:10) for example, I'm using 200 servers
for iSCSI protocol, like my vServer is, my NetApp cluster (18:16) is serving data
to 200 servers, that time, I'll configure more lifts or what I'll do, (18:21) I'll
configure another iSCSI on top of another vServer as well, like that, it's based on
(18:28) your requirement, got it right, yeah, that is fine, Rabin, one more
question, like, can (18:34) you please just make a diagram and explain me how the
client and this iSCSI lift is connected (18:41) in between and how this data has
been served, sure, sure, see, this is my server, okay, (18:50) okay, this is my
LAN, okay, either it will be a dedicated LAN or you can use a VLAN as (19:02) well,
virtual LAN, on network side, this is a network, normal IP network LAN, okay, and
(19:10) my server will be connected to this IP network, one of the switch in the IP
network, everything (19:15) is normal networking configuration, network, same
thing, okay, I have a storage here, my (19:22) storage also, let's assume I have
two controllers, two node clusters, okay, even my storage also (19:30) connected to
network, the same network, okay, and normally, if it is CIFS or NFS, as I told
(19:43) you, the server or the user can be accessed
from a different location as well, but when (19:50) it comes to iSCSI, both of
them will be physically present in the same location, my server and (19:56)
storage, and they should be connected to a dedicated or VLAN or dedicated LAN,
which (20:02) configured for, which is dedicated for iSCSI, okay, so, what I'm
going to do, I'm going (20:10) to configure a vServer on top of my, this one,
cluster, and I have enabled the iSCSI (20:18) services here, okay, and here, these
are the ports connected to my iSCSI network, right, (20:26) so, I'm going to
configure lifts on, iSCSI lifts on top of these ports, okay, for example, (20:33)
if you have a four ports connected, then you can create four lifts, eight ports
connected, (20:37) you can create eight lifts, I mean, based on your requirement,
if you want more performance, (20:43) this is a paths, number of paths, how, I
mean, by you, from four paths, my server will be (20:52) able to get the data from
storage, okay, like this, the same way, on server end also, (20:59) you can make
multiple connections, so that the performance will be increased, like, if (21:04)
you make two connections, yeah, obviously, the performance will be good, right,
okay, (21:10) after that, you know that, right, LAN creation and all, blah, blah,
blah, okay, yeah, I'll (21:19) show you, I'm going to show you now, just I don't
want to discuss those things now, iGroup (21:23) and all, yeah, it's pretty simple,
just like FCP only, but only thing is here, we are not (21:31) using SAN network,
here, we are using a LAN network only, okay, okay, so, we are not using (21:38) any
WWPN, no, no WWPNs here, here, we will be having IQN, IQN is the address, yeah,
IQN, (21:46) which I have already shown you, yeah, this is the address, IQN, yeah,
this is IQN number, (21:55) IQN address of my vServer SVM1, yeah, okay, so, now,
let's do it, now, what we need to (22:04) do, now, we need to establish the
communication between my vServer and client server, my NetApp (22:14) cluster,
vServer and client server, so, what we need to do, just we need to provide the
(22:20) iSCSI IPs to, okay, we need to provide the iSCSI IPs to server team, so,
what they'll (22:27) do, they'll add these IPs on the server, server's iSCSI
configuration, so that it will enable (22:33) the communication between my server
and, I mean, sorry, normal client server and NetApp (22:41) storage vServer, let's
see how to do that, and before that, let me show you, normally, (22:46) when I run
vServer iSCSI, initiator show, will show you empty, initiator means nothing (22:53)
but, if you run this command, you'll be able to see the server IQN numbers here,
normally, (22:58) if the, when the connection is established, but see, now, it is
showing empty, right, (23:03) now, let me show you how, how it will, I mean, I'll
run the same command again after establishing (23:10) the connection, so that
you'll be able to see the server IQN numbers here, okay, so, (23:15) our task is
to, mostly, our task is done, while config, for configuring the iSCSI, we (23:21)
need to just hand, provide these IPs to the server team, so, they'll do it, they,
it is (23:27) their responsibility to establish the communication, okay, if they
are facing any issue, then, (23:32) we'll troubleshoot it, if anything wrong with
the storage end or not, something like that, (23:36) that's it, okay, but, as this
is lab, I'm going to show how to do that, okay, in every (23:41) server, we'll be
having a software called iSCSI initiator, normally, this software (23:48) will be
there in every Windows server, by default, not only a server, even, if you check
(23:54) it in your laptop also, you'll be able to see it, see, this is my normal
laptop, right, (24:01) you'll be able to see it here as well, iSCSI initiator, in
Windows, it will be there by (24:06) default, but, when it comes to other operating
systems, like Linux, Solaris and all, you (24:11) need to install it, you need to
install the package, okay, so, this is iSCSI initiator, (24:19) okay, what I need
to do, first step is, come to discovery, okay, discovery tab here, and (24:25)
there is an option called discover portal, discover portal, just select that, and
here, (24:32) you need to enter the IP address of your iSCSI libs, okay, which I
got from storage (24:38) team, 192.168.0.121, okay, and this is the default port,
which will be used for iSCSI, (24:50) if you are using some customized port, you
can change that, but, mostly, people will use (24:54) 3026 only, okay, that's it,
next, like that, let's add all the IPs, 122, I have created (25:03) four libs,
right, let's add all of them, for example, I don't want to, this is not a
production (25:09) server, so, I don't want to connect this server to NetApp via
all four libs, not required, (25:17) I just want to use two libs only, yeah, then
add two IPs here, you got it, right, so, based (25:22) on the, how many IPs you are
adding, how many iSCSI IPs you are adding here, based on that, (25:27) you will get
number of paths to access the data from storage, you got it, right, Abhijit,
(25:37) okay, Aditya, this is like, you know, you can, here you can decide how
many, through (25:46) how many paths or through how many libs my server has to
communicate with storage, okay, (25:52) this you can decide here, normally, but
here, I just want to use all four libs, that's it, (26:05) done, now, if you go to
targets again, now, I have added all my four libs, right, now, (26:10) if you go to
targets, here, see, previously, it was empty, like, there is nothing here, (26:18)
now, I am able to see some address, so, when I added the, my NetApp, iSCSI IPs
here, and (26:26) when, by using discover portal, my server, iSCSI initiator
software, sent a request to (26:34) NetApp and request NetApp for its IQN address,
so, NetApp will accept that request and will (26:45) share its IQN information with
this server, see, if you, vserver, iSCSI, show, if you (26:57) run this command,
you will be able to see the IQN address of your SVM, right, just compare (27:01) it
with here, same address, see, same address, right, your SVM1 shared its IQN address
with (27:14) server, because server requested its address through these iSCSI paths
to establish the (27:21) communication, but the communication is still not active,
it is showing inactive, right, (27:26) you need to make it active, so, what you
need to do, just select it, click on connect, (27:33) okay, and enable, check on
enable multipath as well, I will tell you little later what (27:38) is multipath,
okay, just enable multipath and click okay, so, now, see, it is showing (27:45)
connected, now, my server can communicate with the iSCSI server, which is
configured (27:55) on top of my vserver called SVM1, which is in cluster 1, NetApp
cluster 1, after that, (28:01) just click okay, that's it, now, let me run the
initiator command again, vserver, iSCSI (28:09) initiator show, now, if I run this
command, I will be able to see the server iSCSI, server (28:16) IQN number here,
see, server IQN number is visible now here, you got it, right, Abhijit, (28:22)
see, it is just like after zoning is completed, you will be able to see the WWPN of
the server, (28:29) right, the same thing here in iSCSI as well, see, this is the
SVM, and through iSCSI 01 (28:35) lift, my server is communicating now, okay, this
is the IQN number of my, what, server, (28:44) Windows server, yeah, like that, and
if you want, I just validate that, let's move ahead, (28:51) yeah, Naveen, the IQN
part was clear with me, but this iSCSI lifts, that part was not (28:59) clear with
me, that's why I asked the question, what do you mean by that, not clear, like
(29:06) to add a server with the help of IQN, that part is clear with me, I can add
that, but (29:14) the, how the connection, let me give you a SAN example, here we
provided the connection, (29:22) right, see, I have added it in the Windows server,
all these IPs, so with the help of (29:28) this iSCSI initiator software, my server
made connections with my storage, okay, so that's (29:35) a server end work, yeah,
that is what I told you, right, this is, whatever we have done (29:41) in this
iSCSI initiator software, everything will be taken care by server team, we don't
(29:46) do that, that is what I mentioned earlier as well, adding these IPs to the
Discover (29:50) portal, everything will be taken care by server team only, only
thing is, we'll provide the (29:55) IPs to them, that's it, our job is done in
iSCSI configuration, it is their headache (30:00) to establish the communication
between that server and our storage, okay, once the connection (30:08) is
established, I'll be able to see the IQN, server IQN here, if I am not able to see
the (30:13) server IQN here, that means there is something wrong, so we need to
troubleshoot it, got (30:18) it right, might be firewall blocking something like
that, you need to check everything, yeah, (30:25) okay, now if you go to
configuration tab, here you'll be able to see the server IQN (30:29) number,
validate it, see, IQN199105, same thing, right, like that, once the connection
(30:37) is established, you'll be able to see the server IQN number here, same
thing, let me (30:41) do it on a different server as well, so that you'll get a
clear picture, let me do it on, (30:45) I have another Windows server, right, which
is a domain controller, let me connect there (30:49) and let me add the iSCSI list
there as well, so that it will establish the communication (30:54) there as well,
okay, now again, same thing, iSCSI initiated, just clicked on it, that's (31:30)
okay, iSCSI services are not configured, it is asking, it asked me, do you want me
to (31:36) configure it, do you want me to enable the iSCSI service, I said yes,
that's it, after (31:40) that, see, nothing here in targets, right, empty, now let
me go to
discovery again, let (31:46) me add the IPs here, 121, one quick question, if the
port number is not right, in that condition (32:08) also, do we see any connection
problem, right, it has to be 3260, yeah, I mean, if you want, (32:17) you can use a
custom port as well, it is not mandatory to use the 3260, which is a default
(32:21) iSCSI port, but you need to work with your networking team to, yeah, okay,
see, here (32:28) for this server, I just added 121 and 123, that means iSCSI 01
and iSCSI 03, so now this (32:36) server will be able to access my storage through
only two ports, two iSCSI lists, okay, like (32:42) that also, yeah, see, now
automatically I am able to see my storage IQ number here, again, (32:47) just to
enable the connection, communication, just click there, click on connect, just
enable (32:53) multipath, check enable multipath, click okay, that's it, see, it is
connected, now (32:59) I will be able to see this server IQ number as well there,
let me run the command again, (33:04) vserver iSCSI initiator show, see, I am able
to see another server, which is dc1.demo.netapp.com, (33:14) dc1 is this server,
right, dc1, like that, okay, so initiator means nothing but server (33:22) address,
okay, initiator means nothing but server address, if it is a iSCSI initiator,
(33:28) then server IK numbers, if you are talking about FCP initiators, then
server WWPNs, that's (33:37) it, let's don't talk about the WWPNs now, okay, let's
talk about them in FCP, yeah, (33:43) this is the way you can establish the
communication between server and storage, any questions? (33:51) No. (33:58) Now,
let's configure the things, okay, let me create a volume, m1, volume name, I will
(34:10) put it as sanwall01, something like that, okay, aggregate, I want to create
it on top (34:19) of it, disaggregate, size 50g or 51g, I will create it, 51gb,
don't worry about the junction (34:33) path, security style and all, because as I
told you, junction path, security style, these (34:39) things are applicable to
only san volumes, but here I am creating a volume for, sorry, (34:45) nas volumes,
sorry, nas volumes, but here I am creating a volume for san, so you don't (34:51)
need to mount this volume to any junction point or you don't need to worry about
the (34:56) security style, okay, I just want to, I don't want snapshots, that's
why I am disabling (35:01) them, just %snapshot, space, none, sorry, 0, I need to
set 0 and snapshot policy also (35:12) I am setting it to none, I don't want
snapshots, that's why, okay, I have created a volume (35:18) with 51gb size.
(35:25) Now, what I am going to do, I am going to create a LUN, sorry, so we cannot
present (35:34) a volume directly to san, san, I mean through san protocol, okay,
so what you need to do (35:42) on top of a volume, okay, you must create another
thing called LUN, okay, LUN, LUN means (35:53) logical unit number, why, why I am
not able to present a volume through san protocol to (36:01) san host, because your
volume will be having security style, okay, security style means (36:08) nothing
but it contains a file system, okay, so file system means this is a file level,
(36:15) it is providing file level access, but when it comes to san, my san uses
block level access, (36:21) okay, that is the reason even volume or qtree, we can't
present volume or qtree through san (36:29) protocols to san host, okay, that is
the reason we must create a LUN, okay, which you will (36:36) be able to understand
it very clearly when we discuss about the differences between san (36:41) and nas,
okay, for now just remember san means block, nas means file, okay, so as volume
(36:49) and qtree are file level, okay, we will be not able to present them to
clients through (36:55) san protocols, because san requires block level access,
okay, just remember that, so (37:01) now what I am going to do, I am going to
create a LUN on top of this volume, okay, LUN (37:06) show, this is the command to
check if you have any LUNs or not, I don't have any LUNs, (37:11) so now I am going
to create a LUN on top of this volume, LUN create vserver name, next (37:21) volume
name, on top of which volume you are creating this LUN, I am creating it on top
(37:25) of sanvol1, okay, what is the LUN name you want to do, I want to use a name
called sanlun, (37:34) okay, sanlun01, something like that, okay, size, I want to
create it with 50 GB, one (37:42) more thing guys, like whenever you need a 50 GB
LUN, you must create a volume with greater (37:49) than that, I mean the size
greater than that, like if you need a 50 GB volume, just create (37:53) your volume
51 GB, 52 GB or 55 GB, something like that and if you want to maintain snapshots,
(37:59) you need to add some more space for snapshots as well, mostly people will
not maintain snapshots (38:06) on top of a sanvol1 where we are hosting LUNs, but
if you have a requirement where you want (38:12) to maintain snapshot, then you
need to create it in that way, I mean like you need to have (38:18) some space for
your snapshots as well on top of the volume, okay, so see here, I am (38:26)
creating a LUN with 50 GB size, OS type, I want to assign it for Windows, right,
Windows, (38:34) okay, if your server is having Windows 2008 or Windows GPT, then
you need to mention (38:41) it specifically, remaining all others, you can specify
just Windows, like that see, NetApp (38:46) supports these many operating systems,
okay, you need to specify which operating system (38:51) you are presenting it to,
next, space reserve enabled, space reserve means, this option (38:59) is nothing
but, okay, thin or thick LUN, like we have a thin volume, thick volume, right,
(39:10) so thin volume what it will do, it will not occupy the space immediately,
thick volume (39:14) what it will do, it will occupy the space immediately from
aggregate, right, the same (39:18) way, if you enable the space reserve, then LUN
will occupy the space immediately from (39:23) the volume, but if you disable it,
then it will be a thin LUN, so which will not occupy (39:30) the space immediately,
okay, when you put some data in the LUN, then only it will slowly (39:35) occupy
the space from the volume, but mostly people recommend to use a thick LUN only,
(39:41) thick only, okay, that is the best practice, better to do that, okay, so I
am enabling (39:47) it, that's it, see, now I have created a LUN with 50 GB, now
see, LUN show, you will be (39:57) able to see, this is my vserver name and this is
the LUN path, okay, online and which is (40:04) unmapped, unmapped means, nothing
but, I have not presented this LUN to any server, then (40:09) only it will show
you like unmapped, okay, what is the type of the LUN and what is the (40:14) size
of the LUN, okay, like that, this path and all you don't need to worry about it,
you (40:18) will get it automatically in the path, okay, now LUN is done, now I
need to present it (40:27) to server side, before presenting it to the servers,
okay, what we need to do, we need (40:33) to create a I group, let me discuss a
little theory here, see, I have two servers now, (40:43) which is composed, another
one is DC, in our example, so now, these two servers has access (40:53) to, they
got access to my storage through iSCSI protocol and these two having IQN numbers,
(41:00) right, IQN, for example, this server IQN address is IQN1, let's assume that
and this (41:06) server IQN address is IQN2, okay, now what, now I need to present
a LUN to both the servers, (41:14) for example, let's assume that these two are in
cluster, let's talk about that a little (41:18) later, so what I am going, first I
want to present a LUN to jump host, for example, now (41:24) what I need to do, I
need to create a group called initiator group, okay, initiator group (41:38) and
you need to add the server IQN in this initiator group, after that what you need,
(41:45) sorry, what you need to do, you need to create a volume, you need to create
a volume and (41:52) you need to create a LUN on top of that volume, you don't
require Qtree, okay, and just map (41:58) this LUN to this iGroup, so that whatever
the server IQN numbers inside this initiator (42:05) group will get access to the
LUN, you got it, right, for example, if I add another server (42:12) which is IQN2,
so along with IQN jump host, DC1 also will get access, sorry IQN2, okay, (42:19)
along with jump host, DC1 also will get access to the LUN, like this, you got it,
right, (42:25) so if you need access, you need to create a initiator group and add
the server IQN number (42:30) into the initiator group and map the LUN to here, for
example, my DC1 don't require access (42:38) to this LUN and DC1 requested a
separate LUN, then create another initiator group, okay, (42:46) iGroup, in
shortcut we will call it as iGroup, okay, and put the IQN number of that server
(42:56) here, just create a volume, create a LUN, map this LUN to this iGroup, so
that this IQN (43:03) number server will get access to this LUN, okay, it is pretty
simple concept and you (43:08) compare it with exports policy, I mean, NFS exports,
this is a pretty simple procedure, (43:15) okay, let me show you how to do that, so
vserver, not vserver, sorry, iGroup, show, (43:32) this is the command to check
iGroups, I don't have any iGroups yet, okay, now let me create (43:38) a iGroup,
iGroup, create, vserver name, iGroup name, I am just creating it as IG1, simply,
(43:48) okay, iGroup1, and which protocol you want to use for this iGroup, see, we
have a FCP (43:55) and iSCSI, right, in SAN, so through which protocol you are
presenting storage to the (44:00) server in this iGroup, I want to, I am presenting
it through iSCSI, right, so I am mentioning (44:05) iSCSI, if you want to use both,
then you need to select mixed, so it will allow, if any (44:12) storage comes
through FCP or iSCSI, both, okay,
but for now, I am selecting iSCSI only, (44:19) OS type, again, you know that,
right, which server you are putting in this iGroup, what (44:25) is the operating
system of that particular server, okay, you need to mention that, Windows, (44:31)
Initiator, here, under Initiator, you need to mention the IQN number of your
server, (44:37) my server IQN number is this, right, which I want to present it to,
that's it, now, I (44:47) have created a iGroup, iGroup show, if you run this
command, see, I am able to see it (44:50) now, vserver name, iGroup name, which
protocol you are using, what is the OS type and what (44:56) is the Initiator, I
mean, IQN number of that server, now, what I need to do, I need to (45:03) map this
LUN to this iGroup, that's it, immediately, once mapping is completed, immediately,
my (45:11) server will be able to see the LUN, okay, I mean, the 50 GB space, so,
the command is (45:20) LUN map, vserver name and LUN path, you need to provide the
LUN path here, yeah, LUN path, (45:30) iGroup name, my iGroup name is IG1, right,
now, it is asking for the LUN ID, LUN ID means (45:36) nothing but identification
number, just like, when you are creating an export policy rule, (45:44) you are
giving a rule index, right, 1, 2, 3, 4, same thing here, whenever you are mapping
(45:50) a LUN to a iGroup, you need to provide an ID called LUN ID, okay, so, now,
I am mapping (45:59) one LUN, after that, I have created another LUN with a 10 GB
size and I want to map that (46:05) LUN also to the same iGroup, that time, I
cannot use LUN, because LUN is already used, (46:11) so I need to select some other
LUN ID, okay, LUN ID means nothing but identification number (46:17) of LUN
mapping, okay, and one more thing guys, LUN ID is very, very important, because
(46:23) on server side, you will be not able to see the LUN path and all, okay, you
will be able (46:28) to see only LUN ID, with the help of LUN ID only, you can
recognize the LUN, okay, (46:36) so, LUN ID will play very crucial role in SAN,
okay, and for now, just I am giving one, (46:42) that's it, now, my LUN is mapped
to, yeah, someone unmuted, yeah, Navin just wanted to (46:52) ask, the LUN ID is a
unique ID number in a SVM or in a cluster, SVM, not SVM also, you (47:03) can say
that it belongs to iGroup, see, generally, not entire cluster actually, for this
SVM, (47:21) this is the unique ID, I cannot use one again, okay, but if I want, I
can use one again for (47:28) another vServer, okay, that means not unique, yeah,
okay, yeah, yeah, okay, now, done, now (47:36) let me run LUN show command, see, I
am able to see, now, it is showing mapped, that means (47:42) because I have mapped
this LUN to iGroup, right, now, if you want to show, that's it, (47:49) our part is
completed, now, we have mapped the LUN, so, automatically, server team will (47:54)
be able to see the LUN, our part is done, only thing what we need to do, just run
this (47:59) command, LUN mapping, show vServer name and LUN path, just enter this
command and it will (48:09) show you, what is the vServer name, what is the LUN
path, to which iGroup it is mapped (48:14) and what is the LUN ID and which
protocol it used, okay, just send this output to the (48:20) server team, so that,
they, in this, from this output, they will be able to see the (48:24) LUN ID and
all, right, so, they will use this as a reference and they will configure the
(48:30) storage on their end, but as this is a lab, I am going to show you, how
they will also (48:34) do it, okay, how they will do it, once everything is
completed, what you need to do, you need (48:40) to log into that particular
server, where we have assigned this, in IG1, we have added (48:45) IQN number,
right, and this IQN belongs to this server, so, I logged into this server, (48:52)
here, just you, what you need to do, you need to open computer management, okay,
this is (48:57) a server, the server path, we don't do this, but I just want to
show you, let it open, (49:05) okay, this is a computer management and in computer
management, you have another thing (49:11) called disk management, which will be
under storage section, if you don't see it, just (49:15) expand the storage, you
will be able to see disk management here, disk management, let (49:22) it load,
okay, now, you are not seeing anything here, right, you are seeing only system
reserved (49:28) and C drive, okay, what you need to do, just go to more actions,
click on rescan disks, (49:38) see, immediately, when I hit rescan, I am able to
see this 50 GB, this is the LUN, which (49:45) we presented, okay, what you need to
do, just to validate that, just click here, go (49:51) to properties, see, you will
be able to see our LUN ID here, see this, see, this is the (50:03) LUN ID, okay,
now, what I am going to do, just bring it online, by default, it will (50:15) be
offline state, just bring it online, now, again, click on initialize disk, once you
(50:26) hit the online, it will become not initialized, what, in order to use it,
you need to initialize (50:31) it, just initialize disk, so here, it will give us
two options, like MBR or GPT, again, (50:39) these are Windows things, like MBR is
a default option and GPT is a advanced partition, for (50:46) example, if you
select MBR, then what you can do, you can put whatever data you want, (50:51) even
if you want to install operating system, you can install it on MBR partition, but
when (50:57) it comes to GPT, you will be able to use it for data or application
purpose only, if you (51:04) want to install some operating system on top of a GPT
partition, which will not allow you, (51:08) okay, so better to select MBR only,
but only thing is, there is a limitation with MBR, (51:14) if you select MBR, the
maximum you can go up to is 2 TB, if you want to expand your disk (51:21) more than
2 TB, then it will automatically convert from MBR to GPT, okay, just click (51:29)
okay, that's it, now it will be initialized and it became online, now just click
here on (51:35) the empty space and click on new simple volume, next, do you want
to create a volume with (51:44) entire 50 GB, I mean, volume is nothing but a
drive, C drive, D drive, like this, it will (51:50) also create another drive or do
you want to create it with some space only, let me do (51:56) it for 30 GB only,
okay, it is in MBs now, let me convert it into MBs, okay, 30 into (52:15) 1024,
30720, I just want to use 30720, so it will use, oh, sorry, so it will use only
(52:27) 30 GB from 51 GB and it will create a disk with 30 GB size, remaining 20 GB
will be free (52:34) only and drive later, I want to use it as N, that's it, click
next and NTFS, don't change (52:45) these things, I told you, as I told you, we
have a different file systems in Windows, (52:50) FAT32, NTFS and all, most of the
people will use NTFS only, unless if you have some requirement (52:56) to do it in
fast, okay, we will do NTFS, for example, this is a new, if you want to put (53:02)
something, like I am keeping it as NetApp, NetApp LUN, okay, LUN something, 01,
okay, (53:16) that's it, click next, finish, see, it created a 30 GB logical disk
with N drive, now if (53:26) you go to my computer, just open my computer here,
click on this PC, see, I am able to (53:36) see the N drive, just like, same thing
like a C drive, right, yeah, just open it, put (53:45) whatever data you want, it
will be copied to your LUN in the backend, okay, this is (53:51) the way how SAN
will work, what is the difference you noticed between SAN and NAS, NAS drive
(54:00) will look like this, like it will, under the drive symbol, it will show you
with a pipe, (54:06) which indicates a network, okay, but SAN drive, it will show
you like a normal disk, okay, (54:13) any questions guys, okay, no Navin, no
questions, and like not today, today or tomorrow, whenever (54:31) you feel time,
can you just note down the steps, how to do the complete task, like we (54:38)
create a LUN, then we create the iSCSI, LIF and then, like what are the steps,
sure, let (54:44) me do it straight away, okay, first one is validate iSCSI
license, that is the first (54:55) step, second step is configure iSCSI on vServer,
third step is create iSCSI LIFs, fourth step (55:21) is enable iSCSI service on
iSCSI LIFs, already enabled, you don't need to do that, but yeah, (55:39) next
thing is normally provide iSCSI LIF IPs to server team, they will do it, okay, from
(55:57) there, it is their task to establish the connection, okay, once the
connection is established, (56:05) validate that, in between the LUN steps, bro,
I'm not yet created the LUN, bro, I'm just (56:17) validating that, I'm able to see
the IQNs on storage end or not, okay, okay, yeah, don't (56:24) jump in like that,
okay, so just validate the IQN address, like once a connection is (56:32)
established, you'll be able to see the server IQNs, right, so that thing, okay,
yeah, yeah, (56:40) okay, this is the sixth step, which I forgot to put it in
front, seventh one, so once validation (56:47) is completed, just create a IGroup,
okay, IGroup with, in IGroup creation, you need (56:55) to mention OS type,
protocol, okay, IQN number of your server, all these things, okay, once (57:08)
IGroup is created, then create volume and LUN, and while creating LUN, you need to
again (57:18) mention OS type, space reserve option, like this, these kind of
things you need to mention, (57:27) once the LUN is created, next step is, map the
LUN to IGroup, okay, while creating, while (57:40) mapping the LUN to the IGroup,
what you need to mention, you don't need to mention anything, (57:45) like you need
to provide the LUN path, IGroup name and all, main important thing is LUN (57:50)
ID, because with the help of LUN ID only, we'll be able to see the LUN, because
without (57:55) LUN ID, you cannot identify the LUN, because no path and nothing
visible from the server (58:01) end, okay, it will be through LUN ID only, okay,
that's it, once the LUN ID is mapped, (58:07) just provide the mapping information
to the server team, so that they'll be able to set (58:13) up the LUN, or utilize
the LUN, whatever, based on their requirement, okay, these are (58:27) Now, what
is, okay, we have 15 more minutes, now server team wants to add this 20 GB also
(58:34) again, so what they can do, they can just extend the volume, extend, I want
to add the (58:41) entire 20 GB, so yeah, that's it, finish, see, now it became 50
GB, now just refresh (58:51) it, see, 49.9, like that, okay, now, what they need
is, like, they want to increase (59:03) the LUN to 100 GB, okay, they want to
increase the LUN to 100 GB, but let me know guys, I (59:11) mean, let me tell you
this, in real time, we will never ever increase the LUN size, (59:18) most of the
organizations will not follow that, okay, increasing and decreasing LUN, (59:24)
nobody will follow that, okay, generally, for example, server team require some
more (59:31) space, what we'll do, we'll create a bigger LUN, for example, now,
they are asking us (59:37) to increase the 50 more GB, right, so what I'm going to
do, instead of increasing the (59:41) LUN, I mean, we can do that, but that is, no
one will follow that, most of the people (59:46) will not follow that, or SAN
experts will recommend that, not recommend that, so what (59:51) is the procedure
is, just create a 100 GB LUN, and give it to them, and ask them to (59:58) move the
data from 50 GB LUN to 100 GB LUN, after that, we'll remove the 50 GB LUN, this
(1:00:04) is the thing which we'll follow, generally, this is the way, when there
is a requirement (1:00:10) to expand the LUN, this is the way we'll follow, but
some organizations, they'll increase (1:00:15) the LUN also, but, which most of the
countries, I know, I have worked in like a 7, 6 organizations, (1:00:23) no one
followed the increasing and decreasing the LUN procedure, okay, that is not best
(1:00:28) practice, but you can do that, pretty simply, actually, so, in order to
increase the LUN, (1:00:33) what you need to do, first, you need to increase the
volume, let's, again, 51 GB I'm adding, (1:00:53) now my volume become 102 GB, just
increase the LUN size now, LUN resize, there is a command (1:01:01) called resize
is the command, vserver name, and LUN path, provide the LUN path here, now
(1:01:10) size, plus 50 GB, that's it, now check LUN show command, see, I'm able to
see my LUN (1:01:23) size now, 100 GB, so, just again, just inform that, you have
increased it, so, what they'll (1:01:30) do, they'll be not able to see it here,
immediately, but, yeah, sometimes you need to rescan it, (1:01:35) sometimes it
will do automatically, see, immediately, it is equal, just what they do, again,
they'll (1:01:41) extract it, that's it, finish, it will become 100 GB, if you go
to this PC, see, I'm able (1:01:54) to see it, 100 GB, like that, pretty simple,
expand, but, we will not follow that, even (1:02:00) I'll also not recommend to
proceed with the LUN expansion, generally, in real time, whenever (1:02:09) this
kind of requirement is there, we'll just give it a new LUN, we'll give a new LUN,
we'll (1:02:13) ask the team to, server team to, or application team to, move the
data from old LUN to new (1:02:18) LUN, after they moved everything, we'll
decommission the old LUN, that is the process we'll follow, (1:02:23) any questions
guys, so far, ok, now, I'll show you how to delete the LUN, now, for example,
(1:02:34) decreasing the LUN, again, decreasing the LUN, no company will do that,
ok, you can (1:02:40) play with volume, you can play with Qtree quotas, decrease
it, increase it, but, never (1:02:46) ever decrease the LUN, which is not at all
recommended, you can increase the LUN, ok, (1:02:52) that's ok, but, decreasing the
LUN is not at all recommended, because, you need to follow (1:02:56) a workflow, to
decrease the LUN, if you missed a single step, which will cause entire data
(1:03:05) corruption in that LUN, so, you'll lose all the data, so, that is the
reason, which, it (1:03:10) is not recommended, but, if you want to do that, you
don't have any choice, then, first (1:03:16) what you need to do, first, server
team has to shrink it from there, shrink, shrink volume, (1:03:23) ok, so, how much
amount of space you guys need to shrink, for example, I want to shrink (1:03:32)
it, I want to remove 30 GB from it, 30 GB means, nothing but, 30720, right, oh, ok,
(1:03:45) enter the amount of space to shrink in MB, let me see, I'm bit confused
here, shrink, (1:03:54) yeah, so, it removed 30 GB again, from, now it became 70,
ok, so, now, you need, now, (1:04:03) server team will give us the confirmation,
like, ok, we have shrinked it 30 GB, now, (1:04:08) you can decrease it, after that
only, you need to do that, again, which is very risky, (1:04:15) ok, resize,
vserver name, LUN path, size, minus 30 GB, yeah, see, that is what it is (1:04:36)
saying, reducing a LUN size without coordination with a host may cause permanent
loss or corruption, (1:04:42) use force flag to size reduction, see, they are using
force flag, you must use the force (1:04:47) flag to do that, even NetApp also not
recommending this, now, LUN show, see, now it became again (1:04:55) 70 GB, you
will be not able to see that extra 70 GB here, rescan that, see, you lost that
(1:05:08) 30, now, again, if you refresh it, it will become 69 point, like that,
this is the way (1:05:16) you can decrease the LUN as well, I mean, reduce the size
of the LUN, but, don't do (1:05:21) that, any questions? (1:05:30) Naveen, one
question, like, while you are explaining the iSCSI one, so, you used a word
(1:05:36) mix, so, could you please explain on that? (1:05:41) Nothing, like, same
LUN, you want to present it to two servers, but, through, one server (1:05:50) is
connected to my NetApp through FCP protocol, another server is connected to my
NetApp through (1:05:55) iSCSI protocol, ok, but I want to present the same LUN to
both of them, then, just use (1:06:01) mixed instead of specifying protocol, so, it
will accept if it is connected to FCP or (1:06:07) iSCSI, it doesn't matter.
(1:06:09) Ok, so, in normal standard and, like, as best practice, we always select
mix, right? (1:06:18) No, no, no, we always specify the one, I mean, it is not like
a best practice and all, it (1:06:24) is based on your requirement, see, normally,
you know, right, you are already working on (1:06:29) this LUN, normally, I have
two servers, ok, just a quick question to you, like, I have (1:06:36) a LUN, from
NetApp prospect, I have very less experience on this LUN part, ok, not on NetApp
(1:06:44) prospect, any LUN, ok, yeah, that's fine, it is common for LUN, so, can I
assign this (1:06:51) LUN to both the servers, these are independent servers, can I
assign a single LUN to them? (1:06:58) Yes. (1:06:59) No, you cannot, you cannot
assign a single LUN to multiple servers if they are individual, (1:07:08) if they
are clustered, then you can do that, because SAN will work at block level, ok,
(1:07:18) in NAS, you can do that, even if they are individual servers or whatever
it is, it doesn't (1:07:23) matter, because SAN will work at file level, but when
it comes to NAS, NAS will work at (1:07:29) block level, so, if these two servers
are, let me, let me finish it, after that you can (1:07:34) tell, ok, if these two
servers are individual servers, then there will be no coordination (1:07:41) while
writing the data to the LUN, ok, so, which will cause to data corruption on LUN,
(1:07:50) so, that is the reason in SAN, nobody will assign a single LUN to
multiple servers if (1:07:56) they are not part of a cluster, but if they are part
of a cluster, then there will be (1:08:01) a quorum, a disk, which will be
involved, and iSCSI 3 service will be enabled, which (1:08:08) will establish the
coordination between these two servers, so, when one server is writing, (1:08:17)
the other server will be blocked automatically from writing the data, that is the
reason (1:08:24) it can avoid the data corruption, but generally, nobody will
allocate it to different servers, (1:08:30) yeah, ok, the concept is very good and
to understand, but let me share you my experience, (1:08:40) like, in IBM SAN, so,
we create this, we make the initiator, every server is a individual (1:08:51)
initiator and this one LUN can be mapped with multiple servers also, no, but
therefore, (1:09:01) like it's been a long time I have not been using that SAN,
they must be part of a cluster, (1:09:07) then only you can do that, so, like as
you explained, so, that was the thing I was thinking (1:09:14) of, yeah, they
should be part of a cluster from server side, then only you can assign (1:09:21) a
single LUN to multiple servers or else it will cause data corruption, I think Bhanu
(1:09:28) also know that, Bhanu, ok, yeah, correct, yeah, ok, Aditya, did you got
it, this is (1:09:38) very, very important, ok, never ever assign a single LUN to a
multiple servers, I mean (1:09:44) individual servers, those two servers are part
of a cluster, then you can do that, ok, (1:09:54) now, last thing, which I am going
to show you, I have 5 more minutes, right, let's do (1:09:58) it, I am going to
add, I have another server, right, vserver iSCSI initiator show, Aditya, (1:10:06)
can you please mute yourself, iSCSI initiator show, I have another server also,
right, (1:10:17) which is DC1, so, what I am going to do, I am going to add this to
the iGroup, iGroup (1:10:27) add vserver name, iGroup name, which is my iGroup name
and IG, for example, let's assume (1:10:34) that I bought a new server and I have
added it to cluster, I mean, not NetApp cluster, (1:10:43) server cluster, ok, that
time only we will add the multiple servers to the same iGroup (1:10:49) as well,
ok, now I have added this to the same iGroup and now if you take a look at the
(1:10:54) iGroup show command, see, I am able to see two IQN numbers here, now, I
have already (1:11:00) assigned, I have already mapped a 70 GB LUN to IG1, right,
the same LUN will be visible (1:11:06) on the second server as well, just let me
login, see, this is my second server, right, (1:11:12) and if I go to my computer,
I will be able to see it automatically, let me take some (1:11:22) time, let me go
to disk management, computer management, go to disk management, let's do (1:11:38)
rescan, see, 70 GB, automatically it will be visible to you, ok, just you need to
bring (1:11:48) it online, that's it, no other configurations required because I
have already configured (1:11:55) on another server, right, and I have added these
two servers to same iGroup, that means (1:12:02) these two are part of a cluster,
it will assume like that, so it will apply the same configurations (1:12:08) and
the LUN will become visible to the second server also automatically, see, but as
these (1:12:15) two servers are not in cluster, I know that, so I just want to show
you, that's why I have (1:12:20) added, but in real time, don't do that, if they
are individual servers, ok, any questions (1:12:31) guys, if you don't have any
questions, then I will stop the session for today. (1:12:38) No, Naveen, thank you.
(1:12:40) Yeah, shallow then, thank you guys, have a nice day. (1:12:45) See you
tomorrow, bye. (1:12:47) Yeah.

You might also like