Memory profiling in Python using memory_profiler

If you use Python a lot then you probably know that many people claim that Python takes up more time to execute. Well, you probably have seen approaches like the total time spent to execute a part of code or something like that but sometimes you need something more than that. What about RAM usage? Nobody really talks about that but it's equally essential. This might look really insignificant but it's actually pretty important while writing code for production. 

We will be using memory-profiler from PyPi. We will also be needing requests to test the functionality. To do so, simply type the following in your terminal

pip3 install memory-profiler requests

Note: If you are working on windows or using a virtual env, then it will be pip instead of pip3

Now that everything is set up, rest is pretty easy and interesting obviously.  Create a new file with the name word_extractor.py and add the code to it. Below is the implementation of the code. Everything is pretty well documented as in-line comments.

# imports
from memory_profiler import profile
import requests


class BaseExtractor:
    # decorator which specifies which
    # function to monitor
    @profile
    def parse_list(self, array):
      
        # create a file object
        f = open('words.txt', 'w')
        for word in array:
            
            # writing words to file
            f.writelines(word)
    
    # decorator which specifies which 
    # function to monitor
    @profile
    def parse_url(self, url):
      
        # fetches the response
        response = requests.get(url).text
        with open('url.txt', 'w') as f:
            
            # writing response to file
            f.writelines(response)

Notice the @profile this is a decorator. Any function which is decorated by this decorator, that function will be tracked. Now, our main code is ready. Let's write the driver code which will call this class functions. Now, create another file called run.py and insert the following code in it.

from word_extractor import BaseExtractor


if __name__ == "__main__":
    
    # url for word list (huge)
    url = 'https://raw.githubusercontent.com/dwyl/english-words/master/words.txt'
    
    # word list in array
    array = ['one', 'two', 'three', 'four', 'five']
    
    # initializing BaseExtractor object
    extractor = BaseExtractor()
    
    # calling parse_url function
    extractor.parse_url(url)
    
    # calling pasrse_list function
    extractor.parse_list(array)

So, basically now we are done. You will notice that parse_url() will consume more memory than parse_list() which is obvious because parse_url calls a URL and writes the response content to a text file. If you open the link, then you will find that the word list is huge. So, now to test your code, simply run the run.py  file. You can do so by typing 

python3 run.py

Note: If you are working on windows or using a virtual env, then it will be python instead of python3

If everything ran successfully, then you should see something like this

An important thing to remember is that memory-profiler itself consumes a significant amount of memory. Use this only in development but avoid it in production.