Python program to extract Email-id from URL text file

Python program to Recursively scrape all the URLs of the website

Last Updated : 26 Mar, 2020

In this tutorial we will see how to we can recursively scrape all the URLs from the website

Recursion in computer science is a method of solving a problem where the solution depends on solutions to smaller instances of the same problem. Such problems can generally be solved by iteration, but this needs to identify and index the smaller instances at programming time.

Note: For more information, refer to Recursion

Modules required and Installation

Requests :
Requests allows you to send HTTP/1.1 requests extremely easily. There’s no need to manually add query strings to your URLs.
```
pip install requests
```
Beautiful Soup:
Beautiful Soup is a library that makes it easy to scrape information from web pages. It sits atop an HTML or XML parser, providing Pythonic idioms for iterating, searching, and modifying the parse tree.
```
pip install beautifulsoup4
```

Code :

from bs4 import BeautifulSoup 
import requests 
   
# lists 
urls=[] 
   
# function created 
def scrape(site): 
       
    # getting the request from url 
    r = requests.get(site) 
       
    # converting the text 
    s = BeautifulSoup(r.text,"html.parser") 
       
    for i in s.find_all("a"): 
          
        href = i.attrs['href'] 
           
        if href.startswith("/"): 
            site = site+href 
               
            if site not in  urls: 
                urls.append(site)  
                print(site) 
                # calling it self 
                scrape(site) 
   
# main function 
if __name__ =="__main__": 
   
    # website to be scrape 
    site="http://example.webscraping.com//"
   
    # calling function 
    scrape(site) 
    

Output :

python-web-scraping

Python program to extract Email-id from URL text file

R

rakshitarora

Improve

Article Tags :

Practice Tags :

python

Similar Reads

Python program to extract Email-id from URL text file

Prerequisite : Pattern Matching with Python Regex Given the URL text-file, the task is to extract all the email-ids from that text file and print the urllib.request library can be used to handle all the URL related work. Example : Input : Hello This is Geeksforgeeks [email protected] rev

Python | Get email alert when the website is up

In this article, we are going to learn how to check whether any website is running or it's down with a simple Python script. We will use Python's requests library for sending 'get' request and 'smtplib' library for sending email notification when the website is up.It means we don't need to check eve

Remove URLs from string in Python

A regular expression (regex) is a sequence of characters that defines a search pattern in text. To remove URLs from a string in Python, you can either use regular expressions (regex) or some external libraries like urllib.parse. The re-module in Python is used for working with regular expressions. I

Web Scraping using lxml and XPath in Python

Prerequisites: Introduction to Web Scraping In this article, we will discuss the lxml python library to scrape data from a webpage, which is built on top of the libxml2 XML parsing library written in C. When compared to other python web scraping libraries like BeautifulSoup and Selenium, the lxml pa

Python | URL shortener using tinyurl API

There are multiple APIs (e.g-bitly API, etc.) available for URL shortening service but in this article, we will be using Tinyurl API to shorten URLs. Though tinyURL has not officially released its API but we are going to use it unofficially. Here, we can put as input any number of URLs at a time and

Python - Remove substring list from String

In Python Strings we encounter problems where we need to remove a substring from a string. However, in some cases, we need to handle a list of substrings to be removed, ensuring the string is adjusted accordingly. Using String Replace in a LoopThis method iterates through the list of substrings and

Python program to find GSoC organisations that use a Particular Programming Language

Currently, it's not possible to sort GSoC participating organizations by the programming languages they use in their code. This results in students spending a lot of time going through each organization's page and manually sorting through them. This article introduces a way for students to write the

Python Get All Values from Nested Dictionary

In this article, we will learn how we can Get all Values from Nested Dictionary in Python Programming. A nested Dictionary in Python is a dictionary that contains another dictionary as its values. Using this, we can create a nested structure where each of the key-value pairs is in the outer dictiona

Python program to convert URL Parameters to Dictionary items

URL parameters are often used to pass data between web pages and converting them into a dictionary makes it easier to work with the data programmatically. In this article, we will learn how to convert URL parameters into dictionary items in Python. Using urllib.parse moduleThe urllib.parse module pr

Extract feed details from RSS in Python

In the article, we will be seeing how to extract feed and post details using RSS feed for a Hashnode blog. Although we are going to use it for blogs on Hashnode it can be used for other feeds as well. RSS means Rich Site Summary and uses standard web formats to publish information that changes frequ