Web Scrapping Using Python

  

Web Scrapping Using Python

Web Scrapping

Web scraping is the process of gathering information from the Internet. Even copying and pasting the lyrics of your favorite song is a form of web scraping! However, the words “web scraping” usually refer to a process that involves automation. Some websites don’t like it when automatic scrapers gather their data, while others don’t mind.

Reasons for Web Scraping

The world offers other ways to apply that surfer’s mindset! Instead of looking at the job site every day, you can use Python to help automate your job search’s repetitive parts. Automated web scraping can be a solution to speed up the data collection process. You write your code once, and it will get the information you want many times and from many pages.

In contrast, when you try to get the information you want manually, you might spend a lot of time clicking, scrolling, and searching, especially if you need large amounts of data from websites that are regularly updated with new content. Manual web scraping can take a lot of time and repetition.

There’s so much information on the Web, and new information is constantly added. You’ll probably be interested in at least some of that data, and much of it is just out there for the taking. Whether you’re actually on the job hunt or you want to download all the lyrics of your favorite artist, automated web scraping can help you accomplish your goals.

Challenges of Web Scraping

The Web has grown organically out of many sources. It combines many different technologies, styles, and personalities, and it continues to grow to this day. In other words, the Web is a hot mess! Because of this, you’ll run into some challenges when scraping the Web:

  • Variety: Every website is different. While you’ll encounter general structures that repeat themselves, each website is unique and will need personal treatment if you want to extract the relevant information.

  • Durability: Websites constantly change. Say you’ve built a shiny new web scraper that automatically cherry-picks what you want from your resource of interest. The first time you run your script, it works flawlessly. But when you run the same script only a short while later, you run into a discouraging and lengthy stack of tracebacks!

Unstable scripts are a realistic scenario, as many websites are in active development. Once the site’s structure has changed, your scraper might not be able to navigate the sitemap correctly or find the relevant information. The good news is that many changes to websites are small and incremental, so you’ll likely be able to update your scraper with only minimal adjustments.

Steps for Scraping using Python

1. Search the URL you want to scrap.

  Here, I am using the following data science page for scrapping

  https://en.m.wikipedia.org/wiki/Data_science



 2.  Inspecting the Page

 Right-click on the webpage, click Inspect, and you’ll see the inside of that site: its source code, the     images and CSS that form its design, the fonts and icons it uses, the Javascript code that powers           animations, and many more.



3. Find the data you want to extract

Here, I am extracting the heading present in the div class

 4. Code:

Create a python file

(Python file can be created in any of the python interpreter like pycharm, jupyter notebook, google colaboratory, Vs code,etc.)

Here, I am using google Colaboratory



5. Run the code and extract the data

      6. Save in the required proper format

   Saved it as data.csv file

This is about Web Scarping using python.

Thank You😊

Click here for the Code





Comments

Post a Comment

Popular posts from this blog

18IT100_Practical_Exam_Work

Image
Task-1: Dataset Description using Orange tool. What is need to be done to improve the accuracy of the classification result of the given dataset? Get the maximum classification accuracy possible by performing the following methods. -->Pre-processing o Encoding o Normalization o Missing value handling o Feature Selection Compare your accuracy with and without applying pre-processing steps. Perform the Classification and visualize accuracy before and after preprocessing in Orange/Python. What is Data Pre-processing? In any Machine Learning process, Data Preprocessing is that step in which the data gets transformed, or Encoded, to bring it to such a state that now the machine can easily parse it. In other words, the features of the data can now be easily interpreted by the algorithm. Encoding Encoding is the process of converting data from one form to another. While "encoding" can be used as a verb, it is often used as a noun, and refers to a specific type of encoded data Nor...
Read more

Practical:10 Getting started with Neo4j and Gephi Tool

Image
This blog contains two different tools , Neo4j and Gephi tool for data visualization in graphical form . Neo4j Neo4j delivers the lightning-fast read and write performance you need, while still protecting your data integrity. It is the only enterprise-strength graph database that combines native graph storage, scalable architecture optimized for speed, and ACID compliance to ensure the predictability of relationship-based queries. In simple words, Neo4j is the MySQL of the graph databases. It provides a graph database management system, a language to query the database, a.k.a CYPHER, and a visual interface with the neo4j browser. Advantages of Neo4j Following are the advantages of Neo4j. Flexible data model − Neo4j provides a flexible simple and yet powerful data model, which can be easily changed according to the applications and industries. Real-time insights − Neo4j provides results based on real-time data. High availability − Neo4j is highly available for large enterprise real-time...
Read more

Practical:11 PREDICTING GENDER AND AGE USING IMAGE DATA IN PYTHON

Image
What is OpenCV? OpenCV is short for Open Source Computer Vision. Intuitively by the name, it is an open-source Computer Vision and Machine Learning library. This library is capable of processing real-time image and video while also boasting analytical capabilities. It supports the Deep Learning frameworks TensorFlow, Caffe, and PyTorch. What is a CNN? A Convolutional Neural Network is a deep neural network (DNN) widely used for the purposes of image recognition and processing and NLP. Also known as a ConvNet, a CNN has input and output layers, and multiple hidden layers, many of which are convolutional. In a way, CNNs are regularized multilayer perceptrons. Age and gender, two of the key facial attributes, play a very foundational role in social interactions, making age and gender estimation from a single face image an important task in intelligent applications, such as access control, human-computer interaction, law enforcement, marketing intelligence and visual surveillance, etc. Her...
Read more