Practical:6 Data-Preprocessing with orange Tool

A guide to data preprocessing using Orange and how to use Python in Orange.

This blog will help you understand how to perform Data pre-processing using Orange, use the Orange library in Python, and integrate Python Scripts in Orange.

What is Data Preprocessing?

When we talk about data, we usually think of some large datasets with a huge number of rows and columns. While that is a likely scenario, it is not always the case — data could be in so many different forms: Structured Tables, Images, Audio files, Videos, etc..

Machines don’t understand free text, image, or video data as it is, they understand 1s and 0s. So it probably won’t be good enough if we put on a slideshow of all our images and expect our machine learning model to get trained just by that!

In any Machine Learning process, Data Preprocessing is that step in which the data gets transformed, or Encoded, to bring it to such a state that now the machine can easily parse it. In other words, the features of the data can now be easily interpreted by the algorithm.

Data discretization 

Continues features in the data can be discretized using a uniform discretization method. Discretization considers only continues features, and replaces them in the new data set with corresponding categorical features:




Continuization

Continuization refers to transformation of discrete (binary or multinominal) variables to continuous. The class described below operates on the entire domain; documentation on Orange.core.transformvalue.rst explains how to treat each variable separately.

  • #python script for Continuization
  • import Orangetitanic = Orange.data.Table(“titanic”)continuizer = Orange.preprocess.Continuize()titanic1 = continuizer(titanic)print(“Before Continuization : “,titanic.domain)print(“After Continuization : “,titanic1.domain)#Data of row 15 in the before and after continuizationprint(“15th row data before : “,titanic[15])print(“15th row data after : “,titanic1[15])




Normalization

Normalization is the process of organizing data in a database. This includes creating tables and establishing relationships between those tables according to rules designed both to protect the data and to make the database more flexible by eliminating redundancy and inconsistent dependency.

Normalization is used to scale the data of an attribute so that it falls in a smaller range, such as -1.0 to 1.0 or 0.0 to 1.0. Normalization is generally required when we are dealing with attributes on a different scale, otherwise, it may lead to a dilution in effectiveness of an important equally important attribute.







Random sampling data 

Random sampling is done by constructing a vector of subset indices (e.g. a table of 0’s and 1’s), one corresponding to each instance, and then passing the vector to the table’s Orange.data.Table.select method.

With randomization, given a data table, the preprocessor returns a new table in which the data is shuffled. Randomize function is used from the Orange library to perform randomization.



Guide To Use Python Scripts In Orange

Python Script is this mysterious widget most people do not know how to use, even those versed in Python. Python Script is the widget that supplements Orange functionalities with (almost) everything that Python can offer.

We will try to replicate the work for Discretization using Python Script. As shown below, we will create two paths in the workflow; one will use the Discretize Widget and then give output; meanwhile, the other path will go to Python Script, where we will write the logic related to discretization.



Discretization using Python Script

Now, two different tables will be generated — one for each path. ‘Data Table’ has output after passing through the Discretize widget, and ‘Data Table (2)’ has the output after passing through the Python Script Widget.





As you can see above, both the Data Tables are similar. This helps in proving that by using Orange, we can also carry out Script Programming along with Visual Programming.

Thank you:)





Comments

Post a Comment

Popular posts from this blog

18IT100_Practical_Exam_Work

Image
Task-1: Dataset Description using Orange tool. What is need to be done to improve the accuracy of the classification result of the given dataset? Get the maximum classification accuracy possible by performing the following methods. -->Pre-processing o Encoding o Normalization o Missing value handling o Feature Selection Compare your accuracy with and without applying pre-processing steps. Perform the Classification and visualize accuracy before and after preprocessing in Orange/Python. What is Data Pre-processing? In any Machine Learning process, Data Preprocessing is that step in which the data gets transformed, or Encoded, to bring it to such a state that now the machine can easily parse it. In other words, the features of the data can now be easily interpreted by the algorithm. Encoding Encoding is the process of converting data from one form to another. While "encoding" can be used as a verb, it is often used as a noun, and refers to a specific type of encoded data Nor...
Read more

Practical:10 Getting started with Neo4j and Gephi Tool

Image
This blog contains two different tools , Neo4j and Gephi tool for data visualization in graphical form . Neo4j Neo4j delivers the lightning-fast read and write performance you need, while still protecting your data integrity. It is the only enterprise-strength graph database that combines native graph storage, scalable architecture optimized for speed, and ACID compliance to ensure the predictability of relationship-based queries. In simple words, Neo4j is the MySQL of the graph databases. It provides a graph database management system, a language to query the database, a.k.a CYPHER, and a visual interface with the neo4j browser. Advantages of Neo4j Following are the advantages of Neo4j. Flexible data model − Neo4j provides a flexible simple and yet powerful data model, which can be easily changed according to the applications and industries. Real-time insights − Neo4j provides results based on real-time data. High availability − Neo4j is highly available for large enterprise real-time...
Read more

Practical:11 PREDICTING GENDER AND AGE USING IMAGE DATA IN PYTHON

Image
What is OpenCV? OpenCV is short for Open Source Computer Vision. Intuitively by the name, it is an open-source Computer Vision and Machine Learning library. This library is capable of processing real-time image and video while also boasting analytical capabilities. It supports the Deep Learning frameworks TensorFlow, Caffe, and PyTorch. What is a CNN? A Convolutional Neural Network is a deep neural network (DNN) widely used for the purposes of image recognition and processing and NLP. Also known as a ConvNet, a CNN has input and output layers, and multiple hidden layers, many of which are convolutional. In a way, CNNs are regularized multilayer perceptrons. Age and gender, two of the key facial attributes, play a very foundational role in social interactions, making age and gender estimation from a single face image an important task in intelligent applications, such as access control, human-computer interaction, law enforcement, marketing intelligence and visual surveillance, etc. Her...
Read more