UrbanPro
true

Learn Big Data from the Best Tutors

  • Affordable fees
  • 1-1 or Group class
  • Flexible Timings
  • Verified Tutors

Search in

Different Data File Formats in Big Data

Raghunandana S K
17 Mar 0 0

Overview

 

In this lesson I will be explaining the different kinds of Data File formats used in Big Data, These are widely used but unspoken of. Anyone aspiring to be a Data Engineer/Data Analyst/ML engineer should be aware of this hidden magic in big data.

 

Why do we even need a data file format? 

 

- Imagine storing all information you get in a random order, Will this help? No, it won't 

- Every company gets 10s and 1000s of GB data every day. If these are not stored in a proper format then understanding this data will be difficult. 

- More time you spend sorting through the data, The company is missing out on the opportunity to retain customers or generate more orders/revenue.

 

This is why data file formats are used in Data lake.

 

Different kinds of file formats present

 

We have CSV,JSON,ORC,Parquet,Avro. These are the leading data file formats present.

Which one to choose from and why to choose them completely depends on the use case of a company.

 

Use-case 1:

If you are looking into total sales data from a table let's say orders, Then this requires 1 column in your table sale_amount to be scanned/quired.

Hence storing this table in columnar format is the best way to go.

File formats to choose: ORC, Parquet

 

Use-case 2 :

If you are trying to identify the customer behavior, What kind of items are customers placing the order? 

To gather this information you have to scan row-level information spanning multiple columns like user_id, item_name, sale_amount.

Here, Storing the data in row-level format is the right choice to make.

File formats to choose: CSV, JSON, Avro 

 

Conclusion 

It's important to know that we don't have one fit-all. The question you must ask "How will this data be used? " Based on this file format should be defined.

0 Dislike
Follow 1

Please Enter a comment

Submit

Other Lessons for You

HDFS Commands - Data Engineering
HDFS commands : HDFS commands will interact with namenode to show results some commands like cat ,tail will interact with datanode to show results HDFS help commands: hadoop fs -help ls list commands...

Ram Kumar | 10 Jun

0 0
0

Skill Enhancement Program | Industry Training | Atreya Associates
“Education is not the learning of facts, but the training of mind to think.” - Albert Einstein Skills are crucial to allow any person to manage their daily lives in this ever-changing world....

4 Key Things to Learn for Data Science
1. Theory:Use Coursera and EdX for theory, concepts, and applications of probability, statistics, linear algebra, calculus, and machine learning.2. Data Visualisation:Tableau and PowerBI are easy-to-use...

CheckPointing Process - Hadoop
CHECK POINTING Checkpointing process is one of the vital concept/activity under Hadoop. The Name node stores the metadata information in its hard disk. We all know that metadata is the heart core...

Silvia Priya | 26/11/2020

0 0
0

An Introduction to Probability and Theory of Counting
In this lesson, we introduce the concept of probability with high school mathematics as a prerequisite. Before we start, I want to make you familiar with some standard terms in probability theory. The...

Sayan Mukhopadhyay | 17/08/2020

0 0
0
X

Looking for Big Data Classes?

The best tutors for Big Data Classes are on UrbanPro

  • Select the best Tutor
  • Book & Attend a Free Demo
  • Pay and start Learning

Learn Big Data with the Best Tutors

The best Tutors for Big Data Classes are on UrbanPro

Book a Free Demo

UrbanPro.com is India's largest network of most trusted tutors and institutes. Over 55 lakh students rely on UrbanPro.com, to fulfill their learning requirements across 1,000+ categories. Using UrbanPro.com, parents, and students can compare multiple Tutors and Institutes and choose the one that best suits their requirements. More than 7.5 lakh verified Tutors and Institutes are helping millions of students every day and growing their tutoring business on UrbanPro.com. Whether you are looking for a tutor to learn mathematics, a German language trainer to brush up your German language skills or an institute to upgrade your IT skills, we have got the best selection of Tutors and Training Institutes for you. Read more