Books, libraries and document sections in the office are a thing of the past. In today’s world, everything is digitalized. With large volumes of data getting added every second, using & accessing it is itself becoming a huge challenge.
The solution that organizations are using to solve this puzzle is “Big Data”.
What is Big Data and Why Big Data now?’
Big Data, as a whole, can be defined as huge amounts of data. Usually measured up to Terabytes or Petabytes (more details on this later).
Source – Reeder assoc
Anything and everything creates Big Data
- Tracking your uber driver location,
- Managing your zomato orders over time,
- Storing the number of calories you’ve burned during your workout,
- The videos you upload,
- The items you’ve added into your online cart,
- Your transaction history etc.
Virtually, everything adds up to Data making it Big Data.
Clearly, big data is now starting to move past being simply a buzzword and a thing of large companies to a term and concept that defines the competitive edge of every firm.
Just how much Big Data are we creating?
In recent years, Data production has increased exponentially. In 2013, we produced 4.4 Exabytes of data according to Wikipedia.
Wow, hold on! What does that actually mean?
Let us take a deep breath and understand in terms of things we normally relate to.
- A byte is one character or a letter
- A kilobyte is 1024 characters or roughly 200 words (maybe a page?)
- A megabyte is a large book of 1024 pages
- A gigabyte is a small library of 1024 such books
- A terabyte is a large library of over million such books
- A petabyte is 1000 such large libraries
- An exabyte is a million such large libraries of million books each.
Source: Digitalized Magazine
So, in 2013, we produced new data worth 4.4 million large libraries!
Is that a lot? This Quora query makes it easy to put things in perspective.
Yes. In all these years, we created physical books of less than 1/10th as much as digital data we produced in 2013!
Thanks to Social Media, IOT, Smartphones, and Smart Living that by 2023, we will generate 10 times what we created in 2013. Now, you see the problem?
Yes. How do we handle so much data and
Even more challenging question is how do we put it to use?
Organizations need a new age workforce to help them achieve it.
One way to think about is that organizations now need “new age librarians” to manage this vast data. “The Big Data Specialists” are the new age librarians of this humungous data being generated. Their job description is thus to process, order, store and retrieve this humungous digital data. But, being a librarian is only one easy job. Then, Big Data Specialists are also wearing the hat of analysts. They analyse the data to extract patterns, trends and associations.
To help us get a glimpse of Big Data and how this industry has evolved around the years, INSOFE’s Big Data Specialist Yugandhar Reddy shared some really interesting tidbits about his “Big Data Specialist” job.
Yugandhar, what makes you fall in love with your work all over every day?
Ans: Big Data is not a new concept. It has been around for many decades and we were able to handle it but on a very very minute scale. But now, the technology has evolved and new tools have been on the rise, upgrades keep happening every day.
Being a part of this revolution by learning and teaching at INSOFE, makes me fall in love with my work every day.
What according to you is the most exciting aspect of your work?
Ans: Being able to learn and teach about new technology being implemented. When I started out as a System’s Engineer in 2004, we were able to handle a maximum of 5 GB data in the form of DBMS or RDBMS. This limit was extended to a maximum of 150 GB when high-end systems were used.
And this was all without the use of Hadoop in that time. By using Hadoop, the data handling capacities improved exponentially to over 1 TB of data. Handling such huge amounts of data and being able to pass on the knowledge about it is the most exciting aspect of my work.
How do you see your job changing over time? From databases to Hadoop to SPARK
Ans: In all of my 15 years of experience working as a Big Data Architect, I saw that the data has undergone changes but in a cyclical order.
The real changes and challenges were triggered when the data began morphing from structured to unstructured. And initially, cloud was a concept interesting to read about but risky to use owing to the fact that we didn’t know how secure it actually is?
But after a few years, everyone’s in the race for cloud computing. Even we at INSOFE, use AWS extensively along with the servers we have in-house. Now, we have various platforms that can help us in handling unstructured data. And we are heavily invested in training students along similar lines at INSOFE.
What kind of future do you see for Big Data specialists? What advice do you have for Big Data Aspirants?
Ans: Firstly, I would like them to acknowledge themselves that they would be choosing a pretty challenging and highly intuitive field that’s not going anywhere at least for the next 20 years. But, they can only call themselves Big Data Specialists until and unless they are experienced in Hadoop.
Adopt practices that lead to immense usage of both Hadoop and Cloud computing. Fall in love with Open Source tools. Equip yourself with Spark, Scala and you will truly emerge as a Global Data Leader.
That’s all for this week. Let us meet next week with another exciting topic of AI and ML
If you are interested in computer programming-related jobs, INSOFE PGP in Data Science offers unparalleled opportunities to transform yourself into a Big Data Specialist in six months and zoom ahead in the career.