Welcome to our Pyspark tutorial series. Before we start, we do want to mention, if you have no idea about Big data and impact it will have in 2019.
you need to know before learning Pyspark?
Pyspark Tutorial series will rely on some concepts which we expect you to be aware of, we are not asking you to be professional but at least more than beginner in following technologies
- Python – Basics required(at least must have programming knowledge)
- Big Data – Concepts required
Again if you are not an expert that is fine, we just want to make you aware that it is the least what is required.
System requirement for learning
pyspark as beginner
Following will be specification of system we will use for this pyspark series
- Windows 7 – 64 Bit
- 8 GB of Ram
- 1 TB of HDD
Who should follow this tutorial series?
Basically anyone with basic knowledge of python(programming knowledge). We don’t differntial but anyone who is comfortable with big data or willing to work on big data is good on our list. But for sake of it, here is the proper list
- Students(always recommended)
- MNC Freshers(will provide you a huge boost)
- Software Developers / Web Developers ( good for you to skyrocket your career growth)
- Big data hackers(who like to play with big data)
What is Pyspark?
Pyspark is the combination of 2 powerful technologies
- Python – A programming language
- Apache Spark – Apache Spark is a unified analytics engine for big data processing, with built-in modules for streaming, SQL, machine learning and graph processing – official website
Now if we can simplify it. With the help of Pyspark framework, we can process our big data and can get meaningful results in different formats like graphs using python or SQL like quries on files like .csv or .json
Now if you don’t understand anything, no need to worry about. Just follow along and carry on.
- Introduction to Pyspark (Current Lesson)
- Installing required software / Setting up
- Working with SparkContext (Coming soon)
- What are
dataframes, working with dataframes(Coming Soon)
- Working with SparkSession (Coming Soon)
- Using Panda with Spark Dataframe (Coming Soon)
AnalysingBig Data (Coming Soon)
- Working on a real working Project (Coming Soon)
- MLib to create Powerful Machine Learning Models (Coming Soon)
- Create a Spam filter using Spark and Natural Language Processing (Coming Soon)
- Use Spark Streaming to Analyze Tweets in Real Time! (Coming Soon)