Sunday, 8 September 2019

Stream the Real-Time data on Google Cloud Platform

Hello Everybody!! Today we will see how GCP components like Pub/Sub, Dataflow and BigQuery can be used to handle streaming the data. There are few kinds of products in market which having similar types of functionalities like amazon kinesis, Apache Kafka etc.

But today, what we'll use? we'll use the Google cloud product for the real-time data processing.

we are going to use Pub/Sub, Dataflow and BigQuery for the real-time data processing.

We will discuss one by one so let’s start ……

Firstly, we need to open Pub/Sub console by searching on search bar of gcp console. Then click on Create topic and put the topic id then click on create topic.

After successfully creation of topic, let's go to the bigquery console under this pubsub_BigQuery dataset we'll create a table called employee table (only table schema). let's have one field called message so this is will be our message will be sending via Pub/Sub.

Now lets come to dataflow, click on create job then give a name of job in dataflow and select a template as Pub/Sub to bigquery then you need to provide all information in the fields(like temporary output location, BigQuery dataset& table details etc.) as mention the below screenshots.

Then click on Run Job button, we are ready to stream the data from Pub/Sub to BigQuery by using Dataflow. So, let’s go to Pub/Sub and click on publish message here we will get to two options to pass the message:

1)message Box: in this box we have pass the value in the form of key-value pairs.

For example:

“message”: “Welcome to My World!!”

2) Just click on Add AN Attribute: Here you can add N numbers of attributes according to you BigQuery table schema and put key-value pairs in the fields.

Note: key-value pair is the input for the empty table which we have created earlier.

After publishing the message, again come to BigQuery console and do query for table you can able to retrieve the data or message which you passed from the Pub/Sub console.