Imagine that you have a large set of data with millions of rows and you're faced with the task of extracting information from the data. How do you make sense of a large set of data? The simple answer to this question is that we do so via data visualization, but what is that, exactly? Data visualization is the pictorial representation of data to figure out what is hidden inside the information. In this three-part series, we'll create a visualization app to extract information and make sense of a large set of data. Querying large chunks of data without proper hardware can result in performance issues. Google provides a solution to this problem in the form of BigQuery. Backed by Google's infrastructure, it enables you to query large chunks of data. We'll be using a dataset from Google BigQuery for our visualization application. In this tutorial, I'll take you through the process of creating a visualization application using Python, Google BigQuery, and the D3.js JavaScript library. We'll be using Google App Engine (or GAE) to host our application. This tutorial assumes that you have a basic knowledge of Python and JavaScript. For creating graphs, we'll be using D3.js. Note also that the code from the above tutorial is available on GitHub, and a demo is hosted on AppSpot.
Getting Started With Google App Engine
Step 1: Downloading and Using The GAE Python SDK
Let's get started by setting up our development environment. Download the Google App Engine Python SDK and extract it. We'll be using the webapp2 framework for Python. From Google's documentation:
A webapp2 application has two parts:
1) one or more RequestHandler classes that process requests and build responses
2) a WSGIApplication instance that routes incoming requests to handlers based on the URL.
The aforementioned points will become more clear when we create our request handler class. Create a directory called PythonD3jsMashup which will be our project directory. Navigate to the project directory and create a file called app.py. This will be our application module. Let's create a request handler class to process the request and build a response, and a WSGIApplication instance to route requests. Here is how app.py should look:
01
02
03
04
05
06
07
08
09
10
11
importwebapp2
classShowHome(webapp2.RequestHandler):
defget(self):
## Code to render home page
## Here is the WSGI application instance that routes requests
application =webapp2.WSGIApplication([
('/', ShowHome),
], debug=True)
When a / request occurs, it's routed to the ShowHome class which renders the home page. Let's write the Python code to display our homepage. First, create a new folder called Templates in the PythonD3jsMashup folder. Inside it, create an HTML page called index.html. It should contain the following code:
Next, navigate to the project directory PythonD3jsMashup and create a file called app.yaml. This file will act as a configuration file for our application. It should look like this:
1
2
3
4
5
6
7
8
9
application: appid
version: 1
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /.*
script: app.application
As you can see, we have defined a number of things in the app.yaml. Most of the parameters are self-explanatory. The handlers directive dictates that every URL request that matches /.* be handled by the application object in the app module. To run the application locally, navigate to the GAE SDK and run the following command:
Next, we'll be deploying our app onto the Google App Engine. In order to deploy the app, first log in to the GAE console:
From the dashboard click on the Create Application button. Next, you'll see a screen similar to the one shown below. Type an Application Identifier and Application Title for the app and click Create Application.
Once the application is registered successfully you'll be greeted with a success message:
Make a note of the red circled text which will be used as application identifier later. Click on the dashboard link. From the left menu in the Dashboard, under the Administration tab, click on the Application Settings. You'll see the following screen:
Make a note of the Service Account Name. We'll need this in the Google developer console. Next, modify the application name to pythond3jsmashup in app.yaml.
1
2
3
4
5
6
7
8
9
application: pythond3jsmashup
version: 1
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /.*
script: app.application
Now to deploy the app to GAE, navigate to GAE Python SDK and type the following command:
To get started with Google BigQuery, log in to the Google API console. You should see a dashboard listing all the available projects in your Google account. Click on the project name PythonD3jsMashup. Make a note of the Project Number, which we'll use while connecting with Google BigQuery. From the left side menu, click on the APIs & auth tab. From the listed items click on APIs. Now you should see a list of available APIs:
BigQuery API is turned OFF. Click on the OFF status to turn it ON. Click on the project name pythond3jsmashup on the left side menu and click Permissions. Click Add Member and add the Service Account Name (which we noted down earlier) with "Can edit" permissions as shown below:
Collecting Data From Google BigQuery
Until now, we created our simple app and deployed it on GAE with BigQuery API enabled. Next, we'll be connecting to one of the freely available datasets on BigQuery. Let's start by creating a new page called Templates/chart.html where we'll show the visualization. Create a method to route the Click Here link on our home page to chart.html. Here is what it looks like:
Now try running the app and on clicking Click here, and you should be redirected to chart.html. Next, we'll try to authenticate our app with Google BigQuery and fetch some data. For authentication, we'll require Google API Python client. Downloadgoogle-api-python-client-gae-1.2.zip and extract it into the project folder. Include the following libraries in app.py.
Second, we need to create a query that we'll be executing against the BigQuery dataset. We'll be using the Shakespeare dataset. Here is the query that we'll be executing against the dataset:
1
queryData ={'query':'SELECT word FROM [publicdata:samples.shakespeare] LIMIT 1000'}
To query the data we'll create a job from the bigquery_service:
The response from the query would be printed out on chart.html. Save all your code and upload the modified code to GAE using appcfg.py update PythonD3jsMashup/. Now if you point your browser to http://pythond3jsmashup.appspot.com/chart you'll see the json response returned from querying the dataset.
Conclusion
In this tutorial, we created a Python application and deployed it on Google App Engine. We connected our application with Google BigQuery and fetched data from the freely available dataset. Next, we'll try to parse the data fetched from Google BigQuery and visualize it using JavaScript library D3.js. In the meantime, please don't hesitate to leave any questions or comments in the feed below.
IPV6 OSPF Configuration on Gns3: This is the simple IPV6 tutorial for configuring IPV6 and OSPF for IPV6 on Gns3. You must enable IPV6 routing on router by using command “ipv6 unicast-routing”, all other command are same like normal IP configuration but just replace IP with IPV6 in all commands. I use the c7200 IOS for this lab. Configure the topology as below: R1 Configuration: R1(config)# ipv6 unicast-routing R1(config)# ipv6 router ospf 1 R1(config-router)# router-id 1.1.1.1 R1(config-router)# exit R1(config-if)# interface Loopback1 R1(config-if)# ipv6 address 2001:211:10:1::1/64 R1(config-if)# ipv6 ospf 1 area 0 R1(config-if)# interface Serial1/0 R1(config-if)# ipv6 address 2001:210:10:1::2/64 R1(config-if)# ipv6 ospf 1 area 0 R2 Configuration: R2(config)# ipv6 unicast-routing R2(config)# ipv6 router ospf 1 R2(config-router)# router-i...
I'm going to discuss how I like to implement a rope-swinging gameplay mechanic. When I started working on Energy Hook I mostly did everything with my own custom code. I believe it's possible to simply go into Unity and use the Configurable Joint to do some very similar stuff, but at the time it wasn't available. I'm pretty sure this has been the right call, anyway, because it gives me control over everything - and I can share it with you. The fundamentals of doing swinging the way I did - using constraints - are actually quite simple. (Hope you're not disappointed when you see what's under the hood.) It works the same whether you're making a 2D game or 3D game, it's just that the vectors are different, so I'll start off with 2D and then discuss some wrinkles when going to three dimensions. This article also assumes you're using a game engine like Unity that can do a lot of the work for you, like raycasting against geometry and orienting a chara...
In this tutorial, we are going to design a set of icons in Adobe Photoshop. An icon set needs to have the same background and theme. For learning purposes, we are going to design a sun icon, an ice flake icon, and an RSS icon. Let's get started. 1. Preparing the Canvas Step 1 Start by making a new file with size 350 px × 350 px . Click the small box next to the Background Contents option to change the new canvas color. Step 2 In the Color Picker dialog box, select grey ( #e0e0e2 ) for the canvas background color. Step 3 It is always a good idea to keep your work structured from the start. Make a new layer group and name it sun . This is where we will place all layers used in the sun icon. 2. Designing the Icon Base Step 1 Use the Rounded Rectangle Tool to draw a rectangle with size 83 px × 64 px and 8 px radius. To get accurate results, use the Properties panel. In this panel, you can simply enter the exact siz...
Comments
Post a Comment