Imagine that you have a large set of data with millions of rows and you're faced with the task of extracting information from the data. How do you make sense of a large set of data? The simple answer to this question is that we do so via data visualization, but what is that, exactly? Data visualization is the pictorial representation of data to figure out what is hidden inside the information. In this three-part series, we'll create a visualization app to extract information and make sense of a large set of data. Querying large chunks of data without proper hardware can result in performance issues. Google provides a solution to this problem in the form of BigQuery. Backed by Google's infrastructure, it enables you to query large chunks of data. We'll be using a dataset from Google BigQuery for our visualization application. In this tutorial, I'll take you through the process of creating a visualization application using Python, Google BigQuery, and the D3.js JavaScript library. We'll be using Google App Engine (or GAE) to host our application. This tutorial assumes that you have a basic knowledge of Python and JavaScript. For creating graphs, we'll be using D3.js. Note also that the code from the above tutorial is available on GitHub, and a demo is hosted on AppSpot.
Getting Started With Google App Engine
Step 1: Downloading and Using The GAE Python SDK
Let's get started by setting up our development environment. Download the Google App Engine Python SDK and extract it. We'll be using the webapp2 framework for Python. From Google's documentation:
A webapp2 application has two parts:
1) one or more RequestHandler classes that process requests and build responses
2) a WSGIApplication instance that routes incoming requests to handlers based on the URL.
The aforementioned points will become more clear when we create our request handler class. Create a directory called PythonD3jsMashup which will be our project directory. Navigate to the project directory and create a file called app.py. This will be our application module. Let's create a request handler class to process the request and build a response, and a WSGIApplication instance to route requests. Here is how app.py should look:
01
02
03
04
05
06
07
08
09
10
11
importwebapp2
classShowHome(webapp2.RequestHandler):
defget(self):
## Code to render home page
## Here is the WSGI application instance that routes requests
application =webapp2.WSGIApplication([
('/', ShowHome),
], debug=True)
When a / request occurs, it's routed to the ShowHome class which renders the home page. Let's write the Python code to display our homepage. First, create a new folder called Templates in the PythonD3jsMashup folder. Inside it, create an HTML page called index.html. It should contain the following code:
Next, navigate to the project directory PythonD3jsMashup and create a file called app.yaml. This file will act as a configuration file for our application. It should look like this:
1
2
3
4
5
6
7
8
9
application: appid
version: 1
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /.*
script: app.application
As you can see, we have defined a number of things in the app.yaml. Most of the parameters are self-explanatory. The handlers directive dictates that every URL request that matches /.* be handled by the application object in the app module. To run the application locally, navigate to the GAE SDK and run the following command:
Next, we'll be deploying our app onto the Google App Engine. In order to deploy the app, first log in to the GAE console: From the dashboard click on the Create Application button. Next, you'll see a screen similar to the one shown below. Type an Application Identifier and Application Title for the app and click Create Application. Once the application is registered successfully you'll be greeted with a success message: Make a note of the red circled text which will be used as application identifier later. Click on the dashboard link. From the left menu in the Dashboard, under the Administration tab, click on the Application Settings. You'll see the following screen: Make a note of the Service Account Name. We'll need this in the Google developer console. Next, modify the application name to pythond3jsmashup in app.yaml.
1
2
3
4
5
6
7
8
9
application: pythond3jsmashup
version: 1
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /.*
script: app.application
Now to deploy the app to GAE, navigate to GAE Python SDK and type the following command:
To get started with Google BigQuery, log in to the Google API console. You should see a dashboard listing all the available projects in your Google account. Click on the project name PythonD3jsMashup. Make a note of the Project Number, which we'll use while connecting with Google BigQuery. From the left side menu, click on the APIs & auth tab. From the listed items click on APIs. Now you should see a list of available APIs: BigQuery API is turned OFF. Click on the OFF status to turn it ON. Click on the project name pythond3jsmashup on the left side menu and click Permissions. Click Add Member and add the Service Account Name (which we noted down earlier) with "Can edit" permissions as shown below:
Collecting Data From Google BigQuery
Until now, we created our simple app and deployed it on GAE with BigQuery API enabled. Next, we'll be connecting to one of the freely available datasets on BigQuery. Let's start by creating a new page called Templates/chart.html where we'll show the visualization. Create a method to route the Click Here link on our home page to chart.html. Here is what it looks like:
Now try running the app and on clicking Click here, and you should be redirected to chart.html. Next, we'll try to authenticate our app with Google BigQuery and fetch some data. For authentication, we'll require Google API Python client. Downloadgoogle-api-python-client-gae-1.2.zip and extract it into the project folder. Include the following libraries in app.py.
Second, we need to create a query that we'll be executing against the BigQuery dataset. We'll be using the Shakespeare dataset. Here is the query that we'll be executing against the dataset:
1
queryData ={'query':'SELECT word FROM [publicdata:samples.shakespeare] LIMIT 1000'}
To query the data we'll create a job from the bigquery_service:
The response from the query would be printed out on chart.html. Save all your code and upload the modified code to GAE using appcfg.py update PythonD3jsMashup/. Now if you point your browser to http://pythond3jsmashup.appspot.com/chart you'll see the json response returned from querying the dataset.
Conclusion
In this tutorial, we created a Python application and deployed it on Google App Engine. We connected our application with Google BigQuery and fetched data from the freely available dataset. Next, we'll try to parse the data fetched from Google BigQuery and visualize it using JavaScript library D3.js. In the meantime, please don't hesitate to leave any questions or comments in the feed below.
As the music begins to play, a soft breeze fills the space with the undeniable smell of magnolia blossoms. She gracefully enters and begins a “tree pose” with her hands in Namaste form. Our female figure is inspired by the Indian culture for its gorgeous use of color and the grace of its yoga formations. I hope you’ll enjoy the journey as we retrace, step by step to enlightenment. 1. Becoming Flesh and Blood Step 1 Our first step will be to bring in our sketch. I've scanned her in at 300 dpi to catch as much of the detail as possible. I'm not as picky about my choice of initial drawing tools, and this particular project was completed with a mechanical pencil. We'll be working with a clean line quality from Adobe Photoshop and now we can begin to work. I'm working in Adobe Illustrator at a size of 8.5 x 11 inches for this specific project. When I'm ready to start my layers, I'll label my first layer as "Sketch" and lock the layer to make su...
In this tutorial, we are going to design a set of icons in Adobe Photoshop. An icon set needs to have the same background and theme. For learning purposes, we are going to design a sun icon, an ice flake icon, and an RSS icon. Let's get started. 1. Preparing the Canvas Step 1 Start by making a new file with size 350 px × 350 px . Click the small box next to the Background Contents option to change the new canvas color. Step 2 In the Color Picker dialog box, select grey ( #e0e0e2 ) for the canvas background color. Step 3 It is always a good idea to keep your work structured from the start. Make a new layer group and name it sun . This is where we will place all layers used in the sun icon. 2. Designing the Icon Base Step 1 Use the Rounded Rectangle Tool to draw a rectangle with size 83 px × 64 px and 8 px radius. To get accurate results, use the Properties panel. In this panel, you can simply enter the exact siz...
This tutorial will not only show you how to launch the contact picker and get results, but also how to use those results in Android SDK 2.0 and above. Getting Started This tutorial will start out simple, but then we’ll get in to some of the technical details of using Contacts with the ContactsContract class, which was introduced in API Level 5. Make sure you have your Android development environment installed and configured correctly. You’re free to build upon any application you have, start a new one from scratch, or follow along using the InviteActivity code in our open source project . We’ll then be adding functionality to allow a user to choose one of their existing contacts and send a canned message to them. We’re going to dive right in, so have all of your code and tools ready. Finally, make sure your device or emulator has some contacts configured (with names and emails) within the Contacts application. Step 1: Creating Your Layout There are two essential form control...
Comments
Post a Comment