91ý

A STEP-BY-STEP GUIDE TO BUILDING AND DEPLOYING GOOGLE CLOUD FUNCTIONS

WhyUse Google Cloud Functions?

is aFunction as a Service (FaaS) that allows engineers and developers to run code without worrying about server management. Cloud Functions scales as needed and integrates with Google Cloud’s operations suite (such as Cloud Logging) out of the box. Functions are useful when you have a task or series of tasks that need to happen in response to an event. Cloud Functions can be invoked from several events, such as HTTP, Cloud Storage, Cloud Pub/Sub, CloudFirestore, Firebase, and in response to Google Cloud Logging events.With Cloud Functions, monitoring, logging, and debugging are all integrated, and functions scale up and down as required. This is especially useful when you want to focus on writing code butdon’twant to worry about the underlying infrastructure. You can write functions in the Google Cloud Platform console or write them locally and deploy using Google Cloud tooling on your local machine. Note that Cloud Functions has a few limitations, such as a nine-minute execution limit. Cloud Functions can be a good choice for short-lived requests that do one specific task in response to another event. Inthisstep-by-step guide,we’llwalk through an example scenario that uses Google Cloud Functions, Google Cloud Storage, Google Cloud Vision API, and GoogleBigQuery.The purpose of the function will be to pass an image to theVisionʱ/AI and obtain information about it for later analysis.We’llset up a cloud function in Python that listensfor a new upload event to a specific Google Cloud Storage bucket.Next, the script willtake that image and pass it to the Google Cloud Vision API, capture theresults,and appendthemto a table inBigQueryfor further analysis.This example will use sample imagesbutcanbe modified to read documents containing text.

Google Cloud PlatformProject Setup

To begin,youwillneed a Google Cloud Platform project with a few APIs enabled. If you are not familiar with setting up a Google Cloud Platform project,to learn how to do so.During this tutorial, we will be usingto deploy our functions andto write the script.You could also useif youprefer.Be sure your environment is set up to develop scripts on the Google Cloud SDK with Python using the linksabove.

Enable APIs

Onceyour project is set up,you’llneed to enable a few APIs, specifically theCloud Functions API, theCloud Vision API, and theBigQueryʱ.To enable those APIs within your project:
  1. 𳦳ٳhamburger menufrom theupper left-hand corner of theGoogle Cloud Platform console.
  2. Navigate toAPIs & Services.
  3. 𳦳.(image 1 below)
  4. On the Library page, search for each API mentioned above and enable them with the bluemanagebutton.You will see anAPI Enabledlabelafter it has been enabled successfully.(image 2)
  5. Ensure that theCloud Functions API, theCloud Vision API, and theBigQueryʱare all enabled.
Image 1:

Image 2:

Set Up Cloud Storage

Onceyour Cloud Platform project isready, create a Google Cloud Storage bucket. If you are not familiar with Cloud Storage buckets, they are uniquely namedareasthat provide fine-grained access toobjects. They can be used for data storage, landing zones for user upload, or archivesfor data that need to be kept forextended periods. You can find more information aboutGoogleCloud Storage. To create a bucket:
  1. Select thehamburger menufrom the upper left-hand corner of the Google Cloud Platform console.
  2. 𳦳Ƿɲ.(image 3)
  3. Select the blue CREATE BUCKETlink at the top.
  4. Follow the creation wizard, keeping note of the bucket name you select.(image 4)
Image 3:

Image 4:

Create aBigQueryTable

Next,you’llneed tocreate atable tocontain metadata about the images that are extracted by the Cloud Vision API.BigQueryis Google’s data warehouse designedfor analyzing petabytes worth of data usingSQL and is often used for historical analysis, time series data, and other scenarios where SQL-like datacanbe retained for a long timewith fast retrieval. Our tablewon’tquite hold petabytes of data, butyou’ll see how easy it is to write to a table using Cloud Functions shortly. OurBigQuerytable could be connected to a data visualization tool (such as Google’s Data Studio) for further analysis. In this example, we will keep it simple by capturing filename, URI, andgenerated labels and landmarks as well asthe confidence that Cloud Vision has in the output.
  1. Ensure you have a project selected in the GCPConsole.
  2. Select thehamburger menufrom the upper left-hand corner of the Google Cloud Platform console.
  3. 𳦳BigQuery.(image 5)
  4. 𳦳CREATE DATASETfrom the left-hand side.(image 6)
  5. Give your dataset a name and leave all other values atdefault
  6. Press the blueCreate Datasetbutton when you are finished(image 6)
Image 5:

Image 6:

It isnow timeto create a table and set up a schema. The schemadefines what type of data we will be placing into our table.Inthiscase, we will have two STRING fields and two REPEATING fields to hold information about our image metadata.
  1. From the Dataset page, press the blueCREATE TABLEbutton.(image 7)
  2. Give the table a name, and make sure that theProject NameandDataset Namematch your current project anddataset.
  3. Leave the remaining values at theirdefault anddefine the schema as in image 8. Note: When adding properties to a repeated field, you will need to press the (+) buttonto add nested fields.
  4. Write down the dataset name and the table name, as we will need them for our script later.
Image 7:

Image 8:

Generate a Service Account Key

If you wish to develop locally prior to deploying the Cloud Function, we recommend you generate a service account key.
  1. Select thehamburger menufrom the upper left-handcorner of the Google Cloud Platform console.
  2. 𳦳APIs & Services.
  3. 𳦳Credentials.(image 9)
  4. Press the blueCREATE CREDENTIALSbutton andselectServiceaccount.
  5. Give the new service account a name and description then pressCreate.
  6. Set theOwnerrole on the serviceaccount andleave the other fieldsblank.
To finish configuration for your environment, see the documentation on. Image 9:
This completes setup of our Google Cloud Platform environment, and we are now ready to begin scripting!

DeveloptheScript

As discussed previously, this script will listen for new uploads to a Cloud Storage bucket andacton the newly uploaded item. In our example today, we will be using photos of world cities, but you could use other photos (such as a scanned invoice) if you were looking to parse other types of data. We will start by creating a newdirectory on our local machine.
  1. Create a new folder
  2. Add some images for testing. We used photos of cities around the world. (image 10)
Image 10:

Next, create a new file in the same directory called main.py.This is the source code for the function you will be uploading to Google Cloud Functions.For now, go ahead and insert the following code:

We import the Google CloudBigQueryand Google Cloud Vision libraries, which contain functions and properties we can use to interact with those APIs. In thisexample, we are responding to aCloud Storageupload event, so we pass two parametersto the function, which are afilevariable and acontextvariable. Thefilevariable holds a reference tothe object that was uploaded, while thecontextvariable can provide us information about the context in which the object was uploaded. For this scenario,we willbe focusing on thefilevariable. Now we willbegin pulling information from thefilevariable and set up some other variableswe’llneed later on. Modify your script to look like this:
Thebucketandfilenamevariables hold information about the bucket and filename strings, and theimage_urivariableconstructs the Google Cloud Storage URI that we will pass to the Vision API shortly.If you are not familiar with the format function, itinjects strings for the braces it finds within a given string.Do not modify theimage_urivariable for thisexample. On line 8, you will need to modify thetable_idvariable to use your Google Cloud Platform project name, your dataset name, and your table name. Refer to the dataset name and table name you wrote down earlier. If you are not sure where to get those, look in theBigQuerysection in the Google Cloud Platform console for your project.Finally, therowvariableholds information about the image, such as labels and landmarks, that will ultimately be passed toBigQuery. Once those variables are set, it istime tocall the Vision API. Modify your script to look like this (I have edited the image to removethereference to thetable_id):
In the new code, we set up aclient object,vision_client,that we will use to call the Vision API. We called the methodannotate_imageon the client and passed it a reference to the Google Cloud Storage object that triggered the upload event. Becausethis function will be contained in the same project as the storage bucket and theBigQuerytable, wedon’thave to worry about authentication. In a production system with multiple projects or environments, you would need to take additional steps to authenticate this code.Note further that we are not checking for an error response of theannotate_imagecall. In a production environment you would want to wrap this in a try block and gracefully handle or retry any errors as appropriate. We are nearly complete with our script.All we need to do now is collect the label and landmark information (along with the Vision API’s confidence level), add it to ourrowvariable, and then append it to ourBigQuerytable(I have edited the image to remove the reference to thetable_id):
The two items to call out here arethat we are rounding up the score to be on a scale of100, andinserting the row as JSON by using theBigQueryʱinsert_rows_jsonmethod.We have now completed our script, and we are ready to deploy it to Cloud Functions for testing.

Deploy the Script

Before we can deploy,let’smake sure Google Cloud Functions can pull down all the dependencies required for our function to run properly. In your directory, create a newfile calledrequirements.txt.This file should be in the same directory as main.py. Withinrequirements.txt, place the following two lines:
One of the ways to specify dependencies is to use the pip package manager for Python within Google Cloud Functions. When the function is deployed, Google Cloud Functions reads the data inrequirements.txtand pulls down those dependencies. For more information on pip and how Google Cloud Functionsuses it, review the documentation. The script is now ready for deployment. Open a terminalwindow andnavigate tothe directory containingmain.py. From the terminal, enter the following command:
The function will begin deploying and should complete within two or three minutes. If the deployment tool detects an error, it will show on the screen.Ifyoureceivean error,check your codeand remediate any issues you find. If you get dependency errors, make sureyou’veupdatedrequirements.txtas in the instructionsabove and that the file is in the same directory asmain.py. Let’sreview the function deployment in the cloud console before we begin to test it.
  1. 𳦳ٳhamburger menufrom the upper left-hand corner of the Google Cloud Platform console.
  2. ClickCloud Functions. A list of all functions will appear on the screen.(image 11)
  3. Verify that the function has a green check mark to the left of its name, and that the bucket name matches the bucket you set up and specified in your deploy command.
Image 11:

Test Your Function

You are now ready to test your function by uploading an image to the bucket.We’lluse thegsutilcommand to upload one of the images. Within the directory containing your test images, execute this commandin the terminal, whereFILE-NAME.JPGandCLOUD-STORAGE-BUCKET-NAMEare the names of the file you want to upload and the Cloud Storage bucket you created, respectively:
You will see output in the terminal as the file is uploaded and when it completes. If you run into an error, check permissions on your system.Once the file is uploaded, it is time to check ourBigQuerytable for the information that Cloud Vision was able to provide for labels and landmarks.
  1. Select thehamburger menufrom the upper left-hand corner of the Google Cloud Platform console.
  2. ClickBigQuery.
  3. 𳦳SQL workspace.
  4. Select theBigQuerytable you created earlier and press the blueQUERY TABLEbutton.(image 12)
Image 12:

In the screen that appears, press the blueRUNbutton and fill out the query. For testing purposes, we can usethe following command:

Make sure to replace the placeholder values in the query with references to your project name, dataset name, and table name. (image 13). Image 13:
The query will take a few seconds to run. Once completed, we can see information aboutall oftheimages we’ve uploaded to our Cloud Storage bucket. This will trigger every time a new image is uploaded, so give it a try by uploading four or five images and re-running the query. We can see the labeldescription, landmark description, and the confidence that the Vision API has in the label and landmark.

Clean Up

Once you are done with the exercise,make sure toclean up. Delete anyBigQuerytables you created, along with any images you uploaded to Cloud Storage, and push your function to a git repository.If you created a service account for local development, consider deleting it now.

Suggested Functionality

This exercise focused on standing up a function to capture information about images uploaded to Google Cloud Storage. However, there are many other capabilities you could add to your function including:
  • Editthe code to capture all text in animage.
  • Storevalues from the Vision API into a SQL Database for later analysis or retrieval.
  • Attach ourBigQuerytableto a data visualization tool such as DataStudio.
  • Add robust logging and retry logic to our code to make it more production-ready.
Wehope this has demonstrated some of the interesting things you can do with CloudFunctions, andencourage you tobuild your own. Documentation & References
SUBSCRIBE
Subscribe to the 91ý I/O Newsletter for a periodic digest of all things apps, opps, and infrastructure.
This site is protected by reCAPTCHA and the Googleandapply.