by googlecreativelab

googlecreativelab / aiexperiments-giorgio-cam

Take a picture to make music with the computer.

451 Stars 101 Forks Last release: Not found 22 Commits 0 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

Giorgio Cam

Take a picture to make music with the computer.


This is an experiment built with machine learning that lets you make music with the computer just by taking a picture. It uses image recognition to label what it sees, then it turns those labels into lyrics of a song.

This is not an official Google product.


Built by Eric Rosenbaum, Yotam Mann, and friends at Google Creative Lab using MaryTTS, Tone.js, and Google Cloud Vision API. Check out more at A.I. Experiments.


The client-side javascript application captures images using WebRTC. When the user hits the shutter button, an image is sent to the server which then returns an array of labels and confidence scores for that image using Cloud Vision. These labels are dropped into a rhyming template to create the next phrase that the computer will speak. To get the audio of that phrase, the client makes another request to the MaryTTS (text to speech) server which returns a wav file of the audio. That audio is then synced to the music using Tone.js.


To build the client-side javascript, first install node and webpack. Then you can install of the dependencies of the project by typing the following in the terminal:

cd static
npm install

Then build all of the files

webpack -p


The back-end uses Google App Engine to serve static content and mediate between the two other back-end services: Google Cloud Vision and MaryTTS.

Google Cloud Vision API

You will need to first enable the API and generate credentials. Under "Key type", use "JSON" and then download the key.json file.

Add your json key to the

section of your
file like so:


Download and install MaryTTS. Then run the MaryTTS Server. Add the IP Address and Port number that MaryTTS is running on to

. The default location is
MARY_TTS_PORT = '59125'

App Engine

To install the dependencies so they can be launched within Google App Engine they will need to be installed into a local folder (documented here).

cd server
mkdir lib
pip install -t lib -r requirements.txt

Then follow instructions on launching your App Engine code.


Racer by Giorgio Moroder


Copyright 2016 Google Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.