Need help with aiexperiments-giorgio-cam?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

462 Stars 98 Forks 22 Commits 2 Opened issues


Take a picture to make music with the computer.

Services available


Need anything else?

Contributors list

Giorgio Cam

Take a picture to make music with the computer.


This is an experiment built with machine learning that lets you make music with the computer just by taking a picture. It uses image recognition to label what it sees, then it turns those labels into lyrics of a song.

This is not an official Google product.


Built by Eric Rosenbaum, Yotam Mann, and friends at Google Creative Lab using MaryTTS, Tone.js, and Google Cloud Vision API. Check out more at A.I. Experiments.


The client-side javascript application captures images using WebRTC. When the user hits the shutter button, an image is sent to the server which then returns an array of labels and confidence scores for that image using Cloud Vision. These labels are dropped into a rhyming template to create the next phrase that the computer will speak. To get the audio of that phrase, the client makes another request to the MaryTTS (text to speech) server which returns a wav file of the audio. That audio is then synced to the music using Tone.js.


To build the client-side javascript, first install node and webpack. Then you can install of the dependencies of the project by typing the following in the terminal:

cd static
npm install

Then build all of the files

webpack -p


The back-end uses Google App Engine to serve static content and mediate between the two other back-end services: Google Cloud Vision and MaryTTS.

Google Cloud Vision API

You will need to first enable the API and generate credentials. Under "Key type", use "JSON" and then download the key.json file.

Add your json key to the

section of your
file like so:


Download and install MaryTTS. Then run the MaryTTS Server. Add the IP Address and Port number that MaryTTS is running on to

. The default location is
MARY_TTS_PORT = '59125'

App Engine

To install the dependencies so they can be launched within Google App Engine they will need to be installed into a local folder (documented here).

cd server
mkdir lib
pip install -t lib -r requirements.txt

Then follow instructions on launching your App Engine code.


Racer by Giorgio Moroder


Copyright 2016 Google Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.