Automate Image Labeling with AI-Powered Firebase Extensions

Introduction

In a world increasingly driven by visual content, the ability to automatically analyze and label images is more crucial than ever. The “Label Images with Cloud Vision AI“ Firebase extension offers a robust solution, automating the task of extracting labels from images using Google Cloud’s Vision API. This extension simplifies categorizing and understanding content within jpg or png images uploaded to Firebase Cloud Storage, making it an essential tool for developers looking to enhance their applications with advanced image processing capabilities.

Google Cloud Vision

Google Cloud Vision API allows developers to integrate vision detection features within applications, including image labeling, face and landmark detection, and Optical Character Recognition (OCR) for text extraction. For more details, refer to the Google Cloud Vision documentation.

Getting Started with the Extension

Before you begin, you’ll need a Firebase project with Cloud Firestore and Cloud Storage configured. Follow these steps to set up:

1. Create a Firebase project
2. Set up Cloud Storage within your project

Once your project is ready, add the Extract Image Text extension.

Extension Installation

You can install the “Extract Image Text with Cloud Vision AI“ extension via the Firebase Console or Firebase CLI.

Option 1: Firebase Console

Visit the Firebase Extension’s page and locate the “Extract Image Text with Cloud Vision AI” extension. Click “Install in Firebase Console” and set up the configuration parameters.

Option 2: Firebase CLI

Alternatively, install the extension using the Firebase CLI with the command:

firebase ext:install googlecloud/storage-label-images --project=projectId_or_alias

Configure Parameters

Properly configuring the “Label Images with Cloud Vision AI” extension is essential for effective functioning. Below is a detailed overview of each configuration parameter:

1. Cloud Functions Location: Description: Choose a location closest to your Firestore database to minimize latency.Example:
   • If your Firestore database is in Europe, you might choose europe-west1 as your Cloud Functions location.
   • For a database in the US, us-central1 would be a suitable choice.
   • Refer to the location selection guide.
2. Cloud Storage Bucket for Images:
   • Description: Specify the Cloud Storage bucket to which you will upload images for labeling.
   • Example: devrel-extensions-testing.appspot.com
3. Paths that Contain Images You Want to Label:
   • Description: Restrict the labeling to specific locations in your Storage bucket by providing a comma-separated list of absolute paths. Wildcard notations for directories in the path can also be used.
   • Example: /users/avatar,/users/photos
     The following photograph is an example of our storage bucket and its paths that we want to label. Please see the following photograph as an example.

1. List of Absolute Paths Not Included for Labeled Images:
   • Description: Specify paths to exclude from labeling in your Storage bucket. This can be a comma-separated list of absolute paths supporting wildcard notations.
   • Example: /users/avatar/thumbs,/users/photos/thumbs
2. Collection Path:
   • Description: Define the path to the Firestore collection where the labels will be written.
   • Example: imageLabels
3. Amount of Label Information to Write to Firestore:
   • Select “basic” if you need just the extracted text for simplicity and storage efficiency.

• Choose “full” if you require detailed annotation data, such as bounding box coordinates and language hints, which can be helpful for advanced use cases.

How It Works

Once installed and configured, the extension automatically labels images:

• Upload Trigger: When a jpg or png image is uploaded to the specified Cloud Storage bucket, it triggers the extension.
• Label Extraction: The Cloud Function activates the Cloud Vision API to analyze the image and extract labels.
• Storing Results: The extracted labels are stored in a Firestore document, with a field indicating the image’s file path in Cloud Storage.
• Accessing Data: To access the labeled data, perform a Firestore query using the file path, allowing you to retrieve and utilize the labels in your application.

Steps to Label Images

Before implementing the “Label Images with Cloud Vision AI” extension in your application, ensure that the Firebase SDK is initialized and you can access Firebase Cloud Storage and Firestore. Here’s a basic setup for your application:

Initial Setup

import { initializeApp } from 'firebase/app';
import { getStorage, getDownloadURL, ref, uploadBytes } from 'firebase/storage';
import {
  getFirestore,
  collection,
  query,
  where,
  getDocs,
} from 'firebase/firestore';

const firebaseConfig = {
  // Your Firebase configuration keys
};

const app = initializeApp(firebaseConfig);
const storageRef = getStorage(app);
const firestoreRef = getFirestore(app);

UI for File Selection

Include an HTML input element for file selection in your application.

<!-- HTML input element for file selection -->
<input type="file" id="image-upload" accept="image/*" />

Add JavaScript to handle the file upload.

document
  .getElementById('image-upload')
  .addEventListener('change', handleFileUpload);

let file;

function handleFileUpload(event) {
  file = event.target.files[0];
}

Step 1: Upload the Image File to Firebase Cloud Storage

Upload an image file to the specific path in Firebase Cloud Storage that the extension configures to monitor.

const imageRef = ref(storage, `/users/photos/${file.name}`);

let uploadedFullPath;

uploadBytes(imageRef, file).then((snapshot) => {
  console.log('Uploaded an image file!');
  uploadedFullPath = `gs://${snapshot.ref.bucket}/${snapshot.ref.fullPath}`;
});

Step 2: Monitor the Labeling Process and Access the Output

The extension processes the image file and writes the labels to Firestore. To access these labels, use a query to find the correct document in Firestore.

const labelsQuery = query(
  collection(firestore, 'imageLabels'),

  where('file', '==', uploadedFullPath)
);

getDocs(labelsQuery).then((querySnapshot) => {
  querySnapshot.forEach((doc) => {
    console.log('Labeled data:', doc.data());

    // Process or display the labeled data as needed
  });
});

Real-Time Updates and Display

To display text as soon as it’s extracted, set up a real-time listener on the Firestore document:

const docRef = collection(firestoreRef, 'imageLabels');

const labelQuery = query(docRef, where('file', '==', uploadedFullPath));

onSnapshot(labelQuery, (querySnapshot) => {
  querySnapshot.forEach((doc) => {
    if (doc.exists()) {
      console.log('Updated text:', doc.data().labels);

      // Update UI accordingly
    }
  });
});

The results of the labeling extension

Conclusion

The “Label Images with Cloud Vision AI” Firebase extension provides a streamlined and robust solution for automatically extracting labels from images, enhancing the capabilities of applications with advanced image analysis.

Its ease of integration and real-time data handling make it an invaluable tool for developers looking to enrich their applications with efficient image labeling and analysis. This extension simplifies complex processes and opens up new possibilities for interactive and dynamic content management in the digital space.