In this tutorial, we will go through the process of running a custom neural network on a Pixetto sensor.  By the end of this tutorial, you will learn:

  • how to set up and train a neural network with Keras
  • how to convert your trained model into the TensorFlow Lite format
  • how to load a TensorFlow Lite model onto a Pixetto sensor

If you already have your neural network model trained and ready to convert, feel free to skip directly to Section 2 (“Converting model to .tflite”). If not, don’t worry! We will quickly walk through how to set up and train your custom neural network with Keras, using MNIST digit recognition as an example. Let us begin!

Training a Keras model

One of the easiest ways to set up and train a neural network model is with Keras, a deep learning library that uses TensorFlow as a backend.

A simple Convolutional Neural Network (CNN) model for digit recognition — a type of image classification — can be created as follows:

# Importing the required Keras modules
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Conv2D, Dropout, Flatten, MaxPooling2D, BatchNormalization

# Creating a Sequential model
model = Sequential()

model.add(Conv2D(32, kernel_size=3, activation='relu', input_shape = (28, 28, 3)))
model.add(BatchNormalization())
model.add(Conv2D(32, kernel_size=5, strides=2, padding='same', activation='relu'))
model.add(BatchNormalization())
model.add(Dropout(0.4))

model.add(Conv2D(64, kernel_size=3, activation='relu'))
model.add(BatchNormalization())
model.add(Conv2D(64, kernel_size=5, strides=2, padding='same', activation='relu'))
model.add(BatchNormalization())
model.add(Dropout(0.4))

model.add(Conv2D(128, kernel_size=4, activation='relu'))
model.add(BatchNormalization())
model.add(Flatten())
model.add(Dropout(0.4))
model.add(Dense(10, activation='softmax'))

Next, we will import our training data from the MNIST handwritten digit database, and pre-process it for training:

import tensorflow as tf

(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()

# Reshaping the image matrices
x_train = x_train.reshape(x_train.shape[0], 28, 28, 1)
x_test = x_test.reshape(x_test.shape[0], 28, 28, 1)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')

# Normalizing the RGB values
x_train = x_train / 255
x_test = x_test / 255

Now with both our neural network defined and our training data ready, we can compile our model and begin training.

# Compiling and training model
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.fit(x=x_train,y=y_train, epochs=10)

After ten epochs, training will complete. You now have a basic Keras neural network model that can perform digit recognition!

Converting model to .tflite

To run your neural network on a Pixetto sensor, you will first need to convert it into a TensorFlow Lite FlatBuffer (.tflite) file. The TensorFlow Python API provides a TFLiteConverter class that can convert a number of model formats into .tflite files.

In total, TFLiteConverter has three class methods, each for converting from a different format:

  • from_keras_model() converts from Keras models
  • from_saved_model() converts from SavedModel directories
  • from_concrete_functions() converts from concrete functions
# Converting the model
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
 
# Saving the TF Lite model
with tf.io.gfile.GFile(‘model.tflite’, ‘wb’) as f:
    f.write(tflite_model)

The sample code above uses the from_keras_model() method to convert our Keras model into a TFLite model.  Then, the TFLite model is written to the ‘model.tflite’ file, which we can upload to a Pixetto. (See the TFLiteConverter Python API documentation here for example usage of the other methods.)

Before converting your model, please keep in mind the following:

  • Make sure that your neural network model takes input in the shape of (1, h, w, c) where h and w are the height and width of your input image and c is the number of channels (e.g. 1 for grayscale and 3 for colored images).
  • If you are converting from a Keras model, make sure that you are importing tensorflow.keras and not keras.  Otherwise, you may receive a ‘Sequential’ object has no attribute ‘_get_save_spec’ error upon conversion.

If you are training your neural network in an online environment like Google Colaboratory, download the .tflite model to your local machine after the conversion completes.

Uploading model to Pixetto

With your .tflite file generated, we will now upload it onto a Pixetto.

Open up the Pixetto Utility application and make sure your Pixetto is connected to your computer.  Under the “Install Neural Network Model” panel, click on the “Model Path” field.  Upload your .tflite file by navigating to it, then press “OK.”

After uploading completes, you will see that the “Function” under the “Configuration” panel has been set to “Neural Network.”  This means that your Pixetto is now configured to run your .tflite neural network model!

Under the same panel, there are some additional configuration options that you can change, depending on the functionalities you want to implement.

  • Object detection algorithm is the algorithm used to detect objects and extract regions of interest before passing them on as inputs to your neural network model.  (For example, “Face Detection” looks for faces as input images, so use this if you are running a facial recognition or classification model.)
  • Maximum display objects is the maximum number of detected objects that are processed and displayed.
  • Minimum confidence of neural network is the confidence needed from your neural network output for the Pixetto to display the classification results.

This is an example of my custom handwritten digit recognition model running on a Pixetto. Notice that the default Edge Detection algorithm is used to localize the digit (found in the green box), while I increased the minimum confidence to 70%.