This project will demonstrate how to use the Pixetto camera’s lane detection algorithm to control a toy car to follow a path. More specifically, this guide will provide a simple Arduino program that takes the output from the camera and uses this information to control a car to stay within the two lines in a lane.

To follow along, you’ll need the following:

  • VIA Pixetto Vision Sensor

  • Toy car with Arduino board

  • Pixetto Utility

This guide will first walk you through the camera setup. Then it will explain how to create a basic car controller. Finally it will provide some guidance regarding fine tuning the controller for your own car.

Setting up the Camera

First, connect your Pixetto camera to a computer with the Pixetto Utility installed. Open the utility and scroll to the bottom of the function selection drop-down on the right. From there, select Lanes Detection as the algorithm to run.

You’ll want to set up your Pixetto at a proper height and angle for detection. For optimal detection, it is recommended to set the camera such that the lanes are visible in the bottom 1/2 of the screen. Below are some examples:

Creating a Controller

The full Arduino code is provided below.


float Kp = 1.0;
float Kd = 0.0;

SmartSensor ss(0,1);
Servo steering_servo;
int x, err, prev_err, cmd;

void setup(){
	x = 0;
	err = 0;
	prev_err = 0;
	cmd = 0;
	pinMode(13, OUTPUT);
	pinMode(3, OUTPUT);
	pinMode(5, OUTPUT);

void loop(){
	if (ss.isDetected()) {
		digitalWrite(13, HIGH);
		analogWrite(3, 45);
		analogWrite(5, 45);
		x = ss.getPosX();
		err = x - 50;
		int d_err = err-prev_err;
		cmd = (int)(90 + Kp * err + Kd * d_err);
		prev_err = err;

We use a servo to control the angle of the two front wheels relative to the car. Its input is an angle between 0 and 180 where 0 is on the left and 180 is on the right. Values to either extreme are not recommended since your servo may burn out from trying to turn to an impossible angle; so you may want to clip the input before turning.

Our rear wheels are each connected to a motor. These motors are controlled through pins 3 and 5. Their input is a percentage of its maximum torque output. We set these to a constant value for now.

Understanding the Code

The setup() function simply sets some default values to the variables and assign pins. This will vary with your system, so don’t worry too much about this. What’s important is that prev_err is initialized to 0.

The controller for the steering angle is within loop(), reproduced below:

Copy to Clipboard

When the Pixetto camera detects a lane, it will calculate a goal point in the center of the two lines. This point is normalized between 0 and 100, representing a percentage of the width going from left to right, and can be obtained by calling getPosX().

Since the goal is to stay in between the two lines, we ideally want the center of the car to match with the center of the lines. Assuming the camera is mounted in the center of your car, this implies that we want the goal point to equal 50. Thus, we know that the error err between our current state and the desired state will be the difference between x and 50. The goal can now be formalized to minimizing the error.

Let’s skip the d_err variable for now and look at the command output cmd. The command output will be an angle for the front wheels to turn to. We would like our controller to have the following property:

The greater the deviation from the center, the more the car turns.

In other words, we want the car to turn proportionally to the error. Since we want err to be negative when we want it to turn left and vice versa, we let err = x – 50. We can simply add the error to the offsetted command and achieve our objective; but we may want to control how much we want the wheels to turn per angle of error. This is done by multiplying err by a constant Kp.

Often times this is not enough and can lead to two issues:

When the car reaches its goal, it may be facing the center of the lane but its orientation may not be parallel with the lines. This will lead to the car oscillating along the center of the path.
When the road is curved, the goal point is constantly moving. If the turn is too sharp, the car could turn too slowly and exit the lane.

Problem 1 is solved by reducing Kp, yet problem 2 is solved by increasing Kp. When both problems occur at the same time, a more powerful controller is needed. So to tackle this, we introduce a second property:

When the car turns too fast, it turns less; and when it turns too slow, it turns more.

We can judge how fast the car is turning by looking at how much the err is changing, or the difference in the error between each step. We set this difference in error d_err = err – prev_err, where prev_err is the previous error saved at the end of each step.

When the car is turning too fast, d_err will be the opposite sign of err. When the car is turning too slowly, d_err will be the same sign as err. If we add d_err to cmd, the car will turn less when it turns too fast and vice versa. Finally we multiply d_err with a constant Kd to control how much we want to modify how much the car is turning.

This results in a final output command of cmd = (int)(90 + Kp * err + Kd * d_err). This is known as a PD controller, which you can read more about here

Tuning the Parameters

There are primarily two parameters to tune for this controller. They are set at the top of the file here:

float Kp = 1.0;
float Kd = 0.0;

  1. Kp: Determines how fast the car will turn in relation to its deviation from the center. If the car turns too little, increase Kp. If the car turns too fast, decrease Kp.
  2. Kd: Determines how much the car turns in relation to how much the error is changing. When going along a straight path, Kd controls how resistant the wheel is to turning. If the car is oscillating, increase Kd. If Kd is too great, the car will turn too slowly.

Going Beyond

An alternative to using getPosX() is to call getLanePoints(lx1, ly1, lx2, ly2, rx1, ry1, rx2, ry2). This returns the endpoints of the left and right lines detected by the camera, sorted from left to right. Using this, you can calculate your own goal point or even make a completely different controller.