December 11, 2014

Debugging Robots

cover

I was recently helping Sara debug an issue with a project for an upcoming book she's writing a chapter in. She was using Johnny-Five and a Spark Core to remotely control a boat with JavaScript.

The Setup

The setup was a sparkfun motor driver and a standard servo hooked to A0 and A1 of the Spark Core.

sparkcore pinout

Sara had one submersible thruster on the boat hooked up to the motor driver and a rudder hooked up to the servo. She was using VoodooSpark with Johnny-Five to remotely control the two devices.

The problem was whenever you turned on the motor the servo would steer all the way to the right and stop responding. Instant Crash! Both components worked fine in isolation. Moving the pins around didn't seem to help. We thought we got it working on a different spark core but if we did it was only for a few minutes and did not work when we installed it in the boat. What on earth was going on?

Since swapping out the components had no effect we were able to reasonably claim the hardware was in working order. Our next thought was maybe we wired it up wrong. We could rule that out by testing the output from the spark core itself. If the output was as expected then the wiring was probably the issue. If the output was bad then we could keep looking up the stack.

Testing PWM output

Both the motor driver and the servo use PWM for their communication. The motor driver uses it to control the speed of the motor, and the servo uses it to determine it's position. In theory, moving the servo should have no effect on the motor and driving the motor should have no effect on the servo.

I unhooked everything from the spark core and took an Arduino I had laying around and programmed it to be a simple pulse timer. It would count how many milliseconds a pulse was and would write it out on the serial port. I don't think it was very accurate but it was good enough to see if the signal on the pins were what we expected. I hooked the Arduino up to the sparkcore's servo pin.

unsigned long duration;

void setup() {
  pinMode(7, INPUT);
  Serial.begin(9600);
}

void loop() {
  duration = pulseIn(pin, HIGH);
  Serial.print("Channel 1: ");
  Serial.println(duration);
}

I used the serial monitor from node-serialport but the one in the Arduino IDE works great too.

I measured the servo's pin and turned on the motor and got this.

Channel 1:1465
Channel 1:1465
Channel 1:1464
// PWM Pin on
Channel 1:61
Channel 1:61
Channel 1:61

Well that's not right. It shouldn't have changed at all. I swapped pins and found the same issue. I tried two servos without a problem!? Tried two motors and that also worked! We only got the problem with a servo and a motor. In any case the problem wasn't in the wiring.

Removing layers of the stack

The full stack looks something like this:

  • Parts and Devices
  • Wiring
  • Spark Core Hardware
  • Spark Core Software running VoodooSpark
  • UDP over Wifi
  • spark-io node library
  • johnny-five node library
  • Sara's boat code

We already ruled out the parts and devices and the wiring. Debugging the Spark Core Hardware seemed daunting, so we decided that it would be easiest to rule out the upper layers of the stack.

After a little bit of hacking we got rid of the boat code and johnny-five and were left with a spark-io example.

var Spark = require('spark-io');
var board = new Spark({ /* connection info /*});
board.on('ready', function(){
  var pwmPin = "A0";
  var servoPin = "A1";

  this.pinMode(servoPin, this.MODES.SERVO);
  this.pinMode(pwmPin, this.MODES.PWM);
  this.servoWrite(servoPin, 90);

  setTimeout(function(){
    console.log('pwm on');
    this.analogWrite(pwmPin, 200);
  }.bind(this), 5000);
});

We booted everythign up and had the same output. =(

We were left with this:

  • Spark Core Hardware
  • Spark Core Software running VoodooSpark
  • UDP over Wifi
  • spark-io node library

I'm going to gloss over the details of exploring the VoodooSpark firmware and listening to serial debug output and monitoring packets on the wifi. I talk about anything noteworthy in the pull requests I opened as we explored this issue. I was convinced that voodoospark was either getting bad data or interpreting it wrong. Neither was the case. I spent about two nights learning a lot but not finding the bug.

We finally decided to write a small Arduino C program to replace VoodooSpark. If it showed the bad output then we'd know the Spark Core Hardware was to blame!

Servo s;

void setup() {
  Serial.begin(9600);
  s.attach(A1);
  pinMode(A0, OUTPUT);
}

void loop() {
  Serial.println("A1 Servo to 90");
  s.write(90);
  delay(5000);
  Serial.println("A0 PWM to 200");
  analogWrite(A0, 200);
  delay(5000);
}

I loaded this program on the spark core and booted up the Arduino. And low and behold, the bug!

The only layer of the stack left!

  • Spark Core Hardware

Spark Core Hardware Internals

At this point we were lost. How do you know what's going on in there? I've heard of JTAG which is way of attaching a debugger to the code on the chip, but I lacked the hardware and knowhow to make that work.

Luckily for us Zach Supalla was watching the issue. He noticed that A0 and A1 are on the same "timing peripheral". He was able to share a diagrams and charts of what's in the spark core's internals.

Even though both the motor controller and the servo use PWM, they use it at different frequencies. The component behind both the pins could produce independent PWM duty cycles (eg. control two devices) but only if they were at the same frequency!

THATS THE CAUSE OF THE BUG!

And the fix? Move the servo to A4 which has a different component behind it. Once we did that everything worked as expected! We left the issue open so we could figure out how to communicate the error via the software in the future.

The final wiring looked like this.
Final and working wiring

You can follow the saga in pull requests;
https://github.com/rwaldron/spark-io/issues/39
https://github.com/voodootikigod/voodoospark/issues/32

And look for the book! Despite the issues we had working with Johnny-Five and the Spark Core is a lot of fun! The authors figured out all this mishegas so you don't have to. ;-) It's called Make: JavaScript Robotics: Building NodeBots with Raspberry Pi, Arduino, and BeagleBone and comes out in March 2015.

The photos are from this years RobotsConf and shot by Matthew Bergman. The firebot and PaddleBoatBot (below) which was built by a 10 year old.

paddleboatbot

Keep hacking!

-Francis

Roborooter.com © 2024
Powered by ⚡️ and 🤖.