Steam Heat

My home has radiators that are 70ish years old. I’m going to guess the air vents are 10-30 years old. It’s supposed to let the air out while the pipe fills with steam. Then and this is the important part, it stops letting anything out of the pipe and keeps the steam in the pipe. Some of our valves weren’t letting the air out, and some was letting everything out.


  1. It’s cold in my room
  2. Our boiler runs out of water too often
  3. Our windows take a constant bath from condensation

So lets fix some things.

This slideshow requires JavaScript.

The vents in the peanut butter jar were old, came with the house. They didn’t quite close when the steam hit them so they whistled a ton. So if you have a PB jar full of vents I recommend installing them at night so nobody is awake to hear them. The next day Sara drove me in the snow to spend a lot of money on valves from a store. I replaced the vents in my bedroom which were clogged. The air couldn’t escape so it couldn’t fill with steam. The one in the bathroom would never close, so while our bathroom was very warm, it also very damp. Notice the rust around the vent in the photo? That’s crazy. The last two had similar problems.

It’s so quiet! I love it! I hope my bedroom is warmer, I’ll find out. I can’t confirm it but I bet the house will heat a little faster, and my hunch is that the air will be much less humid.

I wish I could somehow measure the heat in the house vs outside, the time it takes to warm the house, the humidity, etc etc. Oh wait! I know how!


I’ll use my Nest! Unfortunately they don’t seem to make it easy to use the data. Looks like I may have to get clever.

Money Transferring

I was expressing frustration with moving money around the other day and was asked to share what I knew. This is an incomplete list of ways to move money, and their consequences. Moving money (and in turn getting paid or paying others) is one of the most important yet strangely high-friction things a business has to do. On top of that, most of it is frustrating from a technical standpoint.

I’m not going to cover moving money on behalf of other people, which is worthy of it’s own blog post. I also think it’s one of the more interesting areas for exploration (as far as money is concerned). I’m also going to gloss over specifics around the Automated Clearing House (ACH) and other payment networks. Just know that it’s actually a lot more complicated, with a ton more rules and a few more players then I talk about.


The cover photo is of NYC’s MTA Money Train by “Orange Suede Sofa”.


Cash is the easiest to understand. It’s what we grow up with. It’s a physical object that holds value and it’s instantly convertible either to products or your bank account. The retail currency exchanges take a large cut, but if you’re in a single country it can’t be beat. Because of it’s physical nature there isn’t an accounting system in existence that doesn’t require you to manually track what you spend it on or where it goes.

  • Cost: Low – Usually none unless you have to change currencies
  • Time to clear: Instant
  • Hassle: High – You have to go get it and then physically give it to someone.
  • Risk: High – No fraud protection
  • Traceability: Very little
  • Accounting Cost: High – No way to know who received it or why without keeping your own records


I don’t like checks but they are very common and are almost as cheap as cash. The checks themselves are not actually worth money. I don’t like them because they are unpredictable. If I give you cash, I no longer have it. If I give you a check, I need to ensure I have the money in my account until you deposit the check and it clears. It requires me to keep a separate log of accounting to ensure I don’t accidentally double spend. The Automated Clearing House (ACH) processes is crazy too.

Here’s the general processes:
1. I physically have to write how much money I want you to have on a piece of paper. (Or print it, but that isn’t easier.)
2. I have to physically get it to you. (mail, in person, etc)
3. You have to bring it to a branch of your bank and deposit it. (“Mobile deposit” is great but banks usually only risk around $2000 max on a single mobile deposit.)
4. Your bank scans the check and sends it in a nightly batch processes to the ACH.
5. The ACH does some level of fraud protection that I am unclear of the specifics.
6. The my bank receives a notice of my check and verifies the funds and authorizes the withdrawal.
7. The ACH lets your bank know that it was approved.
8. Our banks reconcile the funds. (Do their own bank to bank transfers.)
9. Sometime in the past few steps your bank updated your account with some or all of the balance.

The worst part for me is that you have around 6 months to deposit the check. Some banks will allow you to try to deposit checks of any age.

  • Cost: Low – Unless there’s a returned check, then fees are levied to both parties around $20-$60
  • Time to clear: Long – 2-4 days to clear plus 0-160 days to deposit
  • Hassle: High – Lots of physical interactions, lots of waiting and often you have to use the US postal system
  • Risk: Medium – Checks aren’t worth money but they can be faked. You can also deposit a “bad check” which can incur you fees.
  • Traceability: High – Every check comes from an account and goes to an account. Accounts usually have names and it shows up as an item in your bank statement.
  • Accounting Cost: Low – It’s easy to see to whom a check was written and the memo field usually says why (eg invoice number). It also usually reconcilable from the bank statements. That is of course they deposited the check.

Wire Transfer

This is pretty easy and can usually be done from an online interface with your bank. In the US it uses the ACH like checks but electronically sends the account information. Internationally it uses IBAN and SWIFT, which are neat. This will usually cost between $10-$40 to both send and receive.

  • Cost: Medium $10-$40 for all parties involved
  • Time to clear: 2-4 days
  • Hassle: Medium – You need address, name, account info, routing info and for the person you’re paying to know what an ACH transfer is.
  • Risk: Medium – The money is usually immediately withdrawn. The medium rating comes from having near zero recourse if you send the money to the wrong place.
  • Traceability: Medium – I’d say high except if you have the information wrong you can’t easily find out who you sent the money to.
  • Account Cost: Low – Same as checks

Direct Deposit

This uses the same infrastructure as checks and is usually free to the sender and receiver. Same hassle as a wire transfer with a little extra because you need a service to assist. I’m hazy on the details.

Online Bill Pay

Your bank will helpfully send a check. They may or may not debit your account and send a bank check. They may or may not fine you for overdraft fees. They may or may not send a check regardless of balance (and then fine you for a bounced check). They may or may not fall back to electronic means with larger companies that have accounts with them. For most small businesses, you’ll get a check in the mail if someone pays you via this method.

I am unaware of the regulation of this feature.

Credit Cards

In the past you’d have to open a merchant account and negotiate fees. These days the merchant account is virtual and you’ll get an ACH transfer for free with your cash minus $0.30 and %3 in a few days. Stripe, Braintree and a few others are great in this space.

  • Cost: Medium – $0.30 and 1.5-5% This can add up for large payments when compared to checks or wire transfers, but if you have volume and are a low risk you can negotiate the fee down.
  • Time to clear: Medium – You’ll get the money in your merchant account instantly, but can’t get access to it for 2-7 days depending on how nice your relationship is with your processor / merchant account and how much risk they think you are. If you’re using a modern payment gateway it’s usually 4 days and you don’t have to have a merchant account.
  • Hassle: Low – Easy to initiate, money usually shows up without issue. Some larger charges will fail for unspecified reasons, but a phone call to the bank will usually clear it up.
  • Risk: Low – There is fraud protection on both sides of the transaction. You get to verify credit card details if you wish before authorizing the charge. If your card is stolen you usually get the charges refunded. If you are receiving money a chargeback can cost you $10-$75 but they are usually rare depending on your business.
  • Traceability: High – Everyone has their name on both sides of the charge.
  • Accounting Cost: Low – It’s like another bank account. You know where your money went and where it came from.

Paypal, Venmo, Square Cash, etc

They use a combination of Credit cards, ACH transfers and debit card refunds to move money around.

  • Cost: Low – They range from free to credit card style fees
  • Time to clear: Medium – Unless you stay within their network you usually need to wait 2-7 days to get your money. You often need to “transfer” it into your bank account via the ACH.
  • Hassle: Medium – If you already have an account on the payment network it might be low, but chances are you or your client won’t.
  • Traceability: Varies
  • Risk: High – There’s a good chance you’ll get your money, there’s a big chance you won’t know the rules of the network or be able to access your money right away. Fraud protection is often amped up very high on these networks and they’ll lock down accounts involved in any transaction of scale, to give them time to check you out.
  • Accounting Cost: Medium – It’s not unlike a credit card processors account but each system has their own reporting and accounting systems. You may or may not have accountability


This is a funny one. Currently there are many exchanges where you can buy and sell bitcoins. Some legitimate, some shady, some with large fees and some with very low fees. Coinbase for example lets you take payment with bitcoin and instantly transfers it to cash like a credit card processor would. It’s fees are only 1% but may not give your customers the best exchange rate. Bitcoin to bitcoin transactions can be made with a transaction fee well under $0.05. Obtaining bitcoins isn’t easy and their value is volatile.

  • Cost: Low – There are no or very low ($0.05 cents to 1%) fees depending on if you’re keeping bitcoin as a currency or using it as a fund transfer mechanism.
  • Time to clear: Medium – Bitcoins will transfer in a matter of minutes – usually less than an hour to be safe. Getting your money from an exchange will either use the ACH or checks and that may take days to weeks.
  • Hassle: High – Coinbase makes it easy but very few people will have bitcoins or a coinbase account. Buying bitcoins without fees is very hard. Storing bitcoins online isn’t recommended and it’s easy to lose them as they can be erased or stolen.
  • Risk: High – Lots of bitcoin based companies have been hacked. It’s not safe to keep large amounts of bitcoins online. There is no fraud protections. Some exchanges will even monitor what your coins were used for and shut down your account if they think you’re part of illicit activities.
  • Traceability: High – There are no names on accounts but every transaction is public.
  • Accounting Cost: High – Try explaining bitcoins to your accountant. The statements from exchanges won’t look much different from a credit card processor however.

Nodebots NYC {2015-001}

Earlier this week we had this year’s first NodeBots NYC event! Andrew and Myself were happy to host. We need to thank for giving us food, space and heat.


As you may know, we share our library of electronics with ScriptEd, a non-profit which teaches programming in under-resourced schools around the city. So we were quite overjoyed when the NYC conference, Empire Node, donated their Arduino kits from their conference this past Fall. These kits will make teaching a ton easier, both in high schools and at our events!

Our event last week had a nice mix of newbie and experienced roboticists. I took a bunch of notes on links that I wanted to send to attendees, so here it goes:

  • More than a few people wanted to buy an Arduino Starter Kit. I can wholeheartedly recommend the $64 Starter Pack for Arduino from Adafruit. It will have everything you need to get started. If you want a little more or less to play with, they also have a $50 and an $85 version.

  • A Johnny-Five Chat can be found on Gitter. It’s a great place to ask questions about Johnny-Five and electronics in general.

  • We had a primer on how to blink a light (the “hello world” of robotics), and combined it with how to read values from buttons and potentiometers.

  • We learned about John Yeh’s Edison Scope, which is an oscilloscope built on top of an Intel Edison, a MSO-28, and NodeJS. It’s open source and amazing. John not only designed the software for this project, but also designed the MSO-28! He built them for his local highschool’s FIRST robotics team and there are currently a few of them on the Internation Space Station!

    Oscilloscopes show changes in voltage over time, and are very useful for seeing what our robots’ brains are outputing. Make has an interesting little video that explains the basics.

  • We talked about the Mesh Thing a bunch, I can’t wait for it to come out.

A few people decided to start playing with LED strips for next month. Adafruit has a great guide to NeoPixels. They’re a ton of fun and might just be the focus of next month’s event.

Once we confirm a few things we’ll be posting our next event, which will be Feb 10th 2015! Watch our Meetup page for more info!


Cover image by Matthew Bergman @ Robotsconf 2014

2014 Year End Review

In 2014 our plan was twofold; Help small business’s dreams come true and to be a model for how we’d like our industry to operate. In 9 months we came close to achiving everything we wanted to. Close enough that I don’t think we were ambitious enough.


In March 2014 after hanging out with the Flatiron BK-000 class for 4 months, I did an open call for applicants. I was starting a new dev shop and I wanted them. My theory was that I could teach them a strong processes to help shape their junior skills and provide them with enough time and feedback to make good software. I did a big interview with whoever showed up. (Too many people at once.) After a day long group project, some individual interviews, and a bunch of coffees. I had a team of 4 developers eager to get started with new projects.

I would be amiss if I didn’t point out that Amy is self taught and did not go through a boot camp.

We started with some intense training with regards to processes, app structure, testing and teaching the team. We had one internal project to get the hang of things. We set a low rate and started looking for client work.

Our first client was the wondrous One Month. They were short on developers and happened to be a coaching client of mine. After a few months helping build out their platform to support their ever growing list of online classes they leaned heavily into finding their product fit. This meant refining what they offered and how they offered it. I’m really excited to see them prosper.

We took a few prototype projects after that. Since the companies haven’t yet brought them to market we can’t share too much. One was in the “social goods” space and I hope to see them launch this year. Another was with a large tech company who wanted to improve their web presence.

The Winnower was a “rescue project” that turned into a partnership. Their mission is to fix academic publishing by turning it into an affordable open processes. Their platform however suffered from all of the most common mistakes you can make when building a new platform from scratch. We spent a lot of time rebuilding the core areas and sticking tests on the features that weren’t worth rewriting yet. The Winnower now commonly appears on the top of reddit’s science section. They are also soon to release a set of blogging tools to help people bring citability to the work they self publish. We’re really happy to be a part of developing their platform and watching it take traction.

We’re currently working on some longer projects. One is an upcoming SaaS app for the food industry (we’ve learned a ton about food) and another is a maintenance project for a startup that’s in the middle of a complete rewrite.

In the non tech side of the business, we spent a lot of time refining our inception and proposal processes. Probably too much time. We’ve tried several approaches and are still not 100% happy with the results. This may be a mismatch with our expectations or the resources of our clients, which usually keeps us from devoting too much time to research before a project. Communicating the value of a research and design period is something I’d like to work on for 2015.

We also left a few really exciting projects on the table. Since there are only a few of us we weren’t always available. This was incredibly frustrating as we’d love to see more of these projects come to life.

We aimed at startups and social good focused organizations and made an impact. I couldn’t have hoped for more.

As for being an example of how a development team should run? I think we knocked it out of the park and can’t wait to keep improving. It should be noted that we’re small, only 3 people. I’ve noticed most problems with development teams happen around 5+ people. So maybe we have it easy.

We started with five people and are now three. Hannah Nordgren playing to her talents is now at ESPN in a frontend focused development role and Gustavo Guimarães was asked by German Chancellor Angela Merkel to join the Alexander von Humboldt Foundation to do research into online education. They’re continuing to do good work and we’re proud of them both.

We did “lunch and learns” for half the year, and have brought it back for this year. We kept our 20% free most weeks and had to actively stop each other from working on those days so we could go learn something. Amy started teaching HTML, CSS and JavaScript at Girl Develop It and was joined by Armando to TA her classes. We’re planning on developing course work for them this year. I started running the NodeBots NYC chapter where we teach people programming through robotics.

Armando recently got covered by Business Insider. They wrote about how he changed his life by learning to code. It reminded us how rare stories like his can be, even if it’s not exactly rare here at Wizard Development.

We started our first open source app, which helps people find their local farmers markets in NYC. We’re working with Megan Taylor and Vico Zabel on that project and hope to launch it in a few weeks. We chose to help Megan build off of an app she had designed for a local meetup and to challenge ourselves. Could we build a useful app in a day? No, not yet. We were pretty happy with how much we accomplished and it was a great use of our 20% time.

Our team

I’m damn proud of what we’ve accomplished last year. It makes thinking about our plans for this year even more exciting. We decided to grow, change our focus, and keep doing the things that we love. We’re really excited about it. More on that in the coming days.

– Francis

Header photo of the Lower East Side by Douglas LeMoine


Rick was driving north on I-95, he had to find an uplink or this whole trip had been for nothing. “Rose, scan the vehicles ahead.” His onboard computer sprang into action attempting to connect to internal networks of each and every car on the road around him. “No joy” she responded. All the cars either were too outdated to allow wireless access or detected their connection attempt and shut it down before they could get in. Rick sped past the cars in front of him, he had to find what he was looking for and these cars weren’t going to help.

It had been three long weeks since he had access to the ‘hub. Way too long for any self respecting keyboard cowboy. Back in 2017 when the loss of net neutrality caused the global collapse the internet as we knew it, all of the east coast had lost their uplinks to the internet. The Silicon Valley was the only place where you could get a reliable public connection and nobody dared to go there anymore. The internet wasn’t dead it was just hard to get to. Large corporations had the only uplinks and provided their own cell and wifi coverage to only their most elite customers. Having cast off the chains of debt and credit cards Rick was not considered an “Elite” customer. As a result it was hard to get online, and he had code to push.

Three quick “BEEP BEEP BEEP” sounded over the music, Rose flashed all the HUD’s alarms. His modified 6LOWPAN radio had picked up packets from a speed trap up ahead. He toggled back on his transponder and slowed down to the regulation 75 MPH. 6LOWPAN was supposed to be the savior of the net, allowing us to put all our devices online in distributed low power mesh networks. At one point you could send packets from New York to Boston via the ip6 mesh. The powers that be had different plans however. Private security had taken over the spectrum. Civilians quickly lost access to private low power networks and if you ran your own you were accused of interfering with national security and shut down.

Once he was sure he was clear of the trap Rick switched off the transponder and increased to hunting speed. This made him invisible to the EZPass speed checkpoints that littered the highway every few miles. They’d have no idea how fast he was going. It randomized his UUID whenever he switched it back on and appeared to be just another vehicle on the road. This got him past tolls and checkpoints. Even in this day and age it took the government a few weeks to even notice anything was out of the ordinary. Any tickets they would want to issue ended up in someone else’s inbox.

“Pay Dirt!” Rose exclaimed. A Fung Wah bus was up ahead and it had a connection to the net! Fung Wah had long since been shut down, but their buses still drove the roads. Their drivers were able to pick up the buses on the cheap after the company went bankrupt and had organized their own routes free of oversight and regulation. It was pretty lucrative. Luckily the bus’s uplink hardware was valuable enough that it remained in service and if you rode the bus you could use it to connect to your own systems. Someone on this bus had it activated and was using it right now!

Rick pulled up alongside the bus in the driver’s blind spot. If the driver or the passenger noticed him, either in real life or cyberspace, he’d be shutdown, reported and locked out from all uplinks until he could cycle his hardware address and that took weeks. Rose started the connection attempt. It was almost too easy. He had his 128 bit address in no time at all, and routes to all the backbones populated almost instantly. It was never this easy. The passenger on the bus either had no security or had seen him coming miles away. “Fuck it, git push.. wha!?” Rick was cut off by a squeal of sound. The whole car shuddered and the smell of magic blue smoke sifted into his nostrils. The safety circuits kicked in and the car drifted slowly to the side of the highway. It stopped, dead. What happened! Had been back hacked? “Rose!”

After what seemed like an eternity Rick was able to get the car to spring back to life. It took flashing the firmware in his starter circuits twice to get the engine running. Rose had detected the hack and shut everything down before the hacker on the Fung Wah bus could get too far. The navigation and scanning circuits were toast however. Not only were they hacked but the hacker had decided to pump emergency current into their subsystems and destroy $30,000 worth of equipment. Rose and the rest of his systems were fine. He could drive. And he better leave before any private security showed up to see why he stopped.

That was close but he couldn’t give up.

“You wont believe this” Rose piped up. It had been two hours and they had almost reached their destination. Rick had feared they wouldn’t be able to push any code tonight. “It’s a MEGABUS”. MEGABUS had been one of the few companies to survive the collapse. Every bus had power and material hookups at every luxury leather seat, and most importantly a corporate uplink connection active 24/7 for use by all passengers. Rick wasn’t going to mess around this time. He killed his lights and activated stealth mode. Rose picked up and identified packets from all 36 of the passengers on the megabus. Most were encrypted but one was downloading a ton of data. Mostly facebook, twitter and netflix. Streaming video on the east coast!? Gluttonous. This guy wouldn’t notice a little extra bandwidth. Rose cloned his MAC and carefully connected to the bus. Connections to twitter and the hub were successful. “I’m in” she reported. “Alright, easy now. cd johnny-five, git push origin master.” Rose connected via SSH to github and started pushing binary blobs of robot control data. “Eta 137 seconds” she reported. Rick decided to tweet.

There was only 30 seconds left when the MEGABUS driver noticed him and pulled hard to the right narrowly missing Rick as he slammed on the breaks. “We’re losing signal!” Rose yelled out as the bus sped away. 20 Seconds! Rick sped up, this time right behind the bus. It slammed on it’s breaks, and broke off towards an exit. Rose quickly deduced the situation “The Fung Wah hacker must have reported us to all uplinks in the area, they knew we were there from the second we pulled up!” Rick yanked on the wheel and slammed the pedal. Even if it meant leaving the safety of the highway he wasn’t going to drop this connection. 5 seconds left! He pulled ahead of the bus and slammed on his breaks. Time for some of the your own medicine! The buses safety circuits kicked in and the bus slammed on it’s brakes skidding to a halt before tapping Rose’s bumper. It would need a visual inspection before it was allowed to drive again. “Git push complete.”

Rick sped off into the night.

Bosty by Richard Cox

Debugging Robots


I was recently helping Sara debug an issue with a project for an upcoming book she’s writing a chapter in. She was using Johnny-Five and a Spark Core to remotely control a boat with JavaScript.

The Setup

The setup was a sparkfun motor driver and a standard servo hooked to A0 and A1 of the Spark Core.

sparkcore pinout

Sara had one submersible thruster on the boat hooked up to the motor driver and a rudder hooked up to the servo. She was using VoodooSpark with Johnny-Five to remotely control the two devices.

The problem was whenever you turned on the motor the servo would steer all the way to the right and stop responding. Instant Crash! Both components worked fine in isolation. Moving the pins around didn’t seem to help. We thought we got it working on a different spark core but if we did it was only for a few minutes and did not work when we installed it in the boat. What on earth was going on?

Since swapping out the components had no effect we were able to reasonably claim the hardware was in working order. Our next thought was maybe we wired it up wrong. We could rule that out by testing the output from the spark core itself. If the output was as expected then the wiring was probably the issue. If the output was bad then we could keep looking up the stack.

Testing PWM output

Both the motor driver and the servo use PWM for their communication. The motor driver uses it to control the speed of the motor, and the servo uses it to determine it’s position. In theory, moving the servo should have no effect on the motor and driving the motor should have no effect on the servo.

I unhooked everything from the spark core and took an Arduino I had laying around and programmed it to be a simple pulse timer. It would count how many milliseconds a pulse was and would write it out on the serial port. I don’t think it was very accurate but it was good enough to see if the signal on the pins were what we expected. I hooked the Arduino up to the sparkcore’s servo pin.

unsigned long duration;

void setup() {
  pinMode(7, INPUT);

void loop() {
  duration = pulseIn(pin, HIGH);
  Serial.print("Channel 1: ");

I used the serial monitor from node-serialport but the one in the Arduino IDE works great too.

I measured the servo’s pin and turned on the motor and got this.

Channel 1:1465
Channel 1:1465
Channel 1:1464
// PWM Pin on
Channel 1:61
Channel 1:61
Channel 1:61

Well that’s not right. It shouldn’t have changed at all. I swapped pins and found the same issue. I tried two servos without a problem!? Tried two motors and that also worked! We only got the problem with a servo and a motor. In any case the problem wasn’t in the wiring.

Removing layers of the stack

The full stack looks something like this:

  • Parts and Devices
  • Wiring
  • Spark Core Hardware
  • Spark Core Software running VoodooSpark
  • UDP over Wifi
  • spark-io node library
  • johnny-five node library
  • Sara’s boat code

We already ruled out the parts and devices and the wiring. Debugging the Spark Core Hardware seemed daunting, so we decided that it would be easiest to rule out the upper layers of the stack.

After a little bit of hacking we got rid of the boat code and johnny-five and were left with a spark-io example.

var Spark = require('spark-io');
var board = new Spark({ /* connection info /*});
board.on('ready', function(){
  var pwmPin = "A0";
  var servoPin = "A1";

  this.pinMode(servoPin, this.MODES.SERVO);
  this.pinMode(pwmPin, this.MODES.PWM);
  this.servoWrite(servoPin, 90);

    console.log('pwm on');
    this.analogWrite(pwmPin, 200);
  }.bind(this), 5000);

We booted everythign up and had the same output. =(

We were left with this:

  • Spark Core Hardware
  • Spark Core Software running VoodooSpark
  • UDP over Wifi
  • spark-io node library

I’m going to gloss over the details of exploring the VoodooSpark firmware and listening to serial debug output and monitoring packets on the wifi. I talk about anything noteworthy in the pull requests I opened as we explored this issue. I was convinced that voodoospark was either getting bad data or interpreting it wrong. Neither was the case. I spent about two nights learning a lot but not finding the bug.

We finally decided to write a small Arduino C program to replace VoodooSpark. If it showed the bad output then we’d know the Spark Core Hardware was to blame!

Servo s;

void setup() {
  pinMode(A0, OUTPUT);

void loop() {
  Serial.println("A1 Servo to 90");
  Serial.println("A0 PWM to 200");
  analogWrite(A0, 200);

I loaded this program on the spark core and booted up the Arduino. And low and behold, the bug!

The only layer of the stack left!

  • Spark Core Hardware

Spark Core Hardware Internals

At this point we were lost. How do you know what’s going on in there? I’ve heard of JTAG which is way of attaching a debugger to the code on the chip, but I lacked the hardware and knowhow to make that work.

Luckily for us Zach Supalla was watching the issue. He noticed that A0 and A1 are on the same “timing peripheral”. He was able to share a diagrams and charts of what’s in the spark core’s internals.

Even though both the motor controller and the servo use PWM, they use it at different frequencies. The component behind both the pins could produce independent PWM duty cycles (eg. control two devices) but only if they were at the same frequency!


And the fix? Move the servo to A4 which has a different component behind it. Once we did that everything worked as expected! We left the issue open so we could figure out how to communicate the error via the software in the future.

The final wiring looked like this.
Final and working wiring

You can follow the saga in pull requests;

And look for the book! Despite the issues we had working with Johnny-Five and the Spark Core is a lot of fun! The authors figured out all this mishegas so you don’t have to. ;-) It’s called Make: JavaScript Robotics: Building NodeBots with Raspberry Pi, Arduino, and BeagleBone and comes out in March 2015.

The photos are from this years RobotsConf and shot by Matthew Bergman. The firebot and PaddleBoatBot (below) which was built by a 10 year old.


Keep hacking!


Migrating Legacy PHP Apps to Heroku


There are lots of ways to host, deploy and maintain applications. We often use Heroku because the tradeoff of control gives you a lot of operational benefits that are expensive to build and maintain. In this post we’re going to be talking about the hurdles around moving a legacy PHP application to Heroku.

Note: It’s been years since I’ve been active with PHP and the rest of my team hasn’t had the pleasure. There’s a good chance I’m missing obvious things. If I get something wrong please reach out on twitter or email and I’ll update this article.

I have a lot to say about Heroku. Their 12 factor app design philosophy speaks to me. Once upon a time, I helped bring 3 of the 12 factors (codebase, logging and disposability) to a large company. We still had many many problems, but even just improving those 3 was a huge boom to reliability and productivity.

Heroku is somewhat similar to the Docker based hosting that’s starting to mature. They were an influence, if not an inspiration, for much of the LXC based systems we’ve been seeing. They enforce strict code and data separation that not only inspires scalable designs but increases security and reliability. Their API lets you easily clone environments for development, testing, and demos. Heroku’s operational support, logging, postgresql, buildpacks, scalability of machines, and ease of use and setup make them a great fit for most early stage projects or projects without a permanent team.

We’ve been tasked with bringing a homegrown tracking and reporting application into a stable state while we start a rewrite. It’s a big ball of nightmares running on a “snowflake” and now it’s ours.

I should also point out that the app isn’t built on WordPress, which has a horrible code and data separation that makes it a nightmare to deploy, debug, and recover from errors.

Administrators usually let WordPress manage itself and set up nightly code and database backups. Then they just hope the backups will work together.

We should also take a good long hard look at what PHP puts in its system configuration (php.ini) vs in its runtime configuration. Many, many of the language-provided functions (eg. mail()) are configured as language options instead of application options. I’m happy to say the worst bits have been deprecated but this is a legacy app so, of course, we’ll have to deal with them anyway.

I’ll break the conversion into the different areas of concern.

Differences between Heroku and “traditional hosting”

This usually comes up with WordPress. Wherein “wordpress just works on my host” and “it doesn’t work” on Heroku. There are several reasons for this.

Heroku maintains an ephemeral filesystem which is reset every time a new dyno starts. Combining this with WordPress’s auto update functionality results in updates and file uploads that just disappear.

Heroku prefers PostgreSQL and you need to go out of your way to get a MySQL database via an addon. Most PHP apps prefer MySQL despite the project’s questionable future.

Traditional web hosts have a single web server. Heroku calls their web servers dynos and easily lets you run multiple ones. There’s no major difference between a single web server and a dyno until you make a second dyno. Since the dynos operate in isolation and don’t share anything, some basic funtionality, including sessions and file uploads, will appear to break. This isn’t an issue with Heroku, as you’d have the same issues with two web servers, but you’ll need to adjust to accommodate it.

PHP version and configuration

Heroku uses Composer for versioning and dependcy management. Chances are, your legacy app doesn’t. Composer is a commandline tool that runs againsts your project’s composer.json config file. For our project we could use the latest version of PHP. We also needed the built in MySQL functions. Initially our file looked like this.

  "require": {
    "php": ">=5.5",
    "ext-mysql": "*"

You’ll need to run composer update after each change to the composer.json to also update the composer.lock file.

Now we needed to tweak some of the php.ini settings. Heroku makes this easy by allowing you to make a .user.ini file in your project that it will load. Ours enabled outputting errors to to the logs and enabled the mysql extensions that we required from composer.

; php config
display_errors = Off
html_errors = Off
log_errors = On
error_reporting = E_ALL & ~E_DEPRECATED
extension =

MySQL Database Setups

Now that we have PHP with the MySQL extension setup we need a database to connect to. The only offering on the Heroku platform is cleardb. They are a pretty good MySQL shop with fantastic support. (I had a decent conversation with with their CTO once about some networking issues.) It bothers me they don’t have machines in Heroku’s data center but this hasn’t been a performance issue for any of these apps yet. An alternative would be using Amazon’s RDS which shares the same datacenter but requires a bit more setup and management.

Once we have a database, we’ll need to connect to it. Heroku addons publish connection info through url schemes made available in the environment variables. While you can look up the connection info, don’t hard code these credentials in your app. Instead parse them into a PHP array (or hash, as most people call them) from the url.

$connection_info = parse_url($_ENV['CLEARDB_DATABASE_URL']);

//database server
define('DB_SERVER', $connection_info['host']);

//database name
define('DB_DATABASE', substr($connection_info['path'], 1));

//database login name
define('DB_USER', $connection_info['user']);

//database login password
define('DB_PASS', $connection_info['pass']);


PHP includes the mail() function which has been the bane of mail services on shared hosting since it’s inception. It makes it very easy to use the server it’s running on to send email. It used to be the school of thought that email was an operating system service. This was back when your user account on a machine was also probably your email account. This is no longer the case. In recent days, applications will connect to remote email servers with dedicated credentials. There’s a lot to be said for centralizing email, as it helps prevent spam. However, you now need the help and permission of a 3rd party to send email reliably, whereas in the past anyone could do it from any computer.

The philosophy aside, Heroku wont let you send email directly from their machines, they require you to use a 3rd party such as Sendgrid. The mail() function wont work.

Sendgrid knows how to send email and offers a litany of addon services to help you do it reliably. Their non-email infrastructure appears to be a mess. Bugs in their website and account creation systems have been a problem for me in the past. However, for the basic use case they usually “just work”.

We opted to use Sendgrid’s PHP api client because talking SMTP isn’t something PHP can do without a library. If we’re going to bring in a library, let’s bring in one that works with the mail service we’re using.

We’ll add it to our composer.json

  "require": {
    "php": ">=5.5",
    "ext-mysql": "*",
    "sendgrid/sendgrid": "2.1.1"

And we’ll change our calls to mail(), to use the SendGrid api.

$sendgrid = new SendGrid($_ENV['SENDGRID_USERNAME'], $_ENV['SENDGRID_PASSWORD']);
$email = new SendGridEmail();
     setFromName("My C00L W3BSITE")->
     addHeader('X-Sent-Using', 'SendGrid-API')->

File uploads

We had none in this application, thank god. Heroku’s ephemeral filesystem allows you to upload files, but you can only store them temporarily on the dyno. You’ll want to upload them to a cloud file storage services such as Amazon S3. If we ever need to figure this out I’ll update this article.


PHP has a $_SESSION varible that holds a PHP Array of data. It’s a superglobal, which is what most languages call a variable in global scope, and can be accessed from anywhere. To use a PHP session you call session_start() and by default it will read/write a unique identifier to a cookie, and store your session data in a temporary file. If you have more than one dyno they can’t read each other’s sessions. One technique to work around this is to store the sessions in memcache. Heroku has a nice article about using memache to store php sessions which we’ll follow.

First, we’ll add the memcache extension to our composer.json and update the lockfile.

  "require": {
    "php": ">=5.5",
    "ext-mysql": "*",
    "ext-memcached": "*",
    "sendgrid/sendgrid": "2.1.1"

And then set up a memcache addon in Heroku. I’m using memcachier today.

Since sessions are handled by PHP itself instead of your app, you’ll need to set up memcache in the user.ini PHP configuration. I added the following lines to do a persistent connection to memcachier’s memcache servers. All the connection info is provided in the environment varibles.

; Session handling

; Use persistent connections
session.save_path="PERSISTENT=myapp_session ${MEMCACHIER_SERVERS}"

You can test session handling by logging into your app and then running heroku restart to restart your dynos, which will destroy the ephemeral filesystem. When it starts back up you should still be logged in.


You’ll want to add a logging add-on such as papertrail to keep a small archive of your logs.


It should also be noted that if you ever get hacked, the server cleanup will be a lot easier, and you’ll never run the risk of undiscovered malicious files laying around.

You’ll now have a much more reliable and easier to manage setup, and can take advantage of all the services that Heroku has to offer.

I hope this helps!


Code Review: Cleaning up the Environment


I’ve been reviewing a lot of projects lately and I thought I should share what I look for when evaluating the code quality and maturity of a project. My team is often brought in to “clean up” an existing system. This might include bug fixes, new features, a rebranding, and scaling issues. This will be the first in, hopefully, a series of posts about our approach. I’d like to write down my findings in hope that if we all learn from these mistakes we’ll see less of them.

If that ever happens we’ll need to find new things to do. Good thing that there’s always work to be done. =)

A note of warning, as projects mature they often obtain and then move away from some of these tenants. However, the principals and purposes often remain. I’m also going to be primarily talking about Rails projects but these aren’t Rails specific issues.

Reduced time to startup

Maybe this should be called reducing complexity to startup. The theory is that the more “magic” one needs to retain in their head in order to run the application, the heavier the overhead on all developers on the project. This also slows down any new developer who wants to get started.

The first place I’m going to look for setup instructions is your Readme file. This is usually full of database setups and dependencies such as command line apps or specific versions of libraries. The older the project the more of these you’ll see. This list is usually out of date (fixing this is a great first commit for a new developer) but it’s also usually missing the “config file” which contains all the keys for all the 3rd party services you need to run the app locally. While I don’t want this file to exist, without it I usually can’t start the app up or load a single page.

Often missing is a good set of development data. Data that you can load into an empty database to allow someone to try out the application for purposes of modifying or adding new features. This is usually done with a rake task. Having this built with code as opposed to loading an SQL or CSV file is preferred. You should make use of factories and the objects in the system. The data will be a lot more likely to be valid if it exercised your code to end up in the database.

You should also make sure to generate users to be used in testing. At the minimum you’ll want an admin and a ‘normal’ user.

Isolation of Environments

Don’t, for the love of god, have your development environments use production api keys or share production services! (or ssl keys, or s3 buckets, or anything else!)

More to the point, don’t hard code any keys! Use environment variables.

On the server this is easy. You can use a gem such as foreman, figaro, .env, or many, many more. The idea is that production credentials should only exist in production. (And maybe also exist in your encrypted password manager.) In development you can load them from a file either before you start your development environments (eg, foreman) or have your app load them from a file if you haven’t already defined them (eg Figaro).

This works great for configuration. How should you connect to your database? Where should you store your uploads? However, this is usually when I wonder why you’re requiring a 3rd party service for your app to function.

Remove dependencies on 3rd party services

Chances are, any third party service you rely on will go down. This means you need a way to function with it down. Maybe you hide that “latest tweet” box, or queue email to be sent later. I advocate for disabling features that rely on 3rd party services if you are unable to connect.

In development you can go a step further and either mock the service or change your app’s behavior to handle not having it. For example, you can store images locally instead of on S3 if S3 isn’t configured. You also don’t actually have to send email or manage mailing lists if you don’t know how to talk to sendgrid or mailchimp.

When you’re developing a feature that relies on a 3rd party it’s fine to talk to it during normal development. However, I bet you can just skip the call and pretend it was successful. This will allow the app to function without knowing any api keys and will help you identify the “mock points” for testing.

RVM: The Ruby Version Manager

RVM-like software exists for every single programming langugage that’s worth a damn. You get to easily isolate your packages and choose a version of the language for the app. Node.js and NPM for example provides isolation of packages by default. For any ruby project a .ruby-version and a .ruby-gemset should already exist. I’d like your app to run in isolation on my dev machine but you should want it to do the same in production. This makes it easier to rebuild the running environment. If your production environment can’t be easily rebuilt, you’re in a world of trouble.


If you’ve done all this, I can type the following and have a mostly functional application.

git clone
cd project
bundle install
rails s

And then I can actually get on to evaluating your code.