Copy/Paste/Pray, or How I (Kind of) Learned to Code in 4 Years and 5 Months*

/

[Reposted from Medium] 

1. Remember that kid in high school, the kid who lugged around a 17 inch XPS and wore massive air-controller headphones and waited in the midnight line for Halo 3 and built a legit robot while the rest of the class struggled with potato versions?

I was not that kid. 

 2. “Be nice to nerds; chances are you’ll end up working for one,” my mother told me, the day I complained about the dandruff flakes of the nerd who sat in front of me in Lang&Comp. “Pretend they’re snow flakes,” she said. But they only looked like snow from a hard squint.

 3. The worst grade I ever received was in gym class, senior year. Dodgeball. Mr. L didn’t appreciate my Clueless-coopted defense. 

The second was in Computer Skills. I believe the skills we covered included Word and PowerPoint. Mrs. A humorless about windings. The end of that class was the end, I thought.

 4. While working at Guest of a Guest, I learn the acronyms CMS, WYSIWYG, HTML. HTML is the ugly alternative to WYSIWYG.

 5. Except sometimes, the WYSIWYG doesn’t give you what you see. I learn <p>, <ul>, <li>, <br>. 

 6. I start my own blog. Wordpress says Sandbox is a blank canvas for CSS artists. My canvas quickly assumes the aspect of a toddler’s finger painting. To make it less ugly, I have to open up the style files, stare down a new beast, CSS, google things like “wordpress change body font css,” copy, paste, and pray. 

 One time, I try to change a function. The entire site disappears, and only reappears after a tearful two-hour conversation with Bluehost. I steer clear of PHP after that. 

7. A few years later, a coworker shows me something called Stylebot. Stylebot lets you click on a page element, change it with a WYSIWYG, look at the new CSS, and paste that into your style sheet. Huzzah! And I learn a little more CSS to boot. 

8. The company works primarily with C# and .NET. I spend about two hours trying to learn the former. 

9. On twitter, I spend a lot of time talking to the people who use, or could use, our software. They tweet about #posho, #bash, #sql. What is MVC? What is a schema? I google, but I know not. 

 10. I teach myself how to use Sharepoint. Sharepoint is like Wordpress, only bigger and inherently ugly. You can make a lot of money helping companies bandaid that ugliness.

 11. Sharepoint is… not my thing. Luckily, the company hatches a spinoff company, and I go to work for that. We start off using Java, HDFS, and HBase. Java is hard and boring, people tell me. What’s another language that can be used to analyze data? Python. MIT Open Courseware has a free course on Python, and I sign up.

 12. And last about three weeks. I feel like I’m using the same copy/paste/pray technique I used with CSS, only this time I really want to learn. Also, there’s no one to answer my prayers. Which brings me to lesson one of Learn to Code: 

There needs to be someone/intelligent thing on the other end. Otherwise you often a) don’t know if you’re getting things right and b) don’t know why you’re getting them wrong. 

13. At my annual review, our CEO tells me I should learn Javascript. I sign up for Codecademy and start their JS track. I get further with Javascript than I did with Python, but a dedicated two weeks turns into a semi-dedicated month turns into a here-and-there two months turns into nada. Here are the good things about Codecademy:

A: web-based interpreter. Aka you don’t have to download anything to get started. Aka no staring at your terminal’s error messages, wondering why ggplot2 won’t install. (But more on that later.)

B: Error checking. You can’t move onto the next part of a lesson until you’ve gotten the code necessary to complete the current one correct. 

 Here are the not-so-good things:

A: Lack of explanation for why your code is wrong. This is understandable, but it’s hard to learn from your mistake if you don’t know why it was a mistake.

B: If you can’t figure out why your code is wrong, you have to go to the forums, and hope that someone has already answered your question, and that their answer works. The Stack Overflow model works because there are experts contributing, and the top answers have explanations, not just code. At Codecademy, the people answering are usually also novice coders, and so can’t do more than say “this worked for me.”

C: The projects accompanying each lesson are generally not compelling. This is a missed opportunity, because the projects are where you really cement new knowledge. Alors, lesson two of Learn to Code:

 Lessons and especially projects should teach you how to build things that have clear real-world applications. Do teach me how to scrape geotagged tweets from twitter. Don’t teach me how to build a Rock, Paper, Scissors game.  

14. At my friend’s behest and 7am cajoling, I go to a Rails Girls Baltimoredev shop on Ruby and Rails. Andddd lightbulb! By the end of the day, I understand what classes, objects, methods, and hashes are, and I have vague graspings of the components of a full stack. I owe this to the fine folks at Rails Gilrs, B’more Rails, and Jumpstart Lab’s excellent “Ruby in 100 minutes” tutorial, which is written in succinct-but-edifying layperson English.

15. Go Ruby! I start Dan Nguyen’s Bastard’s Book of Ruby. Twitter scarfs, at last! Except Dan’s instructions are tailored to the Twitter v1.0 API, not the 1.1 API, which has totally different permissions. Also, what is an API? 

Downside: My scarf isn’t very warm. 

Upside: I know that scarf-making is a thing, and that I want to make one for email data. I want to analyze things like sender/unread rates, time to reply rates, words associated with rapid replies, etc… 

16. I show David General Assembly’s 12 week Web Development Immersive course. He grimaces at the mention of Ajax. “Take this one,” he says, pointing to the Data Science course. 

17. The Data Science courses requests students have some familiarity with Python and/or R. Python it is, then! I return to Codecademy. Same pros, same cons. But: this is my fourth language stab, and finally, I am learning the underlying framework. Just as it was easier for me to learn Spanish after French and Italian after Spanish, it is easier to learn Python after Ruby, and vice versa. 

18. Enter Arabic, aka R. My GA pre work includes this lovely PDF from USCB. Its URL proclaims it to be the BestFirstRTutorial. As my pre work also included the first two chapters of the O’Reilly Code School R tutorial, I can safely say that USCB’s version is not the BestFirstRTutorial if you don’t know statistics. 

 That’s right. R is a statistical programming language, and I don’t know statistics. Not anymore, anyhow. I took AP Stats in high school, which exempted me from math at NYU. Save complicated tip calculations, I haven’t touched any it since. 

So, I spend a weekend looking up distribution functions and what sigma means and is summation the same as sum. The USCB PDF was supposed to take me ~2 hours to get through. It takes, er, significantly longer than that. 

19. Onto ggplot2! ggplot2 is system that lets you decide how you want your R graphs to be plotted. Installing it is supposedly easy, but my terminal will. not. cooperate. ‘install.packages(“ggplot2")’, I say, and it says nothing. 

 20. The last part of my GA pre work is a command line tutorial. Unable to install ggplot2, I move on to the command line. And again, lightbulb! The command line is just the code representation of all the stuff that is on my computer, and the ways I access that stuff. Amazing! I learn how to see what directory my terminal is in, and how to change that directory. Installing ggplot2 requires me to be in the R directory. Once I cd into that, it’s a snap. And this brings me to my third, and final lesson of Learn to Code:

Before you do anything else, learn the command line. 

Alors, this brings me up to present day: muddling through the ggplot tutorial, admiring the pretty pastels, trying to remember what dnorm means. The course officially starts tomorrow, and I’ll be documenting my gleanings along the way. Wish me luck!



*I don’t endorse this method. 

 

How to Build Your First Meteor App and Discover Your Inner Artist

/

I recently gave a demo to the Boston Meteor Meetup group of something amusing I made with Meteor (the framework we use for Gander). The title of the meetup was "Build Your First Meteor App," and Meteor Multidraw, which took about  two and a half hours start to finish, fits that beginner-friendly bill.

Effectively, Meteor Multidraw is a simple real time collaborative drawing application. Everyone shares the same drawing state, and any new drawing will automatically propagate to all other users currently viewing the page. It's just Meteor and d3 -- the latter actually handles drawing points. The source is up on github. 

The application itself is pretty simple, but it shows a bunch of Meteor's features all at once. Since it's really meant to be more instructive than an example of best practices, some of the things done are more for educational effect than for any sort of practical reason. Here's how it works, starting with the d3 piece:

Part 1: Create Your Canvas

// A simple object to handle creating an SVG canvas,
// clearing the screen, and actually drawing our points.
function Canvas() {
  var self = this;
  var svg;

  // Creates the SVG canvas.
  var createSvg = function() {
    svg = d3.select('#canvas').append('svg')
      .attr('width', '100%')
      .attr('height', '100%');
  };
  createSvg();

  // Clears the SVG canvas.
  self.clear = function() {
    d3.select('svg').remove();
    createSvg();
  };

  // Naively draws an array of simple point objects.
  self.draw = function(data) {
    if (data.length < 1) {
      self.clear();
      return;
    }
    if (svg) {
      // This is what actually does the drawing. We're not
      // going to cover d3 in any great detail here.
      svg.selectAll('circle').data(data, function(d) { return d._id; })
      .enter().append('circle')
      .attr('r', 10)
      .attr('cx', function (d) { return d.x; })
      .attr('cy', function (d) { return d.y; });
    }
  };
}

All that we're doing here is creating a Canvas object which can render an SVG canvas and clear itself. Additionally it naively handles drawing points as small circles, given an array of them. Pretty simple. What we're going to do then with Meteor is propagate around these arrays of points so that they can be drawn for everyone. There are two major components to any Meteor app - a server, and a client. The server is generally responsible for maintaining the absolute state of the data, in our case, a collection of points. Let's take a look at it in:

Part 2: Serve It Up

// This is our actual points collection. It's stored in an 
//internal mongo database.
points = new Meteor.Collection('pointsCollection');

// This function will publish a named subscription of points
// that the client can then subscribe to. It returns all points
// currently in the points collection.
Meteor.publish('pointsSubscription', function () {
  return points.find();
});

// This declares a server side method that the client can
// remotely call, which simply removes all of the points
// from the database. This is necessary because only the server
// may remove multiple documents at once.
Meteor.methods({
  'clear': function () {
    points.remove({});
  }
});

Not super complicated. It declares a points collection, blindly publishes a cursor of all the points, and provides a method for removing all of them. Let's look at the client:

Part 3: The Client

// Declaring our points collection on the client side.
points = new Meteor.Collection('pointsCollection');

// Just a reference for our canvas.
var canvas;

// Creates a reactive context around us subscribing to our
// points collection. See the discussion about reactivity 
// below.
Deps.autorun( function () {
  Meteor.subscribe('pointsSubscription');
});

// Runs when Meteor is all set to start. It creates our
// canvas out of the Canvas object we declared above and..
Meteor.startup( function() {
  canvas = new Canvas();

  // Creates a reactive context around us getting all points
  // out of our points collection. Fetch will turn the cursor
  // into an array. We then pass off this array to the canvas'
  // draw method to actually draw all the points.
  // (Not performant!)
  Deps.autorun( function() {
    var data = points.find({}).fetch();
    $('h2').hide();
    if (canvas) {
      canvas.draw(data);
    }
  });
});

// Totally unnecessary, but used to illustrate how
// template helpers work. By calling in {{title}}
// in our template (see below), we write out
// the title of our app.
Template.drawingSurface.title = function () {
  return 'Draw with Me!';
}

// Declares an events hash of all events scoped to
// a particular template. Here we're handling the click
// event on the clear button.
Template.drawingSurface.events({
  'click input': function (event) {
    Meteor.call('clear', function() {
      canvas.clear();
    });
  }
})

// Just some DRY for inserting a point into the points 
// collection based on the cursor position.
var markPoint = function() {
  var offset = $('#canvas').offset();
      points.insert({
      x: (event.pageX - offset.left),
      y: (event.pageY - offset.top)});
}

// Another events hash. This one handles capturing the
// drawing-related events. Just for reference, Session can
// also establish a reactive context, but we're not using that
// here.
Template.canvas.events({
  'click': function (event) {
    markPoint();
  },
  'mousedown': function (event) {
    Session.set('draw', true);
  },
  'mouseup': function (event) {
    Session.set('draw', false);
  },
  'mousemove': function (event) {
    if (Session.get('draw')) {
      markPoint();
    }
  }
});

This is probably the most complicated piece. It handles all of the client side interactions. It's also where we're taking the most advantage of reactivity, the heart of Meteor. By utilizing reactivity, changes to our underlying data will rerun functions that depend on our data. Above, the points.find({}).fetch() is some of our underlying data. Our subscription declares that we're subscribing to data from the server, in this case, all of our points. Whenever the server's data changes, it passes those changes onto the client. When the client's data changes, our collection is automatically updated. The call to points.find({}).fetch() is in a reactive context, and when the client side collection changes the function is rerun, which naively causes all of the points to be drawn onto the canvas, including the new ones, thus updating the overall drawing. When the user themselves draws new points, these are inserted into the collection and then synced back to the server, which adds them to the server side collection, thus triggering all of the above for all other clients. This is what allows us to maintain our drawing state across everyone! Now, last piece, swear, the template:

Part 4: The Template

<head>
  <title>meteor-multidraw-demo</title>
</head>

<body>
  {{> drawingSurface}}
</body>

<template name="drawingSurface">
  <h1>{{title}}</h1>
  <input class='clearButton' type='button' value='Clear'/>
  <h2>Loading...</h2>
  {{> canvas}}
</template>

<template name="canvas">
  <div id='canvas'></div>
</template>

Not much going on. The {{> drawingSurface }} just says render the drawingSurface template here. You can see where we stick in the title as mentioned above. Lastly, we can nest templates too, as seen with the canvas template. Note that the template names here exactly match those used in the client side code above.

And that's pretty much it. You can fork the repository and mess with the code yourself from https://github.com/nwmartin/meteor-multidraw-demo, which also includes a set of directions for running it locally. If you want to mess with it, try making it so folks can draw with different colors or select a different "brush" radius. Most importantly, have fun with it. If you have any questions you can shoot me an email at nickm@ripariandata.com or just ask in the comments. Good luck!

Related posts

4 Resources Every Performance Tester Should Know About

/

Cross-posted from Softartisans

Performance is a widely overlooked feature in the development world. In enterprise development, better performance can correlate directly to company profit. More efficient programs utilize fewer resources; fewer resources cost less money.

On the consumer end, performance can make or break a product. One reason Google has such great market share is the speed at which they deliver their applications and data. The philosophy is pretty simple really: when users have to wait for something, they either get distracted (in which case they forget about you) or frustrated (in which case they’re mad at you).

The bottom line is performance matters. That’s why performance testing is vital to any product and company. 

1. Steve Souders’ Blog

Steve Souders is the head performance engineer at Google and has literally written THE book on web performance. I’ve been on Steve’s RSS feed now for a few years and the lessons and data he presents really seem to stick with me long after I’ve read the post.

2. dotTrace

The obvious use case for profilers like dotTrace is to profile your application when you’ve noticed that something is running much slower than you think it should.  However, it’s also good to simply profile your application from time-to-time and see if anything sticks out or has changed significantly. Maybe you’ll realize that one of your arrays would be better suited as a dictionary, or that you’re duplicating efforts in some parts of your code.

3. NUnit

At SoftArtisans we use NUnit a LOT.  Performance testing is just a small part of that, but by testing performance with NUnit every time we build, we can get a lot of interesting data that shows how performance related to different tasks has changed over time. This gives us the ability to quickly detect and address changes that might have dramatic effects on performance.

4. Math.Net Numerics and Excel 

For sifting through and analyzing all of our performance data that’s generated by NUnit, I look towards the Math.Net Numerics library. It’s got a bunch of great statistics functionality built in with which I can slice up the data.  I then usually throw the end result into Excel (using OfficeWriter of course) for visualization of the data and generating nice charts that I can play around with.

Those are just a few of the resources available to you when you’re beginning your performance testing. I’d love to hear what others are using or have found useful in the past. Let me know in the comments section.

Meteor Showers: 10 Meteor-Powered Apps that Rock

/
 

While Nick predicts that in a few years, Meteor will be a lingua fraca, currently, it's more like Elvish or cryptophasia--that is, used by solo developers for personal projects and used by teams for prototyping (more on that later). That said, the number of public Meteor apps is growing, and their foci are myriad. Below, I've listed ten I find interesting. You can browse through a great many more at madewithmeteor.com, and, as always, if there are some I've missed, please let me know in the comments!

1) JSpot

  • What: An algorithmically-driven dating/friendship/casual encounters app for Jews (and the goyim who love 'em). 
  • Made by: Ted Blackman & Elissa Shevinsky
  • State: public beta

2) Share with 911

  • What: An app that enables real-time communication between schools in lock-down and first responders. Communication formats include text and email broadcasts, check-ins, lost & found information, and an aggregated live stream. 
  • Made by: Erik Endress & Adrian Lanning 
  • State: limited beta

3) Greenqloud

5) Wondergraphs

6) BookTrad.es

  • What: Used academic book marketplace. Currently Booktrade.es only handles the postings; exchange is handled by buyer and seller. 
  • Made by: Sam Verhasselt
  • State: limited beta 

7) EtherPOS

  • What: A point of sale system that supports multiple regions, locations, registers, users, and products. 
  • Made by: Stephen Cannon
  • State: limited beta

8) Pegleg

  • What: Crowdsourced full-length movies from Youtube.
  • Made by: Mina Mikhail
  • State: public

9) Subtitles

  • What: Lets you create caption files for video projects. You can transcribe audio while the video is playing, or import an SRT file. 
  • Made by: Ben McMahen 
  • State: public

10) Gander

  • What: An email app that intelligently sorts incoming messages based on your correspondence history.
  • Made by: Us!
  • State: limited beta

Meteor Resources

/

When we started prototyping Gander using Meteor's open source web app platform, we weren't sure the Shiny New Thing with its CSS/jQuery/node.js/Mongodb stack and symmetric real-time data model had legs capable of carrying our email app into the real world. Our relationship in the two months since hasn't always been angels and butterflies, but it has been fruitful--enough so that we're (for now) planning on keeping it for Gander's live version (still in closed beta, but you can sign up for the waitlist here). 

The following list contains information on Meteor's features, third-party tools, Q&As, things to watch out for, events, and best-practices. They've proven useful to myself and our team; hopefully they are to you as well. If you have further questions, though, don't hesitate to ask them in the comments, on twitter, or on Stack Overflow.  

General Links

Meteor Main Page

  • This includes links to the documentation, lots of other resources, Stack Exchange tags, etc.
  • Worth following on twitter http://twitter.com/meteorjs

Matt DeBergalis talk at Realtime 2012

  • Matt describes an overall view of Meteor and the underlying DDP protocol.

David Greenspan introduces Spark 

  • Demo and brief explanation of reactivity, Live HTML and templates

Debergalis Oct 2012

  • Introduction to Meteor, including authentication and reconnection demos

Tom Coleman's Github Page

https://github.com/oortcloud

  • Includes some potentially useful tools for packaging Meteor, either to include new packages, or to deploy to Heroku instead of meteor.com

EventedMind training videos

  • This guy has produced an awesome set of training videos to learn Meteor. This will be very valuable for newbies to Meteor. The one on DDP is especially useful. Great stuff.

http://andrewscala.com/meteor/

  • Nice simple tutorial for Meteor. 

How to debug Meteor server-side. Client side debugging is typically done using the Chrome Developer Tools. See also https://github.com/meteor/meteor/pull/412

Test Driven Development in Meteor pretty comprehensive SO question about using TDD in Meteor

How Does Reactivity Work Behind the Scenes  another pretty comprehensive SO answer explaining how Meteor's reactivity works.

Some Things to be Aware Of

Minimongo Restrictions

Since data is replicated on the client in 'minimongo', an in-memory copy of the data,  there are some restrictions in the client:
  • The MongoDB _id default cannot be used because it is a BSON object for efficiency. Instead a straight text _id must be used like an RFC4122 UUID
  • Indexes are not available client side. Since minimongo is entirely in memory, this should still be fast, but don't send down 100,000 items and expect fast lookups.
  • Since entire collections are copied, there is no point is sorting server-side. For display purposes, sorting has to be done on the client.
  • Minimongo currently does not allow sorting on subkeys (ie. obj.headers.Subject) (This restriction was removed in Meteor 0.5.3 and later).
  • Minimongo supports only String, Number, Boolean, Array and Object types (ref). Binary dates must be converted to strings.

Meteor uses Fibers, not Node.js Async

Meteor uses fibers, which is debatable. In some ways, it makes it easier to program server-side code. OTOH, this is inconsistent with the majority of node.js Node Packaged Modules. It isn't clear how to mix and match these two different synchronization approaches in a single Meteor app. It also isn't clearly documented how to use fibers in a custom package to avoid blocking and other messiness. Further reading: TechCrunch and the sparse Meteor docs, and this performance related SO question (see also the odd comment from Tom Wijsman at the bottom), Quora.

There's a pretty helpful gist that demonstrates a number of different techniques for handling async behavior and node integration with Meteor.

Mongo Access is Serialized and Synchronous

When looking at the packages/mongo-livedata package, it appears that Meteor puts all Mongo in a fiber, which serializes it. It also uses Mongo's safe mode which is synchronous. This leads to a simpler, more reliable implementation. However, this model will introduce scaling problems under higher loads. Mongo supports asynchronous writes, with callbacks, and as well multiple concurrent calls. It is typically used that way in node.js deployments. The Meteor team will likely have to do some non-trivial rewriting of their driver for higher performance.

Publishing User Defined Fields is Poorly Documented

If you add fields to the user's collection, retrieving them later in the app is poorly documented. On the server, follow the documentation:

Meteor.publish("userData", function () {
 return Meteor.users.find({_id: this.userId}, {fields: {'other': 1, 'things': 1}}); 
});

On the client, subscribe to this collection:

Meteor.subscribe("userData");

Now the new fields will appear in Meteor.users, as in:

Meteor.users.findOne()['other'];
Meteor.users.findOne()['things']; 

So even though the subscription is named differently, the data still shows up in the users collection. If the client does not subscribe to this new collection, the additional fields will not be available. (I had to file a bug report to understand this).

Reactivity is not Guaranteed

We have found two places where Meteor's reactive context does not work as advertised.

  • The subscription callback for completeness. This callback will be invoked before the collection is completely loaded on the client. According to a recent SO question, DeBergalis contradicted the documentation stating "subscribe() takes a callback that will run when the initial [emphasis mine] set of documents are on the client." It seems that the publisher will call OnComplete, triggering the client-side callback, prior to the low level shipping of the full collection. Bottom line: it is non-deterministic to know if the collection is fully populated. UseMeteor.call to invoke something guaranteed, synchronously. 
  • Autorun will not execute when some of the dependent Session variables or collections change.

While we don't know for certain, it seems like reactivity breaks down when there are multiple dependencies. If one reaction changes something that should set off a second reaction, the second reaction does not fire consistently.

As a workaround for the subscription callback, it is always possible to use a separate Meteor.call to the server. The server method could return an aggregate object. When the client Meteor.call callback is invoked, the full results are present in the callback. So this is still asynchronous processing but the results are definitely on the client when the callback is invoked.

Synchronous Usage of Meteor.call Does Not Return Results

When using Meteor.call in the synchronous form, this is just a stub and there will be no return value from the server. Instead you have to set up an asynchronous call to capture the correct result.

Spurious Error Message Due to Mal-Formed HTML in a Template

It took a couple of days to track down a bug in Beaver. The browser's page would crash hard, where even a hard reload was not working consistently. In the browser console, there was a very long stack trace starting with:

Exception from Meteor.flush: Error: An invalid or illegal character was specified, such as in an XML name. at Function.Spark.Patcher.copyAttributes

The problem was due to missing a closing '>' in the HTML of the template. Meteor crashes hard if the HTML is mal-formed in a template. Unfortunately the HTML checking tools will not work on templates, especially partials so this must be done manually and carefully.

<td><input type="checkbox" id="show_subcategories"

 {{#if preferences.show_subcategories}}

 checked="true"

 {{/if}}

 >        <---- This was missing

</td>

No Deny without Allow

Meteor allows deny and allow methods in order to implement security on Insert, Update and Delete operations. While it is possible to have allow methods without deny, it is not possible to have deny without allow. If a given collection has a single deny method, it must have at least one allow method. This is buried in the docs as:

"If you never set up any allow rules on a collection then all client writes to the collection will be denied, and it will only be possible to write to the collection from server-side code."

Open Questions

None at this time.

Closed Questions

What storage limits are there to databases stored on meteor.com?

  • Hosting on meteor.com does not currently have any data caps, but it also has no SLA or guarantees =). If someone abuses the hosting service with unreasonable data or bandwidth usage, we may very well add data caps, but we haven't had a problem yet.

How often does the client do a full refresh? SO claims every 10 ten seconds. T/F?

  • True, but misleading. 10 seconds is the time it takes the server to notice documents added directly to the mongo database by an external process. When clients write to the database through Meteor, other connected clients see it immediately, not 10 seconds later.

Is packaged Mongo 32 or 64 bit?

  • MacOS and Linux x86_64 ship with 64 bit mongo. You'll only get a 32 bit mongo if you are running on a 32 bit linux machine. meteor.com is running 64 bit mongo (hosted by MongoHQ).

Does Meteor play nice with Mongo's Auth package, or does it require no Auth?

  • Meteor's auth works at a level above the database. It does not integrate directly with Mongo's auth.

What ports are required to be opened on a Meteor server besides standard 80/443?

  • Meteor listens for incoming HTTP requests on whatever port you tell it (3000 by default in local development, 80 by default when run via the meteor bundle command). The meteor bundle does not handle HTTPS, that is a feature of the meteor.com hosting. The meteor development mode runner takes an addition internal port and runs mongod which takes 2 more ports. But these ports are not used externally, and should not be open.

According to SO, "$ meteor mongo -U" is valid for only one minute. What's a more reliable way to determine this persistently?

  • That's correct. The one minute thing is a feature, designed to limit the risk of your database being attacked. If you want a permanent URL, though, you can work around this. Your deployed server is passed a URL to a mongo endpoint as the "MONGO_URL" environment variable. You can print 'process.env.MONGO_URL' in server code to see it. Be careful though, because we don't expose a good way to change your database password short of deleting the app and re-deploying.

Can regular Mongo monitoring services be used against Meteor?

  • In practice, it doesn't matter. Any serious application should be using a stable Mongo database hosted elsewhere, like MongoHQ. When deploying to meteor.com, Meteor requires hardcoding credentials in a local copy of Meteor for accessing the MongoHQ instance. Long term, it is preferable to run a Meteor app elsewhere like AWS or Heroku, rather than meteor.com.

Related Projects

Meteor either encapsulates or plays nice with

  • Handlebars (HTML templating)
  • jQuery client JavaScript library for eventing
  • node.js server Javascript execution environment v0.8
  • bootstrap the scaffolding / column layout manager used by Mahogany

The partial list of third party packages for Meteor can be found at https://atmosphere.meteor.com/ (DW: doesn't the atmosphere cause meteors to burn up and vaporize?)

Alternatives to Meteor

See Also

Meteor Google+ which includes links to a bunch more talks.

A Critique of Meteor

/

[Image via Meteor]

We're all pretty excited about playing with Meteor. There are number of interesting things about the technology that allude to great promise. Plus, it is a Shiny New Thing, with lots of venture backing so it seems like the next big thing.

However, even the Shiniest of New Things will have some unvarnished spots. Here are Meteor's:

1. Steep Learning Curve

Meteor relies today on a series of underlying technologies that each in turn have significant learning curves. MongoDB is growing in popularity but not really mainstream. jQuery, node.js, CSS are all web technologies that have been around, but have never been bundled quite the way Meteor does things. Additionally, it takes awhile to wrap a new developer's head around Meteor's symmetric realtime data model. While there is the notion of the client calling the server in a traditional way, that's more for specific actions rather than the typical usage model.

2. Application Spread Across Multiple Tightly Coupled Files

There are various Meteor parts that must in perfect alignment for everything to work well, due to symmetric nature of the data. A change in schema requires changes in the client, server and template code, all of which must be wired-up by hand. Since there is no checking by the tools, any misspellings are ignored and do not produce error messages. Instead, it simply doesn't work, with no clue as to the cause, except perhaps a problem in the Chrome JavaScript console window. This also makes merging parallel development efforts potentially problematic. Which leads me to unvarnished spot number 3:

3. Debugging

Server-side debugging is a real problem. Some people have managed to cobble together a debugger, but it looks painful. Thank goodness for Chrome's developer tools. They are effectively the client side debugger for Meteor. A Meteor developer will spend a lot of time in the Inspector.

4. Excessive Server Round Trips

It appears that the client is round-tripping to the server every 10 seconds. On a mobile device, or over a slow connection, where the locally cached symmetric model would be most useful, utility is questionable. We have not run low-bandwidth tests yet. We'll see. In the meantime, I can't leave a mobile browser window open to a Meteor site as it consumes a lot of resources.

5. Maintenance Burden with Complexity.

As the application grows in complexity and scope, it may be very hard to maintain. All classes and templates are in a single high level namespace so collisions could occur without attention.

It's not all bad, is it?

No - the symmetric model definitely has multiple advantages over the alternatives or writing our own. Meteor is fun for prototyping as changes are effortlessly immediately visible. Whether it is viable in a production environment remains to be seen. The model of realtime rich client web interfaces certainly seems interesting. It's been attempted many times with client side plugins, Flash being the most popular example. Meteor's approach has potential, but lots of hurdles still to cross.

 

How to Embed User Experience throughout a Small Company

/

[Image via Heritage of Japan]

I've always found it weird that so many companies consider "user experience" be solely the domain of UX engineers. Shouldn't everyone be pushing the experience further with their own strengths? Although Riparian Data is still constantly reworking its own culture in relation to user experience, here are a few things we've tried that have helped to embed UX throughout our organization:

1) Get everyone in the organization to see the value of UX-centered design

When people in an organization truly buy into the premise that building better experiences for users will increase the organization's chance for success, the UX work really can begin. Changing the culture to make it driven by people's experiences with your product tends to be diffcult, though luckily for me, some substantial work has already been done. The emphasis on design these days makes convincing those holding traditionally non-design-oriented positions of the user experience's importance a bit easier (also, previous generation of UX and IAers have made huge strides here). It seems like an easy premise to accept and pretty common-sensical, but what tends to be lost is allocating resources to user research and taking into account the findings.

2) Know that  product metrics are rooted in users

If a majority of users who played with our product wouldn't recommend the product to someone, we go back to the drawing board and iterate more fundamentally. If we do well on such user-centered metrics, we iterate on the other observations that we made (difficulties with the interface and other opportunities for growth). Then, once we have a large user base, we can do more quantitative analysis alongside the qualitative testing. By tying product direction directly into user research, people kept thinking and asking about what the user experience would be and what unmet needs we would meet.

3) Start a semi-parallel prototype upstream to determine strategy

Because of the makeup of our team, I am usually able to team up with a developer to make interactive prototypes ahead of the actual, scalable implementation. This works for us since we don't have a fully-developed product yet and are still defining a lot of functionality. Plus, some of our devs love the front-end work (especially as a respite from best programming practices, and the chance to hack on something really quickly to get to testers). Working with a developer means the prototyping goes fastter, and we're able to really collaborate to make something even better. Sometimes an idea was much better in wireframes than actually coded up, and we're able to change it right away. We can validate functionality and whether the design aligns with users' needs before we have to actually implement a more robust back-end for it.

These sort of quick prototypes also create a more direct, tangible connection to what's in the pipeline in the next few weeks. Coworkers feel more connected to people, and understand what's being tested because those are the only things that are coded up. It allows for iteration before production-level code and to give some breathing room for designs. This, however, does necessitate constant communication between the groups, and not becoming a silo of doing quick and dirty prototypes without feeding into the broader product. We're still working out the communication hurdles here a little, but with a bit more elbow grease from my end, we can make this work.

4) Develop an internal lexicon

Like how we develop a lexicon for users, it's super helpful to develop a lexicon internally. It builds a sense of community, as well as a common understanding of what you're talking about. We probably still need to work on this. This can (and should) be different from the lexicon that you use with users, because we're all different people. Usually, developers have different mental models from your user base. Maybe people won't like this, and I'll come under fire for it, but I feel comfortable with using Lorem Ipsum with internal mockups with the developers I'm working with, because we're also constantly talking and reworking the prototypes. It saves me time from finding and making content. Of course, this changes a bit when working with users, but we also develop a test account that's more aligned with most of our target users.

5) Use and update personas

On occasion, I'll find a user behavior trend that I didn't get from the more upfront user research, and will update the personas accordingly. When we started working on Gander, our personas were more general than they are today, as our problem space wasn't as well-defined. That being said, I could be better at embedding personas in our day-to-day conversation, and it's something to constantly work on.

But something cool that happened when our QA person, Mike, started: he had this great idea to walk through the product with each persona when trying to find problems. My findings and recommendations also get tied to personas.

6) Tie marketing lead generation with potential testers

Though I'm pretty sure Claire is sick of finding testers, the generation of leads from marketing helps with finding some testers. Since we don't have a launched product and are a bit lower budget on the user research, the outreach Claire has done has been super helpful. Mind you, it's a bit confusing between people who just want to beta test your product versus people who are willing to sit through a moderated session. And you certainly don't want to blast everyone who's interested in your product constantly about testing it. But, along with recruiting testers, mining the marketing leads and data also helps illuminate what appeals to people and what their current behavior is like.

7) Open sessions to remote observers

It's nice to have observers, both to build empathy with the testers and to see what you've built in use. Our awesome QA guy stops in often and it helps him to be able to put himself in the mindset of actual people. User testing sessions are open to the team as long as the participant knows and consents to the observation.

8) Make sure to test the biggest assumptions first

Especially in earlier stage designs, when strategy is informed strongly by user tests, it's easy to fall into task analysis of smaller things. The first thing that we tested was the basic assumption that it meets user needs. These require far more open-ended and exploratory sessions. The smaller interactions, of course, are noted and worked over in a more exploratory mode. I tend to do more structured tasks when our design is more firmly rooted. This strongly ties into Lean UX, and also informs what prototype to make (what to include, and what to exclude).

9) Advocate for and stick with a collaborative culture

Thankfully, there was already a strong collaborative culture at Riparian when I started. One thing that can help foster this is to share ownership of ideas and to try to exterminate "my idea" or "my design." Although it can be a little irksome when someone offers sketches at first, I take a breath, and try to dig at why-- much like understanding our users-- to incorporate feedback from these stakeholders as well. It might not be from a user perspective, but it comes from a relevant perspective to potentially pull a core concept out of. It's also super helpful as the only UXer to make sure I didn't miss something (which happens in one-week iterations). I'm a firm believer that the more you bounce and shape products with people, the better it gets.

These were some things that I've found helpful so far, and will continue to add to the list as I continue to try to improve how I do things. Let me know below if you have any suggestions!

Behind the Data: Ramon Armen

/

Our Behind the Data series aims to give you deeper insight into the minds and personalities of the developers, marketers, and technical support engineers who make up this eclectic, close-knit group. At work, we craft everything from machine learning algorithms to responsive web applications, race against the machine during hackathons, build architecturally sound beer towers during retros, and paddle down the Charles during the warmer months. And outside work? Outside work, things get all shook up. This week's proof: Ramon Armen, aka our head of R&D, aka psuedo-Canuck, aka spelunker extraordinaire.  1. What do you do? I run the R&D department at Riparian Data. So I make sure projects are going smoothly, synthesize user feedback, do development, help with any technical decisions, and try to find more people for the team (like you maybe?). Outside of work I explore caves, bike, build Theremins, brew beer, ballroom dance, and do SCIENCE, amongst other things. I also try to breathe regularly.  Big fan of breathing.

2. What are you listening to right now? The air conditioning fan. Usually when I start listening to music at work, I immediately need to talk to somebody about something.

3. If you could build any app, what would it be and why? I’m already building one – an app that makes it easier to deal with large volumes of email.  Interested? Outside of that, maybe an app that automatically adjusts settings at home or in the office based on resident behavior, to optimize the building for energy efficiency with minimal manual work required. 4. When you were 5 what did you want to be and why? An astronaut. Not only to you get to ride to space on top of a giant fireball, you might also get to go to another planet and explore places nobody has ever been before. 5. What is your favorite tech blog and why? My favorite blog that is related to technology is Centauri Dreams. But it’s not computer tech. My favorite computer tech site that’s related to blogs is Hacker News. But it’s not a blog. If you merge the two into a single entity, maybe you’d have my favorite tech blog?

6. Fill in the blank. Contrary to popular belief I  ________. Did not grow up in Canada, despite apparently saying “aboot.” 7.

Describe your perfect Saturday afternoon in 10 words or less. Finding new cave passage while having stimulating discussions with friends.

8. When did you last laugh? Watching The Daily Show.  Following politics gives me a laughably desperate sense of despair.

9. What is your worst nightmare? Being injected with a paralytic and forced to watch reality TV while I wither away alone, a shell of my former self.

10. What 3 things would you bring to a deserted island? A pen, a pointy but sturdy log, and a large sheet.  I will let you guess about what I would do with such items.

11. Who would play you in a movie of your life? If somebody is making a movie of my life, it probably means I’m either dead, in which case it doesn’t matter to me but it would be cool to have a corpse playing me, or I’m vain enough to let them make a movie about my life, in which case I would probably want the acting glory all for myself.

12. What is the last thing you bought? Probably a sandwich. But besides food, some new hiking shoes.

13. What are your biggest pet peeves? Any sort of hypocrisy.  And people who don’t turn on their blinker until after they have stopped and/or turned. And people who argue for any sort of grammatical pet peeve on internet forums.

14. Name something that makes you smile. Other smiling people

15. If you could teach a college course what would it be? I would teach a course on how to teach science to kids. Because there are so many cool things to play with and learn aboot, and yet most curriculums I’ve seen do a horrible job of getting that across, to the great detriment of our societal understanding of reality.

Celebrate Surviving Sandy with Board Games and Beer!

/

Whew. You guys--we made it! I think. I mean, I'm writing this from a verrrry crowded temporary coworking space in Midtown west, which is an area I normally avoid like the plague. And it took me, oh, just 2.5 hours to get here. But, n'importe quoi. Sandy is gone, and I can think of no better way to celebrate than to carry on with the same activities that got us through her: board games, beer, and pizza.

As Obama and the Allman Brothers have said, "I'm going to keep on keeping on." And so are we. And so, we hope, are you.

Ze deets:

  • What: Pandemic. Settlers of Catan. Dogfish Head. Pepperonis. Peppers. Poker faces.
  • Where: SA/RD HQ, 3 Brook Street, Watertown, MA, 02474
  • When:  Wednesday, November 7th, from 6-10pm.
  • How much: Free! We are the land of beer and honey.
  • RSVP here.

And, if you'd like to know more about our group of java-spitting, Rails-riding buccaneers, check out our employee profiles. Like what you see? Come work with us!

Strange Loop Emerging Languages Camp Recap: Julia, Grace, Rust, and a Bandicoot

/

strangeloop conference[Image via The Julia Language]

This year, the Emerging Languages Camp was the pre-conference for Strange Loop, the developer conference held annually in St. Louis. Recaps for Strange Loop are forthcoming, but here are my notes from the precon sessions.

Transpiling into Javascript

  • Languages: Javascript, coffeescript
  • Speaker: Jeremy Ashkenas (t gh), Interactive News at the New York Times, creator of coffeescript + backbone.js

This whole "compiling" into another language is pretty new, except for C++ which started out by compiling into c code. Recently though, we've gotten these strong solid VMs to build upon; it's just not worth it to build your own VM.

One way to transpile is is to start by switching one simple symbol, then expand out into new keywords and flow controls.

There's a site altjs.org that lists a whole bunch of languages that compile into js.

Be deliberate about where you deviate and what you preserve – this probably doesn't apply too much to porting an existing language to a different vm though.

Not everything can be straight transpiled. For example adding negative indices to something running on js would require changing all array accesses to be: array[x<0 ? array.length-x : x], which would be pretty unpleasant. If you can figure out at compile time though, you're good to go.

The reason that coffeescript compiles readable code is that it makes you feel safe since you read/debug it.

All languages have this duality where they're trying to be readable to both computers and developers. Transpilers can help us move towards the human-readable side of things.

 

Bandicoot: Code Reuse for the Relational Model

  • Language: Bandicoot
  • Speakers: Ostap Cherkashin (t gh), Bandicoot co-founder and software enginner, and Julius Chrobak (t), Bandicoot co-founder and database engineer

Bandicoot is a new language for dealing with relational algebra and sets. One of the big things they talked about was reusability. It didn't seem to me though that this was any better than ActiveRecord's lazy query syntax or LYNQ, plus you have to use a different language.

 

Elm: Making the Web Functional

  • Language: Elm
  • Speaker: Evan Czaplicki (gh), Elm creator

Elm is a pretty sweet language for doing web development. The most impressive part is the declarative/functional way that it does gui. It has built in markdown support so if you want to throw some text in, you can format it that way.

As a functional language, everything is immutable; however, if you have something like the mouse position, or time, which changes, it's stored in what is called a signal. You then can take any of your normal functions, and use a method called lift to convert that into a signal, with another signal, so you just pass the mouse position or time into your sweet function that you originally used to draw everything, and your app now responds to mouse position. Check out the interactive editor or any of the other examples. They're all simple, and easy. This is a serious thing: http://en.wikipedia.org/wiki/Reactive_programming, but the first time that I've ever really seen reactive programming that blew the other methods out of the water.

The Reemergence of Datalog

  • Language: Datalog
  • Speaker: Michael Fogus (t b), Clojure and Clojurescript contributor, co-author of The Joy of Clojure

Datalog is a pretty cool logic and query language that works with Datomic and Datalus. There's also Cascalog which is a map/reduce version for clojure. Bacwn adds negation to datalog. It looks a lot like sparql to me, but sadly that's directed at RDF, whereas datalog is pointed to more generic types of databases.

Roy

  • Language: Roy
  • Speaker: Brian McKenna (t b), programmer at Precogio, Roy creator

I would describe Roy as Haskell for JS. It has haskell style static typing which is pretty sweet. There's no type output in the compiled JS.  Sidenote: you can have haskell defer type errors until runtime.

Julia: A Fast, Dynamic Language for Technical Computing

  • Language: Julia
  • Speaker: Stefan Karpinski (t gh), Researcher at MIT, Julia co-creator

Julia is a fast, dynamic language for technical computing. Features include:

    • a unified type system (i.e. no int vs Integer like in Java)
    • efficient use of numerical arrays
    • parametric types
    • no inheritance from concrete types
    • operator overloading for everything
    • tries to avoid being built on c/fortran libraries
    • Uses type specialization by the compiler
    • fast
    • did I mention fast

Stefan did a demo wherein he created a parameterized type: ModInt{N} where N is a value parameter (e.g. 11), then implemented + and * in a few lines, and then created a random matrix of ModInt{11}, and raised that to 100,000, and it was instant. There was a chart of performance for other technical computing languages: fortran, R, python, matlab, etc. And we were all like, oh yeah Julia is pretty fast, comparable to fortran and much faster than the rest, and then he pointed out that the graph was logarithmic scale, and all our jaws dropped.

Rust

  • Language: Rust
  • Speaker: Dave Herman (t b), Programming Language Propeller Head at Mozilla

Much like how Google is making Go, Mozilla is creating their own systems language: Rust. They have a haiku:

a systems language pursuing the trifecta safe, concurrent, fast

They believe abstractions should not cost you unless necessary, e.g. if you write a method, and it can be inlined for better performance, they will do that.

They described it as the love child of C++ (for fast), Erlang (for concurrent) and Haskell/OCaml (for safe).

It has multiple types of pointers (and I thought 1 was bad enough): Stack Based, Memory managed/shared, one-owner/unique. The later is necessary for sharing data between different tasks (threads). Each task has its own stack and its own heap (perfect for different tabs in a browser). They have parametric types, that are implemented via code duplication like c++. There's an ARC container that requires deeply immutable type, but there needs to be locking on the reference counter.

Further resources: smallcultfollowing.com/babysteps, pcwalton.github.com

Grace: An Open-Source Educational OO Language

  • Language: Grace
  • Speaker: Michael Homer (gh b), PhD student at the Victoria University of Wellington

Grace (unrelated to our intern) is an open source OO language for education, ideally CS1, CS2 and second year students. It doesn't have any incantations, like how in the first class using java you have to say: "Don't worry about that public static void main" part, because you don't really want to explain scoping and static vs. instance. "Things that are more important come closer to the start of the line" So grace does: var previous:Number := 3.

It allows multipart method names, which I thought was pretty cool: 5.between(3)and(8)

The control structures are just multipart methods, so you can write your own if or while.

There's no null, which was the main question, as to why that was left out, and basically the response was that null causes headaches for new students and experienced programmers alike. This made me wonder why every object is optionally null in languages like Java or C#. It seems pretty rare that I want to return or deal with null, why not make nullable be explicit, like with C# structs.

Elixer: Modern Programming for the Erlang VM

  • Language: Elixir
  • Speaker: Jose Valim (t gh), Lead Developer at PlataformaTec, member of the Ruby on Rails Team, author of Crafting Rails Applications

Erlang VM is groovy and super concurrent, so Elixir was built on top of it, adding nice macros and protocol things that are not in erlang.

Visi: Cultured and Distributed

  • Language: Visi
  • Speaker: David Pollack (t b), Telegr.am CEO, Visi and Lift contributor, author of Beginning Scala

David wrote the Mesa and Integer spreadsheet apps. Spreadsheets are for non-programmers, and visi is supposed to be like that. All the code is markdown, and the visi github pages are actually the tests for visi (supposedly). It's all very light-table-y, where you can just say that something is a source (e.g. x or y or "revenue") and sinks (e.g. "profit") then you have a form on the left with all your sources, and list on your right with all the sinks.

The Emperor May Have Mini-Bros

/

Tech startups are known for their fun perks and welcoming environments--unless you're a female jonesing to code. Christina Nguyen calls for an end to this culture of microaggressions.

In response to Claire's post stating there is a lack of sexist brogrammers in tech field, I wanted to expand on the sexism there is in the tech world. Brogrammers are far more overt, but the enemy of females in tech startups lies in the subtle microaggressions-- especially in conjunction with startups, where entrepredouches run rampant. I have been quite fortunate (or perhaps unconsciously selecting) to work and learn in environments that have gender parity. Riparian Data has been an excellent place to work, and gender has never been the basis of any unencouraging comments or action. That said, myself and a number of my friends have had various problems with other individuals in tech, or even an institutional culture that isn't welcoming to women (having a culture in discussing gender issues is key). The rest of this article examines these examples. We should strive to do more, and not be complacent with the work that has already been made.

Let's get started with the tech startup environment in general. Because technology cred is mostly merit-based, it seems less antagonistic and uses many womens’ collaborative nature in the workplace. Whether or not women learned to prefer collaborating and systemic thinking, women tend to be more comfortable in these sort of environments. Sure there are a good number of females in marketing, business development, sales, and the less tech-orientated dimensions of startups, from my visiting of startups in the San Francisco area. All these assertions (generally merit-based, collaborative environment and have a number of females in non-tech fields) does not mean it is an equal playing field.

The fact is few startups are founded by women and few have women in top tech executive positions (though apparently women make up 9% instead of a mere 3.5% of the Forbes 1000, so I guess, yay?). So, let's just say there are few role models. The female founders that are coming out of the woodwork today are mostly in consumer fields like fashion, education, and media. We might be unconsciously sending a message that women can only succeed in these verticals.

There also are insidious beliefs that women are less competent, have less risk tolerance, that we will bear the brunt of raising a family, and these can't work in concert with starting a successful startup -- despite the fact that women are just as competent and women executives don’t leave the workforce more than men. Male founders will have to figure out the balance of families anyway, so why do women founders need to field these questions regarding family? The Kauffman Foundation study of 549 founders of successful businesses (that were peer-identified) showed that nearly 60% had at least one child and were, on average, 40 years-old. We cannot discriminate on gender as to how family roles and plans play out. Reshaping societal expectations is going to be hard, but it will be worth doing.

So what's wrong with having women staying in the "softer" fields of a tech startup? From my observations, engineers assert superiority over other fields, and can be antagonistic towards women. The current trend is to start companies with founders with technical backgrounds, and it could be difficult to find cofounders or hires without one. There may also be an awful belief that these fields are lesser because women are in these fields (pay goes down when more females enter, and men then may leave thus perpetuating the spiral). In a softer vein, I watched female colleagues go into “softer” departments, with bystanders believing "she wasn't good enough to be a developer." There's almost a necessity to assert that one was good enough to be a developer, interviewed, got the position, and then chose to go into that “softer” department. Otherwise one’s worth is being the diversity hire and fulfilling "lesser" functions of a startup. Which is, frankly, stupid.

These functions are just as important and require a skill set and finesse that engineers might have a difficult time with. Some engineers naively believe they could do marketing or sales if they just put their minds to it. The danger is when they believe the opposite doesn't hold true, and further illustrates this wrongheaded feeling of superiority. Since women are situated primarily in these "softer" fields, the association between gender and competency begins. We can do more to strike down the preconceptions of these fields as binary male and female, which causes an us-vs-them mentality and creation of imaginary superiority.

How about women in engineering? If we are able to foster a diverse environment during the formative years and engineering education, perhaps we are able to also affect tech startup culture. Growing up, I have had a few people comment that I would never make it in engineering because I was a girl (despite the data that shows there is no difference in boys and girls’ aptitude for math), and that I should just go study humanities or "softer sciences." I couldn't wrap my head around people who knew me but decisively judged me as a whole gender. In fact, this perhaps incensed me further to prove them wrong. We've created lots of positive programs and changes around and since the time of my growing up, and the growing number of role models will help.

Having male privilege in engineering means that you don't have to defend why and how you're in an engineering program because your gender (despite the fact that most women in engineering are self-selecting, high-achieving individuals). As a male, your male colleagues treat you like a person: you don’t represent some idea of your entire gender; no one is condescending and defending you because of your gender; you are not consumed by trying to navigate boys trying to get into your pants and avoiding slut shaming. These are all less overt things that we now need to deal with and I'm so thankful for the strides that were made before me. There may be huger problems in other industries, but we're still not a place in engineering where gender isn't a striation that determines opportunity. The perception of engineering is discordant with women's gender identity. I believe that we need to foster persistence by helping women create support networks and allow for time to build confidence in their fit with engineering. We can also encourage girls to tinker at an earlier age (and stop with the binary gender!). Again, we’re still not there yet, and have a ways to go in improving cultures of engineering programs and workplace climates (a sample of 20% females with engineering degrees to 11% in the workplace is abysmal).

Where do we go from here? I'm not advocating for having gender quotas or the like for engineering.  Acknowledge your privilege, modify the subtle cues, and talk about it. Women, we can’t dismiss our colleagues who want to focus on family. For now, we play into the current culture until we are able to change it. And that comes with taking risks: start our own companies where we can create great atmospheres or just speaking out on your own experiences. I've just seen my female colleagues jump ship because they were being uncomfortably hit on. Some are not given enough power and responsibility to make a difference despite their competency to do so. Some feel forced to play male or female stereotypes depending on circumstance. And perhaps it's also the startup culture to feel we're so interconnected that we feel like we can't call people out on the subtle stuff. Because we don’t want to be cast as the girl fixated on the gender stuff, or making it a big deal about gender, or whining about it. But I think all we want is to be treated like equal human beings-- as citizens-- not a different class, not a representation of our entire gender.

On another note (though somewhat outside the scope of this long post)-- we also need to talk about gender as a spectrum and gender as an identity-- and to make sure transgendered people aren't also marginalized. And if we have to, explain to others that equality of opportunity for all demographics is necessary for society to progress. Without the perspectives of markedly different experiences (that tend to be formed by societal constructs of being brought up as a female/transgender/racial minority/disabled person/citizen of a developing country/etc), not only do we miss out on empathy and understanding of different verticals and markets, we lose out on utilizing the full potential of humanity. And for engineers who hate non-optimal solutions, doesn't that just hurt your soul?

Drinks to Code By

/

[Cross-posted from the Softartisans' Blog]

To celebrate the commencement of Oktoberfest  festivities in Munich this weekend (and because it’s always fun to think about alcohol), we’re bringing you a handy-dandy guide to see how your programming languages match up to your favorite drinks.It’s important to be prepared –  If all the programming languages were to suddenly turn into drinks, we should know what sort of world we’d be getting (besides a very drunk one)!

C – Tequila.  It gets the job done quickly, but using it isn’t usually that great of an experience.  Everybody uses it at some point, but not too many stick with it unless there’s no other choice, or they need it to do its thing fast.  And if you don’t use it exactly right, chances are things are going to go horribly wrong.

Ruby – Scotch.  You can use it for years, and still discover strange new undertones you hadn’t noticed before.  And then you add a dash of water and discover everything is changed.  But it’s easy to ruin the magic if you don’t know what you’re doing.

Python – Irish Whiskey.  When first starting, you have two main options – Bushmills or Jameson?  Python 2 or 3?  But once you choose one and go with it, it’s remarkably smooth.  There’s not a huge amount of subtlety, but it’s pretty easy to get into and use.

Java – Vodka.  It’s everywhere.  It can be used in pretty much any situation, and most people are willing to use it when it’s the most convenient thing on hand.  But excluding a group of really dedicated fans, it’s not generally people’s first choice.  It just doesn’t have the same elegance as some of the alternatives.  But it can be mixed with other things pretty easily, for those who can’t stand using it straight.

Scheme – Bourbon.  It’s not everyone’s cup of tea, but once you get used to it, it’s amazingly versatile.  It can be mixed or taken straight, and can be used in most environments.  Some people love it, some people can’t stand it.  It has some tantalizing similarities to other types of whiskey, and yet has its own very distinct character.

C# – Sake.  Hugely popular in one area.  There have been attempts to port it to other areas, but in most cases it’s still more of a quirky specialty item.   Although it looks a lot like vodka/Java, using it is remarkably smooth and feels much different.  When mixed with other things, it will either go incredibly well, or go terribly wrong.

Brainf*ck – Absinthe.  It really only exists because, well, why not?  It’s probably nothing like anything you’ve had before, and using it might be actively damaging your brain, but there’s something oddly fascinating about it that makes it hard to turn away.  Most people wouldn’t use it for serious business, but will give it a try for the novelty.

Javascript – Fermented apple juice.   Is it an alcoholic drink, or an accident?  Nobody knows.  But it’s there, so people use it.  It’s not quite at the same level of sophistication as the alternatives, but maybe if you give it a chance to grow a little more it will get there…

There you have it – picking a programming language is a lot like picking a drink. So choose wisely and enjoy your Oktoberfest weekend! But wait, there are so many languages you left out, you may say. Rather than me listing them all (because frankly, there are a ton), I’ll turn it over to you now.  What have I missed? What’s your favorite language, and what drink does it compare to?

Ruby Wranglers: How We Test the Code that Creates Our Java Processes

/

It is a truth universally acknowledged that a Rails app in possession of a good server must be in want of a Java process

The Ruby on Rails portion of Gander must, from time to time, create Java processes to do some of the backend work that keeps the email flowing.1 There are several easy ways to fork a process from Ruby, like ``, system, and methods in the Process and IO modules. These are great, but start to cause problems when you've got them scattered across a codebase. In particular, writing good tests around such code becomes an enormous headache.

There are several things we want to do when we test code that creates a Java process. We want to be able to write a test that verifies that the right arguments are passed to the process at the appropriate time. We want to write a test that uses the output of that process (usually data written to stdout) without actually creating the process, so that we can decouple testing the Ruby code from the Java code. Finally, we want to write integration tests that create new Java processes and then wait for them to finish, so that we can verify that our Rails app has the correct behavior while the process is running and after it's finished its work (e.g., populating a database). None of the built in methods for creating new processes allowed to do all of these things easily, so we made a, if I do say so myself, pretty slick abstraction layer to help us.

I've simplified the code slightly to remove some of the app-specific details, but here's the idea:

class Java
  def Java.run(detach, service, *argument, options)
    options.each_pair do |key, value|
      unless value.nil? or value == false
        arguments << "--#{key}"
        if value == true
          # do nothing
        else if value.is_a? String
          arguments << value
        else
          raise "The value for #{key} is neither a string nor boolean; value: #{value}"
        end
      end
    end
    dir = "#{Rails.root}/lib/"
    child_args = [{'RAILS_ENV' => Rails.env},
                  'java'
                  *(CONFIG['java_opts']),
                  '-jar', "#{service}.jar",
                  *(arguments)]
    if detach
      pid = Process.spawn(*child_args, :chdir => dir, [:out, :err] => ['java.log', 'a'])
      Process.detach(pid)
    else
      IO.popen([*child_args, {:chdir => dir, [:err] => [:out, :child]}]) do |io|
        io.read
      end
    end
  end
end

This gives us a single point to call into for all our Java needs, that we can easily stub out during tests to do whatever we want. Because run returns the pid for processes that it detaches from, we can also re-attach to the process during integration tests if we want to ensure that we wait around for the entire thing to finish. Some examples:

If we're writing a unit test for something that can spawn a Java process, we might see something like this (using RSpec mocks):

Java.stub(:run)
Java.should_receive(:run).with(true, 'sync', [], :user => 1234)
# or
Java.stub(:run)
Java.should_receive(:run).twice

That lets us check that we're passing the right arguments to the new process or calling it the right number of times, without actually running it. If we're writing a test that depends on the output of some Java code we might do

Java.should_receive(:run).with(any_args).and_return("9876")

So that we can test that the Ruby code does the right thing in response to any particular output from the Java code, without actually creating a new process. Finally, in integration tests we want to let the code actually create a full Java process and launch it. Most of the time the Ruby will detach from the Java process so that it can run in the background and, for example, sync a bunch of data into our database. In our tests we want to wait for the process to finish so that we can verify that our Rails code can work with the data it created properly. All we have to do is capture the pid when we call run and then rejoin the process:

thread = nil
Java.stub(:run) do |detach, service, *args, options|
# found this method name by putting a binding.pry here and ls Java
thread = Java.obfuscated_by_rspec_mocks__run(detach, service, *args, options)
end

# ... start the test, eventually calling something that invokes Java ...

thread.join

# ... the rest of the test happens after the Java code exits

With some help from the always-excellent pry we can gather the pid of the Java process from the internals of our Ruby code (which probably throws it away) and wait on it at the appropriate moment.

1: Why? Well, all the good libraries to work with things like PST files and ActiveSync protocols are in Java. Re-implementing all of that in Ruby is a non-trivial amount of work, and things like parsing large files and syncing a lot of mail need to happen in a background worker process anyway, so why not?

The Emperor Has No Bros

/

The rise of brogramming is the latest in a long line of reasons why more women aren't going into high-tech. One problem: brogrammers don't really exist.

magritte ceci n'est pas une pipe This morning, I started to write a blog post about brogamming. Specifically, it was going to be a lighthearted romp through some of the bro-iest programming jobs on the market. You know, the sort of positions that speak of keggers and crushing code and Vegas retreats and rockstars.

This is not that post.

It is not that post because, to put it simply, there was a dearth of resources. What I figured would be a half hour of keyword-based search on StackOverflow Careers turned out to require quite a bit more digging, most of it unfruitful. Sure, there were a few eye-rollers. Hipster’s perks package includes mustache trimming, a fixie, and $10,000 worth of PBR. Launchrock is seeking a code-slinging, bad-ass Front-End Developer/ Professional Partier (side note: how often do you see those titles side-by-side?). Dropbox has whiskey Friday’s. Twitch.tv employees get to play in Super Streetfighter 4 tournaments.

These descriptions are zany, sometimes silly, and yes, rife with language and objects that we (or at least I) tend to associate with the world of men: cursing, spirits, mustaches, video games.

But the association is, to cop an epithet from one job description, “bullshit.” I know this from firsthand experience. Half of my coworkers are women. Many of them like video games. One has been known to toss out the odd body part on our company blog. Everyone likes whiskey. No one has a mustache, but then again, none of our male developers do either.

That I a) set out sure I would find things that b) I couldn’t find, says something about me, and maybe, something about many women working in tech, which is: we let a few bad apples tell us the whole barrel’s against us. My apples were the booth babes at the SharePoint show. Tasneem Raja’s apple was Matt Von Horn’s “bikini shots and gangbang interviews” talk at SXSW.  Shanley Kane’s was Geeklist’s dancing-girl-in-a-tee-shirt video—and the vitriolic twitter exchange she had with Geeklist’s co-founders, Reuben Katz and Christian Sanz

Are there bad apples in the tech industry? Yep. There are bad apples in every industry. Taking one idiot’s misogynistic pureview to be the voice of a profession does us no favors. A few events caused reporter after reporter to rue the rise of a culture that, from what I can see, doesn’t really exist. Pull quotes like “With the phrase 'brogramming' on the rise (less male-geeks and more cool male testosterone-fuelled coders) women may be finding themselves even more alienated then they previously did” are a problem not only because they’re not true, but because they can become self-fulfilling.

It's our job, we women who work in or are thinking of working in high-tech, to do our research, to not let ourselves be convinced that tone skein of oppression does a sweater make. It's also our job, should we encounter it, to take that skein and stomp all over it. Boots make a bigger impression that yarn.

But honestly, our chances of encountering it: not that high. You know what did crop up often in the job descriptions? Nerf guns. Nerf guns, free food, transportation reimbursements, dogs, Cadillac healthcare plans, and generous vacation policies.

So ladies, if you want to work at an offbeat, nurturing, friendly, and accommodating environment, consider working at a tech startup.

Oh, and by the way, we’re hiring.

Trello Made Awesome Part III: From Backlog to Feature

/
trello verification

trello verification

We use Trello to manage development for Gander. Previously in this series we took a look at a a custom stylesheet and how we organize Trello for each sprint's work. Now we'll talk about how we manage stories before and after their sprint.

All stories begin their life in the backlog. The core of the backlog is a single list with a card for each story that's slated to get worked on at some point in the future. In our weekly grooming meetings we look at upcoming stories and assign story points to the ones that don't have them yet. We also take this time to come up with a "plan of action", a provisional list of tasks (stored as a checklist) for the story. This is a good exercise to get us all on the same page for how much work a story is, and saves time in planning.

We often find that we have a large number of stories grouped around a common end goal, like ActiveSync support. Especially when one of these groups is the most important thing to work on, it obscures everything else in the backlog list, which doesn't help anyone. When this happens we create a new list for the group, and create a placeholder card in the backlog that points to it. This way we can easily set the priority of the whole set of stories and see everything else that's going on.

When it comes time to add things to the sprint we pick the top several things from the backlog and create lists in the sprint board for them. Then we just move the backlog card directly into the sprint board and label it as a story. We take the plan of action and turn them all into task cards which automatically get added to the story's list. If we did our job right during grooming, then we're pretty much done. If there are more or different tasks that the story needs we make cards for those, and then the sprint begins.

Once a story is finished we move it off to the right side of the board. Like keeping done cards at the bottom of their list, this makes it easy to scan the board for what needs to get done. We also make a copy of the purple story card and move that into a verification board. This board has "To Verify" list that holds everything we've done that our product owner hasn't had a chance to do a final inspection on yet, a "Verifying" list that has the things he's currently looking at, and then a list for each sprint that we've completed for the things that were done and verified in that sprint.

The verification board also serves as a checklist of what we have and haven't deployed by using the last color, blue, to mark things that have been pushed to production. This makes it easy to write up change logs.

There we go. That's the entire story of how a card becomes a feature at Riparian Data.

Trello Made Awesome Part II: Lists, Cards, Labels

/
trello development

trello development

We use Trello to manage development for Gander. In Part I of this series we showed off a stylesheet that filled Trello with color. Now we'll talk about how we arrange our main development board to manage our sprints.

The baseline for using Trello to manage software seems to be the official Trello development board. Most other people I've talked to that use Trello do something similar. In this board, lists are for states (idea, in progress, done), cards are for stories (better search, copying a card) and labels are for types (bug, feature, infrastructure). We tried that, and hated it.[1]

Instead, we use lists for stories, cards for tasks, and labels for states. The rest of this post is the details about how that works, but if you just take that sentence and forget everything else, you'll get 90% of what we're doing.

When we plan a story into a sprint it gets a new list ("Frob all the widgets"). The first card in the list is labeled purple (Story) and holds all the information that covers the entire scope of the story: a more informative title, a description of why we need to frob the widgets, the acceptance criteria for when we can tell the widgets have been sufficiently frobbed, how many story points we've assigned to widget frobbing, etc. This card stays at the top of the list and doesn't change much once we create it.

Then we make cards for each task in the story. These all start out with no label (effectively "To Do"). The structure of these is pretty freeform. Some of them might have just a title if the task is simple enough ("Make a FrobController with stub `frob` and `unfrob` actions"). We commonly put checklists on testing tasks to list specific tests that we know ahead of time we need to cover, but it's not required. These cards tend to grow as people work on them. We'll have discussions in the comments, attach mockups or error screenshots to the card and whatever else we need.

Once we start working on a card, it gets a team member assigned to it, and labeled yellow (In Progress). This is when most of the action happens on the card. Once the work is done we turn the card orange (Needs Review) and start a code review. As long as everything goes smoothly, the card gets labeled green (Done) and the task is complete. We move the card to the bottom of the list so that it's always easy to find the tasks that need to get worked on (especially for stories with lots of tasks that grow a scrollbar). If something goes wrong and we can't work on a task it gets marked red (Blocked). Cards never turn red without also getting a comment explaining why they're blocked. If the cause is another task or story, the comment has a link to the card.

And that's about it. When we combine this with the stylesheet from last time, the health of a sprint becomes available at a glance. If the board is turning green, the sprint is going well. If too many things are red, we're being stopped from getting our work done, and need to fix that before doing anything else. Too much yellow means the team is trying to work on too many things at once, too much orange means we've been slacking on code reviews.

1: Which isn't to say that it's bad, or that anyone who organizes their board this way is doing it wrong, but just that it didn't work for us at all. If it works for you, great!

Trello Made Awesome Part I: Life in Technicolor

/
trello-development

trello-development

Our development team adopted Trello several months ago for managing our work on Gander. In this ongoing blog series, we'll talk about how we've made it work for us in several ways that, in our most humble of opinions, are even better than the default setup Trello gives you. Up first, some CSS.

Very nearly every card in our Trello board has a label on it. It's great that Trello lets us do this, but it becomes hard to distinguish the different colors, especially from across the room during standup, or after a long day of hacking on email. Enter Scott's custom stylesheet that turns a Trello board so colorful I've seen peacocks get jealous: (Install it with Stylish or Stylebot or your browser's equivalent.) Now, even with just a glance from across the room, it's easy to see what's going on. Which lists are green, which are red? The board is expressive enough that it even starts to take the place of a burndown chart. Instead of graphing how well we're doing, we can just eyeball the board: is it the right color for this far through the sprint?

We think everyone should be using Trello like this, no matter what else you do. It sounds like a tiny thing, and the first time you see a board full of blocks of color it seems jarring, but trust us, it's great.

(Scott's currently working on a v2 of this stylesheet that can display more than one color per card using some sweet CSS gradients. We promise to update this space when it's done.)

Behind the Data: Nick Martin

/

Our Behind the Data series aims to give you deeper insight into the minds and personalities of the developers, marketers, and technical support engineers who make up this eclectic, close-knit group. At work, we craft everything from Microsoft reporting APIs to mobile email applications, race against the machine during hackathons, build architecturally sound beer towers during retros, and paddle down the Charles during the warmer months. And outside work? Outside work, things get all shook up. Proof: one Nicholas Martin, (b | t) a "zen-lazy" programmer with intergalatic dreams, a fondness for tea-time manners, and a serious weakness for puns.  1. What do you do? I’m a software engineer at Riparian Data. I work on the full stack here, from setting up a new production server to rejiggering a Java application to become a library we can use for something else. Sometimes I make pretty web pages. We work in whatever language suits the task.

2. What are you listening to right now? Ratatat – Spanish Armada

3. If you could build any app, what would it be and why? I’ve been meaning to build the same app for like five years. It’s a site to house all the terrible puns I am subjected to on a near daily basis. I’d call it The Punisher *queue booing*.

4. When you were 5 what did you want to be and why? Astronaut, you get to go to SPAAAAAAACCCCEEEEE!

5. If you were a beer what would you be and why? I’d be an imperial stout. It’s kind of dark, but drink two and everything is funny.

6. What is your favorite tech blog and why? I’m a big fan of Hacker News, mostly for the delicious blend of tech and business news.

7. Fill in the blank. Contrary to popular belief I ________. I don’t really speak English. Perhaps that’s not contrary to popular opinion.

8. Describe your perfect Saturday afternoon in 10 words or less. Hammock, beach, shade, table service.

9. When did you last laugh? I last laughed less than an hour ago.

10. Describe your personal style in one word. Zen-lazy.

11. What is your worst nightmare? My worst nightmare is that recurring one with the giant spiders and mandatory extra high school.

12. If you were a superhero, what would be your superhero name/power? I would be Grokules, a barbarian berserker from the cold north with English tea-time manners! His special power is, uh, making sweet tea. If you tell him his tea is bad though, beware, for he shall become terribly upset, and the pouting will not end for hours. You will never hear the end.

13. What 3 things would you bring to a deserted island? I’d bring a Ka-bar, a safari hat, and sadly, a satphone, with internet. Hopefully I could find food and water.

14. Who would play you in a movie of your life? If lucky, I’d get the most interesting man in the world. It would be an art film, with random action scenes with motorcycles and explosions. Also, there would be much saving of puppies.

15. What is the last thing you bought? I bought a badger brush for my steel razor.

17. Name something that makes you smile. Doggies!

18. If you could teach a college course what would it be? I’m going to have to agree with Sean, software engineering in the real world should definitely be a class everyone takes. Writing code that merely works is a small part of the job. It’s all the processes around that writing that makes said code good. Sadly, many CS and even SE departments are teaching old methods, or not teaching it at all.