Sporadic Writings · Dan Baggott

The importance of re-evaluating habits

2016-08-26T00:00:00Z

A few minutes ago I was writing a unit test when I suddenly experienced a feeling that is most easily described as "joy". And in that flood of joy, I was reminded that it's a really good idea to periodically re-evaluate the habits of your life.

The concept of "habits" often gets a bad rap due to its frequent association with the word "bad" and the inflexibility that habits imply. Both of those are, in my opinion, unfortunate associations. I'm a strong believer in the idea that, with a bit ^[1] of self-awareness and focused attention, your habits can be curated and groomed so that the good ones are encouraged to flourish and the bad ones can be gently weeded out. You can even consciously plant the seeds for new habits.

Back to unit tests. While I do have a deep appreciation of unit testing that perhaps even merits an "I ♥ Unit Tests" sticker, my happiness as it relates to testing is usually most profoundly felt when I'm doing one of two things:

refactoring or otherwise modifying pre-existing code and the code is accompanied by solid test coverage
I'm writing tricky code with lots of edge cases and I've written my tests first and it's helping me to confidently nail the implementation

A few minutes ago, I was doing neither of these two things. Instead, I was writing some very simple code that, although different in its details from what I'll show below was, more or less, exactly what you see below (I changed the specifics to avoid the distraction of domain details):

    @Test
    public void shouldThrowExceptionForNegativeAccelerations() throws Exception {
        assertThatExceptionOfType(IllegalArgumentException.class)
                .isThrownBy(() -> vehicle.accelerate(-1))
                .withMessage("Acceleration must be > 0");
    }

"So, wtf? Where's the joy in that?"

The joy, for me, is in the sheer simplicity and readability of the code and the fact that, above all else, the intent^[2] of the code sings out clear as a bell. The joy, for me, is in writing code using an API that is well-thought out and discoverable via auto-complete. I didn't have to stumble around with documentation or find the right matchers to quickly do what I wanted to do. And the joy, in this specific case, was being able to easily make an assertion around the exception type and the message.

In short, the joy I was feeling was because I was using AssertJ for writing my assertions. The reminder to re-evaluate my habits was because, despite AssertJ being "old news", it's only within the last half-year that I finally stopped to look around and realized that my existing coding habits could be improved by switching to AssertJ. My experience with switching to AssertJ pretty much exactly mirrors the experience I had when, several years ago, I switched to using Mockito for mocks.

And so, of course, now I'm wondering what of my other coding habits need re-evaluating? Any suggestions based on your own experiences of changed habits that brought you more joy (coding or otherwise)?

Notes:

Ok, sometimes a whole lot more than "a bit" ↩︎
Right up there with "does what it's supposed to do", "clarity of intent" is one of the very top things I try to think about when I’m writing or reviewing code. In fact, without the clarity, it's so much harder to even evaluate the code! ↩︎

99.99% uptime in 6 simple steps

2016-08-23T00:00:00Z

The secret to building applications with rock-solid stability is shockingly simple:

instrument the f**k out of your application
monitor the instrumentation
experience problems
fill in any instrumentation/monitoring gaps that would have helped you notice or understand the problem sooner or more easily
fix the root cause of the problem
wait for the next problem

Robust applications do not spring forth fully formed as if from the head of Zeus, they evolve over time. ^[1]

The beauty of this heuristic is its simplicity and that it leverage the fact that you already have to deal with live site problems; it also guarantees that you'll be working on shoring up your defenses against real-world problems and not imagined concerns. Although simple in concept, it takes a surprising amount of discipline to execute on the plan as there's always the pressure to take short-cuts (or outright skip!) steps 4 and/or 5.

For example, how many times have you seen a problem "fixed" by restarting a service? Whatever time would have been spent fixing the actual issue is almost aways more than made up for by the cumulative time, distraction, and fatigue that would otherwise be wasted repeatedly responding to the same problem. ^[2]

Similarly, when you're under pressure to get back to your other project, it's all too tempting to skip adding the instrumentation / monitoring that would have let you instantly understand what the underlying problem was in the first place. Typically, the rationale is something like: "I fixed the root problem, it won't happen again, and so I don't need to add any additional instrumentation or monitoring." ^[3]

The mistake with that line of thinking is that even if the same problem really won't ever happen again, adding the additional instrumentation / monitoring will help you more quickly understand and respond to all problems of that same category. This is a critical point! It's gaining the protection against entire classes of problems that is the key to step 4 and that allows you to, down the road, identify novel problems before they impact users.

As a trivial example, if your JVM process crashed due to a memory leak, after you fix the underlying leak, you should still add the missing monitoring of the JVM's heap size (and possibly GC times, etc) so that all future memory-related issues are recognized before they crash or otherwise impair the JVM.

Having successfully improved a number of applications using this iterative approach, I'm convinced that the short-term costs of investing in improving existing systems is well worth the long-term gains. Live applications that require little to no operational overhead and developer involvement are worth their weight in gold in terms of overall team productivity (not to mention end-user experience!). Interrupting in-progress work to put out a live fire is, besides being unpleasant, horribly inefficient and disruptive to the larger endeavor of succeeding at whatever it is you're trying to do. In the extreme cases, operational fatigue from regularly being in a reactive mode is a complete morale and productivity killer.

What follows are some practical suggestions based on my personal experience. The suggestions are, of course, neither comprehensive (they primarily focus on one aspect of monitoring Java-based applications) nor exclusive (there's lots of other routes you can take). However, I can say that what follows has been validated in critical production systems -- including a core system that now handles millions of searches a day across a catalog of a billion products and through which a majority of company revenue flows. If what I describe is compatible with your ecosystem, I'm confident you'll be happy with the end results.

Avoid Tight Coupling (that includes monitoring!)

One of the challenges of monitoring complex systems that can slowly (or quickly) creep up on you is that complex systems typically require similarly complex monitoring. If you're not careful, you end up distributing deep understanding of your application's internals into your monitoring system. This creates tight coupling between the two systems. Needless to say, tight coupling, whether at a code-level or a systems-level, always creates fragility, is more difficult to maintain, and slows down development. Seemingly trivial changes to your application end up cascading into your monitoring layer. At best, you keep track of the dependencies and carefully make changes to the two systems in concert. At worst, you're unpleasantly surprised by a bunch of alerts going off when you deploy and you end up either rolling back the release, disabling the alerts (hopefully temporarily!), or scrambling to update the monitoring.

Although some amount of understanding of your applications by your monitoring layer is usually unavoidable, one simple way to avoid tight coupling is to make the system being monitored responsible for what constitutes a healthy internal state. When the application is in charge of self-reporting its healthiness, it's free to interrogate deep implementation details without having to explicitly surface those implementation details. And suddenly, it's easy to evolve the monitoring as the implementation details change. Moreover, b/c the code is living in one place, any friction to adding extensive monitoring of the health of your application's internals disappears. For one approach on how to do this, keep reading.

Dropwizard & Metrics

At my current job, we've been implementing a lot of our services using the Java framework Dropwizard. It self-describes itself as a "framework for developing ops-friendly, high-performance, RESTful web services." In this context, what I really like about Dropwizard is the "ops-friendly" part. And, specifically, the fact that it uses the Metrics library to expose a very simple but very powerful /healthcheck endpoint over http. ^[4]

You can read up on health checks but, in short, what they give you is the ability to easily write a "small self-test which your application performs to verify that a specific component or responsibility is performing correctly." The /healthcheck endpoint provides a consistent hook across applications that allows your monitoring systems to interrogate the health of your applications without needing to understand a single implementation detail. Requests to the endpoint return a 200 status code if all of the internal health checks pass and a 500 status code if something is not healthy. You won't be surprised that there's some friendly json to let you know what check(s) failed.

Out of the box, Dropwizard comes with a check to verify that there are no thread deadlocks and, beyond that, you're heavily encouraged to write your own checks. Lots of them. Checks are incredibly simple to write and you can use them to check on the health of anything you can think of... In the past, we've used them for all sorts of things such as:

JCBC connection pool is functioning
Elasticsearch cluster is accessible
AWS S3 Bucket is accessible
AWS Redshift Cluster is available/healthy
Guava-based Service is running
Queues are accessible and not backing up
Batch processing job has not failed
In-memory data is not stale
etc, etc, etc

The real beauty/point here is that you can write specific checks to verify any aspect of your application's functionality, dependencies, etc. Moreover, those checks live within your same application, eliminating coupling across systems and friction during development. Less friction means more checks.

Although a super powerful and useful approach, health checks, by design, aren't an all purpose monitoring system. Specifically, the semantics are pass/fail and there's no concept of "warn" or any (built-in) ability to externally adjust thresholds for alerting. However, it's such an incredibly useful facility that I heavily rely on it to perform core monitoring of the health of the applications I build. You do need to be thoughtful as to what kind of checks you add to the system keeping in mind that they need be unambiguous and actionable red/green checks.

It's worth noting that the Metrics library also exposes a /metrics endpoint with low-level and fairly comprehensive instrumentation of the JVM, service requests, error logging, etc. If you're not already familiar with it, think along the lines of request rates, response times broken out by percentiles, etc. Although I'm not going to focus on that functionality in this post, it's extremely helpful, supports a wide-range of custom metrics, and provides a nice complement to the /healthcheck endpoint. There's also a wide range of "reporters" that allow publishing of metric data into systems like, for example, Graphite. ^[5]

As an example of health checks in action, here's a somewhat fictionalized healthy response illustrating the simple red/green semantics of the healthchecks and the optional inclusion of a detailed human-oriented message (typically you always include detailed messages, stack traces, etc if there's a failure):

{
	"deadlocks": {
		"healthy": true
	},
	"messageProcessorIsRunningHealthCheck": {
		"healthy": true
	},
	"messageQueueHealthCheck": {
		"healthy": true,
		"message": "There are 106 queued messages (want <= 250000) and 10 consumers (want >= 10)"
	},
	"importerServiceIsRunningHealthCheck": {
		"healthy": true
	},
	"merchandiseColorSizeLoaderHealthCheck": {
		"healthy": true
	},
	"rabbitMQHealthCheck": {
		"healthy": true
	},
	"redshiftClusterHealthCheck": {
		"healthy": true,
		"message": "Cluster status is 'available' and maintenance window is UTC 'fri:08:30-fri:09:00' (currently 2016-08-23T16:14:37.497Z)"
	},
	"redshiftDataSourceHealthCheck": {
		"healthy": true
	},
	"s3BucketIsReadableHealthCheck": {
		"healthy": true
	},
	"sqsHealthCheck": {
		"healthy": true
	}
}

Alerting on /healthcheck failures from New Relic

Although the Metrics library provides the /healthcheck endpoint, you still need something to regularly probe the application and alert you if it enters an unhealthy state. At my current company, we use New Relic for a substantial portion of our monitoring. New Relic has nothing out of the box to support this type of monitoring but it does have an extensible plugin framework that you can customize as necessary.

The Dropwizard plugin for New Relic that I wrote provides the bridge between New Relic and your Metrics-enabled applications. The plugin allows you to easily receive alerts via NR whenever your Dropwizard application enters into an unhealthy state. Additionally, you can optionally configure NR to send alerts whenever the rate of 4xx responses, 5xx responses, logged errors, and/or JVM heap utilization exceeds configured thresholds. Thresholds for alerting can be configured on a per application basis within NR.

In addition to providing a basis for alerting on your application, the plugin comes with a fairly rich UI to provide historical data on both your healthcheck failures as well as a whole slew of performance metrics (i.e., the ones exposed via the /metrics endpoint).

Here's one of the views of the New Relic plugin dashboard (you can see samples of the rest over on github):

Peace of Mind

If you're anything like me, then you've probably woken up in the middle of the night with that "oh crap" feeling as you suddenly realize that there's some (perhaps unlikely) edge case failure scenario that you didn't previously consider and which, if it went undetected, would be insidiously bad. Now, when that happens to me, I'm often able to go right back to sleep because I know that the next day I'll simply write a new health check.

Sleep well!

Notes:

Poetic license aside, the truth is that there is some amount of "springing forth fully formed" as there are any number of problems and failure scenarios that can and should! be anticipated and planned for using the benefit of your own and others' experiences. Although outside of the scope of this post, sound design is best baked into your architecture and application from the beginning. ↩︎
Moreover, sometimes the underlying cause of an intermittent crash also has less obvious non-intermittent symptoms such as a general performance degradation that you only discover in the process of fixing the intermittent crash! ↩︎
Guilty as charged although I try very, very hard to avoid that pitfall! ↩︎
While the Metrics library comes bundled with Dropwizard, it is a separate library and can be used in any Java-based application -- run embedded Jetty or similar if your application does not already service http requests and you want the http endpoints. ↩︎
I should also add that, because of the significant operational benefits, I've taken to using Dropwizard for non-service applications as well as service applications. Not only are the /healthcheck and /metrics endpoints and the reporting of metrics into systems like Graphite invaluable, but having an easy hook for adding runtime operations via Tasks is awesome! I've used it to create simple admin APIs for things like temporarily pausing a processor, throttling processing rate, etc ↩︎

Jekyll & GitHub == continuously deployed website in 10 minutes

2016-08-16T00:00:00Z

This site has since moved off GitHub Pages: it's now built with Eleventy and served from AWS. The post is as I wrote it in 2016.

I very recently got around to building this website. Now that it's up and I'm thinking back on the process, I'm finding myself surprised/pleased that the part that took me the longest was figuring out how I wanted to host the site. I'm sufficiently thrilled with the simplicity and benefits of where I ended up that I wanted to share.

For context, I should first say that my core requirements for a site are very modest. Primarily, I want a site where I can share some professional information about myself and make the occasional blog post to share the ah-has! and general ups and downs of writing software. Secondarily, I wanted to avoid paying anything more than a nominal service fee for hosting the site. And, of course, I wanted to bring normal engineering standards to the process of maintaining the website.

After consider a number of options (eg Django hosted on DigitalOcean), I ended up using GitHub Pages coupled with Jekyll.

This is hardly new or groundbreaking technology and there are already plenty of great tutorials out there, so I won't talk about any of that except to say that it was very, very easy to get a site up and running.

Instead, I want to share why I personally love this approach:

Free (as in no work) Continuous Deployments: GitHub provides tight integration with Jekyll and you get automatic builds and deployments of your site on check-in. As soon as you merge into master, your change is live. Not having to think about building or deploying the site (not to mention maintaining a server) is invaluable.
Trivial Site Previewing: Jekyll makes it super easy to preview changes to your website locally before pushing your changes via the jekyll serve --watch command which allows you to view your site on localhost. "Edit and refresh" makes for rapid iterations.
Familiarity of Markdown: I'm writing this blog post in simple and familiar markdown which means I really don't have to concern myself with anything other than the content.
Versioned Content: Github makes drafting (and backing up) changes very easy. For example, I made a "feature" branch for this post and I can commit and push to that branch as often as I want. I can take as long as I want to finish the post and have as many in-progress posts as I want without ever worrying about losing them. And all of that happens automatically b/c it is based on my normal development practices.
Free: I was willing to pay some nominal amount for website hosting so it's the icing on the cake that this approach is also free.
Jekyll: Jekyll is new to me but the learning curve for doing simple things is not at all steep and my sense is that, if I ever need it, it affords a considerable amount of flexibility and functionality.
Simple: Once you're set-up, doing something like adding a blog post consists of adding a single file with some metadata (date, tile, categories, and tags) and the markdown-based content. Its hard to imagine it being easier.

If you're thinking about putting together a website or already have one and are experiencing any pain in maintaining it, I highly recommend this approach. As someone who was already using GitHub, it's been a 100% friction free experience for me as it dovetails perfectly with the tool sets and development habits that I already have. Create your website using standard tools and everything else happens automatically and painlessly.

AAE

2012-12-06T00:00:00Z

This is based on an email that I originally sent out to fellow engineers at my company. I probably would have written it somewhat differently for a wider audience (I ony edited it slightly for readability) but, that being said, I strongly believe that acronyms (however well-intentioned) are, at best, not super helpful and, more often, are extremely obfuscating.

I'll just come right out and say what I think: when writing code, acronyms (and other abbreviations) are evil and should be avoided like a plague of blood-sucking demon bats.

Here's the (reasonable) arguments for acronyms/abbreviations that I can think of:

They are shorter to type
They take up less screen space and therefore make the code more compact and easier to read

That's not very many reasons and I don't think either of these reasons are good enough reasons to justify their use. Modern IDEs have auto-completion and so you don't actually have to type out variable names. So, generally speaking, typing a long variable name is just as fast as typing a short one. The second reason is a little bit more compelling to me. However, I think that whatever benefit is gained by having more compact code is more than offset by the loss of clarity in the code itself. There's just no getting around it: acronyms and abbreviations simply don't make sense unless you know what they mean and our goal should be to make our code as easy to understand as possibly can. The best code self-documents itself by using clear and informative names that don't require any domain-specific knowledge.

For example, consider this real-life code I came across detailing two properties on a "member" user object:

public bool AllowMss { get; set; }
public bool AllowFpis { get; set; }

What does that mean? I have no idea! The only person who knows is someone who is already fairly familiar with the code. How could a new engineer (or even someone who has worked at that particular company for many years but is coming to the code for the first time) possibly be expected to know what it means?

Or consider some more real-life code for an e-commerce site that had has a concept of virtual money called "cafe cash": if you look in checkout-related code, you'll discover places where the abbreviation 'CC' can stand for either credit card or cafe cash. The inconsistent use even happens within the same class! But, even if CC were consistently used to always mean "credit card", someone who is looking at the code and knows that there's also a concept of "cafe cash" is still going to wonder whether the CC might stand for cafe cash. I would go farther though: even for code bases where there is no such thing as cafe cash, what's the clearer name?

PayByCC or PayByCreditCard?

You can't get much clearer than PayByCreditCard! Everyone will instantly understand that. Even out of context. Sure, a lot of people will guess by context that PayByCC means PayByCreditCard but there are going to people who need to hunt around in the code to figure that out / make sure their guess is correct. And, even if you know what it means, you are still mentally translating PayByCC into PayByCreditCard in your head every time you look at it. I say there's no reason to make yourself or anyone else perform that extra cycle!

What's a pnlPoliticalContribution? What's a CbaEmail? I'm sure that CBA seemed totally obvious when the code I found it in was written but it's not at all obvious to someone who didn't work on that particular project. What's cbStoreSpecial mean? Is a taxId referring to the id of a taxonomy node or taxes? These are all needless opportunities for ambiguity and confusion and therefore heartache in working with the code and bugs for end-users.

So, in my opinion, when should you use acronyms, abbreviations, or otherwise shortened names in your code? I say never do that unless:

you would expect someone interviewing for a job to know what the name means (id, url, http, api, util, svg, etc)
the scope of the variable name is extremely limited (a small number of lines of code) and using a shortened name very significantly improves readability and then only do so judiciously (i.e. still don't do it)
for universal conventions such as an index variable name like i

There's probably a small number of other reasonable exceptions besides those three. But, generally speaking, I really think that using full names is a great and simple way to write better code.

ps Of course, I'm ignoring the importance of code compaction for things like javascript and css where the code is sent across the network. But that's what compression and minification are for!