Author Archives: Kaiser Leib

vi(m)

I do not know how to use text editors. I grew up with notepad, did HTML in notepad++, and by the time I had to do anything serious IDEs were widely available. I’d run Linux on my home machine for a while out of stinginess, since Windows costs if you don’t get the manufacturer crapware bundle, but during that period the most technical thing I did was an abortive effort at a reddit/metafilter/facebook aggregator in Python, and for that I used IDLE.

My current job handed me a Macbook Pro, and vim has been my instinctive tool to edit bash configs or .gitignores or what-have-you. The process for this is easy: I say “vi filename” at the terminal, and I get this little editor that does things I’ve come to expect, in which I can move around with arrow keys as well as hjkl end press ‘enter’ to get a newline.

I just installed Debian on my home machine in what seems to be a fit of masochism. Debian comes with vi. I typed “vi ~/.bashrc,” thinking to edit some options and make zsh prettier. What I got was evidently a miniature real vi, not the vim I’m used to. I pressed the left arrow key and it wrote a capital D under my cursor, even though I wasn’t in insert mode, and it was weird.

I installed vim using apt, which got me the ‘vim’ command, but didn’t replace vi. I tried adding

to my .bashrc, which also did not work.

Fortunately, the very last answer on the superuser post I was trying to follow actually solved the problem: I had to edit /etc/vim/vimrc.tiny and change

to

Success is a Local Maximum

In 2011, I interviewed with Dropbox.

Wait, that’s not far enough back.

In 2003, this one girl in high school decided she didn’t want to date me. I was also invited to apply at MIT and Yale and the like, based on my PSAT scores. Eventually I went to the University of Montana. I promptly lost my National Merit Scholarship on account of my new-found affinity for whiskey.

I got a job in tech support/dorm network administration, at the so-called “DirectConnect Office.” The DCO was great, until it wasn’t: a supervisory promotion went to exactly the wrong person, and I had to quit. I went to work at one of those body shops that calls itself an “IT Consulting” firm, and made $30,000 a year as a software engineer, until I burned out from the stress of billing clients at a rate I felt was unjust.

Being out of work wasn’t too hard. I sent emails to everyone I’d met who might be able to hire me, and I had a job less than a week later, doing DoDAF architecture for the Army’s intelligence center. Twice the salary still felt like a failure, since I was no longer a software engineer.

Through contractual weirdness, I had three different bosses at any given time, and the company on my pay stub switched even though the people I worked with stayed the same. During that period, I had interviewed at Dropbox, and they “decided not to move forward at this time:” they correctly assessed that I didn’t care about Dropbox as an app and was only in the interview hoping for a job, that I’d never be a true believer in the cause.

Eventually I managed to convince the Army and my employer to let me work remotely, and I did that for a year and a half. Then a senior developer position opened up and I decided that I needed to return to software.

Unfortunately, a few years of tech support and a year of maintenance contracts followed by four years in an unrelated industry don’t lend themselves to the “senior” title. The engineering division still hired me, but only because someone’s boss’s boss owed my boss a favor, and my boss liked me, and so I got to be a staff-level software developer, using my TS/SCI to make games in the Unity engine which would be used to train the Army and the FBI.

It was terrible. They didn’t want to let me keep working remotely. They wanted me to move to Georgia. The codebase was inscrutable. I quit. I bought a motorcycle and went rock climbing and drank too much.

Before I ran out of savings, SoFi fell into my lap. I got lucky. The interviewers liked me. I was competent enough to get the job.

If everything had worked the way I’d wanted it to, starting when I was sixteen, I’d have four kids and be working in a very small town, probably doing remodeling or cutting down trees. If I’d gotten in to MIT or Harvard, I don’t know precisely what the outcome would’ve been, but a friend of mine failed out of Brown and the Yale guy from the wrestling team was selling cars last I checked. If I’d had the supervisor promotion or if the local contracting gig had worked out, I’d still be where I was, being paid mostly in the splendor of Missoula scenery. If the Army contracting gig had worked out, I’d be making an above-average wage in Augusta, eating good food on the cheap and living an upper-middle-class existence.

SoFi just raised a billion dollars at a $4 billion valuation. I’ve got a very small piece of that. My salary is a rounding error on the value of my stock options.

Success is a local maximum. If you never fail, you can never achieve anything greater than what’s in front of you at the moment. If you always get what you want when you want it, you never have to learn to create any value. I’m not taking any credit for the good things that have happened to me; if anything, I’m doing better because i was incompetent earlier in life. Maybe I’ll continue to fail. But if I do, maybe my continued failure will allow me to find better and better things later. One can only hope.

Breaking (and Fixing) Play’s CSRF Protection

Cross-site request forgery (CSRF) is sort of the opposite of cross-origin request scripting. In a CSRF attack, the attacker coerces a victim’s browser to make a GET or POST request to another site – no PUTs or DELETEs or other actions. If your site follows best practices, GETs don’t change any information, and so a CSRF GET request is relatively benign. That still leaves POST requests vulnerable.

The Play Framework provides protection against CSRF attacks by placing a token in the user’s session, a copy of which is submitted along with every POSTed form. If the attacker grabs the page on which the form appears and stores the CSRF token he sees in the form, the victim’s browser will have a different token generated when it loads the page, and the two won’t match – Play will respond to such a POST with the message

or

and evaluation of the POST request short-circuits.

In our world-facing application, we store some information regarding our customer’s behavior in their session, including the type of product for which they’re in the process of applying. That lets us look up the correct product ID in the database, lets us easily verify that the customer owns the application for which they’re requesting the information, and in some circumstances allows us to route them directly to the relevant page rather than having to visit our landing page first.

Now, if the customer does visit the landing page, or if the same browser visits the registration page, we have to clear their session to avoid leaving the wrong kind of application information in place. In fact, if they’re visiting the registration page, it’s probably because they’re a new user on the same machine, and so we should just clear the session entirely.

That makes perfect sense, if we’re only thinking about the session information we use for business logic in the rest of the app. Unfortunately, the session also includes that CSRF token we mentioned before. The browser keeps the registration page cached, as it does with most pages, and includes the CSRF token. The server, on the other hand, wipes out the user’s session information, and so if you hit the registration page without logging out first, you’ll have a CSRF token mismatch and Play won’t let you register.

We saw this problem infrequently, and it seemed always to go away on its own. One day we noticed that it seemed to be happening to customers referred by one of our corporate partners. “Oh, that makes sense,” we thought, “something is wrong with the corporate partner token.” We totally disregarded the capitalized “CSRF” and got stuck on the registration token we were handing to our prospective customer’s employer, encouraging the guy who handles that system to fix his obviously broken code.

One day, I was looking at an unrelated issue, and needed to create a few accounts in quick succession. Immediately I saw the “missing token in request body” error, and I could then reliably reproduce it. The reason we saw it more frequently from our corporate partner was that they’d often have a spouse or domestic partner register from the same machine: it was correlated to the corporate partnership, but otherwise unrelated.

The eventual fix was easy. Instead of saying something like:

I instead did:

The redirect() forces the browser to forget its caching and make a new request to the page, which re-generates the CSRF token since the existing session had been cleared. You could do something similar with web server configuration, telling the browser not to cache anything, but this way we can handle it in code and retain the benefits of caching in all other situations.

The CORS of the matter: Cross-Origin Requests in Play 2.4.x

Building RESTful APIs can make it easier to enforce certain decoupling rules, but it often imposes an additional infrastructural burden, depending on your tech stack. Lately, I’ve been struggling with some of the inconsistencies between our Bamboo-built Docker containers and the locally-compiled version of what should be exactly the same software: it’s not that the bytecode is different, but the boundary between the separately-Dockerized MySQL database and my code’s container was enough to give Hibernate fits until I figured out the necessary library import.

Furthermore, even serving requests via HTTP can be different. One potential pitfall is in CORS filtering, wherein the browser itself protects us from what appears to be a different (possibly malicious) domain. My testing in Paw, an OS X REST client, worked just fine. My coworker tried to hit the deployed API from Chrome and got no result. The confusion was brief, but it’s still a bit of friction.

The linked blog post above describes the concept effectively, but it’s targeted at an older version of Play, using Scala. We’re using Play 2.4 and Java.

Fortunately, once the term ‘cors’ is in your googlesearch toolbox, you’re able to find Play’s documentation on the subject.  In order to open the API wide for development purposes, I added the following to our application.conf:

Which allows us to make a browser request and see the expected JSON without having to fire up Paw.

Java 8 LocalDate and LocalDateTime StreamCorruptedException

Java 8 finally brought the long-awaited equivalents of Joda’s LocalDate and LocalDateTime classes to the core Java API. They’re available in the java.time package. Unfortunately, JPA had been a little slow to pick up on the changes, and so persisting the new LocalDate and LocalDateTime properties of your entities is kind of a pain.

JSON serialization of your entities is actually pretty straightforward, following the example set in this blog post by Chris Ritchie.

Persistence, on the other hand, relied until recently on a custom converter, as described here by Adam Bien. Running locally, we were able to make good use of a converter on our current project. Unfortunately, for reasons that are still unclear to me, we started seeing StreamCorruptedExceptions when we went to run our app inside a Docker instance and talk to a MySQL instance inside a different Docker instance.

I attributed this problem to a number of different sources, since “StreamCorruptedException: Invalid Header” doesn’t hit on anything date-specific on the first few pages of Google. I gutted the entire object model and removed all the @ManyToMany and @ManyToOne relationships. I removed our convenience methods. I re-configured the build plan and our Play! configuration files. Nothing worked.

At some point, I took the “invalid header” reported in the exception and put it through a hex-to-ascii converter. The header was always the same, and it was always the binary representation of “2015,” which led me back to examining our LocalDate and LocalDateTime behavior, including the converters.

I gutted the converters, and had them return empty strings, nulls, or the current date’s string representation, which I thought would allow us to at least get objects out of the database with garbage dates attached to them.  No such luck. I copy-pasted other people’s converters from a few different sources. I even chased our Serializers around, since the StreamCorruptedException eventually manifested as a SerializationException. Nothing worked.

I called in some of the more senior people from our team. They shrugged. I had hoped it would be obvious.

This morning, having slept, I got up to take another crack at it. Now that I know what to look for, the solution is obvious, and a bunch of blog posts and stackoverflow answers are being updated to include it: you can just add a dependency on the hibernate-java8 library, and you’ll be good to go. In SBT, that looks like this:

I spent the better part of three days chasing that around. I’m not totally comfortable with the fact that the converters never worked, but at least we can keep moving forward on this project.

Action Screws, Autoboxing, and Five Years’ Experience

Most jobs want experience, even those jobs classified as “entry level.” This is especially true in the software development field, and it’s easy to take that as an insult, particularly if you just spent four years getting a bachelor’s degree, and thousands of hours tinkering.

Using professional experience does a few things for an employer. In the first place, it guarantees at least a modicum of human normalcy, which is important: we have to be able to communicate with our coworkers and treat them appropriately long enough not to get fired. That problem is hardly unique to software, but it’s exacerbated both by the need to communicate requirements clearly between varyingly technical stakeholders and by the tendency towards introversion. Some really smart people who are able to develop software very quickly might be unable or unwilling to understand the customer’s needs or treat them with respect, and that can make even the best engineer less effective.

Experience also allows a developer to dive in and begin work without substantial ramp-up on documentation. Regardless of whether you start with the model, the view, the controller or the database layer, knowing what those concepts are and how they interact allows you to begin by actually writing code instead of heading over to frameworkofthemonth.com and scratching your head and watching youtube tutorials just to get the ball rolling.

Finally, experience provides an understanding of the gotchas in a given language. I think this is the real reason why we see job posts asking for five and ten years’ experience in J2EE or .NET or whatever: it’s not that a less-experienced developer won’t be able to write code, or even that he won’t be able to establish a sensible architecture: it’s that he won’t understand the edge cases and sometimes non-intuitive behavior of certain features of the language or framework.

A funny quirk of the Java language bit me pretty hard earlier this week. The computer has a bunch of different ways to represent numbers, all of which have tradeoffs. In college, everything was an ‘int,’ an integer represented with 32 bits of binary. This lets us accurately present anything between -2,147,483,648 and 2,147,483,647, and that’s enough for most toy or academic purposes. In professional settings, we end up using a lot of ‘long’ numbers, which have 64 bits and can therefore represent numbers between –9,223,372,036,854,775,808 and 9,223,372,036,854,775,807, or as we doctors like to say, “enough.” Little-i ints and little-l longs are just the numbers themselves, but we also have big-I Integers and big-L Longs, which are bigger objects representing those numeric values with some verbs attached – we can ask a Long how many zeroes are at the end of it, but a long doesn’t even know what a question is, much less the answer.

Despite that, it makes sense to be able to ask the computer whether a Long and a long represent the same number, and you can do just that, by saying “if l == L.” This works just fine for values less than 127. Since my little fake test environment has fifty-ish users, I was able to check whether the users had the same ID using ==.

But when you get above 127, Java starts acting weird. It doesn’t automatically convert a Long to its long value when faced with the == question. Instead, it compares the memory addresses, a process akin to asking two people where they heard some information and deciding that the information must be different if it came from a different source. Why should it work for 126 and not 130? I don’t know. Some former Sun employee knows. I’d much rather have the == operator never work on Longs than have it work sometimes and then fail when I commit code to the world-facing web site of a multi-billion-dollar company.

The quirky, foibly, five-years-matters experience doesn’t only apply to developers, either.

My precision rifle is a Steyr SSG 69 PII. A friend of mine called and mentioned that there was one for sale and that the seller was having trouble moving it, so I bought the thing, and ran it without issue as it came from the box. But being the way I am, I had to take it apart, and when I put it back together it wouldn’t reliably strip the next round from the magazine.

I misdiagnosed the problem, and bought a ten-round factory magazine to replace the five-rounders it came with. Steyr’s factory mags are fragile and pretty terrible, but the ten-rounders were no better than the five. Then I swapped the plastic trigger guard and magwell for an aluminum CDI Precision replacement, which offered both better durability and the ability to accept AICS magazines (evidently I got one of 30 or so that were ever produced, which is cool). The rifle didn’t behave any better.

Having exhausted my nonexistent expertise and my patience (and done a terrible job at a match in the meantime) I took the rifle to a gunsmith in Missoula. He proceeded to sit on the thing for seven months, during which I moved. I got the rifle back from him, and the firing pin wouldn’t even stay cocked – it was missing a spring (which I replaced with a 69-cent part from Ace Hardware) and a set screw (which Steyrarms sent me for free).

A friend of mine started getting into handguns, and did a transfer through a kitchen table FFL who turned out to be a pretty competent gunsmith. Having spent some time messing with the rifle himself, my friend suggested that we could take it over and see whether the man’s expertise would be any use, and we did.

So we drove the several miles to the guy’s house, and he greeted us in his wonderful German accent and told us some brief stories and described his NFA business and corrected my out-of-date numbers on what a registered M16 costs. Then he looked at the rifle.

“Oh, yeah, I know what’s wrong,” he said. “Steyr, I don’t know why they do this, but the plastic stock is very flexible. You have to get a torque wrench when you install the action screws. Otherwise the stock flexes and it’s just terrible. You should get a McMillan stock. In the meantime, I can do it for you, but it’s just as easy if you do it yourself, and my shop rate is $60 an hour.”

Just like the Long != long issue I had, this is something that I could technically have known, something I might even have read at some point before. But because neither of these had caused me specific difficulties in the past, I didn’t remember them. In both cases, it took the oversight of more experienced people to diagnose the problem – not only that, it took a specific sort of experience, and in both cases the first expert I approached couldn’t solve the problem.

So general experience and education are great, and it’s important to have theoretical understanding of the factors at play. But vendor, environment and product-specific experience also matter. We can laugh at the HR guy who writes “10 years experience with Node.js” on a job posting, but the concept is sensible: ten years offers a great many opportunities to run into trouble, and that’s true even if you’re getting one year of experience ten times.

Squats, Rifles, and Javascript

I’m a pretty good marksman. I’m not great. I won’t win any sort of formal competition. But I’m an okay teacher, and I have a few hundred hours of coaching experience under my belt. People on the internet listen to what I have to say, sometimes. There has never been a time in my life when I wouldn’t have told you that I’m a “good shot,” but it’s only pretty recently that it’s been true.

In order to get good, I had to admit that I didn’t know anything, go back to basics, and take a class. Fortunately, I didn’t know that it was a class, or I wouldn’t have gone. I figured that I’d show up, get a “rifleman” patch on the first target, and have everyone ‘ooh’ and ‘ah’ over how great I was, and then I’d go home secure in the knowledge of my greatness. I didn’t. It took me days of consistent effort to learn enough that I could consistently keep rifle bullets within a 4 arc-minute cone.

Similarly, I’m not only strong, but well-versed in personal training and the factors which contribute to myofibrilar and sarcoplasmic hypertrophy. I’ve had periods of religious gym attendance for the past ten years, always including squats and bench press, with deadlifts for the past 8. And I have relatively little to show for it.

I did my first proper full-depth squat in December. I had to admit to myself that my squat form and progress had been unacceptable and start squatting more frequently with greater depth at a lighter weight in order to develop the necessary flexibility. That meant swallowing my pride and lifting a lighter weight than anyone else in the gym, less than 100 lbs. I’m still not perfect, and I’m not as strong as I want to be. But I’m doing it right.

And now I have a job where I need to do Javascript. I am FizzBuzz Javascript proficient. I am not good enough that just diving in and trying to do Node.js or Angular.js in production makes sense, especially not when we’re looking at our relatively uncommon architecture. Javascript allows you to pass arbitrary-length argument lists. That’s Javascript 101. I didn’t know that.

Part of the reason I’m so bad at Javascript is that I’m a real programmer, and I don’t need to learn no stinking for loops. So I’d find a worthwhile tutorial, and the tutorial would start with the basics, and I’d get bored and give up and copy-paste and cargo-cult the workable DOM manipulations and lightboxes. This isn’t 2009, and that’s not a great idea for productivity any more.

So I’m going through Eloquent Javascript. And it’s boring and it’s slow but mercifully it’s got that inline editor stuff that’s really super cool. And I hope I get to where I need to be.

An arcade-style shoot-’em-up in Unity

It was the appeal of video games that got me into programming in the first place, really, as was the case for much of my CS131 class at the University of Montana. It turns out that making video games has almost nothing to do with playing video games, so many of my classmates didn’t continue with the program. I made a Tetris clone once, but that was the extent of my ‘game development,’ since PHP and C# CRUD apps are worth money to business and government customers and games aren’t.

But since I wanted a well-defined problem in an understood space, Unity beckoned. They provide top-notch tutorial projects, one of which is building an arcade-style “space shooter,” of the sort I loved to play as a kid.

UnitySpaceShooter

 

The tutorial walks you through creating a player spaceship, adding the ability to shoot lasers, and providing some asteroids as hazards to avoid and destroy. However, the assets they provide with the tutorial also include an enemy spacecraft, and in the interests of illustrating that I can do more than just follow directions I decided to add him to the game as well.

Again, this isn’t perfect. The idiomatic design pattern for the project is to spawn the lasers a short distance in front of the spaceship using a GameObject to hold the offset, and I spawned the enemy laser by using an offset in a script. You can shoot down the enemy lasers since I re-used the collision detection from the asteroids, which is either cool or silly. And it might be nice if the enemy ship moved around, or if I used the assets provided to create some variety among the asteroids. If this were a production game, we’d want progressive difficulty and probably a persistent high-score system. As it stands, though, I’m happy with the project for what it is.

You  can play the game here, or you can follow the tutorial to create your own game.

Why I’m giving up on the CKII thing

I’m giving up on the CKII thing.

I hit a roadblock which isn’t a show-stopper: the topology used by the game is an 8 bit-per-pixel indexed bitmap, and .NET prefers to edit 32 bit-per-pixel bitmaps. The conversion between the two is well-understood, so that’s not the end of the world.

However, I’ve just interviewed for a couple of jobs in a row which didn’t care about my degree and wanted to see specific experience with the frameworks they use. Now that’s understandable, perhaps even commendable… except that it puts me in kind of a rough spot. There are zero jobs for CKII modders.

You could argue that showing the persistence to finish a nontrivial project is important, and you’d certainly be right about that. (I’ll finish the next one, promise.) But I think my efforts are probably better spent working on a well-documented problem domain that’s obviously relevant to business needs than on a trial-and-error effort to make a glorified toy.

Furthermore, there’s apparently something with the new patch where it’ll crash if your mod doesn’t do rivers and adjacencies between islands properly. There are the problems associated with creating dynamic and interesting characters. There was the sudden realization that I can’t just leave the ocean provinces at the same size as the land provinces, and that the vanilla game seems to have each land province border one ocean province: how would I go about drawing workable ocean provinces dynamically? Simply squishing them together into kingdoms wouldn’t work, because any given county might border two ocean kingdoms, and then it’d crash.

I’m gonna go write a web app.

Procedural Map Generation for Crusader Kings II, Part 4: Markov Names

All of those counties, duchies, kingdoms and empires need names to go with the shiny new borders we’re drawing.

The easiest way to do procedural naming is with something called a Markov chain, a mathematical concept which models changes from one state to another based only on the current state, and not on any ‘memory’ of what happened before (which is not to say that a Markov chain doesn’t use RAM!). Markov chains have applications to other areas, notably in finance, Baseball, and biological population studies, but their most readily apparent application is to generating pronounceable random text.

It works like this: you look at the last N characters of your text, and then at the characters which are allowed to follow them in your sample words, choosing the next character from those among them at random. (The first character is chosen at random from all the characters used to start words in your sample.) We call N the ‘order’ of the Markov generator. Given a vocabulary of English text, Markov generators of order 4 or higher will regularly produce real words, and only rarely give output to make spellcheck unhappy.

I grabbed a Markov generator from the guys at Silicon Commander Games, and found its features to match exactly with my requirements. They were using it to name characters, but naming locations works just as well. I then copied the lists of counties, duchies, kingdoms and empires from the CKII wiki, massaged them as necessary to make comma-separated text files, and fed them to my program. There are only 15 empires in the vanilla game, so I added some more seed data, such as the “Romulan Empire.”

Here are the first few counties for one particular world seed:

Duchies:

All the Kingdoms:

Empires:

As you can see, the Empires are a little weird. I’ll add some custom logic to improve the imperial nomenclature later. The word list for kingdoms doesn’t seem to be long enough, since we’re seeing a lot of kingdom names from the vanilla game. It’s expected that you’ll see a few over time, but seeing Sicily, Arabia, Greece, Hungary, Italy, Denmark and Mesopotamia on my first run indicates that we’d be seeing multiple vanilla kingdom names in every game.