Author Archives: Kaiser Leib

Touch Events: Doodling from Mobile Browsers

Mouse events are not exactly the same as touch events, although they’re similar. That meant that my websocket canvas drawing app only did half its job on mobile pbrowsers: you could see the lines drawn by people on desktop browsers, but you couldn’t draw anything back. Here’s how to hook up a very simple touch event handler. MDN has a more complete reference.

First, we need to hook up the event listeners:

Then we write the event handlers. Since we want touch events to do the same thing mouse events are already doing, I extracted the mouse event handlers’ method bodies, and had both the touch event and the mouse event call the same function:

 

Note that our handleTouchMove function refers to an array of touches, e.changedTouches, using only the 0th element. The changedTouches array supports multi-touch, which is incompatible with our single-pen doodling paradigm.

Deploying a Spark application built with SBT to Heroku

Heroku is a cloud application platform that takes care of many operational concerns – since I have to serve as my own devops for extracurricular projects, that’s pretty appealing. Spark has a tutorial for Heroku deployment, but Spark’s tutorials assume you’re using Maven, and I’m using SBT. Fortunately, Heroku provides an SBT plugin and a reference. Between the two of ’em, we can figure out what to do to deploy a Spark app (in this case, my websocket canvas doodler) to Heroku.

  1. Sign up for a Heroku account, and install the Heroku toolbelt.
  2. Set up your SBT and Spark app.
  3. Add the sbt-heroku and sbt-native-packager plugins to plugins.sbt. Note that the version of sbt-native-packager is different from that described in the Heroku reference:
  4. Add this method to your Spark app, and call port(getHerokuAssignedPort()); in your main() method to determine the port assigned by Heroku:
  5. Create your Heroku app, using the -n flag to avoid adding a git remote:
  6. Add enablePlugins(JavaAppPackaging) and herokuAppName to build.sbt, replacing “your-app-123” with the name Heroku assigned your app:
  7. Finally, deploy the app:

Having completed these steps, you can navigate to your app at http://your-app-123.herokuapp.com, replacing “your-app-123” with the name assigned in step 4.

Spark Doodle: <canvas> and websockets

The source code for this post is on github here. You can play with the app on Heroku here. This app was based on David Ase’s chat tutorial and on Code and Coffee’s node.js example.

Websockets allow duplex communication over a single TCP connection – in other words, they let the web browser constantly speak and listen to a server instead of doing everything through atomized HTTP requests. Before websockets, if you wanted (for instance) a chat app in the web browser, you’d use polling to constantly ask the server whether anything changed. Websockets allow the server to push information to the browser without waiting to be asked.

In this post, I’ll show you how to use websockets to build a simple shared doodling app. Users will see all the lines that other users draw while they are logged in, but they won’t see lines that were drawn before they arrived, nor will they see their own lines reappear if they close and re-open the browser window. The server will use Spark, and the client will be pure, no-library-having Javascript.

The main() method of a Spark application traditionally sets up the routes, but here, all our requests go to the index.html page in /src/main/resources/public, as indicated by the staticFileLocation() call. The webSocket() call sets up the websocket, running at “/doodle” and handling responses using the DoodleWebSocket class. Finally, I put the list of sessions and the broadcastMessage() methods inside this Application class.

Our websocket handler implementation is spartan indeed! Spark uses Jetty’s websocket implementation directly, including Jetty’s annotations. All we do here is add and remove sessions and call the Application controller to broadcast our message out to the clients.

That’s all the java code for this application!

We need some HTML to give ourselves a canvas to draw on:

We have a <canvas> and we include the js source.

Finally, the javascript:

The conceptual implementation is directly from Code and Coffee’s post: if the mouse has moved in the past 25 milliseconds and the mouse button is clicked, we broadcast the instruction to draw a line between our mouse pointer’s current position and the previous position.

The meat of my change is:

  • Switch from a socket.io socket to a plain vanilla websocket
  • add the line to re-create the websocket when the mouse is clicked
  • add the proxy function to wait for the socket to be ready before sending a message.

Re-creating the websocket on mouse clicks is to resolve an issue where the websockets would time out on heroku. The proxy function is there because otherwise we’d be trying to broadcast messages to a socket that wasn’t open yet.

Writing a Java Enum quickly using Excel

Today I had to translate a large code table supplied in a PDF, something like this:

A123 Require more vespene gas
A124 PC Load Letter
A125 You got your swagger back

Hundreds and hundreds of rows, into a Java enum:

That’s tedious to type. If I was strong with macros in a real text editor, maybe, I’d be able to do it quickly in Real ProgrammerTM fashion. But I’m not. I am pretty comfortable with Excel, so I did the following:

  • Copy-paste the table page by page from the PDF into an Excel spreadsheet (right click -> paste special -> plain text), cleaning up any formatting irregularities and extra row breaks.
  • Write a formula, =LEFT(A1, 4) to get the enum constant, and drag the formula cell down the table.
  • Another formula, =MID(A1, 5, 999) to get the String member for the enum value.
  • Copy the resulting cells, and right click -> paste special -> values over the top of themselves, so the next step works with the text instead of the code used to generate it.
  • Finally, =CONCAT(B1, "(", CHAR(34), C1, CHAR(34), ")," to build the line of Java code.
  • Copy the resulting cells and paste directly into Eclipse.

It’s not an especially clever hack, but it sure saves some typing.

Interview Stories: Edulog

I’ve interviewed for a number of jobs. Retail and dishwashing always got me to ‘yes,’ because I can be charming enough to overcome any burden of inexperience for those positions, and high turnover reduces the expectations for new hires anyway. Technical interviews are harder and apt to make for more interesting reports, so they make up the stories I’ll tell.

The first real tech interview I had was during college. They wanted a technical analyst, someone who wouldn’t have to be strictly responsible for writing software but would necessarily have to understand something about the business domain and would be able to understand the code that was being written at least well enough to report progress back to program managers. In retrospect, I can see that this position would merit more technical competency and a greater burden both of time and of experience than I had when I applied for it as a college sophomore, but because I was a college sophomore at a time, I didn’t know any better than to try.

Edulog is a company in Missoula, MT. They specialize in schoolbus routing software, designed originally by a PhD Mathematician. I imagine that it’s mostly a nearest-neighbor optimization on a grand scale, but the scale  implies interesting problems and the necessity of fulfilling business needs probably means that the entire use case is more difficult than just the core algorithm. Anyway, they advertised for an opening that would allow applications for CS students, which I was.

I didn’t get the job, and here’s why.

  1. My resume was weak. I don’t mean that I lacked relevant experience; it was an entry-level position which would’ve been squarely within the realm of my expertise if I had a semester of database courses under my belt. I was flippant and arrogant in my description of cashiering and dishwashing, and took the interiewer’s ‘yes’ as a matter of course, because my charisma had always worked for me in the past.
  2. I was grossly unqualified for the actual position, despite the sole listed requirement of being a student of Computer Science at the University of Montana. The employer didn’t know how to properly discriminate among the applicant pool, or they just needed to interview some poor saps in order to justify their eventual internal hire; either way, my lack of professional experience and skills with MySQL should have excluded me from the interview.
  3. I made a bad impression, joking about the movie ‘Office Space’ and making light of the position’s relevance.

In this case, I think the interview process actually worked really well, for all that it would’ve been nice to have the extra money during college. My performance wouldn’t have been up to par, I wouldn’t have been happy with the job, and driving to and from the office daily would’ve eaten up time that could’ve been better spent drinking whiskey and doing bench presses at the rec center. Despite that, I wanted them to hire me – so here’s what I should have done differently.

  1. I should have developed a basic understanding of what’s expected of entry-level tech people. I sure had a handle on 101-level Java, and I knew that there was such a thing as a database, but taking a few hours to arrange a half-hour lunch with anyone who had industry experience would’ve shown me the enormous gaps in my skill set.
  2. I should have found someone (anyone) to review my resume. It was very clearly written by a 19-year-old who thought he was better than the interviewer, better than anyone, really. I’d never been turned down for a job and I’d never been rejected before and it showed.
  3. I should’ve asked more questions and spoken less during the interview itself. Every word that comes out of a candidate’s mouth is a potential reason to disqualify him from consideration, and i had nothing but shortcomings at the time.

Again, it’s ultimately for the best that I never worked at Edulog. My technical skills still needed serious development, and the internal hire their rejection letter indicated they went with was better suited, regardless of my strengths. Today I’d hire a call center worker from the company over my adolescent self, despite the fact that the former wouldn’t ever eventually be able to fill the “software engineer” role: they needed a business person with domain knowledge, not a can-do introductory tech worker.

I don’t fault myself for having applied, and I don’t really fault the company for having advertised the way they did. I hope they didn’t open a similar position with the same advertisement later, having learned from the candidate pool during my hiring round that you can’t take Java 102 and perform as an analyst.

vi(m)

I do not know how to use text editors. I grew up with notepad, did HTML in notepad++, and by the time I had to do anything serious IDEs were widely available. I’d run Linux on my home machine for a while out of stinginess, since Windows costs if you don’t get the manufacturer crapware bundle, but during that period the most technical thing I did was an abortive effort at a reddit/metafilter/facebook aggregator in Python, and for that I used IDLE.

My current job handed me a Macbook Pro, and vim has been my instinctive tool to edit bash configs or .gitignores or what-have-you. The process for this is easy: I say “vi filename” at the terminal, and I get this little editor that does things I’ve come to expect, in which I can move around with arrow keys as well as hjkl end press ‘enter’ to get a newline.

I just installed Debian on my home machine in what seems to be a fit of masochism. Debian comes with vi. I typed “vi ~/.bashrc,” thinking to edit some options and make zsh prettier. What I got was evidently a miniature real vi, not the vim I’m used to. I pressed the left arrow key and it wrote a capital D under my cursor, even though I wasn’t in insert mode, and it was weird.

I installed vim using apt, which got me the ‘vim’ command, but didn’t replace vi. I tried adding

to my .bashrc, which also did not work.

Fortunately, the very last answer on the superuser post I was trying to follow actually solved the problem: I had to edit /etc/vim/vimrc.tiny and change

to

Success is a Local Maximum

In 2011, I interviewed with Dropbox.

Wait, that’s not far enough back.

In 2003, this one girl in high school decided she didn’t want to date me. I was also invited to apply at MIT and Yale and the like, based on my PSAT scores. Eventually I went to the University of Montana. I promptly lost my National Merit Scholarship on account of my new-found affinity for whiskey.

I got a job in tech support/dorm network administration, at the so-called “DirectConnect Office.” The DCO was great, until it wasn’t: a supervisory promotion went to exactly the wrong person, and I had to quit. I went to work at one of those body shops that calls itself an “IT Consulting” firm, and made $30,000 a year as a software engineer, until I burned out from the stress of billing clients at a rate I felt was unjust.

Being out of work wasn’t too hard. I sent emails to everyone I’d met who might be able to hire me, and I had a job less than a week later, doing DoDAF architecture for the Army’s intelligence center. Twice the salary still felt like a failure, since I was no longer a software engineer.

Through contractual weirdness, I had three different bosses at any given time, and the company on my pay stub switched even though the people I worked with stayed the same. During that period, I had interviewed at Dropbox, and they “decided not to move forward at this time:” they correctly assessed that I didn’t care about Dropbox as an app and was only in the interview hoping for a job, that I’d never be a true believer in the cause.

Eventually I managed to convince the Army and my employer to let me work remotely, and I did that for a year and a half. Then a senior developer position opened up and I decided that I needed to return to software.

Unfortunately, a few years of tech support and a year of maintenance contracts followed by four years in an unrelated industry don’t lend themselves to the “senior” title. The engineering division still hired me, but only because someone’s boss’s boss owed my boss a favor, and my boss liked me, and so I got to be a staff-level software developer, using my TS/SCI to make games in the Unity engine which would be used to train the Army and the FBI.

It was terrible. They didn’t want to let me keep working remotely. They wanted me to move to Georgia. The codebase was inscrutable. I quit. I bought a motorcycle and went rock climbing and drank too much.

Before I ran out of savings, SoFi fell into my lap. I got lucky. The interviewers liked me. I was competent enough to get the job.

If everything had worked the way I’d wanted it to, starting when I was sixteen, I’d have four kids and be working in a very small town, probably doing remodeling or cutting down trees. If I’d gotten in to MIT or Harvard, I don’t know precisely what the outcome would’ve been, but a friend of mine failed out of Brown and the Yale guy from the wrestling team was selling cars last I checked. If I’d had the supervisor promotion or if the local contracting gig had worked out, I’d still be where I was, being paid mostly in the splendor of Missoula scenery. If the Army contracting gig had worked out, I’d be making an above-average wage in Augusta, eating good food on the cheap and living an upper-middle-class existence.

SoFi just raised a billion dollars at a $4 billion valuation. I’ve got a very small piece of that. My salary is a rounding error on the value of my stock options.

Success is a local maximum. If you never fail, you can never achieve anything greater than what’s in front of you at the moment. If you always get what you want when you want it, you never have to learn to create any value. I’m not taking any credit for the good things that have happened to me; if anything, I’m doing better because i was incompetent earlier in life. Maybe I’ll continue to fail. But if I do, maybe my continued failure will allow me to find better and better things later. One can only hope.

Breaking (and Fixing) Play’s CSRF Protection

Cross-site request forgery (CSRF) is sort of the opposite of cross-origin request scripting. In a CSRF attack, the attacker coerces a victim’s browser to make a GET or POST request to another site – no PUTs or DELETEs or other actions. If your site follows best practices, GETs don’t change any information, and so a CSRF GET request is relatively benign. That still leaves POST requests vulnerable.

The Play Framework provides protection against CSRF attacks by placing a token in the user’s session, a copy of which is submitted along with every POSTed form. If the attacker grabs the page on which the form appears and stores the CSRF token he sees in the form, the victim’s browser will have a different token generated when it loads the page, and the two won’t match – Play will respond to such a POST with the message

or

and evaluation of the POST request short-circuits.

In our world-facing application, we store some information regarding our customer’s behavior in their session, including the type of product for which they’re in the process of applying. That lets us look up the correct product ID in the database, lets us easily verify that the customer owns the application for which they’re requesting the information, and in some circumstances allows us to route them directly to the relevant page rather than having to visit our landing page first.

Now, if the customer does visit the landing page, or if the same browser visits the registration page, we have to clear their session to avoid leaving the wrong kind of application information in place. In fact, if they’re visiting the registration page, it’s probably because they’re a new user on the same machine, and so we should just clear the session entirely.

That makes perfect sense, if we’re only thinking about the session information we use for business logic in the rest of the app. Unfortunately, the session also includes that CSRF token we mentioned before. The browser keeps the registration page cached, as it does with most pages, and includes the CSRF token. The server, on the other hand, wipes out the user’s session information, and so if you hit the registration page without logging out first, you’ll have a CSRF token mismatch and Play won’t let you register.

We saw this problem infrequently, and it seemed always to go away on its own. One day we noticed that it seemed to be happening to customers referred by one of our corporate partners. “Oh, that makes sense,” we thought, “something is wrong with the corporate partner token.” We totally disregarded the capitalized “CSRF” and got stuck on the registration token we were handing to our prospective customer’s employer, encouraging the guy who handles that system to fix his obviously broken code.

One day, I was looking at an unrelated issue, and needed to create a few accounts in quick succession. Immediately I saw the “missing token in request body” error, and I could then reliably reproduce it. The reason we saw it more frequently from our corporate partner was that they’d often have a spouse or domestic partner register from the same machine: it was correlated to the corporate partnership, but otherwise unrelated.

The eventual fix was easy. Instead of saying something like:

I instead did:

The redirect() forces the browser to forget its caching and make a new request to the page, which re-generates the CSRF token since the existing session had been cleared. You could do something similar with web server configuration, telling the browser not to cache anything, but this way we can handle it in code and retain the benefits of caching in all other situations.

The CORS of the matter: Cross-Origin Requests in Play 2.4.x

Building RESTful APIs can make it easier to enforce certain decoupling rules, but it often imposes an additional infrastructural burden, depending on your tech stack. Lately, I’ve been struggling with some of the inconsistencies between our Bamboo-built Docker containers and the locally-compiled version of what should be exactly the same software: it’s not that the bytecode is different, but the boundary between the separately-Dockerized MySQL database and my code’s container was enough to give Hibernate fits until I figured out the necessary library import.

Furthermore, even serving requests via HTTP can be different. One potential pitfall is in CORS filtering, wherein the browser itself protects us from what appears to be a different (possibly malicious) domain. My testing in Paw, an OS X REST client, worked just fine. My coworker tried to hit the deployed API from Chrome and got no result. The confusion was brief, but it’s still a bit of friction.

The linked blog post above describes the concept effectively, but it’s targeted at an older version of Play, using Scala. We’re using Play 2.4 and Java.

Fortunately, once the term ‘cors’ is in your googlesearch toolbox, you’re able to find Play’s documentation on the subject.  In order to open the API wide for development purposes, I added the following to our application.conf:

Which allows us to make a browser request and see the expected JSON without having to fire up Paw.

Java 8 LocalDate and LocalDateTime StreamCorruptedException

Java 8 finally brought the long-awaited equivalents of Joda’s LocalDate and LocalDateTime classes to the core Java API. They’re available in the java.time package. Unfortunately, JPA had been a little slow to pick up on the changes, and so persisting the new LocalDate and LocalDateTime properties of your entities is kind of a pain.

JSON serialization of your entities is actually pretty straightforward, following the example set in this blog post by Chris Ritchie.

Persistence, on the other hand, relied until recently on a custom converter, as described here by Adam Bien. Running locally, we were able to make good use of a converter on our current project. Unfortunately, for reasons that are still unclear to me, we started seeing StreamCorruptedExceptions when we went to run our app inside a Docker instance and talk to a MySQL instance inside a different Docker instance.

I attributed this problem to a number of different sources, since “StreamCorruptedException: Invalid Header” doesn’t hit on anything date-specific on the first few pages of Google. I gutted the entire object model and removed all the @ManyToMany and @ManyToOne relationships. I removed our convenience methods. I re-configured the build plan and our Play! configuration files. Nothing worked.

At some point, I took the “invalid header” reported in the exception and put it through a hex-to-ascii converter. The header was always the same, and it was always the binary representation of “2015,” which led me back to examining our LocalDate and LocalDateTime behavior, including the converters.

I gutted the converters, and had them return empty strings, nulls, or the current date’s string representation, which I thought would allow us to at least get objects out of the database with garbage dates attached to them.  No such luck. I copy-pasted other people’s converters from a few different sources. I even chased our Serializers around, since the StreamCorruptedException eventually manifested as a SerializationException. Nothing worked.

I called in some of the more senior people from our team. They shrugged. I had hoped it would be obvious.

This morning, having slept, I got up to take another crack at it. Now that I know what to look for, the solution is obvious, and a bunch of blog posts and stackoverflow answers are being updated to include it: you can just add a dependency on the hibernate-java8 library, and you’ll be good to go. In SBT, that looks like this:

I spent the better part of three days chasing that around. I’m not totally comfortable with the fact that the converters never worked, but at least we can keep moving forward on this project.