Amazingly bad APIs

I actually don't hate the Java language (at least it's killing off c++), but every time I use it I'm blown away by how hostile the APIs are.

For example, let's say that we want to scale an image file. That's a fairly common thing to do, so it's probably only one line of code, right? Wrong. It turns out that they haven't even bothered to include a reasonable API for doing this. Instead, you have to take the awkward APIs that they do provide, and then write about 50 lines of code to do what you really want. This article that explains what the "right" way is.

Once the image is scaled, you'll want to save the smaller version too. Luckily, that's only a few lines of code. Of course that's only if you're happy with the default settings -- what if you want to use a different JPEG quality level? Here are the docs -- see if you can figure it out (the solution involves 4 additional classes).

For comparison, here is some Python code which loads an image, scales it, and then saves it at a non-default quality level:

from PIL import Image

i = Image.open("/tmp/c.jpg")
i.thumbnail([220, 133], Image.ANTIALIAS)
i.save('/tmp/c-thumb.jpg', quality=90)

Doing the same thing in Java requires closer to 100 lines, not because the language is bad, but because the APIs are terrible.

How did this happen? My theory is that the people who made the Java APIs just sat in a room making stuff up (and possibly drawing UML diagrams). Meanwhile, the Python people were actually writing software and creating libraries to make their life easier.

There is a lesson in this: If you are building a platform, you should also be writing applications for that platform, and the platform should be designed to make life very simple for those apps. It's my understanding that Ruby on Rails came out of a process such as this (they extracted the common tasks out of their web apps). If you just sit around staring at the clouds while writing your APIs, you'll probably end up with something bad.

I also have a second theory about what happened to Java: it attracted people who like to write piles of code filled with endless abstractions. When I worked at Google, one of the engineers on our Java mailing list suggested that we ban the keyword 'new'! I was puzzled by this suggestion, but after a little research I learned that one of the hot fashions in Java is to use factories instead of 'new' (not that factories are bad, but "sometimes good" does not imply "always best"). Better yet, you don't reference the factories directly (not enough abstraction), instead you have some framework that injects the factories into your classes according to what is written in an XML configuration file. In this way, you can make your code perfectly unreadable, but more importantly, it is very abstract.


Update: Several people have interpreted this post to mean that I'm opposed to the factory pattern. That is, of course, silly. Factories, when the situation calls for them, are fine. For example, the python code above is using a factory. What harms code is the blanket application of the factory pattern in every possible situation. To put this another way, Tabasco sauce is delicious on eggs, but I still don't put it on ice cream. (but don't think I haven't tried!)

Update Two: It's not about Java, really! My point here is that your APIs should emerge from the needs of real applications, and that they should make common tasks super-easy. Bad APIs are common in many languages, Java just happens to be the language that I'm using right now.

Why didn't I simply hunt down+install some third-party image library? Because that's a hassle too (probably more work than copying code off of some web page), and all I want is to generate nice looking thumbnails.

Flash, Silverlight, and JavaFX all look nice, but what I really want is better file upload

One of the most annoying limitations when writing web apps comes from the <input type="file"> form element, which is the standard way of uploading files.

Here are a few of the problems with <input type="file">:

• You can only select a single file per input element, and there is no way to select entire folders.
  
• Web browsers just pop-open the standard file open dialog, which is often not very good. For example, the dialog probably just shows a list of files, not thumbnails. This can make it difficult to pick the correct file if they all have names like IMG_3485.JPG, and it's therefore easy to upload the wrong file.
  
• There is no way to actually read the data locally without first sending it to the server and then having the server send it back.
  
• There is no way to do any local processing on the data, such as scaling images to make them smaller.
  

Flash apparently solves the multiple file problem (kind of), but it's my understanding that it still has all of the other problems (I'd love to be wrong about this though).

Java applets can actually do most of these things if they are signed and the user agrees to allow the applet access to their filesystem. Unfortunately, Java applets don't actually work, and after 11 years of not working, I'm not expecting that to change. Facebook has a nice image uploader applet that lets you browse your photos locally and then scales them before uploading to make the whole thing go fast. The only problem, other than the annoying "do you want to trust this applet" prompt, is that it crashed my browser the second time I tried it.

To me, the great thing about flash is that it fills in basic capabilities missing from the browser, such is video, sound, camera and microphone access, and they did it with a reasonably small and stable browser plugin that I'm not afraid to allow onto my computer. A few months ago, I accidentally upgraded one of my computers to Flash 9 -- I don't remember how exactly, I just remember that it was simple and non-eventful.

Here's my idea: I want to create a small, reliable browser plugin that does for file uploads what flash does for audio and video. It should work with all the modern browser/os combinations, and install with minimal effort and fuss (browser restarts avoided if possible). It would be free and open source, but hosted in a common location for easy install.

Popular Ajax libraries such as jQuery, YUI, and Dojo can add support in such a way that people who don't have the plugin just get the regular browser upload control (and maybe a link to install the plugin), while those who do have the plugin get a better ui. Clearly, getting people to install a new plugin isn't easy, but once any site convinces them, their experience automatically gets better on all other sites that support the plugin. There isn't a chicken&egg problem because the plugin simply provides an enhancement to the already existing (but very poor) browser functionality, so sites can easily add support without fear of losing people who don't have the plugin (unlike building a whole app in silverlight, or whatever).

I'd like to hear from anyone with experience writing browser plugins. How difficult is the technology side of this? (making it small, reliable, and stable -- crashing browsers is not ok) Also, are you interested it writing this plugin? I'm serious about this, and willing to pay quite a bit of money to make it happen.

Here are a few more details on how I imagine the plugin working:

It would not have any in-browser UI -- it would simply be an object that can be called from javascript and that would pop open a nice file browser with thumbnails and all that (similar to windows explorer, perhaps). It would also provide upload-related utility functions, such as image resizing. Since the file access is read-only and limited to what the user selects, security shouldn't be a big problem, assuming of course that the plugin implementation is secure (and it must be).

Here is some JS illustrating how I imagine using this plugin (UploadHelper is the object provided by the plugin):

// Have the user choose files or directories to upload.
// Returns undefined if user clicks Cancel. Doing this with a
// callback instead of a return value would also be ok.
var files = UploadHelper.chooseFiles(
              {filter:'*.jpg,*.gif,*.png', allowDirectories:true});
if (files) {
  var s = 'You selected: ';
  for (var f in files) {
    var file = files[f];
    s += (file.name + " (" + 
          file.imageinfo().width + "x" +
          file.imageinfo().height + ", original size = " + file.size);

    files[f] = file.scaleImage({maxwidth:100, maxheight:100, quality:5});
    s += " scaled size = " + files[f].size + ") ";

    // If I could somehow insert this image on the current web page
    // too, that would be really awsome.
  }
  alert(s);
  
  // using jquery XMLHTTP wrapper
  $.ajax({
    type: "POST",
    url: "/upload",
    data: UploadHelper.encode(files, 'multipart/form-data'),
    contentType: 'multipart/form-data'
  });
}

var files = UploadHelper.chooseFiles({filter:'*.txt'});
if (files) {
  // We can also read the content of the selected file
  // The ability to get chunks of a file also means that it's possible
  // to restart failed uploads, and to chunk large files into multiple
  // POSTs.
  for (var f in files) {
    var file = files[f];
    var d = file.getData(0, 50); // get the first 50 bytes
    if (file.size > 50) d += "...";
    alert(files[f].name + ": " + d);
  }
}

Update:

Several people have suggested that I should instead try to get this feature added to web browsers or (better yet) the web browser standards. That's a fine long-term solution, but realistically it will take years to happen. I would rather have an immediate (months, not years) fix in the form of a plugin, and then let the web browsers add it as a standard feature on their own schedule. This is somewhat similar to the path followed by XMLHTTP.

Sriram Krishnan of Microsoft makes a few good points:

• The standard file open dialog in windows already has a thumbnail view. Unfortunately, this feature is hidden under a completely unremarkable icon and despite having used windows for many years, I've never seen it before today. This is a great example of why simply adding a feature isn't good enough -- it also needs a good ui or else it may as well not even exist. Furthermore, not existing seems to be the status of the thumbnail view on OSX -- hopefully that will be fixed in 10.5.
• This feature request could be implemented in Silverlight. That's great news! MS should write and release a little Silverlight control that provides these functions to JS. I would be very hesitant to write (or rewrite) my apps in Silverlight, but if I could simply embed a little Silverlight module the same way that I would a flash mp3 player, then I'd definitely use that. This would be a very effective way for MS to get people started installing and using Silverlight (start with something small and optional).

Silverlight sounds pretty nice. Google or Adobe had better have an open alternative on the way, or else MS may be able to recapture the application platform.

Update Two:
I believe that within a few months we will have a few reasonable solutions to this problem!

When people don't know how to change something, they often start searching for a way to justify failure, rather than thinking about how they could try doing something different to make it work.
...
If we pretend that you can teach anyone anything, we'll find out where it's not (yet) true. But if we think that when someone isn't learning it means they can't be taught, no one will even try.
...
If you take the attitude that anything is possible, you'll find that a lot of things that were previously thought impossible actually do become possible.

Page 125

Beliefs, intelligence, and failure

From the book "Hard Facts, Dangerous Half-Truths, and Total Nonsense" (via this blog, via an email from a friend):

A series of studies by Columbia University's Carol Dweck shows .... that when people believe they are born with natural and unchangeable smarts ... [they] learn less over time. They don't bother to keep learning new things.

People who believe that intelligence is malleable keep getting smarter and more skilled ... and are willing to do new things.

That's a pretty important fact. It's also an excellent example of how your success is determined by your beliefs/mindset/frame. If you believe that you can't get smarter, you won't!

I searched around a bit and found another interesting quote:

...beliefs about the nature of intelligence have a significant impact on the way they approach challenging intellectual tasks: Students who view their intelligence as an unchangeable internal characteristic tend to shy away from academic challenges, whereas students who believe that their intelligence can be increased through effort and persistence seek them out.

Students who hold an "entity" theory of intelligence agree with statements such as "Your intelligence is something about you that you can't change very much." Since they believe their intelligence is fixed, these students place high value on success. They worry that failure-or even having to work very hard at something-will be perceived as evidence of their low intelligence. Therefore, they make academic choices that maximize the possibility that they will perform well. For example, a student may opt to take a lower-level course because it will be easier to earn an A. In contrast, students who have an "incremental" theory of intelligence are not threatened by failure. Because they believe that their intelligence can be increased through effort and persistence, these students set mastery goals and seek academic challenges that they believe will help them to grow intellectually.

Not only does a belief in fixed intelligence prevent learning, it also makes people more risk averse! It's easy to see how some cultures or micro-cultures make people successful, while others keep them mired in failure.

I expect that this same basic principle applies to most things, whether it's skills with computers or skills with people. Obviously "nature" plays some role, but beliefs, knowledge, and determination are usually more important. Furthermore, I suspect that you can learn to enjoy these activities, and that finding the fun is the key to mastering the skill.

And you thought all the good domain names were taken?

Using only the magnetic letters stuck to my refrigerator, we were able to discover these promising, yet unregistered domain names!

spitbomz.com
xombiz.com
lisp.tv (imagine the media empire that could be built around that!)
xzombi.com
QykDr.com (quick doctor)
nevrqyk.com
WakiDr.com
FuziXombi.com
phuzix.com
mowbul.com
t-mubal.com
WakyXombieFujPHD.tv


And venturing beyond the refrigerator, we also found these gems:
un4tun8.com (I went ahead and registered this one)
4nic8.net
t-mobull.com
grrrrrrrr8.com (that has eight 'r's)

There's a goldmine in replacing 'for' with '4' and 'ate' with 8.

Don't limit yourself to good ideas...

Whose reality are you living in? Whose reality would you rather live in?

Mental frames are the biased and limited way in which information is perceived or understood. Because the human brain is inherently biased and limited, we are always in some mental frame. That frame determines how we relate to and understand reality. As far as you can tell, that frame is reality.

This is why the world seems better when we are happy, and worse when we are sad. It is also a reason that people on "the other side" of an issue seem so stupid, misinformed, and out of touch with reality. Not surprisingly, mental frames are often discussed with respect to politics. However, their scope and importance goes well beyond that -- they ultimately determine not only our perceptions, but our whole mindset about what is valuable, practical, or dangerous, and what behaviors are responsible and acceptable.

If someone pursues their passions, are they boldly living life to the fullest, or are they simply being frivolous and irresponsible? The answer, of course, depends on your frame.

Perhaps you might prefer to be objective about reality and escape these limiting frames. Unfortunately, that's not really an option, at least not while we are stuck with these monkey brains inside our heads. Although it is nice to attempt objectivity, we must accept that we are human, and therefore limited. To deny that and claim true objectivity is to deny the truth and be stuck in a very limiting and annoying frame.

Mental frames help to explain why some people needlessly stay in bad jobs, bad relationships, or other bad situations -- in their reality, it makes sense. Clearly, some frames are better than others (from my perspective :).

Here's the good news: You can switch frames!

Many influences can shift your frame, such as reading books or taking a walk in the park, but one of the most powerful influences is the people around us. We tend to synchronize frames with our friends, family, co-workers, and anyone else we encounter (including the people on tv), though obviously some of those people are more influential than others.

Therefore, if you want to change your reality, change your surroundings. Find people with a more attractive reality, and live with them. This is very important. When you spend a lot of time with people, their reality becomes your reality.

For example, if you are interested in startups, but work in a big company, you are in danger. If you stay there too long, you will be drawn into the big company frame shared by the other "lifers". Startups will all seem too risky, frivolous, or impractical, and you'll spend the rest of your life in that big company (and posting bitter comments on TechCrunch).

Similarly, if you dream of pursuing some other career or lifestyle that is not shared by the people around you, then you either need to accept that it's not going to happen, or you need to change your situation.

Please note: I'm not suggesting that you completely cut off contact with people outside of your desired frame (that's how cults operate, btw). To the contrary, it's good to keep in contact with a wide variety of people -- that will help provide perspective and keep you from becoming trapped in your new frame (which might not be as great as it seemed from the outside). What I am suggesting is that your target frame should have significant representation in your life (like 50%), or at the very least, some minimal representation (5%) so that it doesn't completely fade from your reality.

Update: To better understand frames, think of a time when you were excited about an idea or possibility (or anything else), but when you shared that thought with someone else, they somehow ridiculed, doubted, or otherwise criticized it. How did that make you feel about the idea? Were you a little less excited, more doubtful, or perhaps even somewhat embarrassed about it? If so, you've just entered further into their frame -- their reality is becoming your reality. If you want to nurture your dreams, it's better to share them with people whose frame is compatible with the dream.

Read my previous post on reality. What am I doing to your frame?

Update Two: One of the commenters on news.yc suggests that if everyone really were in their own frame/reality, then it would be impossible to communicate or build products for other people. This is a good point, and it would be true if our frames were completely disjoint. Fortunately, they are not -- we always have something in common. However, the more our frames differ, the more trouble we have communicating. This is why it can be so difficult to communicate with a broad audience, such as on a blog, and why I've decided to write primarily for those with similar frames (because it's easier).

For more thoughts on how your mental frame affects your life, there are some interesting posts on Steve Olson's blog (though he uses the term 'belief system').

The secret to making things easy: avoid hard problems

That may seem obvious, but in my experience most engineers prefer to focus on the hard problems. Working on hard problems is impressive to other engineers, but it's not a great way to build successful products. In fact, this is one of several reasons why YouTube beat Google Video: Google spent a lot of time solving technically challenging problems, while YouTube built a product that people actually used (using PHP and MySQL, I think, which is not at all technically impressive).

For me, the most effective method of getting things done quickly is to cheat (technically), take a lot of shortcuts, and find an easier way around the problem (and before anyone jumps in with some comment about security or bank transactions, there are obviously a few exceptions). You only need to think ahead enough to avoid painting yourself into a corner, or have a plausible plan for escaping the corner. There's always an easier way -- work lazier, not harder. Note that this doesn't preclude doing things that SEEM difficult -- easy solutions to important problems that LOOK really hard are the best.

I was reminded of this while replying to the comments on news.yc in response to my post on disks and databases. Whenever anyone mentions the possibility of not using a conventional database, a lot of people will immediately reply that databases solve a lot of very difficult problems, and that you shouldn't put a lot of work into reinventing the wheel. These people are, of course, correct.

The thing is, a lot of those difficult problems are irrelevant for 99% of products. For example, "real" databases can handle transactions that are too large to fit in memory. That was probably a really important feature in 1980. Today, you can buy a computer with 32GB of memory for around $5000. How many GB transactions do you suppose Twitter performs? My guess is zero -- I suspect that their average transaction size is closer to 0.0000002 GB (messages are limited to 140 characters).

I want to be perfectly clear about one thing though: I'm not advising you to ditch your database! If your database is plenty fast, then the easiest thing to do is probably "nothing", and that's what I advise you do. If, however, your db is getting slow or overloaded, then you need to do two things:

1. Understand the problem
   
2. Fix the problem

The correct solution to your problem will depend on your situation. For example, if you have some data that's very important but doesn't change very often (username and password), and some data that gets updated continually but doesn't have to be correct (last active time or hit counters), then a simple solution would be to leave the important data in your database and move the less important data into something really simple but less reliable.

Want an example of "simple but less reliable"? Here's one (in one or two easy steps):

1. All updates go in to memcached, but not the database
2. (optional) A background process occasionally copies entries from memcached to the db. Without this, the values will be completely lost when memcached restarts.