Hacking Hacker News, now in JS!

Do you like reading Hacker News? A lot of people do, but they usually get the same problem after a while: There are just too many stories submitted each day.

If you really want to read them all, you will spend a large amount of your copious free time reading stuff you don't actually care about. Joel Grus, a future famous author, had the same problem and found a solution: Training a naive Bayes classifier to recognize interesting and boring stories.

If you haven't already, you should really take a look at his post, complete with code. Here is what he does in a nutshell:

1. Grab new story submission data from the unofficial HN API.
2. Rate the submissions by classifying them using only this data: Title, link, submitter.
3. Sort the stories by interestingness.
4. Post them to a blog.

Voila, you know have your own HN RSS feed, sorted by interestingness.

I really like the idea of adding some kind of automatic filtering to a news site. Maybe my tastes changed or HN did, but I just don't find most of the stories posted there to be that interesting anymore.

Instead of reading the RSS feed, I usually just read the HN homepage. This will of course filter out a lot of boring stories already. But there are still many stories I don't want to read that do show up. I don't want to read stuff about Occupy on HN, I already read BoingBoing. I also don't need too many stories about Apple, I can just read Folklore.org.

So, being the copycat that I am, I adapted Joel's idea to my habits: I also wrote a naive Bayes classifier, a way to train it and to rate stories. There is one rather big difference between Joel's version and mine, though: This one is a JS bookmarklet that augments the HN homepage, running completely in your browser.

Adding a classifier to the HN homepage

Writing a naive Bayes classifier is quite easy, and you can probably do it in a day or two. I wrote a rather generic one in JS, the code is on Github. Don't look too hard at the code, I am a JS novice. If you clone the code and try the demo page, you can train a small classifier yourself and play with it a bit.

Writing the algorithmic part is easy, the complicated part is actually packaging it into a useful bookmarklet:

• How do I get the text data for training and rating?
• How do I preserve this data across browser sessions?
• How do I control this from the HN homepage?

Luckily, these things are all very doable in today's Web 2.0 world.

To get the data in, I use the very handy ViewText, which will reduce any web site to it's main text content. Think of it as Readability with an API. I use JSONP to retrieve the content of a story and then use this for training or rating. The results will still contain some markup, but as this is evenly distributed among interesting and boring stories it should not affect the rating.

The training data is saved as a simple hash with this structure: {"apple": {"interesting": 5, "boring": 2}}. This means that in all the documents used for training, the word "apple" appeared 5 times in interesting ones and 2 times in boring ones. So if you now find the word "apple" in a new document, the document is probably interesting.

This hash (and a bit of additional info) is saved in the browser using DOM Storage, which is a bit like cookies on steroids. I just read the saved hash into memory when the bookmarklet is executed and save the hash after every additional training. It's a simple solution and I guess it will come in handy more often.

Integration with the HN homepage is also not that hard: Just add some icons after every story for training as good, training as a bad and giving the current rating. This is easy with jQuery and did not take me too long.

And that's pretty much it! If you want to try it yourself, just drag the HNBayes bookmarklet to your bookmarks bar, visit HN and click on it. You should now have the additional buttons for training and rating. Rate some stories as good or bad by clicking thumbs up or thumbs down, then see what the classifier thinks by clicking on the eye.

UPDATE: It seems that Posterous will not let me post links with javascript inside them. Nice. While I figure this out, just copy this into a bookmark:

javascript:(function(){var script = document.createElement("script"); script.src="https://raw.github.com/rogerbraun/HNBayes/master/hnbayes.js"; document.body.appendChild(script);})();

If you want to see the code, look here.

Final remarks

It's wonderful what one can do today with some of the newer additions to browsers, like DOM storage. Things that I previously thought were pretty much impossible client-side now seem to be almost trivial. JS has really come a long way since the days of alert() and document.write().

The concept of rating links automatically is, of course, not restricted to HN, but could be extended for any other site. I like that, with this solution, you can have a personalized experience while still not giving anyone your data. On the other hand, your ratings are locked away in your browser: Right now, there is no way to sync it except copying the stored hashes.

Some pages will not rate or train with this bookmarklet. This is because ViewText can not read them and will give an invalid response. I have seen this happen with stories on WSJ, HN itself and with PDFs. I don't really plan to fix this, as most sites will work just fine, but who knows...

If you like or dislike this approach (and my execution of it), you may want to leave a comment! I'd like to hear your thoughts.

Permalink | Leave a comment  »

Some of my friends are learning Javascript this year, thanks to the great folks at Codeyear. Of course, the one thing everybody wants to do is put random dinosaur images on every page. But how can you do that?

Searching for random images

You probably know Google Image Search, or Bing Image Search. They can deliver somewhat random images for pretty much anything you'd want. Just search for something and get a picture - nice!

Now you all know how to search via the usual HTML interface, but these services often also have an API. For example, here is the documentation for Googles Image Search API. It's deprecated, so don't use it for anything serious, but it should be fine for us.

So what's so special about using an API? The nice thing about it is that it gives us just what we need and nothing more: A Javascript object with all the information we want - picture url, picture caption, thumbnail... We don't need all the other stuff that is only there for humans to look at. But how do we access this information with an API?

Accessing the Google API is done via HTTP GET, so you can just copy the API-URIs into your browser and you will see the returned Javascript object. The API has a base URI - https://ajax.googleapis.com/ajax/services/search/images - and takes at least two parameters: v, which is the version number of the API and 1.0 in our case. And q, which is the query string - the keywords you want to search for.

So an URL like https://ajax.googleapis.com/ajax/services/search/images?v=1.0&q=apple will return the first set of results for a search for the keyword "apple". Take a look at the response in the browser, and you will recognize the familiar Javascript Object Notation. Or JSON, as we will call it from now on. Now let's use JS to get this data.

Ajax

If you know a bit about the Interweb 2.0, you probably have heard of the term "Ajax", which means "Asynchronous Javascript and XML". Luckily, it has really nothing to do with XML and does not have to be asynchronous. We can also use it to get JSON data - much easier to work with! Essentially, an XMLHttpRequest object can do everything your browser does: GET data, POST data, with or without parameters, whatever you want. It's an easy way to access data from other web pages from your JS without changing the whole page.

Right now you are surely thinking: "So I just use this XMLHttpRequest object, make the request to my API and use the response in my Javascript code". That's the right idea, but there is a small detail that will prevent this... XMLHttpRequests are subject to the same origin policy. What this means is that, essentially, you can only use it when requesting something from your own domain. Well, I guess you don't host your stuff on googleapis.com, so there seems to be no way to do a request to the API from Javascript. But of course, there is.

JSONP

No, there are parts of HTML which predate the same origin policy. And luckily, the script tag is one of them. You may have noticed that you can use js files from anywhere in the web just fine in your script tags. This is often used to load some kind of standard js, like jQuery, from someone else's server. But now you know the dark secret: You can include ANY javascript from ANYWHERE in the web with a script tag. Just build a script tag with document.createElement("script"), point it to some js file and append it to your document. It will execute just fine.

Now we know how to execute any random Javascript from somewhere else. But how can we use this to retrieve data? We don't want to execute anything, we just want our search results! Now here comes the clever part. Let's say our API usually returns this object:

{caption: "Cute Cat", url: "http://cutecats.com/cat.jpg}

So we have a hash with a caption and an URL. This is a valid Javascript object, but that does not help us. We want that object, but we want it as a parameter to a function we define that does all kinds of stuff with it. So if our function was called "addPictures", we would want something like this:

addPictures({caption: "Cute Cat", url: "http://cutecats.com/cat.jpg});

This would give our self-defined function the response data we want. And that's exactly what a JSONP request does. The Google API takes another optional paramater, "callback". This defines the name of the function that will be wrapped around the response object. Now your inserted script tag will run the remote Javascript, which will call your locally defined function to handle the response.

You can now circumvent the same origin policy. It's wrong, I know. But it feels so right.

Some words about the name

So why is this technique called JSONP? Well, it's supposed to mean "JSON with padding". This is such a bad name. It's not padded, it's wrapped in a function call! And it's not even JSON! It could actually be any kind of Javascript. It should rather be called something like "script tag request insertion" or whatever. I know I am complaining too much, but why give such a misleading name to this technique? Whatever. Now you know what it really does.

An example: Doing an Image Search

Look at this gist that does an image search with Google's API and shows the first four result images.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

<!DOCTYPE html>
<html>
  <head>
    <title>JSON-P Example</title>
    <script src="jsonp.js"></script>
  </head>
  <body>
    <h1>A simple JSON-P example</h1>
    <p>This is a Google Image Search.</p>
    <input id="q" />
    <button id="search">Search</button>
    <div id="images" />
  </body>
</html>

Now, this uses bare Javascript. You might want to use something like jQuery or Ender in your real projects, but this is for educational purposes.

Take a look at the doJsonP function. This actually inserts the script tag and appends it to the document body, so it will get the remote JS and execute it. It also binds the callback function to the global variable "jsoncallback" so it can later use this function name for the JSON wrapper.

So, when you call doJsonP(someUrl, someFunction), your function "someFunction" will be called with the response of the request to someUrl. In our callback function we just append the four result images to our document so we can see that everything works fine. But you could, of course, do anything you can imagine.

That's it. Now you know everything you need to know about JSONP to finally write your "Dinosaurify" bookmarklet. When you are done, be sure to post a link the comments!

Permalink | Leave a comment  »

Here's the backstory.

I noticed that Codebrawl does not yet support Github Flavored Markdown in Gist comments. Gist comments get loaded client-side via the Gist API and are then rendered with showdown.js. Showdown is a great project, but it does not support the extensions Github uses. This really sucks for gist comments which contain code, as they are rendered as regular Markdown-style code spans, which ignores newlines and indentation and makes them completely unreadable.

So I forked the project, added the necessary 30 lines of code and made a pull request. I "tested" my code the stupid way: Look at the output in the browser. This made me feel somewhat uneasy. I want this code to be useful, but how should anyone know that I did not break anything? I decided to add some tests. I have done testing in other languages, but never in Javascript. How do you do this?

Testing your Javascript

How can you test Javascript? JS usually runs in the browser, so you could just make an html page that runs your test code. The downside is that you need a browser and you need to refresh it. I'd rather work only on the command line for this project, as my JS does not really do anything that needs a DOM. It just translates one string to another. So what about testing it in node.js or something?

There is a really nice looking BDD testing framework for Javascript that also supports node.js: Jasmine by Pivotal Labs. It looks pretty much like what I want. It has an RSpec-like syntax and supports all kinds of run options. Having said that, I could not get it to run at all, at least not in under an hour. This was probably my own fault. Still, it drove me to think about what I actually needed and if I could just write it myself.

I really just want two things for this: RSpec-like syntax and RSpec-like output. I like the descriptive "it should do somethign", but I don't need all this beautiful "things.should be_wonderful" stuff, a simple assert_equal would be enough. 

Here is what I came up with.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

var assert_equal = function(a, b) {
  if(a === b){
    return {passed: true};
  } else { 
    throw { message: ("Expected:\n\"" + a + "\"\nGot:\n\"" + b + "\""), passed: false}
;
  }
}

var indent = function(text) {
  return "  " + text.replace(/\n/g, "\n  ");
}

var it = function(name, code) {  
  var res = {passed: true}
  try {
    code();
  } catch(er) {
    res = er;
  }

  if(res.passed){
    console.log("\033[1;32mPassed: " + name + "\033[0m");
    return true;
  } else {
    console.log("\033[1;31mFailed: " + name + "\n\033[0m");
    console.log(indent(res.message));
    return false;
  }
}

This allows you to write tests like this:

1
2
3
4
5
6
7
8
9
10
11

it("should make paragraphs out of simple texts", function() {
  var text = "Hallo";
  var html = converter.makeHtml(text);
  var expected = "<p>Hallo</p>"
  assert_equal(html, expected);

  text = "Hello\n\nHow are you?";
  html = converter.makeHtml(text);
  expected = "<p>Hello</p>\n\n<p>How are you?</p>"
  assert_equal(html, expected);
});

Looks kinda okay! Now to run the tests, just run the specs through node.js. Here is a screenshot.

Nice! How does it look if it fails?

Also nice! I think I'll use this for now.

This is of course not a full featured solution. Right now, it only has an assert_equal method to test things - you could easily add more methods, though. I found that this mini-framework works quite well for my DOM-less algorithmic javascript, without introducing any external dependencies. If you have a similar problem, you might want to check out my showdown fork at https://github.com/rogerbraun/showdown and see how I use this.

Have fun!

Permalink | Leave a comment  »

The selective color effect, implemented with ChunkyPNG

So, I won Codebrawl #10. I participated the previous time that ChunkyPNG was featured and I really enjoyed using it. ChunkyPNG is a pure Ruby library for handling PNG files. One of the nice features it has is that it gives you direct read-write access to the pixels, so you can just load an image, do a map or fold over the array of pixels and then save it right back to a new PNG. ChunkyPNG is well documented and gives you several easy methods to work with pixels and colors, so be sure to take a look. 

When I started to work on this Codebrawl, I thought about two ways to do this: Try to detect each crayon as an object and keep one of them in color, or take a reference color and just keep the colors that are similar to it. Object detection would be great, of course. Just changing colors based on reference does not work in cases were you want to, for example, keep a flag in color. Okay, it does work when you use it on a Japanese flag... Just using a color reference and a distance does work great for the example image that the Codebrawl uses, though, and implementing object detection is a much harder problem than mapping over colors.

I think I know how to iterate over an array, so all I was missing was a distance function for colors. But how do you actually compare colors?

How to measure the distance between colors

There are several ways to get a value that tells you how much alike two colors are. If you have an RGB pixel, you could just take the red, green and blue values as coordinates in a three dimensional space and calculate the distance there (and I guess you know how to do that).

I did not actually implement this, as it does not represent similar colors in the way we would see them. Think about it, the color (255, 0, 0) and (127, 0, 0) are both just shades of red, but they would have the same distance as (255, 0, 0) and (255, 128, 0), which introduces green to the mix and looks pretty different overall.

This also tells us something about what kind of representation we want: One where only the color counts, but not the brightness (or luminance). Skipping through Wikipedia, I first noticed rg chromacity space.

rg chromacity

rg chromacity is a simple way to remove intensity information from your colors and only keep the proportions of red, green and blue. You normalize the values to be between 0 and 1, with r + g + b always adding up to one. It is called rg chromacity because only the red and green components are needed to describe a color, as the blue component is always b = 1 - (r + g). For example, rgb(255,0,0) is rg(1,0), as is rgb(127,0,0).

You now have a 2d space, so distances can be easily calculated. Using this to decide which colors to keep gives somewhat satisfactory results, at least for some of the crayons. But in the end, it just was not as good as I had hoped. Part of this surely is the way this color space actually looks. It is somewhat uneven, as the distance between pure red, rg(1,0), and pure blue, rg(0,0), is one. But between red and green, rg(0,1), it is the square root of two!

By just measuring the distance, I am essentially cutting a circle of colors I want to keep out of this triangle, but colors in this space are not uniformly distributed. I had a lot of ideas how to counter these problems: Make colors spread from a reference color if the neighbors are similar, make negative cuts in this space by specifying colors that should always be made gray, etc... What I really needed was a better distance function for my colors. So let's use a grown-ups color system, HSV.

HSV

If you go looking for color spaces, you will encounter HSL and HSV pretty soon. They use hue, saturation and lightness (or value, respectively) to define a color. If you take a look at some pictures of this color space, you will see that the hue component looks pretty much like what we need.

Now, calculating the hue is more complicated than calculating rg chromacity colors. If I understood it correctly, you make a hexagon of colors, put red at 0°, green at 120° and blue at 240° and then calculate where your color lies. You can see the formula I used in the code.

Calculating the distance is just subtracting one hue from the other. We have to do it two times, though, as the hue is circular and has red on both ends, so the shorter distance counts. This gives good results for most colors. I could not get the yellow and red crayon to seperate perfectly... This may be because I did not find a good reference color, or just that the dark yellow in the tip of the crayon and the light red are actually too similar and this approach won't work at all. You should take a look at the other entries and see how well they did.

Better color distance

I tackled the problem of color similarity from a rather primitive point of view: Numerical values of single pixels. As it turns out, this is not enough to mirror human color perception. Take a look at this: http://gizmodo.com/5839481/the-most-wicked-optical-illusion-ive-seen-so-far. Both spirals have the same color, but you would never think that if you didn't know or check. As far as I know, no algorithm that works for cases like this exists, yet.

There is a standard for measuring color difference by the International Commission on Illumination (sounds good, right?). It takes into account how humans perceive color, so it should give you the best results (if you are human, that is). I did not try it, though, as it seems somewhat complicated and I wanted to keep this entry short and to the point. Peplin did it like this and his code is much more complicated, but still quite readable. One problem with this approach is that you have a lot of magic numbers that you get from the standard definition. These are (at least for me) hard to visualize and I find it hard to grasp their meaning.

Anyway, here are my pictures and my Ruby code.

See the full gallery on Posterous

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

module ChunkyPNG::Color
  # See http://en.wikipedia.org/wiki/Hue#Computing_hue_from_RGB
  def self.hue(pixel)
    r, g, b = r(pixel), g(pixel), b(pixel)
    return 0 if r == b and b == g 
    ((180 / Math::PI * Math.atan2((2 * r) - g - b, Math.sqrt(3) * (g - b))) - 90) % 360
  end

  # The modular distance, as the hue is circular
  def self.distance(pixel, poxel)
    hue_pixel, hue_poxel = hue(pixel), hue(poxel)
    [(hue_pixel - hue_poxel) % 360, (hue_poxel - hue_pixel) % 360].min
  end
end

module SelectiveColor
  # Really simple, just change the pixels to grayscale if their distance to a
  # reference hue is larger than a delta value.
  def to_selective_color!(reference, delta)
    pixels.map!{|pixel| ChunkyPNG::Color.distance(pixel, reference) > delta ? ChunkyPNG::Color.to_grayscale(pixel) : pixel}
    self
  end
end

BONUS: Coffeescript version

I also wrote a Coffescript version of the same algorithm which you can use to quickly check the effects of changing the reference color or the color distance. Porting it over from Ruby was pretty straight-forward, except for the strange interface for pixels that the HTML5 canvas element uses and the fact that modulus is broken in Javascript (WTF?!). I tried to take a functional approach, but I am not a Coffee/Javascript programmer, so this code is rather bad. If you are interested anyway, you can see the code at GitHub.

You can find this version ready to use at http://severe-autumn-9391.heroku.com/index.html. Use the slider to set the color distance that will still be colored, and just click on any point in the image on the right to set it as the reference color. Have fun!

Permalink | Leave a comment  »

Remember when we talked about Maxixe? We talked about how to use it to train a very simple segmenter and split a sentence without spaces into single words. But now you will probably want to try it yourself, for real, in your big enterprisey next big thing! But how do you actually do that?

Ingredients

Maxixe needs three things to build a segmenter:

1. A large corpus of text data. This should be as close to your production data as possible.
2. A set of n-gram counts from this corpus. For example, all 2-grams, 3-grams and 4-grams from the corpus are counted and then saved as the index.
3. A threshold that controls how lhigh the score for a possible segmentation has to be for it to be cut. Simply put, the lower this is, the more segments you will get.

The corpus

So how do you get hundres of megabytes of text in your target language? This can actually be pretty easy! For example, Wikipedia has dumps of their database for all languages, and some other gals and guys already figured out how to get a plain text corpus from them. This can be a bit much and you will probably want to try only a subset of Wikipedia. Also remember: After using Wikipedia as a corpus, your segmenter will be trained to split encyclopedia articles, but it will not necessarily be well equipped to split blog comments or 4chan memes. 

Another source can be Project Gutenberg, which has a lot of free books in plain text format, ripe for the taking. Similar pages are available for more specialized areas. For japanese, there is Aozora Bunko, for classical japanese you can take a look at the Japanese Text Intiative. Just type "$favorite_language text" in Google or DDG and use whatever comes up. Okay, if you need Chinese, I searched for that, too.

Anyway, you now have your corpus. AWSM! What to do next?

Building your index

When you use the segmenter, you can actually change the threshold value anytime you like (you should not, though). But you will have to decide in advance which n-gram count indexes you want to have, as these have to be pre-computed. So which n-grams should you index?

I know that every time I write "n-gram", someone stops reading. What is this thing? An n-gram is just a string of length n. So let's say we have the string "nyan" and want all 2-grams that are in it. Easy! Just start at the beginning and take two characters each time. So for "nyan" we get ["ny", "ya", "an"]. The 3-grams would be ["nya", "yan"]. 

Now, remember that the Tango algorithm that Maxixe uses depends on the fact that n-grams that are part of words or words themselves are more likely to occur in a corpus than n-grams that are not. A simple example: If you have a large corpus, your 3-gram count for "cat" and "dog" will probably be rather high, but the count for "qxc" will be rather low, as a word would need to end with "qx", begin with "xc" or even contain the string "qxc" completely if this 3-gram could be found in the corpus.

So, which values for n should you choose? This is not an easy question. Generally speaking, small n values will lead to small segments and vice versa. But sometimes you may, for example, need n-grams for n = 2, 4 to get the best recognition performance. You will just have to test it.

Over the threshold

There is this last parameter you need, the threshold t. This is the cutoff value. If any position between two characters in the string has a calculated probability value that is larger than the threshold, it will be split at this point. Why do you need this? Usually, positions are split if the calculated value for the current position is a local maximum, which means that it is more likely that a word boundary lies on the current position then on the one directly before or after it. But sometimes, this just isn't enough. A simple example is the word "a", as in "not a simple example". To split this string correctly, the positions right before and right after "a" have to be split. But they are next to each other, so they can not both be local maxima, as this would mean they are each larger than the other. If you are a mathematician, you can now construct an algebra were this is possible, but we normal people will just have to set this to some value. As values can only vary between 0 and 1, 0.5 seem like a good natural compromise. But if you want to get the best value, you will have to test it, too.

Optimize

Guess what? I wrote a method to help you test which values for n and t are the best! You should probably do a "gem install maxixe" as soon as you can reach your terminal to get the new version.

I stole this idea from the paper, too. The idea is to split a small number of sentences (5 to 50) by hand and use these to figure out the parameters that work best for your use case. Now, the authors of the paper are proper scientists, so they used precision and recall to measure recognition performance. But I wrote this in 2 hours after work, so I will measure recognition performance by edit distance: diff the pre-segmented strings with the ones split by the algorithm, add up the edits needed to make them the same and use this as a score. A score of 0 means that there was no difference between your pre-segmented sentences and the ones split by the segmenter, so this is a perfect score. The higher the scores, the worse the word recognition.

Here is the example from maxixes readme:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

pre_segmented = [["MYDOGISINTHEHOUSE", "MY DOG IS IN THE HOUSE"],
                ["FOURNICEDOGS", "FOUR NICE DOGS"],
                ["MYCATLIKESMYDOG", "MY CAT LIKES MY DOG"]]
index = Maxixe::Trainer.generate_corpus_from_io([2,3,4,5], "ILIKEMYDOG
THISHOUSEISMYHOUSE
MYDOGISSONICE
WHOLIKESDOGSANYWAY
CATSANDDOGSUSUALLYFIGHT
INMYHOUSETHEREAREFOURDOGS
IWANTAHOUSEFORMYDOG")

optimal = Maxixe::Trainer.optimize(index, pre_segmented)

# optimal is now {:score => 0, :t => 0.5, :n => ["2","4"]}

As you can see, you will have to build an index first. This should contain any values for n you could possible want. The optimizer will try any combination of values for n and t and calculate the score. Then, it will give you the smallest result for score, n and t, in that order. The above example has the best recognition performance if you have n-grams with n = 2,4 and a threshold of 0.5. If we used a real corpus, we would now have a segmenter with nicely tuned parameters, ready to segment a lot of sentences.

BTW, there are more combinations of parameters that will give you a score of 0 for this example. If you want to see these, take a look at Maxixe::Trainer.check_recognition. 

What's next?

By now, you should have a working segmenter, finely tuned for your exact use case. So, you made a segmenter for biblical Greek? Or for Cicero's work? Why not make a gem out of it, so others can use it? I will do the same, so stay tuned for more...

Permalink | Leave a comment  »

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

require "romkan"

class WadokuEntry
  def initialize(entry_as_string)
    @str = entry_as_string
  end

  def kana
    @str.split("\t")[1] # The reading is the second column in the file
  end

  def romaji
    kana.to_roma
  end
end

Permalink | Leave a comment  »

As this entry had a really stupid permalink by mistake, it can now be found at http://rogerbraun.net/using-picky-for-search-and-profit

Permalink | Leave a comment  »

[UPDATE: I wrote a follow up article on how to train your segmenter and tune the parameters!]

Recently, I wanted to use Picky for a dictionary app (and did, see WadokuJT). This worked great after fixiing some small problems (which I will talk about in a later post). 

Now, to use Picky, you simply have to tell it which data to index, and then it will perform some operations on it to make it searchable. One of these operations is the tokenization: The data is split in tokens which are then indexed seperately. This is rather straight-forward for western languages: Just split the string data on spaces and punctuation. So a string "I love my cat" would give you these tokens: ["I", "love", "my", "cat"].

Now, not all languages work this way. Many asian and other languages have scriptio continua, so they just write word after word after work, with no spaces (and sometimes, no punctuation). You can even get this with western languages, for example latin. So, if you want to tokenize text in these languages, you will have to find out where the word boundaries are.

In my case, Japanese, there are already several software packages that can split text into words, for example morphological analyzers like MeCab, ChaSen and Juman. These do more than we need, as they also analyze sentence structure and morphology of the words. Also, they are all rather large pieces of software that contain hand-written dictionaries and grammar rules to parse natural-language sentences. Further, they are often somewhat hard to install and run, the documentation often being available only in Japanese. And finally, if your language is not Japanese, but say, Chinese, or Linear B, you will have to take a completely different approach unless you want to rewrite the dictionary and grammar. I wanted something universal, written in Ruby, that I could just require from Picky.

So i went to Google Scholar and looked at the results for "japanese segmentation". I found a paper called "Mostly-unsupervised statistical segmentation of Japanese kanji sequences" by Ando and Lee. As it turns out, the algorithm described in this paper (called TANGO) is not really specific to japanese, and you can use it to segment pretty much any language.

So i wrote a gem, Maxixe, that implements this algorithm. Here is an example of what you can do with it.

Say you have a training file containing these unsegmented sentences: 

ILIKEMYDOG
THISHOUSEISMYHOUSE
MYDOGISSONICE
INMYHOUSETHEREAREFOURDOGS
IWANTAHOUSEFORMYDOG

You can use these to train the segmenter and split a sentence. It will detect word boundaries by itself.

hash = Maxixe::Trainer.generate_training_data([3,4], "dogs")

s = Maxixe::Segmenter.new(hash)

s.segment "MYDOGISINTHEHOUSE"
 => "MY DOG IS IN THE HOUSE"

Nice, isn't it? So how does this work? 

The algorithm is based on the idea that if you count any n-grams (substrings of length n) in a text, n-grams that are words or parts of word will appear more often than those which are just random strings created by two adjacent words. So what you do is use your training data to build a hash which contains the count of all n-grams in this corpus. Then, when you try to segment a string, you calculate if it is more probable that you are between two words or on a word, by comparing the count of n-grams to the left and right of the current position and the count of the n-grams that overlap with the current position. Then, after getting a numerical value for all positions, you split on all local maxima and if the value is above a certain threshold. I suggest that you read the paper yourself if you are interested. Once you grasp the underlying idea, it is quite easy to understand.

There are three things here that affect the segmentation performance: The corpus you use, the kinds of n-grams you use and your threshold. The corpus part is straightforward: Use a large amount of text that is similar to the one you want to split. The n-grams and threshold are more complicated. The above example works if you use a threshold of 0.5 and generate n-grams for n = 3,4, but you can easily find other values which will give the wrong results. Sadly, you will have to figure out which parameters are the best by segmenting a small number of sentences by hand and checking which parameters give the results most similar to your test data. I plan to add support for automating this to Maxixe, but right now, you will have to do it yourself.

And that's it! So now, if you ever need to tokenize some text that does not have spaces, give Maxixe a try. The code is small and pure Ruby, so be sure to take a look under the hood.

Atraente_chiquinha_pixinguinha.ogg Listen on Posterous

Permalink | Leave a comment  »

• inugami.txt
• kanji_statistics.rb.txt

Permalink | Leave a comment  »

Permalink | Leave a comment  »

Permalink | Leave a comment  »

Permalink | Leave a comment  »

Pre-2k

def year_for_printing(year)
      return year % 100
    end

2000+

def year_for_printing(year)
      if year 
Man hat also wohl einfach einen Zweig hinzugefügt, der bei den Jahren ab 2000 eine Null zum Ergebnis hinzufügt. Der Code gibt richtig aus: "WS 99/00". Das wird zehn Jahre gut gehen! Nämlich bis zum Jahre 2010...

 Hochspekulativer Zwischenschritt

Der folgende Schritt erscheint einfacher möglich wenn folgendes innerhalb dieser zehn Jahre passierte: Jemand hat sich die Funktion angeschaut und gesehen, dass sie den unnötigen Fall "vor dem Jahr 2000" abdeckt und diesen entfernt. Die neue Funktion sah also so aus:

    def year_for_printing(year)
      return "0" + (year % 100).to_s
    end

Die Argumentation ist zwar noch schlüssig wenn dieser Schritt fehlt, wirkt aber mit ihm wahrscheinlicher.

2010, 2011

Als das Jahr 2010 anbrach wurde die Ausgabe wieder falsch: Der Code gibt jetzt "010" zurück. Das steht aber offensichtlich nicht auf den Ausweisen. Ich schätze dass dem Programmierer der Auftrag gegeben wurde, den Fehler für das Jahr 2010 zu beheben. Er hat die Funktion wohl so abgeändert:

    def year_for_printing(year)
      if year == 2010
        return year % 100
      else
        return "0" + (year % 100).to_s
      end
    end

Dieser Code gibt richtig aus: "WS09/10". Es scheint also alles seine Richtigkeit zu haben! Aber beim Jahr 2011 gibt dieser Code auch genau das aus, was jetzt auf unseren Karten zu lesen ist: "WS10/011". Und das hat anscheinend noch niemand gemerkt. Oder nicht für wichtig genug gehalten.

Hier mein Bugfix für den Code. Ich gehe mal davon aus dass das System ein Embedded-System ohne Datumsfunktionen ist (denn diese sollte man benutzen, wenn sie vorhanden sind!).

    def year_for_printing(year)
      res = year % 100
      if res 

Dieser Code gibt für jedes Jahr den richtigen Wert aus. Wenn ihr das Spiel gerne weitertreiben wollt, dann ersetzt einfach "year == 2010" durch "year >= 2010" und ihr habt wieder Code der nur die nächsten 10 Jahre richtig ist.

Habt ihr eine andere Erklärungsmöglichkeit?

Permalink | Leave a comment  »

Permalink | Leave a comment  »

Was ich 'die Sünde gegen den heiligen Geist genannt habe — die Anmaßung des dreiviertel Gebildeten —, das ist das Phrasendreschen, das Vorgeben einer Weisheit, die wir nicht besitzen. Das Kochrezept ist: Tautologien und Trivialitäten gewürzt mit paradoxem Unsinn. Ein anderes Kochrezept ist: Schreibe schwer verständlichen Schwulst und füge von Zeit zu Zeit Trivialitäten hinzu. Das schmeckt dem Leser, der geschmeichelt ist, in einem so „tiefen" Buch Gedanken zu finden, die er schon selbst einmal gedacht hat. (Wie heute jeder sehen kann — des Kaisers neue Kleider machen Mode!)

Wird die Normativität der dualistisch angelegten Geschlechter-Matrix aufgehoben? Inwiefern ist eine Diversifizierung der Männlichkeits- und Weiblichkeitsbilder festzustellen? Wie wirkt sich die Pluralisierung der Lebensstile aus? Was bedeutet die zunehmende Differenzierung innerhalb der Gruppen der Frauen und Männer und welche Bedeutung haben dabei Machtrepräsentationen im Spannungsfeld der Intersektionalität?

Gewöhnlich glaubt der Mensch, wenn er nur Worte hört, es müsse sich dabei doch auch was denken lassen.

Permalink | Leave a comment  »

Permalink | Leave a comment  »

def get_best_matches(phrase)
      $KCODE = "u"
      Translation.all.map{|item| [::Text::Levenshtein.distance(phrase,item.source),item.source,item.target]}.sort_by{|item| item[0]}
end

Permalink | Leave a comment  »

def insert_into_sorted_helper(sorted, element, rest)

  if rest.size == 0 then
    sorted << element   
  elsif rest[0] >= element then
    (sorted << element).concat(rest)
  else
    insert_into_sorted_helper(sorted << rest[0], element, rest[1..-1])
  end 

end

# Insert an element into a already sorted list. The resulting list stays sorted.
def insert_into_sorted(sorted, element)
  insert_into_sorted_helper([],element,sorted)
end

def recursive_sort_helper(sorted, unsorted)

  if unsorted.size == 0 then
    sorted
  else
    first = unsorted[0]
    rest = unsorted[1..-1]
    recursive_sort_helper(insert_into_sorted(sorted, first), rest)
  end
end

# Sorts a list
def recursive_sort(unsorted)
  recursive_sort_helper([],unsorted)
end

Permalink | Leave a comment  »

Der folgende Beitrag ist eine Reaktion auf einen Gastbeitrag bei Astrodicticum simplex, daher sollte man sich diesen vorher durchgelesen haben.

Permalink | Leave a comment  »

Schreibung [Lesung] /Bedeutung1/Bedeutung2.../Bedeutungen/

input = File.open("jlpt-voc-4-extra.utf")

output = File.open("jlpt-converted.tab","w:UTF-16LE")

input.each do |line|
  puts line
end

input.each do |line|
  schreibung = line.split(" ")[0]
  puts schreibung
endNun gibt unser Code die Schreibung aus! line.split(" ") gibt ein Array zurück, also eine Liste von Stückchen. Diese Stückchen lassen sich über [Nummer] ansprechen. Da bei 0 angefangen wird zu zählen ist das erste Stückchen also [0].

Die Lesung zu filtern ist schon etwas schwerer. Die Lösung:input.each do |line|
  schreibung = line.split(" ")[0]
  lesung = line[/\[(.*)\]/,1] || schreibung
  puts schreibung + " " + lesung
end
Das sieht kompliziert aus, ist aber eigentlich auch einfach. Das zwischen den eckigen Klammern bei line ist ein regulärer Ausdruck. Reguläre Ausdrücke sind speziell dafür gemacht um Textstücke herauszufiltern. In Ruby sind reguläre Ausdrücke immer von Slashes (/Ausdruck/) umgeben, der eigentliche Ausdruck in unserem Fall ist also \[(.*)\]. In regulären Ausdrücken passen bestimmt Zeichen auf bestimmte andere Zeichen. Wenn ein Zeichen passt wird es herausgefiltert. Alphanumerische Zeichen passen auf sich selbst. Beispiele:"Hallo, wie geht's?"[/Hallo/] => "Hallo"
"Hallo, wie geht's?"[/llo/] => "llo"
"Hallo, wie geht's?"[/Bundespräsident Horst Köhler/] => nilGibt es keine passenden Zeichen gibt der Ausdruck nil zurück, was in Ruby "kein Ergebnis" bedeutet. Uns interessieren aber nun beliebige Zeichen, die sich zwischen zwei eckigen Klammern befinden. Wir wollen also etwas wie [irgendwas]. Leider habe eckige Klammern schon eine besondere Bedeutung in regulären Ausdrücken, d.h. sie passen nicht auf sich selbst. Eckige Klammern passen auf \[ und \]. Unser neuer Ausdruck ist also \[irgendwas\]. Jetzt müssen wir nur noch irgendwas als regulären Ausdruck schreiben. Wir wollen eine beliebige Anzahl von beliebigen Zeichen filtern. Ein Punkt (also".") passt auf genau ein beliebiges Zeichen. "[a]"[/\[.\]/] würde also "[a]" zurückgeben. Auf "[aaaa]" würde es aber schon nicht mehr passen! Zum Glück gibt es auch den Ausdruck *, der "beliebig viele vom vorherigen Zeichen" bedeutet. .* bedeutet also "beliebig viele beliebige Zeichen. Genau was wir brauchen! [/\[.*\]/] passt also auch auf [わたし] und gibt in diesem Fall das selbe zurück."赤い [あかい] /(adj) red/(P)/"[/\[.*\]/] würde "[あかい]" zurückgeben. Um die Klammern los zu werden machen wir eine Gruppe aus den "beliebigen Zeichen", indem wir normale Klammern um sie herum machen. Jetzt gibt uns der Ausdruck mehrere Gruppen zurück. Mit ",1" sagen wir ihm, das wir die zweite Gruppe haben wollen. Die erste Gruppe ist alles was passt (also "[あかい]") und hat die Nummer 0. Unsere zweite hat die Nummer 1, also ist unser endgültiger Ausdruck [/\[(.*)\]/] und wird so verwendet:lesung = line[/\[(.*)\]/,1]Manche Einträge haben aber in der JLPT-Datei gar keine Lesung! Katakana-Wörter stehen nur so da, unser Ausdrück würde also nil zurückgeben, da keine passenden Stücke gefunden werden. Dafür ist nun  || schreibung da. Es bedeutet "Wenn das vorherige nil war, dann nimm das folgende". Wenn also keine Lesung in der JLPT-Datei steht wird einfach die Schreibung als Lesung genommen. Wenn man sich nun noch mal den Code oben ansieht wird man merken dass er gar nicht so schwer ist wie zunächst gedacht. Die Bedeutung bekommt man genau so heraus, darum hier das fertige Programm:input = File.open("jlpt-voc-4-extra.utf")
output = File.open("jlpt-converted.tab","w:UTF-16LE")
input.each do |line|
  schreibung = line.split(" ")[0]  
  lesung = line[/\[(.*)\]/,1] || schreibung
  bedeutung = line[/\/(.*)\//,1]
  output.puts "#{schreibung}\t#{lesung}\t#{bedeutung}" unless line[0] == "#" || line.strip == ""
end
output.close
input.close
puts wird nun auf output angewendet und schreibt darum alles in eine Datei. \t ist das Zeichen für Tabulator und das ganze wird nur in die Datei geschrieben wenn das erste Zeichen nicht "#" ist oder die ganze Zeile leer ist.

Permalink | Leave a comment  »