eclecticlogic.github.io

Fluent Interfaces without Recursive Generics

2016-12-31T00:00:00+00:00

Fluent interfaces are an API design style that allow you to be expressive while being terse. Consider the following pseudo example API to split a string.

Splitter splitter = new Splitter(",");
splitter.setOmitEmptyStrings(true);
splitter.setTrimOutput(true);
List<String> parts = splitter.splitToList(input);

That is four lines packed with a lot of noise. The two setters could have been written as parameter-less methods but I wanted to stick to Javabean specifications here. Now compare this with the fluent version of Splitter from Google Guava.

List<String> parts = Splitter.on(",") //
	.omitEmptyStrings() //
	.trimResults() //
	.splitToList(input);

Still four lines but now each line clearly conveys intent in a very “fluent” dialogue like manner. So armed with an understanding of the elegancy that fluent interfaces bring to the table, you set out to create a fluent version of your new animal API. You have written up the following classes with fluent API:

public class Animal {

    public Animal eat(Food food) {
        food.consumedBy(this);
        return this;
    }
    
    
    public Animal run(int distance) {
        // some logic here
        return this;
    }
}

public class Dog extends Animal {
    
    public Dog bark() {
        // logic
        return this;
    }
    
    
    public Dog chaseTail() {
        // logic here
        return this;
    }
}

Now you attempt to use your newly created fluent API and write up the following client code:

public void someMethod() {
	Animal a = new Dog().eat(bone).bark(); // Cannot resolve method bark ???
}

You are suddenly perplexed. Why can’t you call method bark()? It is because eat returns an instance of Animal, not Dog and so even though the runtime instance created is a Dog, the static type checking only knows that you have an instance of Animal. This particular situation is easily resolved by switching the two methods:

public void someMethod() {
	Animal a = new Dog().bark().eat(bone);
}

In a real world API, we may not have the luxury of switching methods like this (what if the dog really only likes to bark after eating!) and requiring particular ordering of calls makes for an ugly API. The standard solution in such a case is to introduce generics with a particular twist - recursive generics.

public class Animal<A extends Animal<A>> {

    public A eat(int food) {
        //food.consumedBy(this);
        return (A)this;
    }


    public A run(int distance) {
        // some logic here
        return (A)this;
    }
}

public class Dog extends Animal<Dog> {

    public Dog bark() {
        // logic
        return this;
    }


    public Dog chaseTail() {
        // logic here
        return this;
    }
}

// in some method ...
Animal<Dog> dog = new Dog().eat(3).bark();

Problem solved? Well, almost. Until you realize that if you know include Animal as a member of any class, you bring along baggage - in the form of recursive generics that you have to apply to the container class (or method) and this can easily mushroom into very complex generic definitions. There is an alternate way of building fluent interfaces with inheritance but without generics by accepting a slight compromise. We do this by introduce a very specific typing method that allows us to switch the static type.

public class Animal {

    public Animal eat(int food) {
        //food.consumedBy(this);
        return this;
    }


    public Animal run(int distance) {
        // some logic here
        return this;
    }


    // A way to cast our return type into a specific sub-type.
    public <T extends Animal> T typed() {
        return (T)this;
    }
}

public class Dog {
	...
}

// caller
Animal dog = new Dog().eat(3).<Dog>typed().bark();

While not perfect, this technique allows us to switch between methods of the parent or child class and not incur the added design-time overhead of generics.

Curve Fitting

2015-07-17T00:00:00+00:00

Here is a simple technique to modify linear data points to a curve of your choice. If your function looks linear like this:

or even has an inverse linear form like this:

And you want to instead map it to a curve that accentuates the initial values and suppresses the tail:

or has an exponential drop at the end points like this:

then, Cubic Bezier curves can be used to achieve the desired transformation. A Cubic Bezier curve is given by:

B(t) = (1 - t)³P₀ + 3(1 -t)²tP₁ + 3(1 -t)t²P₂ + t³P₃

Here P₀, P₃ are the end-points of the curve and P₁ and P₂ are control points. A Cubic Bezier curve, at the starting point is always tangential to the line segment P₀, P₁ and in the direction of P₀, P₁. At the endpoint, the curve is tangential to P₂, P₃ and is in the direction of P₂, P₃. You can play around with various Cubic Beziers shapes at desmos.com.

When implementing a Cubic Bezier curve, the first oddity you will encounter is the fact that unlike polynomial equations, the Bezier does not give you y-coordinates as a function of x-coordinates. Instead, it independently gives you x and y coordinates as a function of a parameter t which varies from 0 to 1. When t = 0, the curve is at point P₀ and when t = 1, the curve is at point P₃. Worse, when you vary t linearly, you do not get linear increments of x. Instead, you get linear increments along the curve’s length. In fact, it is this property of Bezier curves that makes them suitable as easing functions for animations. By constructive a Cubic Bezier with steep starting and ending slopes, you can get a slow start in the x axis followed by an acceleration in the middle and a gradual slowdown towards the end. But for mapping a traditional function of the form y = f(x), we would like to transform the Bezier to a similar function.

There is no simple mathematical way to achieve this given expressions like t² and t³ in the equation. Instead, one can approximate a solution by doing a binary search along t, computing the corresponding x until we are within a tolerance limit. At that point, the value of y can be computed. The gist below shows a Java program to do that. In addition, the program allows you to pre-compute y = Bezier(x) for x = (start, end) in specified increments.

The class above can be extended to create a composite Cubic Bezier curve fitting algorithm that combines multiple Cubic Bezier curves to model any shape you want to create.

Groovy Oddities

2015-02-17T00:00:00+00:00

Groovy is a popular language for the JVM with a syntax that includes and extends Java’s. Groovy’s appeal comes from the extended syntax - optional semicolons, type-inference (strictly speaking, Groovy is typeless), literals for lists and maps, closures, operator overloading, etc. However, these extra features are not always intuitive in their interaction, leading to some syntactic oddities. Here is a list of oddities that I’ve encountered. This is not an exhaustive list. If you have run into other issues, please do leave a commment below.

Null safety in every step

The null-safe operator allows the in-lining of nullability checks. So instead of:

if (student.name != null) {
	lower = student.name.toLowercase()
}

You can write

lower = student.name?.toLowercase()

The variable lower will be null if name is null. Armed with this powerful operator, you set about to show your newly acquired skill. You are writing a function that accepts Account objects. You know that sometimes the account may be null (if a spurious username is provided). However, if an account is non-null, you know that it has a non-null createdOn field. Your task is to get the month from this field. You write something like this:

void accumulateMonths(Account account) {
   def month = account?.createdOn.month
   if (month) {
	  // Do something with month
   }
}

You run the program and upon receiving a null Account object, you see an exception!

Exception in thread "main" java.lang.NullPointerException: Cannot get property 'month' on null object
at org.codehaus.groovy.runtime.NullObject.getProperty(NullObject.java:57)

Why did Groovy not stop when it evaluated account?. ? Why is it complaining about not being able to get the month property? The exception was thrown because we incorrectly interpreted the semantics of the ?. operator. The null-safe operator in fact does not abort the current chain of calls on encountering a null. It is better translated to the construct below.

def a = account == null ? null : account.createdOn
def month = a.month
if (month) {
}

The null-safe operator is a close-cousin of the ternary operator, not the if condition!

Applying null-safe retrieval to map entries.

In Groovy, you can access map entries as map[key] and list entries as list[index]. So if we had a Map<String, List<Integer>>, and we want to make sure a map entry is not null before we access the list, should we be able to use the null-safe operator?

class NullSafeWithMap {

    static void main(def args) {     
        Map<String, List<Integer>> myMap = ['a': [1, 2, 3], 'b': null, 'c': [4, 5, 6]]
        println (myMap['b']?[0]) // Syntax here here.
    }
}

Unfortunately you will get a syntax error since Groovy thinks the ? is part of the ternary operator. This is because the null-safe operator is ?. not just a ? Putting a period after the question-mark will not help either.

myMap['b']?.[0] // Still a syntax error.

If you replace the list subscript operator with the get() method, the code will work:

println (myMap['b']?.get(0))

An equation to get NPE

Here is a brain teaser. What is the output of the program below?

static void main(def args) {
     println 4 + 5 * 3
}

Alright, that’s not much of a brain-teaser. The answer is obviously 19. Turns out, what I wanted was really (4 + 5) * 3 so that I get 27. That’s easy:

static void main(def args) {
	println (4 + 5) * 3
}

Think the answer will be 27? Think again! Lost in the focus on the math equation is the fact that you were able to write println 4 + 5 * 3 in the first place because Groovy lets you drop parenthesis if you have a single argument and the call is unambiguous. So instead of printing out the product of 4 + 5 and 3, Groovy is attempting to multiply the result of println (4 + 5) and 3. println has a return type of void which when forced into a return value becomes null. So at runtime, Groovy throws an exception stating that it cannot call multiply() on a null object.

Operator precedence rules

Groovy supports operator overloading and provides several pre-defined operators to make your code succinct, such as the << operator to add to lists.

List fruits = []
fruits << 'Apples' // fruits = ['Apples']

While writing your code, you come across a case where if citrus is true, you want to add ‘Oranges’ but otherwise you want to add ‘Apples’. Eager to show off your newly acquired Groovy skill, you write the following:

List fruits = []
boolean citrus = ... 
fruits << citrus ? 'Oranges' : 'Applies'

println fruits

As you extol the virtues of Groovy to your colleagues, you look at the output and are horrified to find: [true]

What just happend? Well, turns out the operator precedence puts the left-shift operator at a higher precedence than the ternary operator. So the expression you really wrote is:

(fruits << citrus) ? 'Oranges' : 'Apples'

In Groovy, a list has a true value if it is non-null and non-empty. So ‘Oranges’ was selected as the value of the expression and sent to … /dev/null? To fix the issue, you have to write

fruits << (citrus ? 'Oranges' : 'Apples')

Block scoped local variables

Seasoned programmers develop habits to prevent certain types of errors. One of those habits is to use blocks to scope local variables so that copy-paste of the code (hopefully confined to some hastily written unit test) doesn’t result in redefinition of the variables, which invariably leads to an attempt to rename the variables, which occasionally results in one rename being left off, which results in a hard-to-find bug. So we write

{
   int a = 5;
   callSomeMethod(a);
}
{
   int a = 10;
   callSomeMethod(a);
}

There are more legitimate uses for the block-scope in methods, such as to ensure that logic that follows does not rely on some previous variable. When translated to Groovy, Java code with blocks fail because Groovy thinks that you have defined closures and therefore skips over them. The workaround?

1.times {
   int a = 5;
   callSomeMethod(a);
}
1.times {
   int a = 10;
   callSomeMethod(a);
}

The above construct is arguably a hack, but it works nevertheless.

Map literal oddities

Groovy supports map literals. This allows you to create a map with a compact and readable syntax.

Map<String, Integer> myMap = ['a': 25]
myMap['b'] = 35
int d = myMap['a']

Armed with this new-found knowledge, you proceed to construct a map as follows.

String name = 'joe'
Map<String, Integer> myMap = [name: 25]
...
int value = myMap[name]

You run your program to discover a runtime exception saying that null cannot be converted to int! Upon further inspection with a debugger, you discover that your map is ['name': 25] not ['joe': 25]. What happened? Turns out you can’t have variables in a map initialization. To achieve the intent above, you have to use a GString variable or enclose the variable in parenthesis.

String name = 'joe'
Map<String, Integer> myMap = ["$name": 25] // or [(name): 25]

Map initialization cannot take the return values of functions or enums either. These also need to be turned into a GString expression or wrapped in parenthesis.

Primitive arrays

If you argue that Java’s primitive arrays are an aberration, you win. Now lets deal with with the real world problem presented by the fact that many java api have been written to accept primitive arrays. Since Groovy uses [] to represent lists, how do you define a primitive array instead?

Given the java method

public class SomeClass {

    public static void soStuff(int[] values) {
        for (int i : values) {
            System.out.println(i);
        }
    }
}

the following Groovy code will fail.

def values = [1, 2, 3]
SomeClass.soStuff(values) 

To coerce values into a primitive array, use the as operator with <type>[] type definition.

def values = [1, 2, 3]
SomeClass.soStuff(values as int[])

Type-lessness

Groovy supports duck typing. Yes, but what kind of duck allows this?

static int abc = new AtomicInteger()

Remember that in Groovy, type declarations in your code are completely ignored. So the compiler will happily compile the above. Groovy 2.0 introduced specific annotations, @TypeChecked and @CompileStatic to alleviate some of these bugs. There are some caveats to be aware of when using these annotations. Refer to this presentation for a good overview.

BigDecimal everywhere

Sometimes features built with good intent can have unintended consequences. The next three topics present examples of such features in Groovy. Groovy converts all division to BigDecimal. This is especially useful in financial services apps. However, it converts all division to BigDecimal as I found out. I was writing code to determine the number of threads to use to process a list of objects. I wanted the thread count to be one-tenth the list size but with a minimum of 1 and a maximum of 50.

int threads = Math.max(1, Math.min(list.size() / 10, 50))

When I ran the program, I got an exception stating, Cannot resolve which method to invoke for [class java.math.BigDecimal, class java.lang.Integer] due to overlapping prototypes between ...

Uh? I was expecting integer division and Groovy instead offered a BigDecimal and couldn’t find an appropriate Math.min() method. The code needs to be rewritten to

int threads = Math.max(1, Math.min((list.size() / 10).intValue(), 50))

Groovy Truth

Groovy evaluates truth as many things - null objects are false, empty lists are false, zero is false. Takes you back to the days of C/C++. People bemoaned the unexpected evaluations of clever expressions in C/C++. Java came along and eschewed all cleverness. People bemoaned the verbose conditional expressions in Java. Groovy came back with C/C++ style truth evaluation.

In a payment app, we want to make sure the amount has been defined. So we write a test of nullability.

if (!amount) {
	throw new IllegalArgumentException(...
}

Except somehow, the amount ends up being exactly zero (lets say this is the amount to be charged and a coupon made the amount zero). Well, the code unexpectedly fails.

Groovy allows you to test for nullability and continue an operation by using the safe-navigation operator ?. This is an operator that is frequently abused (its use masquerading for thoughtful analysis of nullability) and sometimes results in side-effects too.

Consider the following condition

if (!someObject.someProperty?.taxable) {
}

The author’s intent was that if taxable is false, some piece of code should be executed. However, if someProperty itself is null, in this particular case, the code should not have been executed. The author realized that someProperty is a nullable field and therefore came up with the above construct. If someProperty is null, the expression evaluates to true! The actual condition required was

if (someObject.someProperty && someObject.someProperty.taxable) {
}

Language philosophy

Groovy introduced a number of new language constructs to the JVM and offered a terse and powerful language to program in. Groovy is particularly suited to unit-tests and DSLs and frameworks such as Grails. What is not apparent in the power of Groovy is a trade-off that has been made between discipline and power. Programmers can abuse the power of Groovy with greater ease than Java and the result is bad code that is tersely defined versus bad code that is verbosely defined. The ease of use of maps and lists in Groovy leads to their excessive use often at the expense of proper Object-Orientedness.

Its been an interesting journey since the days of C, C++. C and C++ allowed all kinds of clever programming tricks and shortcuts. Developers grew tired of the side-effects of the tricks and unfortunately, at least with Java, the language designers pivoted to the other extreme. Groovy in some respects represents a swing back to the C/C++ paradigm. Language designers are now attempting to strike the right balance between concise expression vs. correctness, static type vs. verbosity and terseness vs. comprehensibility. Some of the newer languages like Dart, Swift, Kotlin and Ceylon exemplify such a design trend. Ceylon in particular is an exciting JVM language that is based on the philosophy that code is read more often than written. Ceylon has one of the most exciting type-systems and I for one am eagerly looking forward to using it.

High Performance Postgresql in Java with Pedal

2015-01-09T00:00:00+00:00

Overview

Postgresql is one of the most popular open-source databases in use. It has good standards support and an impressive feature set - rich data types and advanced runtime management. The Pedal framework for Java enables fast inserts (we are talking orders of magnitude faster) into Postgresql through the Copy command directly at the JPA entity level. In this article, we provide an overview of the Pedal framework and show how to use the Copy command.

The Pedal framework consists of three libraries.

pedal-dialect enables dialect (i.e., database) and provider (e.g., Hibernate) level features such as retrieval of schema name, mapped table name given a JPA entity, user-types (arrays, bit strings), etc. It also provides support for the Copy command in Posgresql. pedal-tx is a Java-8 only framework that allows for transaction demarcation using Java 8 lambdas, transaction attached storage and transaction attached pre/post commit lambdas. In addition it also provides a DAO abstraction layer with support for fluent JQL/HQL and native queries. With pedal-loader data population scripts for db-unit tests can be written in a Groovy DSL while working at the JPA entity level (with mapped column types, object-level foreign-keys, etc).

Using the Copy Command

To enable copy support, first create an instance of com.eclecticlogic.pedal.dialect.postgresql.CopyCommand, ideally setup as a Spring-bean. It requires access to com.eclecticlogic.pedal.provider.ProviderAccessSpi which can be configured by creating a com.eclecticlogic.pedal.provider.hibernate.HibernateProviderAccessSpiImpl.HibernateProviderAccessSpiImpl passing it a reference to the EntityManagerFactory. Using Spring with Java-based configuration, the code would look like this:

    @Bean
    HibernateProviderAccessSpiImpl hibernateProvider(EntityManagerFactory factory) {
        HibernateProviderAccessSpiImpl impl = new HibernateProviderAccessSpiImpl();
        impl.setEntityManagerFactory(factory);
        return impl;
    }

    @Bean
    public CopyCommand copyCommand(ProviderAccessSpi provider) {
        CopyCommand command = new CopyCommand();
        command.setProviderAccessSpi(provider);
        command.setConnectionAccessor(new TomcatJdbcConnectionAccessor());
        return command;
    }

The connection accessor is specific to the connection pool you are using. It is used to get a handle to the underlying JDBC-4 compliant Postgresql native connection. Pedal-dialect ships with support for the following connection pools:

To create one for an unsupported connection pool, implement the com.eclecticlogic.pedal.connection.ConnectionAccessor interface.

So what does the code to actual insert rows using the CopyCommand look like? Its simple. Create instance of your entity and copy them into a CopyList<T>. Then pass the CopyList instance to the copy command. Here is the test code in pedal-dialect:

        CopyList<ExoticTypes> list = new CopyList<>();

        // The copy-command can insert 100k of these per second.
        for (int i = 0; i < 10; i++) {
            ExoticTypes et = new ExoticTypes();
            et.setLogin("copyCommand" + i);
            BitSet bs = new BitSet(7);
            bs.set(1);
            bs.set(3);
            bs.set(4);
            et.setCountries(bs);
            et.setAuthorizations(Sets.newHashSet("a", "b", "b", "c"));
            if (i != 9) {
                et.setScores(Lists.newArrayList(1L, 2L, 3L));
            } else {
                et.setScores(Lists.<Long> newArrayList());
            }
            et.setStatus(Status.ACTIVE);
            et.setCustom("this will be made uppercase");
            list.add(et);
        }

        copyCommand.insert(entityManager, list);

The CopyCommand supports a subset of JPA features. Check here for the current supported feature set.

For a moderate row size (100 to 200 bytes), 100k inserts using EntityManager.persist() took 34 seconds while the CopyCommand took 1.4 seconds! That is a 24x speed-up. Make sure you read up on the limitations of the copy feature in Postgresql. In addition, the Copy command will bypass JPA/Hibernate interceptors. So don’t expect Envers to work with it.

SMTP-Bit - An email protocol to fight spam using Bitcoins.

2014-12-20T00:00:00+00:00

Core Idea

Email spams and spoofs are a constant nuisance for all. While filters grow in sophistication every year, the fraud perpetrators are always devising new ways around it. So here is a simple proposal to put an end to spam and scam artists (or at least make it an expensive proposition). Proof-of-work (e.g. Hashcash) and Bitcoin based spam filters have been proposed before. This protocol is simply a refinement of the latter.

Email is perhaps the most pervasively used protocol on the Internet with approx 200 billion sent per day. It is unrealistic to expect the installed infrastructure supporting emails to be upgraded to a new protocol easily. Therefore any solution must work with the existing SMTP protocol and offer an option to upgrade to a newer protocol gradually.

The SMTP-Bit protocol associates wallets with email addresses to protect against spam. An email message is required to identify the wallet associated with the sender’s email, a signature to prove that the sender owns the wallet and the public key for the wallet. Before accepting an email, the server checks the public ledger for a transaction from the sender’s wallet to the recipient’s wallet (the transaction needs to transfer a minimum of 1 Satoshi). If it finds such a transaction, the email is accepted. Otherwise it is rejected. To make the verification of the transaction from the public ledger more efficient, the email message can include the transaction id or the receiver’s mail server can store a cache of transactions involving the recipient’s wallet. The transactions can be specially tagged to make them easier to locate from the public ledger.

With such a setup, the bitcoin transaction from party A to party B serves as a trust identifier. If party A uses a different email address, they can continue to communicate with party B based on the previously established trust relationship. Spam filters based on explicit trust (called permission based filters) have been tried before. ChoiceMail is one such example. The downfall of these filters has always been the need to trust a third-party with confidential information. The Bitcoin protocol with its trust-less blockchain algorithm removes this hurdle.

Bootstrapping

For such a protocol to work, we need to provide an efficient mechanism for the recipient’s wallet to be discovered. We would ideally like the registry to be publicly accessible and not under any single party’s control. The Bitcoin transaction ledger can serve as such a registry. To associate a wallet with an email address, the user could be required to tag a transaction from the wallet to any other wallet with the email address (or multiple addresses). Such a transaction could be part of the process of setting up an email account in the first place. It may seem that having a publicly accessible email address would be folly. However, if the protocol is successful at stopping spam, such a public registry would not prove detrimental and in fact could promote legitimate communication.

The protocol will face a chicken-n-egg hurdle to get going. One will not have an incentive to setup an SMTP-Bit enabled email address while others haven’t. It would be rather lonely. However, if the value proposition of not having to spam is worthwhile, it may prove to be sufficient an incentive for multitudes to join.

Spammers and Spoofers

A spammer will need to send you a Satoshi before he/she can send you an unsolicited email. This makes it somewhat expensive to send out bulk emails. Upon receiving the spam, the recipient can simply add the sender’s wallet to a black-list thereby prohibiting future spam.

Domain level wallets

The protocol can be extended to allow the association of a domain-level wallet (perhaps as a TXT record in the DNS entry). This will allow corporations to establish trust with individuals or between corporations without having to enable it on a per-email address basis.

Subscriptions

Marketing departments wanting to send you emails would need to establish a transaction to your wallet. You can unsubscribe by simply blocking their wallet.

High Profile Emails

The SMTP-bit protocol could also open up celebrity emails to the general public. Lets say - I’m going to pick a dead, not-generally-well-known person - Geerhardus Vos wanted his email to be publicly available but he wanted to restrict people from sending him a flood of emails. He could set up a higher transaction threshold for unknown wallets and create a white-list of wallets of his known friends and contacts (folks would then want to give a celebrity their wallet address when making an acquaintance). Nothing prevents an average Joe from setting a higher threshold either - chances are folks will not be willing to pay a higher premium to email your average Joe (or Jane!).

Loss or compromise of Wallet

If your wallet is compromised, the situation is not too different from your email address being compromised. You simply have to ask others to black-list your old wallet and re-establish trust against your new wallet.

Other Considerations

When all 22 million bitcoins are issued, miners will need to be supported by transaction fees. In such a setup, 1 Satoshi may not suffice. A simple solution then is to use 2 Satoshis and pay out 1 Satoshi as the transaction fee.

Dynamic dependency injection with Spring

2014-12-05T00:00:00+00:00

Out of the box, Spring provides various techniques to define your beans and their dependencies - XML files, Java annotations and since version 4.0, even using groovy scripts. Furthermore, you can control the creation of the beans using FactoryBean and bean post-processors. Still, there are times when it would be very convenient to create an instance of some random class and have its dependencies satisfied at runtime without Spring knowing anything about that class a priori.

If you are willing to accept a few limitations - only bean dependencies, not simple scalar properties, and only those annotated with @javax.inject.Inject, then the following code creates a dependency injector that you can use to dynamically “bind” random objects.

The dependency injector itself should be a spring bean and available to your code that wants to inject something else. In addition, you need to have an instance of AutowiredAnnotationBeanPostProcessor. This bean is automatically included if you have the annotation-config directive in your Spring context file:

	<context:annotation-config />

You can now use the dependency injector to bind dependencies at runtime as follows:

	public class MyObject {
		@Inject SomeOtherObject other; // SomeOtherObject is spring managed

        ...
    }


	...
    MyObject withDependencies = dependencyInjector.bind(new MyObject());

Of Slurpees and Bean Burritos

2014-11-11T00:00:00+00:00

Groovy has some classes with very interesting names - ConfigSlurper, XmlSlurper and JsonSlurper. I wonder if the engineer who wrote this class had a 7-Eleven slurpee addiction!

I can imagine ConfigSlurper (and its cousins) making a slurping noise as it executes (slurps?), complete with a louder noise as it ends - similar to an annoying kid sucking at the last bit of drink from an empty cup filled with ice. One has to wonder about the wisdom behind calling a class Slurper in a professional software library. But perhaps such concerns are not valid when dealing with a language called Groovy. The software world has a penchant for coming up with interesting names. Apart from culinary and beverage related names such as Java, Beans, Cocoa and Guava topped with Sugar and eliciting a Yum response, we also have reptilian and warrior names like Python, Boa and Ninja. Indeed a simple perusal of Github leads to encounters with Pigshell, Nightmare and Phantom. Not to be left behind, the hardware world is re-calibrating from sophisticated names like Centrino and Ivy Bridge to Raspberry Pi and Beaglebone.

In object oriented design, coming up with a good name is a hard problem. In fact, most of the arguments in OO design are related to naming objects. As a professor once said, there are just two problems in Computer Science:

Naming objects
Cache invalidation and
Off by one errors.

Having had my share of endless arguments over class names, I decided to investigate open source software to see what words appear in class names. I wrote up a program to download the source jars for the latest version of all Java libraries available via Maven Central. An hour later I had 60,000 odd jars downloaded on my hard-drive and ready to analyze. Mining Maven central programmatically is interesting enough to deserve an article of its own. Stay tuned for that. I wrote a simple program to then read the jars and parse all class names into its constituent pieces. I owe the parsing of the names to a clever bit of regex that I found on Stackoverflow. Its only fitting that the answer was by a user named poly-gene-lubricants!

Here are some findings from parsing the names:

It looks like we software engineers are an ego-centric bunch. While I found 2,487 references to “My” in classnames, I only found 4 references to “Your.”
The most common word was … Type! A whopping 107,246 times. But that is from a total of 6,906,643 words.
We seem to be doing a good job of separating interfaces from implementation as evidenced by the close runner up - Impl - with 102,786 references.
We like our fruits - Fruit (18), Apple (133), Berry (36), Banana (12), Peach (2), Mango (35) and Grape (23). And we have some die-hard screaming fans of APPLE (23).
We do not like our vegetables. Perhaps those who’s mothers fail to impress upon them the importance of eating vegetables end up as engineers? The first lady may want to take note! Broccoli (0), Spinach (0), Squash (5). Who are these wierdo squash lovers?
But we do like meat! Meatball (1), Meat (2), Chicken (58), Beef (2). I shudder to use the Chicken library.
Oh and we like cheese with that (36).
Some of us secretly wish we were in the Restaurant business: Cook (19), Waiter (84), Server (17,166), Host (4,772) and Janitor (30). I found no reference to Hostess, proof positive that we do have a gender problem in tech!
If we started a zoo, we’d find: Pandas (1), Monkeys (20), Tigers (83), Dogs (132), Cats (351), Snakes (95), Donkeys (5), Zebras (26), Horses (28), Cows (68), Elephants (19), Giraffes (10) and Lions (21).
Is there a lot of workplace violence associated with software development? There really ought to be more - Punch (39), Beat (49), Box (4,993), Poke (6) and Kick (20).
Among unusual names, I found:
1. Valorizer - since nerdy engineer and valor never appear in the same sentence together.
2. Dateless - a reference to the bespectacled geek in the office.
3. Datefind - the geek’s attempt to cure the above stated malady.
4. Autodate - When the geek resorts to chatting with an IM bot.
5. Ntpdate - What a geek does to schedule the one accidental date he lands.
6. Undated - The result of the one accidental date.
7. Unblack - pseudonym for Michael Jackson?
8. Pockuito - A delectable Polish sausage recommended with Prosciutto.
9. Sweetener - A class used to make you sign the employment agreement.
10. Cobbled - A general reference to the way most software is put together. Shame! only one guy was willing to admit it.
11. Monotonous - Denotes a class with a lot of copy-n-paste!
12. Seniority - Hooray! We don’t have an age problem in IT. Oh wait! There was only one reference to it.
13. Switcheroo - Evidence we didn’t pay attention during English class.
14. Duplciate - More evidence we didn’t pay attention during English class.
15. Boogaloo - A disruptive software solution to the Ghostbusters.
16. Anonymizing - What the author of Boogaloo did.
17. Absolutechildren - Remnant of someone’s militaristic upbringing.
18. Delinquency - Unfortunately not of the juvenile (0) type.
19. Greenish - a sophisticated word in the male vocabulary for Teal.

There are 2,760 class names that are greater than 50 characters in length. Among the largest are:

“Has This Type Pattern Tried To Sneak In Some Generic Or Parameterized Type Pattern matching Stuff Anywhere Visitor” (from Apache ServiceMix aspectJ 1.8.0)
“Web Get Number Of Parent Process Instances With Active User And Activity Instance Expected End Date” (from Bonita Server 5.10.2)

I would have loved to be a fly on the wall when these names were being discussed!

The raw class component names data is available as a CSV file.

Solving Wordament

2014-10-17T00:00:00+00:00

Statutory Disclaimer: This post is simply an exercise in algorithm development. It is not a warrant to attempt to cheat while playing Wordament. Any attempt to cheat the game will get you booted. The booting will be well deserved and this blog takes no responsibility for it.

Game Time

Wordament is a very popular word puzzle that is available on all the mobile app stores. The object of the game is to identify as many words as possible from a 4x4 grid of letters. Words are formed by sliding your finger on the letters. You can form words by moving in any direction as long as you don’t repeat the letters. You get points for your words - longer words earn more points and there are special bonuses for uncommon words and words that contain letter sequences that are specifically called out.

A first time casual player will be simultaneously astounded at the number of words identified by the winners of any given round and embarrassed by their own dismal performance. In fact, the winners are jaw-droppingly fast. Have a look at this video of a face-off (or is that word-off?):

Some of us speed-challenged folks, especially the geeks at heart may consider instead the challenge of algorithmically finding all the words. The problem is relatively straightforward and with adequate setup may even make a great interview question. If you do use this as an interview question, I would strongly suggest letting the candidate program on a computer with an IDE/language of their choice and not on a whiteboard.

Word Bank

First we need a database of valid words. Here are a couple of sources to get such a list of words:

SIL English Wordlist
SCOWL dictionary

At a high-level, our task will be to navigate the grid adding letters to a sequence and testing if what we have is a valid word. Our challenge is to:

Ensure that we navigate all permutations of the letters.
Not visit any letter more than once.
Determine if the sequence of letters we have is a valid word.

The last step is straightforward. We can simply load the entire list of valid words into a HashMap and check to see if the word we’ve assembled is valid. However, for our purpose, we are going to eschew the HashMap based approach and instead construct a tree of valid words such that we can more efficiently figure out if a word is valid or not. From a practical computational standpoint, this extra optimization is not going to make a material difference. However, the optimization does have academic value.

From Word Bank to Word Tree

We will construct our word tree by creating nodes that have two properties:

A letter value.
An index of up to 26 child nodes, one per letter that can follow the current node.
A flag that indicates if the path from the root to the current node represents a valid word.

We use two classes to construct the nodes, a NodeLoader and the Node itself. The code for these classes is shown below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
public class NodeLoader {

	
	public static Node init() throws IOException {
        // LineIterator and FileUtils are from apache commons io.

		LineIterator iterator = FileUtils.lineIterator(new File("src/main/resources/dict.txt"));
		
		Node root = new Node();
		root.setWord(false);
		
		while (iterator.hasNext()) {
			String line = iterator.next().toLowerCase().trim();
			createNodes(line, root, 0).setWord(true);
		}
		
		return root;
	}
	

	private static Node createNodes(String line, Node current, int index) {
		if (index != line.length()) {
			Node next = current.createOrGet(line.charAt(index));
			return createNodes(line, next, index+1);
		} else {
			return current;
		}
	}
}

Notice that root node does not represent a letter. In fact, the Nodes don’t explicitly represent a letter. The letter representation is implicit from their position in the parent. For example, the sequence from the root node of the first child node, its second child node and its third child node represent the letter sequence ‘abc’.

The Node is a simple class of mostly boilerplate code (please Oracle, if you are listening, at least add Lombok style @Getter/@Setter annotations to Java 9?)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
public class Node {

    private boolean word;

    private Node[] branches = new Node[26];


    public boolean isWord() {
        return word;
    }


    public void setWord(boolean word) {
        this.word = word;
    }


    public Node getNode(char c) {
        int i = c - 'a';
        return branches[i];
    }


    public Node createOrGet(char c) {
        Node n = getNode(c);
        if (n == null) {
            n = new Node();
            branches[c - 'a'] = n;
        }

        return n;
    }
}

“Wording” it together

Armed with our word-tree, we can now write the grid input and traverse logic. For reading in the input, we will require the user to type in four lines of letters with each line containing 4 letters. We can then parse the letters and put them into a two-dimensional array:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
        	Node root = NodeLoader.init();
        Scanner scanner = new Scanner(System.in);
        char grid[][] = new char[4][4];
        String parts[] = new String[4];

        parts[0] = scanner.nextLine();
        parts[1] = scanner.nextLine();
        parts[2] = scanner.nextLine();
        parts[3] = scanner.nextLine();

        for (int i = 0; i < 4; i++) {
            for (int j = 0; j < 4; j++) {
                grid[i][j] = parts[i].charAt(j);
            }
        }

With the grid initialized we now have to traverse the grid. Our grid traversal must consider every location in the grid as a starting point. As we traverse the grid and identify valid words, we add it to a set (thus eliminating duplicates). Once the words have all been identified, we can sort them by length and print them. At a high level, assuming a “traverse” method, the code required is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
       for (int i = 0; i < 4; i++) {
            for (int j = 0; j < 4; j++) {
                traverse(grid, new boolean[4][4], i, j, root, "");
            }
        }

        List<String> temp = new ArrayList<String>(wordsFound);
        Collections.sort(temp, new Comparator<String>() {

            public int compare(String o1, String o2) {
                return o2.length() - o1.length();
            }
        });
        for (String word : temp) {
            System.out.println(word);
        }

The variable wordsFound is simple HashSet. The only thing left to flush out is the traverse routine. Grid traversal is easily written as a recursive routine. Our job is to consider the current location, add the letter to the sequence by grabbing the appropriate child node of the current node (indexed by the letter) and seeing if the node represents a complete word. If it does, we add it to our list of valid words and then try to traverse by moving in each of the 8 different directions we can head in (left, right, up, down, diagonally to the top-right, top-left, bottom-right and bottom-left).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
/**
     * @param grid The grid of letters representing the game.
     * @param visited A grid of indicators telling us which letters have already been visited.
     * @param i Our current location in the grid. x-axis
     * @param j Our current location in the grid. y-axis
     * @param node The node we have just visited.
     * @param current The currently formulated word.
     */
    private static void traverse(char[][] grid, boolean[][] visited, int i, int j, Node node, String current) {
        char c = grid[i][j];
        String newStr = current + Character.toString(c);
        Node next = node.getNode(c);

        if (next != null) {
            visited[i][j] = true;
            if (next.isWord() && newStr.length() > 2) {
                wordsFound.add(newStr);
            }

            if (j < 3 && !visited[i][j + 1]) {
                traverse(grid, visited, i, j + 1, next, newStr);
            }
            if (i < 3 && !visited[i + 1][j]) {
                traverse(grid, visited, i + 1, j, next, newStr);
            }
            if (j > 0 && !visited[i][j - 1]) {
                traverse(grid, visited, i, j - 1, next, newStr);
            }
            if (i > 0 && !visited[i - 1][j]) {
                traverse(grid, visited, i - 1, j, next, newStr);
            }
            if (i < 3 && j < 3 && !visited[i + 1][j + 1]) {
                traverse(grid, visited, i + 1, j + 1, next, newStr);
            }
            if (i < 3 && j > 0 && !visited[i + 1][j - 1]) {
                traverse(grid, visited, i + 1, j - 1, next, newStr);
            }
            if (i > 0 && j > 0 && !visited[i - 1][j - 1]) {
                traverse(grid, visited, i - 1, j - 1, next, newStr);
            }
            if (i > 0 && j < 3 && !visited[i - 1][j + 1]) {
                traverse(grid, visited, i - 1, j + 1, next, newStr);
            }
            visited[i][j] = false;
        }

    }

Here is a sample run. The letters entered are:

abcd
efgh
ijkl
mnop

This gives us the following words:

knife
plonk
glop
mink
jink
fink
fab
ink
fin
lop

The complete code is available at Github. The code comes with a plea to use it to further academic interest in such puzzle solvers and not to get cheap thrills from cheating in the real game.

Groovier Groovy DSL

2014-09-26T00:00:00+00:00

DSLs allow a programmer to tailor-make keywords and syntax to express a higher abstract representation of a problem domain. Recognizing the power of DSLs, most modern JVM languages have taken to support them. Scala, Groovy and the up-and-coming Ceylon all support writing DSLs.

Recently I needed to build a Groovy DSL to facilitate some complex data setup for executing automated scenario tests. The GroovyBuilder mechanism wasn’t to my liking as it seemed too closely tied to object instantiation and the canonical constructor syntax. While researching other options I came across Steven Devijver’s post on creating Groovy DSLs. Steve’s technique works by defining a closure to bootstrap the script execution and redirecting method resolution using delegate modification and prioritization. The idea while intriguing had two aspects that didn’t fit well with my requirement:

There wasn’t an easy way to supply the closure to a self-contained “DSL executor.”
I would have to provide closure definitions for every top-level method I wanted to have in my DSL.

My ideal solution would involve giving a self-contained DSLExecutor object a script filename and a handler object that had the method definitions for the script. I decided to dig through Spring’s GenericGroovyApplicationContext that provides a DSL for Spring context configuration and the groovy.lang.Script class for inspiration and clues to achieve this. Spring’s Groovy context loader takes advantage of the fact that the groovy.lang.Script class, using an overridden invoke method, attempts to see if the Binding object has a closure definition of the same name as the method that just failed to resolve:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
public Object invokeMethod(String name, Object args) {
    try {
        return super.invokeMethod(name, args);
    }
    // if the method was not found in the current scope (the script's methods)
    // let's try to see if there's a method closure with the same name in the binding
    catch (MissingMethodException mme) {
        try {
            if (name.equals(mme.getMethod())) {
                Object boundClosure = getProperty(name);
                if (boundClosure != null && boundClosure instanceof Closure) {
                    return ((Closure) boundClosure).call((Object[])args);
                } else {
                    throw mme;
                }
            } else {
                throw mme;
            }
        } catch (MissingPropertyException mpe) {
            throw mme;
        }
    }
}

Combining ideas from Steve’s post, Spring and the groovy.lang.Script implementations, I realized that by supplying the groovy.lang.Script’s metaclass with a methodMissing closure, I can effectively route unmatched methods to any arbitrary object. The result is a simple standalone DSL executor:

The DSLExecutor.execute() method accepts a script filename and a handler object. The script is resolved using Spring’s resource loader (you can therefore prefix the filename with classpath: or other prefixes that Spring supports). The handler object is expected to have methods for your constructs in the DSL. The execute() method returns a map of variables returned by the script.

A simple example of a handler is shown below:

1
2
3
4
5
6
7
class MyHandler {

   def printMe(String name, Closure cl) {
		println "Hello $name"
        cl(name)
   }
}

The above code allows a simple DSL of the form:

1
2
3
printMe "Dude" {
    println it
}

We can extend the handler paradigm by making it hierarchical, wherein the handler delegates processing of some methods to a more specialized handler. For example, if our DSL had a section dealing with users, we may want to be able to write a construct such as:

1
2
3
4
5
6
7
8
9
10
printMe "Dude" {
   println it
}

users {
    tom = create "Tom"
    login tom {
        it.setRememberMe()
    }
}

We may want to segregate the responsibility for all user related methods to a UserHandler class. This is easily achieved within the primary handler by setting the delegate on the closure supplied to the users method.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
class UserHandler {
   // ... various methods
}

class MyHandler {

   def printMe(String name, Closure cl) {
	println "Hello $name"
        cl(name)
   }

   def users(Closure c) {
       UserHandler handler = new UserHandler()
       c.delegate = handler
       c.resolveStrategy = Closure.DELEGATE_FIRST
       c()
   }
}

We thus have yet-another-DSL construction paradigm using Groovy. I hope other folks find this useful and are able to create some elegant tailor-made DSLs with it.

Classpath scanning with Spring

2014-09-01T00:00:00+00:00

Spring has a number of utilities that the framework ships with. One of them is a classpath scanner provided by ClassPathScanningCandidateComponentProvider. That is a long name and a mouthful. The class allows you to enable auto-discovery features in your own framework or code. Here is a simple example to demonstrate its use:

The constructor takes a boolean parameter to apply default filters or not. Default filters are based on annotations for @Component, @Repository, @Service and @Controller. Once you have your provider, you give it filters to zero-in on the types you are interested in. Filters implement the TypeFilter interface and Spring helpfully ships with a number of useful ones:

AnnotationTypeFilter: filters based on annotations on the class.
AssignableTypeFilter: filters that match based on superclass or interface.
RegexPatternTypeFilter: matches a fully qualified class name against a regular expression.

Spring also has an AbstractClassTestingTypeFilter that allows you to write your own matcher against class metadata that spring provides for each Class/Interface that it finds.

Once you have your filters defined, you simply give the scanning provider a root package to search within and it returns you a list of BeanDefinition objects for matches. You may incorrectly assume, because of the name BeanDefinition, that this is restricted to beans in a Spring context. Thankfully, it is not, as that would severely restrict the utility value of this class. The BeanDefinition is simply a container of useful information about the class including the Class instance.

Introducing Whisper

2014-08-25T00:00:00+00:00

Its common these days for Java based projects to use a logger like Logback and it is also very common to have an Email appender delivery error emails to your development or support team. This is not only common, but it is also good practice. The enabling of timely feedback can not only help correct issues in a production environment as soon as they happen, but it also allows you to respond to users of the system who were affected by the issue proactively.

A side-effect of such proactive monitoring that is often overlooked is that a simple email appender can very easily result in a flood of emails if something critical (such as user database) goes offline or if a often-used functionality stops working. Logback has a DuplicateMessageFilter that one can configure to suppress duplicates but it is too simplistic. Once it starts to suppress the message, there doesn’t seem to be a mechanism to revive the logging.

Enter Whisper

Whisper is a generic appender written to solve precisely this problem. However, the architecture of Whisper allows you to throttle any other “endpoint” appender - SMTPAppender being one of them. In other words, Whisper is an appender that sits in between your log statement and the final appender. It feeds the final appender as long as a per-message threshold is not breached. Once breached it suppresses all messages until idle rate is achieved.

Whisper is available via Maven Central:

1
2
3
4
<groupId>com.eclecticlogic</groupId>
<artifactId>whisper</artifactId>
<packaging>jar</packaging>
<version>1.0.3</version>

The Whisper binary and source code can be downloaded from the Github page.

Getting started

The Whisper code ships with a sample logback configuration (under src/sample/resources). But here is the configuration for use with Logback to handle the most common case of suppressing ERROR messages that are sent via the email appender.

To configure the Whisper appender, first you must configure two other appenders - the regular email appender for ERROR level logs and a second email appender for sending the suppression Digests when suppression kicks in.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<appender name="errorEmail" class="ch.qos.logback.classic.net.SMTPAppender">
    <filter class="ch.qos.logback.classic.filter.LevelFilter">
        <level>ERROR</level>
        <onMatch>ACCEPT</onMatch>
        <onMismatch>DENY</onMismatch>
    </filter>
    <smtpHost>ADDRESS-OF-YOUR-SMTP-HOST</smtpHost>
    <to>EMAIL-DESTINATION</to>
    <from>SENDER-EMAIL</from>
    <subject>TESTING: %logger{20} - %m</subject>
    <layout class="ch.qos.logback.classic.PatternLayout">
        <pattern>%date %-5level %logger{35} - %message%n</pattern>
    </layout>
</appender>

<appender name="errorDigest" class="ch.qos.logback.classic.net.SMTPAppender">
    <smtpHost>ADDRESS-OF-YOUR-SMTP-HOST</smtpHost>
    <to>EMAIL-DESTINATION</to>
    <from>SENDER-EMAIL</from>
    <subject>%X{whisper.digest.subject}</subject>
    <layout class="ch.qos.logback.classic.PatternLayout">
        <pattern>%date %-5level %logger{35} - %message%n</pattern>
    </layout>
</appender>

As you can see Whisper adds a special entry into the Mapped-Diagnostic-Context (MDC) to convey an appropriate email subject for the error digest.

Next we configure the Whisper appender itself.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<appender name="whisper"
    class="com.eclecticlogic.whisper.logback.WhisperAppender">
    <!-- Filter out non error logs -->
    <filter class="ch.qos.logback.classic.filter.LevelFilter">
        <level>ERROR</level>
        <onMatch>ACCEPT</onMatch>
        <onMismatch>DENY</onMismatch>
    </filter>
    <!-- This is the name of the logging category to use to send out error digests. This is associated with the 
    errorDigest appender. -->
    <digestLoggerName>digest.appender.logger</digestLoggerName>
    <!--  suppressAfter specifies the criteria to enter suppression. The example below says that if 3 errors of the same kind
    are encountered within a 5 minute window, then suppression should kick in. -->
    <suppressAfter>3 in 5 minutes</suppressAfter>
    <!-- expireAfter specifies how much of silence the logger must see between messages before stopping suppression. --> 
    <expireAfter>4 minutes</expireAfter>
    <!-- digestFrequency specifies how often error email digests should be sent containing statistics on messages 
    suppressed -->
    <digestFrequency>20 minutes</digestFrequency>

    <!-- The pass-through appender for the normal case when suppression is not in-force. -->
    <appender-ref ref="errorEmail" />
</appender>

The units for the suppressAfter and expireAfter elements can be seconds (s, sec, or secs), minutes (m, min or mins) and hours (h, hr or hrs).

The digest logger name is then associated with the digestAppender and the whisper appender is included in the list of default appenders.

1
2
3
4
5
6
7
8
<logger name="digest.appender.logger" level="error" additivity="false">
    <appender-ref ref="errorDigest" />
</logger>

<root level="debug">
    <appender-ref ref="whisper" />
    <appender-ref ref="fileAppender" />
</root>

Whisper has been released under the Apache License 2.0. Download the library or fork the code and give it a shot. I’d love to hear your feedback.