We’ve officially moved to Libera.chat.

It’s official: ##java has moved to libera.chat, and along the way changed its name from ##java to #java.

We tried not to do this, we really did. But with a gap in trust between IRC channels and IRC staff, it was bound to happen.

So what did happen?

Well, we’ve been trying to refer people to the channel blog (this site!) regularly so they could keep up with what was going on, on a
fairly regular basis. We had already moved the bot to the Libera.chat
network, because of the gap in trust, but we tried to not violate the
intent on Freenode to directly advertise a “competing network.” But these notices were fairly rare, maybe five a day.

But around the time of one of the notices, a user pointed out the migration fairly explicitly, in support of another user’s question. This was entirely innocent of ill will, mind: one user on the channel supporting another.

A few hours later, one of the IRC staff changed the channel topic and removed access for the entire set of ops, claiming the channel was abandoned and had no controls in place. The ops who were in the channel (and watching, I might add) immediately protested, because this was an entirely unwarranted action almost literally out of nowhere.

To the staff’s “credit,” they offered to restore access once the ops protested the incursion. But… remember that gap in trust?

That gap in trust was because the ops were concerned that an arbitrary decision might be made concerning the channel, and that’s exactly what happened. The staff even used the innocent user supporting another user, as part of an active conversation, as justification. “This is spam!,” was the claim, “and the ops aren’t asserting control!” — and neither claim was true.

We now have access back for ##java on Freenode. However, we’ve set the channel to be moderated (i.e., +m), to prevent possible spam; if you want to be able to talk in ##java on Freenode, you’ll need to message an op to get it.

But the channel has moved to Libera.chat (again, see ##java), along with the most invested and active of its users, as well as the bot. If you want what Freenode ##java used to be, you should migrate.

(And if you really want to, you can still join ##java – it’ll redirect to #java.)

Announcement about Migration Away From Freenode

This is an account from one op, based on casual observation. It is offered without lots of logs, and hopefully without emotion; your mileage and understanding may differ, but this view seems to be pretty consistent with what we know, so chances are it’s close to true if not factually true.

So what happened to Freenode to make us consider migration?

Basically, an ircop sold the domain for freenode to a VPN provider, who then added ads to the website against the staff policy, and then…

The VPN provider (Andrew Lee) claimed that he owned Freenode itself (as opposed to just the domain) and threatened to sue the staff (which, as a set of volunteers, didn’t have money or organization to fight this) so he basically horned his way in… and the staff quit.

So he replaced them with 4chan ops, and changed a lot of longstanding freenode policies along the way. Among them were considerations for speech considered harmful (such as hate speech towards religions or specific people groups or other protected classes.) 4chan isn’t exactly a haven for values that Freenode ##java considers important; choosing stewards from 4chan doesn’t inspire confidence.

Here’s a tweet that mentions some of this. Also see the “WTF FAQ,” a page kept up by an op on a similar programming channel.

It’s fair to argue that free speech includes speech that some consider aggressive, but Freenode policy was made to reduce social friction, and had done so fairly successfully. Changing those rules without specific stimuli feels like a harbinger of a migration to more harmful practices, an IRC server where channels are subject to behavior against which there is no official defense at the network level.

Thus, Freenode’s core values are changing, literally, and that’s not necessarily a desired condition.

The new owner has done very little to engender trust during the transition, including what started the transition. When you start off with “Hey, I own the domain, so that means I own the service, right?,” that sends all kinds of messages to the community that you don’t want the community to hear.

What’s more, the new staff is incredibly defensive about owning the network, to the point where they’re forcibly taking over channels that don’t appreciate the possibility of being taken over. This hasn’t affected ##java or the directly-associated channels yet, but it has affected some of the channels that are implicitly associated.

So What Are We Gonna Do About It?

Well… we’re going to move the channel to a new network. We never really had any other recourse outside of the good graces and intent of Freenode staff, and that has always worked, because everyone’s been pretty benevolent. But with a wholesale change in staff, it’s hard to predict what the new staff’s intent is or how responsive they’ll be, and what we do know indicates a lot of defensive posturing and not a lot of effort to engender trust.

What we‘re doing is “moving away from Freenode,” to an alternative server. There are a few main candidates: EFNet (which has a #java already), Undernet (which … also has a #java), DALNet (which not only already has a #java, but underwent its own political chaos a few years ago and is under autocratic control), OFTC (which hosts the OpenJDK channel, among many others), and Libera.chat, which was created largely by the outgoing Freenode staff as a response to Andrew Lee taking over.

Of these, the latter seems the most palatable. So that’s literally what we’re doing: we’re moving to a new server.

This is not an act of defiance or even active defense; it’s just a move that seems wisest based on the level of trust we have for the new staff and organization behind Freenode. It’s done mostly to protect the idea behind ##java and its associated channels, and it’s done without malice.

We’re going to maintain a presence on Freenode as long as we’re able to, but the “main effort” for community development will be on the new server.

Hibernate Ignores Entity Listeners

I switched a codebase from EclipseLink as JPA provider to Hibernate and found out that with Hibernate you can not use inheritance in event listener classes. The JPA specs do not go into much detail on this topic so JPA providers are bound to handle this different from one another.

I separated my entity listeners from my entities even though it is possible to add methods annotated with e.g. @PostPersist directly in the entity class. The entity listeners themselves were less than trivial but only slightly so: I created a generic base class for all entity listeners and specialized subclasses for each entity.

public class EntityListenerBase<T> {
	public void postPersisted(T object) {
	protected abstract Event createPostPersistEvent(T object);

public class ThingEntityListener extends EntityListenerBase<Thing> {
	protected Event createPostPersistEvent(Thing thing) {
		return new ThingAdded(thing);

Adding the entity listeners to the entity is straight-forward: Just add the @EntityListeners annotation to the entity and point it to the listener.

public class ThingEntity implements Thing {

EclipseLink had no trouble at all using event listeners like these. The problems started after I switched the JPA provider to Hibernate: at some point I realized that apparently the app was not generating any events anymore in response to persistence events. It was like the entity listeners weren’t even there!

After a longer debugging session into how Hibernate manages its event listeners I discovered that my previous approach for the event listeners was no longer viable, for two reasons.

The first reason is that Hibernate insists on the annotated event methods being declared directly in the entity listener class. Methods inherited from super classes are not scanned for event annotations! This is different from the behaviour when methods in the entity class itself are annotated; in that case, super classes annotated with @MappedSuperclass would be used to locate event listener methods as well. However, this is not the case for entity listener classes.

Upon moving the postPersisted method down to the subclass I stumbled upon the second reason: for every event annotation Hibernate only allows a single method in the entity listener class to be annotated with it. Now, because the original postPersisted method has a generic parameter and the subclass has a fixed type parameter, the compiler creates two versions of the method, one with Object as the parameter type, and one with the actual type I want, Thing. One of the methods is marked as synthetic but both get the annotation, and Hibernate doesn’t like that. Not. One. Bit.

So, the solution is simple, even if it means a little bit of code duplication over all the entity listeners: just remove the type hierarchy from the event listeners.

public class ThingEntityListener {
	public void postPersist(Thing thing) {
		SomeEventBus.sendEvent(new ThingAdded(thing));

One could argue that this is actually even better; less class hierarchy, and all relevant listener methods are right there in the class, increasing visibility and reducing complexity.

WebMvcConfigurationSupport Mangles ISO8601 Timestamps in Spring Boot

I was working on a test in my Spring-Boot app and noticed that the timestamps in JSON output were sometimes formatted incorrectly. Luckily I was able to identify the issue and fix it.

During an integration test I noticed that an expected JSON output of "2019-08-26T08:22:21Z" was actually 1566807741.000000000 even though in other tests in my application the automatic conversion from java.time.Instant to an ISO8601-formatted string was working just fine. As is the default for Spring-Boot, Jackson’s ObjectMapper was used for the conversion. So what could make Jackson work correctly in one case but not another?

First I thought that I did something wrong with configuring so I did a lot of googling in order to find out what I was doing wrong. I added jackson-datatype-jsr310 to my project’s dependencies, I disabled Jackson’s “write dates as timestamps” serializer feature in four different ways but the error would persist. I dug into how the ObjectMapper used by Spring was created; maybe the feature was not disabled correctly? No, it was disabled just fine but Jackson still would not format the Instant properly.

During my research on how to create your own ObjectMapper and have Spring use it, I stumbled upon this question. It had nothing to do with my immediate problem but it mentioned that when using a @Configuration class extending WebMvcConfigurationSupport in conjunction with @EnableWebMvc Spring-Boot’s auto-configuration would be disabled. A disabled auto-configuration could explain an ObjectMapper that can not format Instants properly!

As it turns out, I had a SwaggerConfiguration class that extended WebMvcConfigurationSupport. Following the hints in the StackOverflow post I replaced the superclass by WebMvcConfigurerAdapter only to discover that it was deprecated. Fixing that was easy, though, as the interface WebMvcConfigurer could (and should!) be used instead. And this made my integration test work…

…although at this point I am still not 100% sure why. I am not using @EnableWebMvc anywhere (just @SpringBootApplication) so I do not yet know why the SwaggerConfiguration class extending WebMvcConfigurationSupport would disable Spring-Boot’s auto-configuration mechanisms. Spring’s usually excellent documentation is rather unclear as to what happens if you only do one of those things.

I also don’t know why the Jackson mapper in the working test had its “write dates as timestamps” serializer feature disabled but the one in the other test had not. From my understanding it must have been disabled so that Jackson’s jackson-datatype-jsr310 module was used—it was already a part of my dependencies without me knowing so explicitely adding it to my project’s build file did not actually change anything.

And even though my mental model of some things involving Spring has been improved by getting this to work I will not investigate this any further. My integration tests work and that shall be good enough for me, for today.

Help! My Java Locale is Wrong in JDK11!

We discovered this issue when a user from the Netherlands reported that, upon upgrading to Java 11, his application now reported that the week started on Sunday. This is incorrect for the Netherlands, but the real issue was the change of behavior.

With JDK11, Java now loads locales differently. https://www.oracle.com/technetwork/java/javase/documentation/java11locales-5069639.html#providers details the loading behavior for JDK11, and how it supersedes previous versions. Notably, CLDR, or classloaded entries, now go before COMPAT, or things included with the JDK. This can lead to incorrect behavior with complex classpaths.

To supersede this setting, the java.locale.providers system setting is used. It’s a comma-separated list of values, where the possible values are the providers in the blog post above. The default value is CLDR,COMPAT,HOST,JRE where JRE is also COMPAT. The value that I would have expected is COMPAT,HOST,CLDR, where it uses the built-in locales first (from JDK9!), then the host locales, then the classloader locales. Setting -Djava.locale.providers=COMPAT will cause the JDK11 locale loader to act like the JDK9 locale loader.

Jackson-databind and Default Typing Vulnerabilities

Today, GitHub sent out security notices to owners of projects using old jackson-databind versions (older than and 2.9.5). These notices pertain to this issue. I have talked about its relevance before on IRC, but since it is getting more attention now, I will describe it here again.

The Problem

The “bug” comes from using the so-called default typing. This feature allows a user to deserialize subclasses (or even Object) without specifying the full possible type hierarchy. Consider this model:

interface I {
class A implements I {
    private int i;
class B implements I {
    private boolean b;

Now, if we wish to serialize the interface I, you usually need to specify some sort of type info. This is typically done through an annotation on the interface:

@JsonTypeInfo(use = JsonTypeInfo.Id.NAME)
@JsonSubTypes({@JsonSubTypes.Type(value = A.class, name = "A"), @JsonSubTypes.Type(value = B.class, name = "B")})
interface I {

If we now serialize an object of type I, we get a result like this:

new ObjectMapper().writerFor(I.class).writeValueAsString(new A())

Deserializing this JSON works as expected.

Default typing

This annotation-based registration is fine for small use cases, but can get cumbersome if the types are in different modules, or there are just a lot of them, or it would be bad style to reference them from the parent class. Normal Java serialization does not have this problem (it just carries the actual dynamic class name with it), so this could be a barrier for adoption of Jackson for previous Java serialization users.

The problem here is that we want people to migrate from Java serialization. There are lots of reasons, most of them compelling.

Enter default typing. With default typing, we don’t need the type info annotations at all:

new ObjectMapper().enableDefaultTyping().writerFor(I.class).writeValueAsString(new A())

(Ignore the fact that this is now suddenly an array – this is one of the ways jackson may include type info)
The idea is simple: Include the full class name in the json, and you can simply get the proper class at runtime! Sounds good, right?

The Vulnerability

Well, turns out this is not that good of an idea. Java serialization does a very similar thing (though it still requires the named class to be serializable, which jackson doesn’t) and this has lead to what feels like a third of all serious security vulnerabilities in Java applications, ever. The problem lies with the fact that Java classpaths are often massive, and allowing any class on that classpath to be deserialized at will can be disastrous since it exposes a huge attack surface. If you can get any class on the classpath to execute code when deserialized with jackson, you have successfully achieved remote code execution.
This is exactly what happened with Jackson. Some classes that were common on user classpaths could be deserialized to execute arbitrary code. The fix for this issue is basically a blacklist of a few of these classes that could be exploited. Blacklists are not a solution, though, and since this first list, the list has been amended several times. The maintainers are playing whack-a-mole here, and in my opinion it is a waste of everyone’s time to be adding all exploitable classes to this list.

The Solution

Our experience with this same issue in the Java serialization world tells us not to deserialize untrusted data. Luckily, Jackson is much more secure than Java serialization – if you don’t use default typing. The only acceptable solution to this issue in the long run is: do not use default typing to deserialize untrusted data. Default typing is rarely necessary or even a good idea.
Unfortunately, online resources saying this are sparse. Default typing is an “easy” solution, and many people simply do not have the security awareness to see the issue with it – they will stumble over stackoverflow answers such as this one and simply enable default typing to easily serialize Object fields. The documentation of Jackson also doesn’t highlight this as much as it should.
Two alternatives to default typing exist in Jackson:

  • Normal, annotation-based typing as shown above. This allows you either to use the full name as with default typing, or even specify your own name for greater compatibility (you can later change the class name without affecting the serialized representation). This is the “standard” solution, and the appropriate one for most use cases.
  • Should you not know the possible subtypes of a class you wish to deserialize in advance, you can use the rich registerSubtypes API to dynamically add the types you desire. These types could be detected through an existing module system you are already using, or using something like SPI.

All in all, I am a bit dissatisfied with the attention this issue has gotten. The issue is a security vulnerability by design, and anyone using default typing should have been aware of it. Luckily, default typing is not on by default. I do not have statistics but I would be surprised if many people used it or knew of its existence in the first place, and so I find the attention GitHub has given this a bit over the top – the biggest thing these notifications will spread is uncertainty about Jackson, so this article was an attempt at clearing up what it’s actually all about.

OpenJDK 11 on Fedora

Interesting in running the current version of Java (and its compiler) on your Fedora instance? It’s easy:

sudo dnf install java-11-openjdk-devel
sudo alternatives --config java # select Java 11
sudo alternatives --config javac # if necessary, select java 11

Note that this is still labeled as the “early access” version as of this writing – but the early access version is virtually identical to the released version.

The JavaChannel Podcast, Episode XVII (17)

The XVIIth podcast is live! … one wonders why the Roman Empire didn’t last much longer, doesn’t one. One also wonders how long one can refer to oneself in the whatever-person this is. (It’s… second-person? No, folks, it’s “third person.” English grammer fumbling and lesson over.)
In this podcast, we chase a few things: the discussion topic centers back around on Python vs. Java, and why one might encounter one over the other in certain situations.
We also jump down the gullets of a lot of varied topics, only some of which are Java, but most are at least sort-of relevant:

  • Expanded switch statements in java! … Stuff we can look forward to in Java 12. Java’s looking more and more like Kotlin all the time.
  • Hibernate with Kotlin – powered by Spring Boot describes, of all things, using Kotlin, Hibernate, and Spring Boot together. (The summary has a good laundry list of things to pay attention to.)
  • Another Java Enhancement Proposal: the Java Thread Sanitizer. It would provide a dynamic data race detector implementation for Java and native JNI code utilizing ThreadSanitizer in OpenJDK with interpreter and C1 support… and while it’s great that tools like this might exist, it’s fantastic that they’re apparently rarely needed if they show up twenty years after Java’s release.
  • The Left Hand of Equals is not strictly java related, but still interesting. It mostly centers around what it means for one object to be equal to another. I wonder if the channel blog should have recommended LINKS as well as books….
  • TomEE: Running with Systemd is a pretty fair walkthrough, and should be applicable to other app servers as well… that is, if appservers are still relevant in the world where microservice containers do be existin, mon.
  • https://brilliant.org/ is a kinda neat site for math and logic skills. If you use it, don’t forget to take the question surveys, because that’s how it’ll improve.
  • How to Be a Person Who Can Make a Damn Decision is the first of at least two annoying links for Andrew – this one says it’s “how to be a person who can” but actually mostly documents people being those kinds of people. We also have the resurgence of the podcast drinking game; take a shot whenever game theory is mentioned. However, the article doesn’t really have a lot of takeaways, apart from pointing out that the ability to make a decision quickly is probably a worthwhile skill to have.
  • OpenPDF 1.2.1 has been released. Joe didn’t even know about this library. No idea how useful it is; this release doesn’t look like a big one from the surface, but still: the more libraries out there, the merrier, right? (Unless they’re logging libraries.)
  • 7 Scientific Benefits of Reading Printed Books is the second annoying link for Andrew. It goes over some, uh, tenuous reasons print books are worth reading, some of which were taken exception to. Joe thought it was worth thinking about when e-books are ALL the rage for programming topics…
  • Other tangential topics:
    • https://hmijailblog.blogspot.com/2018/08/you-could-have-invented-lmax-disruptor.html I hated reading this, so I stopped.
    • https://perens.com/2018/08/22/new-intel-microcode-license-restriction-is-not-acceptable/ Apparently Intel was saying not to publish benchmarks, which is kinda gross. However, worth noting is that after the initial scraping of this article, Intel backed down and changed the police. Way to go, Bruce Perens!

The JavaChannel Podcast, vol 16

It’s only been six months, so it’s finally time for a new podcast. This one doesn’t even pretend to go over the mountains of killer content from ##java since the last podcast – it focuses on some of the more recent links, and that’s it. Well, apart from talking about the Java ecosystem a bit, especially in contrast with Python, an upstart language that’s making a lot of headway lately thanks to a giant upsurge in data science applications.

(A bit of irony: the very first paragraph in the podcast says it’s only been “four months” when it’s actually been six. Yikes.)

But there are some interesting links, and here are the ones the podcast focused on!

This was written with the new editor plugin for WordPress, called “Gutenberg.” It’s a lot like Medium.com’s editor. It’s effective for writing… unless you have any actual features you want in the text.