Skip to main content

Migrating older extensions

Migrating from Ant to Maven​

Why are we doing this change?​

Ant is a fairly old (antique?) build system that does not incorporate any dependency management. By migrating to Maven we are making it easier for developers to extend OpenRefine with new libraries, and stop having to ship dozens of .jar files in the repository. Using the Maven repository also encourages developers to add dependencies to released versions of libraries instead of custom snapshots that are hard to update.

When was this change made?​

The migration was done between 3.0 and 3.1-beta with this commit: https://github.com/OpenRefine/OpenRefine/commit/47323a9e750a3bc9d43af606006b5eb20ca397b8

How to migrate an extension​

You will need to write a pom.xml in the root folder of your extension to configure the compilation process with Maven. Sample pom.xml files for extensions can be found in the extensions that are shipped with OpenRefine (gdata, database, jython, pc-axis and wikidata). A sample extension (sample) is also provided, with a minimal build file.

For any library that your extension depends on, you should try to find a matching artifact in the Maven Central repository. If you can find such an artifact, delete the .jar file from your extension and add the dependency in your pom.xml file. If you cannot find such an artifact, it is still possible to incorporate your own .jar file using maven-install-plugin that you can configure in your pom.xml file as follows:

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-install-plugin</artifactId>
        <version>2.5.2</version>
        <executions>
            <execution>
                <id>install-wdtk-datamodel</id>
                <phase>process-resources</phase>
                <configuration>
                    <file>${basedir}/lib/my-proprietary-library.jar</file>
                    <repositoryLayout>default</repositoryLayout>
                    <groupId>com.my.company</groupId>
                    <artifactId>my-library</artifactId>
                    <version>0.5.3-SNAPSHOT</version>
                    <packaging>jar</packaging>
                    <generatePom>true</generatePom>
                </configuration>
                <goals>
                    <goal>install-file</goal>
                </goals>
            </execution>
            <!-- if you need to add more than one jar, add more execution blocks here -->
        </executions>
      </plugin>

And add the dependency to the <dependencies> section as usual:

com.my.companymy-library0.5.3-SNAPSHOT

Migrating to Wikimedia's i18n jQuery plugin​

Why are we doing this change?​

This adds various important localization features, such as the ability to handle plurals or interpolation. This also restores the language fallback (displaying strings in English if they are not available in the target language) which did not work with the previous set up.

When was the migration made?​

The migration was made between 3.1-beta and 3.1, with this commit: https://github.com/OpenRefine/OpenRefine/commit/22322bd0272e99869ab8381b1f28696cc7a26721

How to migrate an extension​

You will need to update your translation files, merging nested objets in one global object, concatenating keys. You can do this by running the following Python script on all your JSON translation files:

import json import sys

with open(sys.argv[1], 'r') as f: j = json.loads(f.read())

result = def translate(obj, path): res = if type(obj) == str: result['/'.join(path)] = obj else: for k, v in obj.items(): new_path = path + [k] translate(v, new_path)

translate(j, [])

with open(sys.argv[1], 'w') as f: f.write(json.dumps(result, ensure_ascii=False, indent=4))

Then your javascript files which retrieve the translated strings should be updated: $.i18n._('core-dialogs')['cancel'] becomes $.i18n('core-dialogs/cancel'). You can do this with the following sed script:

sed -i "s/$.i18n.(['"]([A-Za-z0-9/\-])['"])[['"]([A-Za-z0-9-_])["']]/$.i18n('\1/\2')/g" my_javascript_file.js

You can then chase down the places where you are concatenating translated strings, and replace that with more flexible patterns using the plugin's features.

Migrating from org.json to Jackson​

Why are we doing this change?​

The org.json (or json-java) library has multiple drawbacks.

  • First, it has limited functionality - all the serialization and deserialization has to be done explicitly - an important proportion of OpenRefine's code was dedicated to implementing these;
  • Second, its implementation is not optimized for speed - multiple projects have reported speedups when migrating to more modern JSON libraries;
  • Third, and this was the decisive factor to initiate the migration: its license is the MIT license with an additional condition which makes it non-free. Getting rid of this dependency was required by the Software Freedom Conservancy as a prerequisite to become a fiscal sponsor for the project.

When was the migration made?​

This change was made between 3.1 and 3.2-beta, with this commit: https://github.com/OpenRefine/OpenRefine/commit/5639f1b2f17303b03026629d763dcb6fef98550b

How to migrate an extension or fork​

You will need to use the Jackson library to serialize the classes that implement interfaces or extend classes exposed by OpenRefine. The interface Jsonizable was deleted. Any class that used to implement this now needs to be serializable by Jackson, producing the same format as the previous serialization code. This applies to any operation, facet, overlay model or GREL function. If you are new to Jackson, have a look at this tutorial to learn how to annotate your class for serialization. Once this is done, you can remove the void write(JSONWriter writer, Properties options) method from your class. Note that it is important that you do this migration for all classes implementing the Jsonizable interface that are exposed to OpenRefine's core.

We encourage you to migrate out of org.json completely, but this is only required for the classes that interact with OpenRefine's core.

General notes about migrating​

OpenRefine's ObjectMapper is available at ParsingUtilities.mapper. It is configured to only serialize the fields and getters that have been explicitly marked with @JsonProperty (to avoid accidental JSON format changes due to refactoring). On deserialization it will ignore any field in the JSON payload that does not correspond to a field in the Java class. It has serializers and deserializers for OffsetDateTime and LocalDateTime.

Useful snippets to use in tests:

  • deserialize an instance: MyClass instance = ParsingUtilities.mapper.readValue(jsonString, MyClass.class); (replaces calls to Jsonizable.write);
  • serialize an instance: String json = ParsingUtilities.mapper.writeValueAsString(myInstance); (replaces calls to static methods such as load, loadStreaming or reconstruct);
  • the equivalent of JSONObject is ObjectNode, the equivalent of JSONArray is ArrayNode;
  • create an empty JSON object: ParsingUtilities.mapper.createObjectNode() (replaces new JSONObject());
  • create an empty JSON array: ParsingUtilities.mapper.createArrayNode() (replaces new JSONArray()).

Before undertaking the migration, we recommend that you write some tests which serialize and deserialize your objects. This will help you make sure that the JSON format is preserved during the migration. One way to do this is to collect some sample JSON representations of your objects, and check in your tests that deserializing these JSON payloads and serializing them back to JSON preserves the JSON payload. Some utilities are available to help you with that in TestUtils (we had some to test org.json serialization before we got rid of the dependency, feel free to copy them).

For functions​

Before the migration, you had to explicitly define JSON serialization of functions with a write method. You should now override the getters returning the various documentation fields.

Example: Cos function before and after.

For operations​

Before the JSON migration we refactored engine-dependent operations so that the engine configuration is represented by an EngineConfig object instead of a JSONObject. Therefore the constructor for your operation should be updated to use this new class. Your constructor should also be annotated to be used during deserialization.

Note that you do not need to explicitly serialize the operation type, this is already done for you by AbstractOperation.

Example: ColumnRemovalOperation before and after.

For changes​

Changes are serialized in plain text but often relies on JSON serialization for parts of the data. Just use the methods above with ParsingUtilities.mapper to maintain this behaviour.

Example: ReconChange before and after.

For importers​

The importing options have been migrated from JSONObject to ObjectNode. Your compiler should help you propagate this change. Utility functions in JSONUtilities have been migrated to Jackson so you should have minimal changes if you used them.

Example: TabularImportingParserBase before and after.

For overlay models​

Migrate serialization and deserialization as for other objects.

Example: WikibaseSchema before and after

For preference values​

Any class that is stored in OpenRefine's preference now needs to implement the com.google.refine.preferences.PreferenceValue interface. The static load method and the write method used previously for deserialization should be deleted and regular Jackson serialization and deserialization should be implemented instead. Note that you do not need to explicitly serialize the class name, this is already done for you by the interface.

Example: TopList before and after