Blog

In our effort to engage with local OpenRefine community and foster peer-to-peer connections, the latest Toronto OpenRefine Meet-Up (on November 17) focused on actual use cases, presented by OpenRefine user from various backgrounds. The goal to demonstrate how OpenRefine plays in a specific contexts and data environment, rather than drilling down on specific functionality. This article provides a summary of the night and highlight of each talk.

Learn the basics of OpenRefine with our Foundation course available on the tranzform course platform.

Stefan Urbanek when laying the foundation for the school of data program at the Open Knowledge, presented the following Data Processing Pipeline going from:

The main motivation behind RefinePro is being able to commit developer time to OpenRefine (since I am not a developer myself) and have the project leave its stagnant stage. RefinePro motivation isn’t to make millions or get some prestige by stealing other people work – there is faster and less risky way to do that than creating a start-up on a niche market.

Following my article on enabling parallel processing for OpenRefine: Spark vs Akka, I drafted a road map to integrate OpenRefine with Akka.

Andrey from SpazioDati developed Refine on Spark in an attempt to process larger dataset which is good. However it fell short in some areas and I wanted to benchmark it with an other parallelism engine like Akka.

In the two articles of this series of three we have seen how corporations leverage Open Source Software (OSS) to build new solution and how an open source strategy foster in-house and community based innovation. It is a new world for software development and related business-building strategies, a world where even enterprise software projects leverage open source components for their foundation. The freedom to extend or even fork the project to explore new options can lead to unexpected innovation and usage, when done right. Open licenses mean that contributors – including yourself – have no restrictions on tweaking code and tailoring it to match requirements or customizing it to specific needs within an organization or industry. However to ensure continuity and sustainability of the project, the community now need to scale, gain the critical size and drive broad adoption.

In the first article of this series of three on open source strategy, we seen how Open Source Software (OSS) is now part of corporations strategy and the benefit of building an ecosystems. By sharing your code and vision openly you enable other expand your solution, adapt to their specific environment. It is not rare to see unexpected innovation sparking in this process. In this article we will see the steps to create such innovation friendly environment and take example the OpenRefine community.

In its 2014 annual survey on the Future of Open Source, Black Duck indicated that 56 percent of corporations expected to contribute to open source software solutions, more than ever before. Most of those companies were already using open source software internally, but wanted to go a step further and contribute back through comments, bug reports, or by subsidizing developer time.

The last five weeks have been full of learning and surprises (good and bad). We have a better understanding on how Refine is used and how it behaves with different browsers and data set sizes. Some things worked as expected, other completely broke. This post highlights the coming technical points we plan to address to improve RefinePro.