Blog

Following my article on enabling parallel processing for OpenRefine: Spark vs Akka, I drafted a road map to integrate OpenRefine with Akka.

Andrey from SpazioDati developed Refine on Spark in an attempt to process larger dataset which is good. However it fell short in some areas and I wanted to benchmark it with an other parallelism engine like Akka.

In the two articles of this series of three we have seen how corporations leverage Open Source Software (OSS) to build new solution and how an open source strategy foster in-house and community based innovation. It is a new world for software development and related business-building strategies, a world where even enterprise software projects leverage open source components for their foundation. The freedom to extend or even fork the project to explore new options can lead to unexpected innovation and usage, when done right. Open licenses mean that contributors – including yourself – have no restrictions on tweaking code and tailoring it to match requirements or customizing it to specific needs within an organization or industry. However to ensure continuity and sustainability of the project, the community now need to scale, gain the critical size and drive broad adoption.

In the first article of this series of three on open source strategy, we seen how Open Source Software (OSS) is now part of corporations strategy and the benefit of building an ecosystems. By sharing your code and vision openly you enable other expand your solution, adapt to their specific environment. It is not rare to see unexpected innovation sparking in this process. In this article we will see the steps to create such innovation friendly environment and take example the OpenRefine community.

In its 2014 annual survey on the Future of Open Source, Black Duck indicated that 56 percent of corporations expected to contribute to open source software solutions, more than ever before. Most of those companies were already using open source software internally, but wanted to go a step further and contribute back through comments, bug reports, or by subsidizing developer time.

The last five weeks have been full of learning and surprises (good and bad). We have a better understanding on how Refine is used and how it behaves with different browsers and data set sizes. Some things worked as expected, other completely broke. This post highlights the coming technical points we plan to address to improve RefinePro.

It all started in June 2011 when we opened OpenRefine Tips and Recipes blog. Over the last three years we documented Google Refine and then OpenRefine functionality with concrete examples and screenshots and listed the best materials written by others. It started as a personal notebook and grew into a knowledge base of over 110 articles coverings more than 80 topics.

We are extremely happy to announce that RefinePro has signed a partnership with the Data Scientist Training for Librarians (DST4L) allowing all librarians who take the course to have access to the RefinePro platform for the next eight months (until June 2015). Through their RefinePro accounts, they will have access to their projects from multiple computers and be able to work on the latest version of OpenRefine.

Over the last five years, OpenRefine has built a robust platform, to which many developers have contributed plugins and extensions useful for their own audiences. That list of plugins and reconciliation services grows month after month, demonstrating that the community is active and thriving, with a healthy and expanding user base.

One of the big news in the industry this month was CrowdFlower raising $12.5 million in funding to support its growth. CrowdFlower is like a souped up Amazon Mechanical Turk with a very nice API and well-thought-out back end for job editors. I couldn’t agree more when Mark Sullivan say: