Datavolo is proud to announce the release of a GitHub Action designed to help with Continuous Integration of Apache NiFi Flows and make reviewing of changes between two flow versions as easy as possible.

At Datavolo, collaboration on the Flow Definitions is done by the use of registry clients directly connecting to code repositories. We currently provide two options:

• GitHub Registry Client
• GitLab Registry Client

The idea of a Registry Client is to connect NiFi directly to a repository and use the repository as a way to store and version flow definitions. A demo is available on our Youtube Channel where the GitHub Registry Client is used as an example to understand the flow versioning capabilities with NiFi.

Branching

Let’s consider a relatively simple approach for branching:

• There is a main branch for what is deployed in the production environment
• There is a dev branch for the working being done in the development environment
• Additional feature-xxx branches are created when working on the new feature of the flow

A fairly common strategy is to create a feature branch from the dev branch when a new feature needs to be developed in an existing flow. This allows for multiple individuals to work on different features for the same flow, at the same time.

To do that, the following steps would be executed:

• In the code repository used by the configured Registry Client, the user would go create the feature branch.
• Once done, the user can go back into the NiFi UI and import the flow from the feature branch as a new process group.
• At this point, the user can go in the process group and start working on the changes and commit changes whenever required.

Once the final changes for the feature are done and the last commit landed on the feature branch, it would be time to go into the code repository and open a pull request from the feature branch against the dev branch.

A demo is available on our Youtube Channel where we discuss the concept of branching with the GitHub Registry Client.

Pull Requests and reviewing changes

In NiFi, a flow definition is a JSON file and comparing two JSON files might not be the easiest thing to do. When opening a pull request, it is expected from the author to clearly describe the changes that are submitted but the reviewers of the pull request would also look at the differences between the two JSON files to accept or not the submitted changes.

To help with reviewing changes, Datavolo provides a GitHub Action that will compare the two flow definitions and automatically add a comment to the pull request with a human readable description of the changes.

To configure this Github Action, create a file .github/workflows/flowdiff.yml in the repository used to version your NiFi Flow definitions. The content of the file can be found in the Datavolo Flow Diff GitHub Action page.

When filing a Pull Request to review changes for a new flow version, the GitHub Action will be automatically triggered to compare the two versions and comment the Pull Request with a comprehensive description of the changes.

Please check this video for a demo of this feature:

This GitHub Action is free to use by anyone and contributions are welcome!

Datavolo is working on a lot of new features to help with CI/CD pipelines when it comes to Apache NiFi, so stay tuned and have a look at our documentation.