Hortonworks announces availability of HDF 3.1

Hortonworks, a provider of global data management solutions, has announced general availability of Hortonworks DataFlow (HDF) 3.1. The launch of HDF 3.1 enhances operations and developer productivity, and delivers stronger integration and interoperability between HDF and Hortonworks Data Platform (HDP). This new version of HDF delivers a single open-source tool set that integrates governance, security and management across the entire data lifecycle from the edge to analytics to real-time decisions.

“As more and more organizations embrace the value of capturing data in motion, they are also devising more ambitious objectives and sophisticated approaches to maximizing the utility of that data and leveraging the insights provided,” said Scott Gnau, Chief Technology Officer, Hortonworks. “This newest release improves ease-of-use and time-to-value by enabling developer productivity while also expanding enterprise interoperability.”

The new HDF 3.1 offers organizations the following enhancements:

Improved management of data flows to speed up application development and operations:

Apache NiFi Registry, a new Apache sub-project now included within HDF Enterprise Management Services, facilitates the development, management and portability of data flows. The core to its functionality is the ability to abstract data flow schemas and programs to enable users to track and monitor data flow changes at a more granular level. Data flow schemas are stored in a shared repository that allows for easy sharing on a global basis as well as versioning of schemas.

Through this, the export and import of data flows allow easy porting and enables smooth migration of data flows from one environment to another. The functionality improves the storage, control, and management of versioned flows, further shortening the software development life cycle and accelerating application deployment to achieve faster time to value.

Increased developer productivity:

HDF 3.1 adds new capabilities to improve streaming data operations in Hortonworks Streaming Analytics Manager (SAM). In the new SAM “Test Mode”, developers can experiment in the creation of SAM apps using mock data and create unit tests for SAM Apps integrated into their continuous integration and delivery environments.

In addition, the new SAM Operations Module gives users the ability to easily test, debug, troubleshoot, and monitor the deployed applications, making the operations of a running application as easy as building the application. These efficiencies will allow developers to reallocate their time toward value-generating initiatives in application creation.

Increased operational efficiencies through automation:

Today’s announcement also includes improvements to Apache Ambari and Apache Ranger that automate the processes for managing Apache NiFi resources. Users can now easily add a new NiFi node to an existing cluster without manually updating node information. Improvements to Apache Ranger allow administrators to quickly define group-based policies for NiFi resources. The elimination of repetitive manual process streamlines the operations to achieve optimal efficiency and utilization of people.

Advanced security and message handling with Apache Kafka 1.0:

With this release, HDF now supports Apache Kafka 1.0, setting a foundation for manageability, visualization, and navigation around Kafka clusters catering to the distinct needs of various personas in the subsequent release. The interoperability between HDF and Kafka also enables advanced security options and more stringent message processing semantics including support for message headers and transactions.

Kafka 1.0 is now fully integrated with all HDF services including:

Apache Ambari support for Kafka 1.0 – Users can now install, configure, manage, upgrade, monitor, and secure Kafka 1.0 clusters with Ambari.

Apache Ranger support for Kafka 1.0 – To enhance data governance and lineage, users can now manage access control policies using resource or tag-based security for Kafka 1.0 clusters.

New NiFi and SAM processors for Kafka 1.0 – New processors in NiFi and Streaming Analytics Manager support Kafka 1.0 features including message headers and transactions.

Integrated governance and security for enterprise deployments:

When HDF is co-located with Hortonworks Data Platform (HDP), HDF will now be able to integrate with Apache Atlas for governance, Hortonworks SmartSense, and Apache Knox for security to provide better manageability and access of data as well as toolsets across the platform. This integration creates opportunities for:

Cross-component lineage view at dataset level with Atlas

Easier data collection process with SmartSense for proactive support and troubleshooting

Convenience of single sign-on capability with Knox as a standard security gateway

The integration also allows Atlas to obtain meta information of NiFi data flows to enable holistic data governance across data at-rest and in-motion for compliance.