FATE & KubeFATE v1.7.0 released! Flexible & decoupled FATE-Flow, new compute & storage engine support, tons of algorithm enhancements and MUCH MORE!


wfangchi@...
 

Hi,

We are very excited to introduce the recently released FATE 1.7.0 and KubeFATE 1.7.0! It is a major release with significant architectural improvements, lots of new features, enhancements and bug-fixes. Welcome to check this out!

One of the major changes in FATE 1.7.0 is FATE-Flow, the
multi-party federated task security scheduling platform for federated learning end-to-end pipeline, is now in an independent repository, which enables a more flexible model of working with all the different components.

Release Notes:
FATE 1.7.0: https://github.com/FederatedAI/FATE/releases/tag/v1.7.0
FATE-Flow 1.7.0: https://github.com/FederatedAI/FATE-Flow/releases/tag/v1.7.0
KubeFATE 1.7.0: https://github.com/FederatedAI/KubeFATE/releases/tag/v1.7.0

Major Features and Improvements in FATE 1.7.0

FATE-ARCH

  • Support EggRoll 2.4.0
  • Support Spark-Local Computing Engine
  • Support Hive Storage
  • Support LocalFS Storage for Spark-Local Computing Engine
  • Optimizing the API interface for Storage session and table
  • Simplified the API interface for Session, remove backend and workmode parameters
  • Heterogeneous Engine Support: Federation between Spark-Local and Spark-Cluster
  • Computing Engine, Storage Engine, Federation Engine are set in conf/service_conf.yaml when FATE is deployed

FederatedML

  • Optimized Hetero-SecureBoost: with gradient packing、cipher compressing, and sparse point statistics optimization, 4x+ faster
  • Homo-SecureBoost supports memory-based histogram computation for more efficient tree building, 5x+ faster
  • Optimized RSA Intersect with CRT optimization, 3x+ faster
  • New two-party Hetero Logistic Regression Algorithm: mixed protocol of HE and MPC, without a trusted third party
  • Support data with match-id, separating match id and sample id
  • New DH Intersect based on PH Key-exchange protocol
  • Intersect support cardinality estimation
  • Intersect adds optionally preprocessing step
  • RSA and DH Intersect support cache
  • New Feature Imputation module: can apply arbitrary imputation method to each column
  • New Label Transform module: transform categorical label values
  • Homo-LR, Homo-SecureBoost, Homo-NN now can convert models into sklearn、lightgbm、torch & tf-keras framework
  • Hetero Feature Binning supports multi-class task, higher efficiency with label packing
  • Hetero Feature Selection support multi-class iv filter
  • Secure Information Retrieval supports multi-column retrieval
  • Major training algorithms support warm-start and checkpoint : Homo & Hetero LR, Homo & Hetero-SecureBoost, Homo & Hetero NN
  • Optimized Pailler addition operation, several times faster, Hetero-SecureBoost with Paillier speed up 2x+

Fate-Client

  • Pipeline supports uploading match id functionality
  • Pipeline supports homo model conversion
  • Pipeline supports model push to FATE-Serving
  • Pipeline supports running jobs with specified FATE version

FATE-Test

  • Integrate FederatedML unittest
  • Support for uploading image data
  • Big data generation using storage interface, optimized generation logic
  • Support for historical data comparison
  • cache_deps and model_loader_deps support
  • Run DSL Testsuite with specified FATE version


Major Features and Improvements in FATE-Flow 1.7.0

  • Independent repository instead of all code in the main FATE repository
  • Component registry, which can hot load many different versions of component packages at the same time
  • Hot update of component parameters, component-specific reruns, automatic reruns
  • Model Checkpoint to support task hot start, model deployment and other
  • Data, Model and Cache can be reused between jobs
  • Reader component supports more data sources, such as MySQL, Hive
  • Realtime recording of dataset usage derivation routes
  • Multi-party permission control for datasets
  • Automatic push to reliable storage when model deployment, support Tencent Cloud COS, MySQL, Redis
  • REST API authentication

Major Features and Improvements in KubeFATE 1.7.0


KubeFATE v1.7.0 supports FATE v1.7.0 and brings the following updates:

  1. Update the examples come with docker-compose from DSL v1 to DSL v2;
  2. Update KubeFATE and FATE's Ingress deployment to support Kubernetes v1.22;
  3. Update all examples of the Jupyter Notebook;
  4. Fix some bugs.

The deliverables include:

  1. The Docker Compose v1.7.0;
  2. The FATE v1.7.0 chart;
  3. The Exchange v1.7.0 chart.
  4. The KubeFATE CLI and KubeFATE server image v1.4.2;

Note: The KubeFATE CLI and KubeFATE service do not need to upgrade. Import new charts following the guide: Manage charts in KubeFATE manually

Cheers!