Date   

Please help solve a problem problem about DataTransform

wforever5620@...
 

please help to answer a question: I use 'MD5' to desensitize the 'id' field of the data, the generated data is 32 bits, and then after the 'DataTransform' component processing, I found that the id and the original data id is not the same.


Re: FATE standalone

Tianjian Chen
 

For Windows 10 users who want to try a linux docker image, there is a work-around scheme.

1. Make sure you have wsl2 installed and activated ( yeah, even if wsl2 was installed, it is not a default option ) 
2. Install latest docker desktop, and designate the docker backend  to wsl2. Docker Desktop WSL 2 backend | Docker Documentation
3. Enjoy your linux docker image on Windows 10

//Bests
Tianjian


Re: FATE standalone

Haining Zhang
 

hi Mojgan,

FATE currently only supports Linux because some AI/ML libraries depend on Linux to work. 

Henry

On 2022/7/18 7:36 PM, mojgan.hashemian@... wrote:

Hi Henry,

Thanks for your prompt response. 

I missed that, sorry. Do you have any similar guide for Windows users? Does FATE work at all on Windows OS? 

Many thanks.


Re: FATE standalone

mojgan1987@...
 

Hi Henry,

Thanks for your prompt response. 

I missed that, sorry. Do you have any similar guide for Windows users? Does FATE work at all on Windows OS? 

Many thanks.


Re: FATE standalone

Haining Zhang
 

hi Mojgan,

Thanks for your question.

It the document, it states that the supported OS is

CentOS Linux

It does not run on Windows.

Henry


On 2022/7/18 5:55 PM, mojgan.hashemian@... wrote:

Hello.

I am following this guide. After running "bash init.sh init" at step 3.3 I get this error: "Not support this kernel."

I have downloaded version 1.8.0 and running on Windows 10.

You help is much appreciated. 

Kind regards


FATE standalone

mojgan1987@...
 

Hello.

I am following this guide. After running "bash init.sh init" at step 3.3 I get this error: "Not support this kernel."

I have downloaded version 1.8.0 and running on Windows 10.

You help is much appreciated. 

Kind regards


Re: Federated Learning on iOS

Agnik Banerjee
 

Hi Tianjian,

I checked with the OpenMinded team but unfortunately SwiftSyft hasn’t had support in over a year and there are no plans to return to the project. Thanks for the suggestion though!

Sincerely,

Agnik Banerjee


On Wed, Jun 29, 2022 at 11:11 PM Tianjian Chen <chentianjian@...> wrote:

On Thu, Jun 30, 2022 at 10:33 AM <agnik.banrjee@...> wrote:
Hi everyone, I'm Agnik, an AI intern at the Wall Lab in Stanford University's School of Medicine. Our team is attempting to implement federated learning on iOS. Currently, we have one professional iOS app developer leading this process. If you would like to help us out/have any advice please reach out to me. Although we were intending to use this system for a private research project, we can first work on an open-source demo so it is available to everyone. Any help would be amazing! 


Re: Federated Learning on iOS

Tianjian Chen
 

On Thu, Jun 30, 2022 at 10:33 AM <agnik.banrjee@...> wrote:
Hi everyone, I'm Agnik, an AI intern at the Wall Lab in Stanford University's School of Medicine. Our team is attempting to implement federated learning on iOS. Currently, we have one professional iOS app developer leading this process. If you would like to help us out/have any advice please reach out to me. Although we were intending to use this system for a private research project, we can first work on an open-source demo so it is available to everyone. Any help would be amazing! 


Federated Learning on iOS

Agnik Banerjee
 

Hi everyone, I'm Agnik, an AI intern at the Wall Lab in Stanford University's School of Medicine. Our team is attempting to implement federated learning on iOS. Currently, we have one professional iOS app developer leading this process. If you would like to help us out/have any advice please reach out to me. Although we were intending to use this system for a private research project, we can first work on an open-source demo so it is available to everyone. Any help would be amazing! 


Re: ERROR: Unauthorized variable: host_uuid

Keye Zhou
 

The problem is that the configuration protocols have been updated. The model used early configuration version, and it is kind of hard for me to modify them.


Re: ERROR: Unauthorized variable: host_uuid

Keye Zhou
 

The problem is that the configuration protocols have been updated. The model used early configuration version, and it is kind of hard for me to modify them.

Haining Zhang <hi@...> 于2022年5月17日周二 10:33写道:

The FATE 1.2.0 is a very early version and it is no longer supported. Could you upgrade to the latest version (v1.8) to have a try?

Henry


On 2022/5/17 1:09 AM, Zhoukeye11@... wrote:
I'm using Fate1.2.0 and deployed a model https://github.com/chengziqiang/FL_DrugDiscovery. When training the given model there is an error. Logs are shown below: 
  • ValueError: Unauthorized variable: host_uuid
  • 160 "2022-05-16 16:53:14,549 - task_executor.py[line:132] - ERROR: Unauthorized variable: host_uuid"
  • 161 Traceback (most recent call last):
  • 162 File "/fate/fate_flow/driver/task_executor.py", line 119, in run_task
  • 163 run_object.run(parameters, task_run_args)
  • 164 File "/fate/federatedml/model_base.py", line 75, in run
  • 165 self._init_runtime_parameters(component_parameters)
  • 166 File "/fate/federatedml/model_base.py", line 47, in _init_runtime_parameters
  • 167 self._init_model(param)
  • 168 File "/fate/federatedml/nn/ecfp_nn/enter_point.py", line 127, in _init_model
  • 169 super(ECFPNNClient, self)._init_model(param)
  • 170 File "/fate/federatedml/nn/ecfp_nn/enter_point.py", line 64, in _init_model
  • 171 enable_secure_aggregate=secure_aggregate)
  • 172 File "/fate/federatedml/framework/homo/procedure/aggregator.py", line 191, in with_role
  • 173 return Host().register_aggregator(transfer_variable, enable_secure_aggregate)
  • 174 File "/fate/federatedml/framework/homo/procedure/aggregator.py", line 170, in register_aggregator
  • 175 transfer_variables).create_cipher()
  • 176 File "/fate/federatedml/framework/homo/procedure/random_padding_cipher.py", line 60, in create_cipher
  • 177 uuid = self.generate_uuid()
  • 178 File "/fate/federatedml/framework/homo/sync/identify_uuid_sync.py", line 60, in generate_uuid
  • 179 self._uuid_transfer_variable.remote(obj=_uid, role=consts.ARBITER, idx=0, suffix=ind)
  • 180 File "/fate/federatedml/transfer_variable/base_transfer_variable.py", line 117, in remote
  • 181 return self.remote_parties(obj=obj, parties=parties, suffix=suffix)
  • 182 File "/fate/federatedml/transfer_variable/base_transfer_variable.py", line 72, in remote_parties
  • 183 parties=parties)
  • 184 File "/fate/arch/api/transfer/standalone.py", line 59, in remote
  • 185 self._remote_side_auth(name=name, parties=parties)
  • 186 File "/fate/arch/api/transfer/__init__.py", line 271, in _remote_side_auth
  • 187 if self._role not in self._authorize.authorized_src_roles(variable_name=name):
  • 188 File "/fate/arch/api/transfer/__init__.py", line 209, in authorized_src_roles
  • 189 self._update_auth(variable_name)
  • 190 File "/fate/arch/api/transfer/__init__.py", line 199, in _update_auth
  • 191 raise ValueError(f"Unauthorized variable: {v_name}")


Re: ERROR: Unauthorized variable: host_uuid

Haining Zhang
 

The FATE 1.2.0 is a very early version and it is no longer supported. Could you upgrade to the latest version (v1.8) to have a try?

Henry


On 2022/5/17 1:09 AM, Zhoukeye11@... wrote:

I'm using Fate1.2.0 and deployed a model https://github.com/chengziqiang/FL_DrugDiscovery. When training the given model there is an error. Logs are shown below: 
  • ValueError: Unauthorized variable: host_uuid
  • 160 "2022-05-16 16:53:14,549 - task_executor.py[line:132] - ERROR: Unauthorized variable: host_uuid"
  • 161 Traceback (most recent call last):
  • 162 File "/fate/fate_flow/driver/task_executor.py", line 119, in run_task
  • 163 run_object.run(parameters, task_run_args)
  • 164 File "/fate/federatedml/model_base.py", line 75, in run
  • 165 self._init_runtime_parameters(component_parameters)
  • 166 File "/fate/federatedml/model_base.py", line 47, in _init_runtime_parameters
  • 167 self._init_model(param)
  • 168 File "/fate/federatedml/nn/ecfp_nn/enter_point.py", line 127, in _init_model
  • 169 super(ECFPNNClient, self)._init_model(param)
  • 170 File "/fate/federatedml/nn/ecfp_nn/enter_point.py", line 64, in _init_model
  • 171 enable_secure_aggregate=secure_aggregate)
  • 172 File "/fate/federatedml/framework/homo/procedure/aggregator.py", line 191, in with_role
  • 173 return Host().register_aggregator(transfer_variable, enable_secure_aggregate)
  • 174 File "/fate/federatedml/framework/homo/procedure/aggregator.py", line 170, in register_aggregator
  • 175 transfer_variables).create_cipher()
  • 176 File "/fate/federatedml/framework/homo/procedure/random_padding_cipher.py", line 60, in create_cipher
  • 177 uuid = self.generate_uuid()
  • 178 File "/fate/federatedml/framework/homo/sync/identify_uuid_sync.py", line 60, in generate_uuid
  • 179 self._uuid_transfer_variable.remote(obj=_uid, role=consts.ARBITER, idx=0, suffix=ind)
  • 180 File "/fate/federatedml/transfer_variable/base_transfer_variable.py", line 117, in remote
  • 181 return self.remote_parties(obj=obj, parties=parties, suffix=suffix)
  • 182 File "/fate/federatedml/transfer_variable/base_transfer_variable.py", line 72, in remote_parties
  • 183 parties=parties)
  • 184 File "/fate/arch/api/transfer/standalone.py", line 59, in remote
  • 185 self._remote_side_auth(name=name, parties=parties)
  • 186 File "/fate/arch/api/transfer/__init__.py", line 271, in _remote_side_auth
  • 187 if self._role not in self._authorize.authorized_src_roles(variable_name=name):
  • 188 File "/fate/arch/api/transfer/__init__.py", line 209, in authorized_src_roles
  • 189 self._update_auth(variable_name)
  • 190 File "/fate/arch/api/transfer/__init__.py", line 199, in _update_auth
  • 191 raise ValueError(f"Unauthorized variable: {v_name}")


ERROR: Unauthorized variable: host_uuid

Keye Zhou
 

I'm using Fate1.2.0 and deployed a model https://github.com/chengziqiang/FL_DrugDiscovery. When training the given model there is an error. Logs are shown below: 
  • ValueError: Unauthorized variable: host_uuid
  • 160 "2022-05-16 16:53:14,549 - task_executor.py[line:132] - ERROR: Unauthorized variable: host_uuid"
  • 161 Traceback (most recent call last):
  • 162 File "/fate/fate_flow/driver/task_executor.py", line 119, in run_task
  • 163 run_object.run(parameters, task_run_args)
  • 164 File "/fate/federatedml/model_base.py", line 75, in run
  • 165 self._init_runtime_parameters(component_parameters)
  • 166 File "/fate/federatedml/model_base.py", line 47, in _init_runtime_parameters
  • 167 self._init_model(param)
  • 168 File "/fate/federatedml/nn/ecfp_nn/enter_point.py", line 127, in _init_model
  • 169 super(ECFPNNClient, self)._init_model(param)
  • 170 File "/fate/federatedml/nn/ecfp_nn/enter_point.py", line 64, in _init_model
  • 171 enable_secure_aggregate=secure_aggregate)
  • 172 File "/fate/federatedml/framework/homo/procedure/aggregator.py", line 191, in with_role
  • 173 return Host().register_aggregator(transfer_variable, enable_secure_aggregate)
  • 174 File "/fate/federatedml/framework/homo/procedure/aggregator.py", line 170, in register_aggregator
  • 175 transfer_variables).create_cipher()
  • 176 File "/fate/federatedml/framework/homo/procedure/random_padding_cipher.py", line 60, in create_cipher
  • 177 uuid = self.generate_uuid()
  • 178 File "/fate/federatedml/framework/homo/sync/identify_uuid_sync.py", line 60, in generate_uuid
  • 179 self._uuid_transfer_variable.remote(obj=_uid, role=consts.ARBITER, idx=0, suffix=ind)
  • 180 File "/fate/federatedml/transfer_variable/base_transfer_variable.py", line 117, in remote
  • 181 return self.remote_parties(obj=obj, parties=parties, suffix=suffix)
  • 182 File "/fate/federatedml/transfer_variable/base_transfer_variable.py", line 72, in remote_parties
  • 183 parties=parties)
  • 184 File "/fate/arch/api/transfer/standalone.py", line 59, in remote
  • 185 self._remote_side_auth(name=name, parties=parties)
  • 186 File "/fate/arch/api/transfer/__init__.py", line 271, in _remote_side_auth
  • 187 if self._role not in self._authorize.authorized_src_roles(variable_name=name):
  • 188 File "/fate/arch/api/transfer/__init__.py", line 209, in authorized_src_roles
  • 189 self._update_auth(variable_name)
  • 190 File "/fate/arch/api/transfer/__init__.py", line 199, in _update_auth
  • 191 raise ValueError(f"Unauthorized variable: {v_name}")


FATE & KubeFATE v1.8.0 released! Huge performance improvements, security enhancements, updated algorithms, optimized scheduling and testing frameworks, better high availability support and MORE!

wfangchi@...
 

Hi,

We are super excited to announce the new major release of FATE and KubeFATE - v1.8.0. After months of hard work, we have added numerous new features and improvements in this release, including performance improvements in many commonly-used components, added some most wanted features in a number of algorithms, further enhancement to the deployment and operation scenarios using KubeFATE and more. It is time to upgrade your FATE to the latest version!

Release Pages:
FATE 1.8.0: https://github.com/FederatedAI/FATE/releases/tag/v1.8.0
FATE-Flow 1.8.0: https://github.com/FederatedAI/FATE-Flow/releases/tag/v1.8.0
KubeFATE 1.8.0: https://github.com/FederatedAI/KubeFATE/releases/tag/v1.8.0

Major Features and Improvements in FATE v1.8.0

FederatedML

  • Add non-coordinated-version Hetero Linear Regression, based on integrated Hetero GLM framework, with mixed protocol of HE and SPDZ
  • Homo LR support one-vs-rest
  • Add SecureBoost-MO algorithm to speed up multi-class classification of Hetero & Homo SecureBoost, 1.5x-5x faster
  • Optimize Hetero SecureBoost Predict Transmission Data Size,reduce 75% bandwidth consumption if tree's max depth is small
  • Speed up DH Intersection implementation, 30%+ faster
  • Optimized Quantile Binning gk-summary structure & split point query,20%+ faster, less memory cost
  • Support weighted training in non-coordinated Hetero Logistic Regression & Linear Regression
  • Merge Hetero FastSecureBoost into Hetero SecureBoost as a boosting strategy option

Fate-ARCH

  • Adjustable task_cores for standalone FATE
  • Enable Eggroll option to make computing output "IN_MEMORY" by default

Fate-Test

  • Include Paillier encryption performance evaluation
  • Include SPDZ performance evaluation
  • Optimized testsuite printout
  • Include examples data upload and mnist download
  • Provide pipeline to dsl convert tools

Bug-Fix

  • Fix bug for SPDZ when using default q_filed
  • Fix multiple get problem of SPDZ
  • Fix bugs of recursive-query homo feature binning
  • Fix homo_nn's model aggregation problem
  • Fix bug for hetero feature selection when using federated filter but some party's feature is empty.

Major Features and Improvements in FATE-Flow v1.8.0

  • Optimize the model migration function to reduce user operation steps;
  • Add version compatibility check in component center to support multiple parties to use different versions;
  • Add data table disable/enable function, and support batch delete disable table

Major Features and Improvements in KubeFATE v1.8.0

KubeFATE v1.8.0 supports FATE v1.8.0 and brings the following updates:

  • Support specify ingress class names #523 ;
  • KubeFATE support liveness and readiness #552 ;
  • Adjust the file structure of the Chart to facilitate the addition of new module #526 ;
  • Solve the warning of potential security vulnerabilities in Github pages #519 ;
  • and other bug fixes #501

For the feature list details, please refer to: KubeFATE v1.8.0 Feature List


Cheers!

 


Re: My test hetero lr job has stopped 28 percent, but no erros appeared.

David Bauer
 

Your job is stalled on the FateFlow server in the intersection stage.  For this stage it is critical that your id fields in the data on each federate are logically correct, but it looks like you are using an ootb example.  Did you modify the model?  Can you check the job logs on the FateFlow server side?  Specifically the debug logs for the intersection stage, and the fateflow_*.log at the top level of the log directory. 

David


On Fri, Mar 18, 2022, 11:57 PM BearDream <laxzhang@...> wrote:

The lr job is canceled, but the statistic job also is. The log is below.


This is cluster job log.




This is host job log.

No any errors...


Re: My test hetero lr job has stopped 28 percent, but no erros appeared.

BearDream
 

The lr job is canceled, but the statistic job also is. The log is below.


This is cluster job log.




This is host job log.

No any errors...


Re: My test hetero lr job has stopped 28 percent, but no erros appeared.

dylanfan(范涛)
 

This doesn't look normal, please the error logs


My test hetero lr job has stopped 28 percent, but no erros appeared.

BearDream
 

Task executor: Host, guest, and ARBITER.
host and guest are on two machines respectively.

This is the command of job submission.
flow job submit -c test_hetero_lr_job_conf.json -d test_hetero_lr_job_dsl.json

Running 13 hours. May I know if that's normal?




Can someone tell me how secure_information_retrieval defines the method and operation of retrieving data?

BearDream
 

Everyone pays attention to the use of the FATE framework to achieve federated deep learning, but few people talk about the usage of federated statistics. Can a small partner share it?


FileNotFoundError: [Errno 2] No such file or directory: '/proc/59518/stat'

bingquan.hbq@...
 

I ran into this error and got no idea, who can help me out? I am using Kubefate 1.7.0 and using fate_test 1.6.1 to do the homo_lr benchmarking test: 
fate_flow_schedule_error.log:1:[ERROR] [2022-03-10 05:49:01,087] [202203100548509151460] [22:140305212581632] - [federated_scheduler.task_command] [line:246]: failed to execute federated task reader_0 command(status/running) detail: 
fate_flow_schedule.log:74:[WARNING] [2022-03-10 05:49:00,994] [202203100548509151460] [22:140305212581632] - [federated_scheduler.federated_command] [line:273]: an error occurred while /party/202203100548509151460/reader_0/202203100548509151460_reader_0/0/guest/10000/status/running the job to role guest party 10000: 
fate_flow_schedule.log:81:[ERROR] [2022-03-10 05:49:01,087] [202203100548509151460] [22:140305212581632] - [federated_scheduler.task_command] [line:246]: failed to execute federated task reader_0 command(status/running) detail: 
fate_flow_schedule.log:207:AttributeError: _cache
fate_flow_schedule.log:220:FileNotFoundError: [Errno 2] No such file or directory: '/proc/59518/stat'
fate_flow_schedule.log:370:AttributeError: _cache
fate_flow_schedule.log:383:FileNotFoundError: [Errno 2] No such file or directory: '/proc/59518/stat'
fate_flow_schedule.log:535:AttributeError: _cache
fate_flow_schedule.log:548:FileNotFoundError: [Errno 2] No such file or directory: '/proc/59518/stat'
fate_flow_schedule.log:639:[WARNING] [2022-03-10 05:49:26,140] [202203100548509151460] [22:140305212581632] - [federated_scheduler.federated_command] [line:273]: an error occurred while /party/202203100548509151460/guest/10000/status/failed the job to role guest party 10000: 
fate_flow_schedule.log:643:[WARNING] [2022-03-10 05:49:26,171] [202203100548509151460] [22:140305212581632] - [federated_scheduler.federated_command] [line:273]: an error occurred while /party/202203100548509151460/host/9999/status/failed the job to role host party 9999: 
fate_flow_schedule.log:647:[WARNING] [2022-03-10 05:49:26,202] [202203100548509151460] [22:140305212581632] - [federated_scheduler.federated_command] [line:273]: an error occurred while /party/202203100548509151460/arbiter/9999/status/failed the job to role arbiter party 9999: 

1 - 20 of 84