Skip to content

Federated LoRA Fine-Tuning

Plato now provides first-class support for parameter-efficient LoRA fine-tuning of HuggingFace causal language models.

Quick start

The reference configuration configs/HuggingFace/fedavg_opt_lora.toml launches a single-round federated run with one client using gradient accumulation to fit facebook/opt-125m on the srivatsavaasista/textgenerator-ds-mini dataset.

uv run python plato.py --config configs/HuggingFace/fedavg_opt_lora.toml

Key configuration points:

  • data.datasource: LoRA enables the LoRA-aware HuggingFace datasource and lets you set data.text_field, data.max_length, and dataset splits.
  • algorithm.type: fedavg_lora exchanges only LoRA adapter weights between clients and the server.
  • parameters.lora configures rank, target modules, dropout, and related hyperparameters consumed by peft.
  • trainer.gradient_accumulation_steps keeps GPU memory usage manageable when fine-tuning larger models.

Customising your run

  • Datasets – Change data.dataset_name or provide data.dataset_config and data.text_field for other HuggingFace corpora.
  • Models – Update trainer.model_name to any HuggingFace causal language model. When parameters.lora is present the model registry wraps the network with a LoRA adapter automatically.
  • Optimisers – Tune parameters.optimizer or swap to an alternative optimiser strategy if needed.

Because the LoRA integration lives inside the standard trainer, you can still combine callbacks, composable strategies, or alternative aggregation rules exactly as with other Plato workloads.