1. Enable Verbose Debugging
Before digging deeper, ensure Trufeds exposes all internal states (like pre-aggregation tensor shapes, worker heartbeats, and serialization overhead). Add this to the very top of your entry script:
import trufeds
# Set log level to DEBUG for both Orchestrator and Workers
trufeds.set_log_level(trufeds.logging.DEBUG)
# Automatically export state JSONs and stack traces on crash
trufeds.enable_crash_dumps(path="./trufeds_dumps", include_weights=False)2. Version & Compatibility Matrix
Most aggregation failures stem from backend mismatches between the Server and Client nodes. Check your stack against our supported targets.
| Trufeds Version | Hardware / CUDA | PyTorch | Notes |
|---|---|---|---|
| v2.1.x (Latest) | CUDA 11.8 - 12.x | 2.0+ | Full GPU tensor aggregation. Requires safetensors. |
| v2.0.x | CUDA 11.7+ / CPU | 1.13 - 2.0 | Introduced hybrid CPU/GPU offloading. |
| v1.5.x (Legacy) | CPU Only | 1.10 - 1.12 | Deprecated (Uses Pickle) |
Common Error Codes (FAQ)
Symptom: The orchestrator waits for client updates, but nodes throw a TimeoutError during the initial gRPC handshake.
Solution:
This is usually caused by unexposed Docker ports or restrictive firewalls. Ensure TRUFEDS_PORT (Default: 8080) is open.
If clients are loading massive local datasets, the handshake might legitimately time out. Increase the timeout in your config:
server_config = trufeds.Config(client_timeout_sec=600)Symptom: RuntimeError: Gradient shapes do not match global model during the FedAvg phase.
Solution:
A client is training a model with a different architecture (e.g., mismatched linear layer sizes due to different local class counts) than the server expects.
Run trufeds.verify_model_sync(server_model, client_model) before kicking off training to catch hash discrepancies.
Symptom: UnpicklingError: invalid load key or SafetensorError.
Solution:
Since Trufeds 2.0, we default to safetensors for zero-copy deserialization and security. If you have legacy nodes running Trufeds 1.x, they are sending pickle payloads.
Recommendation: Upgrade all nodes to 2.x. If impossible, force legacy mode on the server:
trufeds.set_backend(trufeds.backends.PICKLE_LEGACY) # Not recommended!Symptom: Workers crash after exactly N communication rounds with CUDA OutOfMemory.
Solution: Trufeds keeps previous model snapshots in VRAM for fast differential rollbacks by default. If your edge devices have limited VRAM, change your Worker Policy to swap to RAM:
worker_policy = trufeds.WorkerPolicy(keep_snapshots_in_vram=False)Direct Developer Support
Completely stuck? As the core maintainer of Trufeds, I'm happy to help you out directly. Please generate a system dump before opening a ticket.
Before you ask
- Are Server and Client on the exact same Trufeds version?
- If using CUDA, is PyTorch compiled with the matching NVCC version?
-
Did you run
python -m trufeds.doctorin your terminal?
Handy CLI Commands
Check network topology:
trufeds-cli ping --all
Force purge cache (Fixes stale weights):
trufeds-cli cache clear