Deployment¶

Docker Installation¶

The ICOS Intelligence Layer can be deployed using Docker containers. While the core installation remains consistent with deliverable D4.1, this version introduces new environment configurations to customize API timeout and resource allocation.

Download the Docker image from this link and run the container using the command below:

docker run --network host -it --rm -p 3000:3000 -p 5000:5000 \
-e BENTOML_CONFIG_OPTIONS='api_server.traffic.timeout=600 runners.resources.cpu=0.5 runners.resources."nvidia.com/gpu"=0' \
docker_name:latest serve

This command launches the ICOS Intelligence Layer Coordination API server, accessible via http://0.0.0.0:3000.

Trustworthy AI Module – Deployment¶

Explainable AI¶

• Deployment: Packaged inside the Intelligence Layer Docker container.
• Access: Trigger using the start_mlflow_ui() API endpoint (runs on port 5000).
• URL: http://127.0.0.1:5000
• Dependencies: MLFlow 2.13.2, Python 3.10, BentoML 1.2.16

Prediction Confidence Scores¶

• Tech Stack: Python 3.10, Scikit-learn
• Deployment: Returned as part of model prediction responses.

Model Monitoring (Drift Detection)¶

• API Endpoint: POST /detect_drift
• Description: Activates drift detection using Jensen-Shannon Divergence.
• Input: Boolean flag (true to enable, false to skip).
• Output: Structured report detailing drift metrics and anomalies.

Federated Learning¶

• Tech Stack: Python 3.10, Flower FL, PyTorch
• Model: LSTM for time-series metrics (CPU, memory, power)
• Pipeline: Scaphandre → Prometheus → OpenTelemetry → dataClay
• Export Format: BentoML
• Deployment: Via Docker Compose, supporting multi-service orchestration
• Retraining: Supports periodic or on-demand loops based on data availability

AI Analytics Module – Deployment¶

AIOps with MLFlow¶

• Access: Launch via the start_mlflow_ui() API endpoint (port 5000)
• URL: http://localhost:5000
• Dependencies: MLFlow 2.13.2, Python 3.10, BentoML 1.2.16

Model Compression¶

Compression options can be configured during model training by submitting a request to /core/analytics_train with the parameters below:

{
  "pytorch_model_parameters": {
    "hidden_size": 64,
    "num_epochs": 50,
    "quantize": true,
    "distill": false
  }
}

Supported configurations include: - Quantization: Dynamic quantization applied post-training using PyTorch - Distillation: Trains a teacher-student model pair with a smaller student model - Combined: Applies distillation followed by quantization to produce an INT8-optimized model

Trained models (INT8 or FP32) are stored in the Intelligence Layer model registry.

System Requirements¶

The following packages and versions are required for successful deployment of the ICOS Intelligence Layer container: