Azure AutoML Complete Reference Cheatsheet

Introduction: Automated Machine Learning in Azure

Azure Automated Machine Learning (AutoML) is a capability that automates the time-consuming, iterative tasks of machine learning model development. It enables data scientists, analysts, and developers to build high-quality models with greater efficiency while ensuring model explainability. AutoML democratizes the machine learning process with an easy-to-understand interface that simplifies experimenting with different algorithms, hyperparameters, and feature engineering techniques.

AutoML Capabilities & Supported ML Tasks

Key Components & Interfaces

Azure ML Studio (UI)

• No-code interface for creating AutoML experiments
• Built-in data exploration and visualization
• Experiment monitoring and results analysis
• Model explanation and interpretation tools
• One-click model deployment options

Python SDK

• Programmatic interface for AutoML
• More granular control over experiment settings
• Integration with existing ML workflows
• Two versions available:
  • SDK v1 (legacy): azureml-train-automl
  • SDK v2 (current): azure-ai-ml

Azure CLI with ML Extension

• Command-line interface for AutoML operations
• Suitable for automation and CI/CD pipelines
• Consistent interface across different environments

AutoML Workflow: Step-by-Step Process

1. Data Preparation

Requirements:

• Tabular data (CSV, Parquet, Excel, etc.)
• Target column for supervised learning
• For time series: date/time column and time-dependent variables

Key Considerations:

• Minimum 100 rows (ideally 1000+)
• Balanced classes for classification tasks
• Handle missing values appropriately
• Consider data sampling for large datasets

2. Experiment Configuration

Key Settings:

3. Feature Engineering Options

Automated Features:

• Missing value imputation
• Encoding categorical variables
• Feature scaling and normalization
• Feature selection
• Feature extraction (PCA, etc.)

Time Series-Specific Features:

• Lag features
• Rolling window statistics
• Holiday detection
• Seasonal decomposition

4. Compute Resources Configuration

5. Monitoring & Managing Experiments

Key Monitoring Metrics:

• Run status and duration
• Model performance metrics
• Resource utilization
• Algorithm leaderboard
• Training progress

Management Actions:

• Cancel underperforming runs
• Clone successful experiments
• Compare multiple experiment results
• Enable early termination for inefficient runs

AutoML Configuration: Python SDK v2 Examples

Basic Classification Example

# Import necessary libraries
from azure.ai.ml import automl, Input
from azure.ai.ml.constants import AssetTypes
from azure.ai.ml.automl import (
    ClassificationPrimaryMetrics,
    ClassificationModels,
)

# Configure the AutoML job
classification_job = automl.classification(
    compute="aml-cluster",
    experiment_name="credit-card-fraud-detection",
    training_data=Input(type=AssetTypes.MLTABLE, path="./training-data"),
    target_column_name="Fraud",
    primary_metric=ClassificationPrimaryMetrics.ACCURACY,
    allowed_models=[
        ClassificationModels.LOGISTIC_REGRESSION,
        ClassificationModels.RANDOM_FOREST,
        ClassificationModels.XGBOOST
    ],
    enable_model_explainability=True,
    validation_data=Input(type=AssetTypes.MLTABLE, path="./validation-data"),
    max_trials=20,
    training_parameters={"n_cross_validations": 5},
    limits={"timeout_minutes": 120}
)

# Submit the job
returned_job = ml_client.jobs.create_or_update(classification_job)

Time Series Forecasting Example

# Import libraries
from azure.ai.ml import automl, Input
from azure.ai.ml.constants import AssetTypes
from azure.ai.ml.automl import (
    ForecastingPrimaryMetrics,
    ForecastingModels,
)

# Configure the forecasting job
forecasting_job = automl.forecasting(
    compute="aml-cluster",
    experiment_name="sales-forecasting",
    training_data=Input(type=AssetTypes.MLTABLE, path="./sales-data"),
    target_column_name="Sales",
    time_column_name="Date",
    time_series_id_column_names=["Store", "Product"],
    primary_metric=ForecastingPrimaryMetrics.NORMALIZED_ROOT_MEAN_SQUARED_ERROR,
    allowed_models=[
        ForecastingModels.PROPHET,
        ForecastingModels.ARIMA,
        ForecastingModels.EXPONENTIAL_SMOOTHING
    ],
    forecast_horizon=30,
    training_parameters={
        "time_series_settings": {
            "featured_lag": {"lag_length": 12},
            "rolling_window_settings": {"window_size": 3}
        }
    },
    limits={"timeout_minutes": 180}
)

# Submit the job
returned_job = ml_client.jobs.create_or_update(forecasting_job)

Model Evaluation & Interpretation

Key Evaluation Metrics by Task

Classification:

• Accuracy, precision, recall, F1-score, AUC
• Confusion matrix
• Precision-recall curve
• ROC curve

Regression:

• RMSE, MAE, R², Spearman correlation
• Predicted vs. true values plot
• Residual histogram

Forecasting:

• RMSE, MAPE, R², normalized RMSE
• Forecast vs. actual values
• Forecast with confidence intervals

Explainability Features

Global Explanations:

• Feature importance
• Permutation feature importance
• Partial dependence plots

Local Explanations:

• SHAP values for individual predictions
• “What-if” analysis
• Individual conditional expectation plots

Visualization Tools

Model Deployment & Operationalization

Deployment Options

Deployment Process

1. Register the model in Azure ML workspace
2. Create inference configuration:
   • Scoring script (entry script)
   • Environment configuration
   • Input/output schema
3. Configure deployment target
4. Deploy model as web service or package
5. Test the endpoint with sample data
6. Monitor performance and data drift

Example Deployment Code (SDK v2)

# Register the best model from the AutoML run
best_model = ml_client.models.create_or_update(
    Model(
        path=f"azureml://jobs/{returned_job.name}/outputs/artifacts/paths/model/",
        name="credit-card-fraud-model",
        description="Credit card fraud detection model from AutoML",
        type="mlflow_model"
    )
)

# Create an online endpoint
endpoint = ManagedOnlineEndpoint(
    name="fraud-detection-endpoint",
    description="Endpoint for credit card fraud detection",
    auth_mode="key"
)
endpoint = ml_client.online_endpoints.begin_create_or_update(endpoint).result()

# Deploy the model to the endpoint
deployment = ManagedOnlineDeployment(
    name="fraud-model-deployment",
    endpoint_name=endpoint.name,
    model=best_model.id,
    instance_type="Standard_DS3_v2",
    instance_count=1
)
deployment = ml_client.online_deployments.begin_create_or_update(deployment).result()

# Set the deployment as the default
endpoint.traffic = {"fraud-model-deployment": 100}
ml_client.online_endpoints.begin_create_or_update(endpoint).result()

Optimization & Best Practices

Performance Optimization

Cost Optimization

• Use low-priority VMs for non-critical workloads
• Implement auto-scaling (min nodes = 0)
• Set appropriate timeout for experiments
• Enable early termination policies
• Use smaller compute for initial experiments, scale up for final models
• Consider serverless compute options (preview)

MLOps Integration

Version Control:

• Dataset versioning
• Environment versioning
• Model versioning
• Experiment tracking

CI/CD Pipeline Integration:

• Automated data validation
• Model training pipelines
• Model evaluation and registration
• Deployment approval workflows
• A/B testing strategies

Monitoring:

• Model performance monitoring
• Data drift detection
• Endpoint health monitoring
• Resource utilization tracking

Common Issues & Troubleshooting

Feature Limitations & Considerations

• Time series forecasting requires adequate historical data (recommend 10× forecast horizon)
• Deep learning models require larger datasets (10,000+ rows recommended)
• Some models have maximum feature limitations (e.g., tree-based models)
• Computer vision and NLP tasks in AutoML have preview limitations
• Cross-region data transfer may impact performance
• Consider data privacy and compliance requirements
• Model deployment may require additional authentication and networking setup

Azure AutoML vs. Other Approaches

Resources for Further Learning

Official Documentation

• Azure AutoML Documentation
• AutoML SDK v2 Reference
• Best Practices for AutoML
• AutoML Samples GitHub Repository

Training & Workshops

• Microsoft Learn AutoML Modules
• DP-100 Exam: Designing and Implementing a Data Science Solution on Azure
• Azure Machine Learning Workshop

Community Resources

• Azure Machine Learning Community
• Azure ML GitHub Discussions
• Stack Overflow Azure ML Tag