Need help with AzurePublicDataset?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

Azure
385 Stars 73 Forks Other 63 Commits 5 Opened issues

Description

Microsoft Azure Traces

Services available

!
?

Need anything else?

Contributors list

Overview

This repository contains public releases of Microsoft Azure traces for the benefit of the research and academic community. There are currently two classes of traces:

  • VM Traces: two representative traces of the virtual machine (VM) workload of Microsoft Azure collected in 2017 and 2019, and one VM request trace specifically for investigating packing algorihtms.
  • Azure Functions Traces: representative traces of Azure Functions invocations, collected over two weeks in 2019, and of Azure Functions blob accesses, collected between November and December of 2020.

We provide the traces as they are, but are willing to help researchers understand and use them. So, please let us know of any issues or questions by sending email to our mailing list.

Quick links by paper:

  • Traces (2017)(2019) for the paper "Resource Central: Understanding and Predicting Workloads for Improved Resource Management in Large Cloud Platforms" (SOSP'17)
  • Traces (2019) for the paper "Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider" (ATC'19)
  • Traces (2020) for the paper "Protean: VM Allocation Service at Scale" (OSDI'20)
  • Traces (2020) for the paper "Faa$T: A Transparent Auto-Scaling Cache for Serverless Applications" (SoCC'21)

VM Traces

The traces are sanitized subsets of the first-party VM workload in one of Azure’s geographical regions. We include jupyter notebooks that directly compare the main characteristics of each trace to its corresponding full VM workload, showing that they are qualitatively very similar (except for VM deployment sizes in 2019). Comparing the characteristics of the two traces illustrates how the workload has changed over this two-year span.

If you do use either of these VM traces in your research, please make sure to cite our SOSP’17 paper "Resource Central: Understanding and Predicting Workloads for Improved Resource Management in Large Cloud Platforms", which includes a full analysis of the Azure VM workload in 2017.

Trace Locations

  • AzurePublicDatasetV1 - Trace created using data from 2017 Azure VM workload containing information about ~2M VMs and 1.2B utilization readings.
  • AzurePublicDatasetV2 - Trace created using data from 2019 Azure VM workload containing information about ~2.6M VMs and 1.9B utilization readings.

Azure Traces for Packing

  • AzureTracesForPacking2020 - This dataset represents part of the workload on Microsoft's Azure Compute and is specifically intended to evaluate packing algorithms. The dataset includes:

    • VM requests along with their priority
    • The lifetime for each requested VM
    • The (normalized) resources allocated for each VM type.

If you do use the Azure Trace for Packing in your research, please make sure to cite our OSDI'20 paper "Protean: VM Allocation Service at Scale", which includes a description of the Azure allocator and related workload analysis.

Azure Functions Traces

Function Invocations

  • AzureFunctionsDataset2019 - These traces contain, for a subset of applications running on Azure Functions in July of 2019:

    • how many times per minute each (anonymized) function is invoked and its corresponding trigger group
    • how (anonymized) functions are grouped into (anonymized) applications, and how applications are grouped by (anonymized) owner
    • the distribution of execution times per function
    • the distribution of memory usage per application

If you do use the Azure Functions traces in your research, please make sure to cite our ATC'20 paper "Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider", which includes a full analysis of the Azure Functions workload in July 2019.

Functions Blob Accesses

  • AzureFunctionsBlobDataset2020 - This is a sample of the blob accesses in Microsoft's Azure Functions, collected between November 23rd and December 6th 2020. This dataset is the data described and analyzed in the SoCC 2021 paper 'Faa$T: A Transparent Auto-Scaling Cache for Serverless Applications'.

Contact us

Please let us know of any issues or questions by sending email to our mailing list.

These traces derive from a collaboration between Azure and Microsoft Research.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.