Log Request/Response Payloads in Azure API Management

Ilias el Matani

Introduction

Errors are inevitable in the realm of software development. When our internal application started receiving HTTP 502 error codes while interacting with Azure API Management Services, I was thrust into a scenario that required me to dig deeper and understand the root cause of the errors. The challenge was that all traffic was encrypted (HTTPS - SSL/TLS), which meant we had to find a way to capture both the request and response data. The solution was to use request and response payload logging in Application Insights for Azure API Management Services. This blog post will guide you through the process of how to log the request and response payload in Azure API Management.

Prerequisites

Before we begin, please ensure you have the following prerequisites:

  • An Azure account
  • A Log Analytics Workspace environment set up
  • An Application Insights instance connected to your Log Analytics Workspace

If you haven't set these up yet, refer to the Microsoft Azure documentation for guides on setting up an Azure account, Log Analytics Workspace, and Application Insights.

Steps to Enable Payload Logging

Once you have the prerequisites sorted out, follow these steps:

  1. Go to the Azure portal and navigate to your Azure API Management Instance.
  2. Click on your API and then scroll down to Monitoring.
  3. Under Monitoring, you will find Application Insights. Click on the '+ Add' button.
  4. Select your Application Insights instance from the dropdown menu. Uncheck the options "Use as default" and "Enable basic logging for all APIs", and also uncheck "Add availability monitoring".
  5. After adding Application Insights, click on APIs under the APIs menu and choose the API for which you want to add logging.
  6. Click on the operation for which you want to add logging.
  7. Scroll down to Diagnostics logs and check Enable.
  8. Select the Application Insights instance as the destination.
  9. At the bottom of the page, click on Advanced Options.
  10. Check to enable Frontend Request, Frontend Response, Backend Request, and Backend response.
  11. For each of these, under Headers to log, add the value X-Forwarded-For and under Number of payload bytes to log (up to 8192), specify a value up to 8192 bytes.
  12. Click save.

Steps to Query Payload Logs in Application Insights

  1. Navigate to the Application Insights
  2. Under Monitoring click on Logs.
  3. To view the last 10 results, run the following query:
requests 
| where url == "https://exampleapi.azure-api.net/api/resource"
| where timestamp >= ago(1h)
| order by timestamp desc
| take 10