Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

🏠 Back to Blog

Azure Express Route

Overview

  • Azure ExpressRoute lets you extend your on-premises networks into the Microsoft cloud over a private connection facilitated by a connectivity provider.
  • With ExpressRoute, you can establish connections to Microsoft cloud services, such as Microsoft Azure, Office 365, and Dynamics 365.
  • ExpressRoute connections do not go over the public Internet, and offer more reliability, faster speeds, lower latencies, and higher security than typical connections over the Internet.
  • ExpressRoute connections typically have redundant connectivity from the partner network into the Microsoft Edge

Benefits

  • Layer 3 connectivity between your on-premises network and the Microsoft Cloud through a connectivity provider.
  • Connectivity can be from an any-to-any (IPVPN) network, a point-to-point Ethernet network, or a virtual cross-connection through a connectivity provider at a co-location facility.
  • Connectivity to Microsoft cloud services across all regions in the geopolitical region.
  • Global connectivity to Microsoft services across all regions with the ExpressRoute premium add-on.
  • Built-in redundency in every peering location for high availability.

Private vs. Public Peering

Peering refers to the connection between two networks for traffic exchange.

  • Private peering allows remote networks to access Azure vNets and resources connected to those vNets, such as infrastructure and PaaS services.
  • Public peering allows remote networks to access Microsoft Cloud services such as Office 365 and Azure Platform services.

ExpressRoute Components

  • On-prem devices: Devices located physically within an organization’s premises

  • Customer Edge (CE) router: The on-premises router that connects to the service provider’s edge router.

  • Provider Edge (PE) devices CE Routers: These are devices used by providers to connect to the CE router.

  • Partner Edge devices facing Microsoft Edge routers: These are devices used by ExpressRoute service providers to connect to Microsoft Edge routers

  • Microsoft Edge Routers: These are redundant pairs of routers on the Microsoft side of the ExpressRoute connection.

  • ExpressRoue vNet Gateway: This service connects an ExpressRoute connection with an Azure vNet.

  • Azure vNet: A virtual network in Azure that can be connected to an ExpressRoute circuit.

    ExpressRoute Components

ExpressRoute Connectivity Models

When architecting an ExpressRoute connection, you can choose from two different connectivity models:

  • Provider Model: The provider model connects a remote network to Azure using a third-party provider. To establish this connection, we need to work with the provider to set up the connection. Depending on the service offering the ExpressRoute partner provides, we have up to 3 connectivity options that we can implement:

    • Cloud Exchange co-location

      • This involves moving our infrastructure into a data center where the ExpressRoute partner has a presence. We can then order virtual cross-connections to the Microsoft network. The cross-connect could be a layer 2 or layer 3 connection.

    • Point-to-point Ethernet connection

      • This involves working with an ISP that provides single-site layer 2 or layer 3 connectivity between the remote network and the Azure vNet. The key point with this option is that connectivity is for a single customer site.

    • Any-to-Any IPVPN connection

      • This option leverages ISP-provided MPLS connectivity to connect multiple customer sites with the Microsoft cloud network. This model is recommended for customers with existing MPLS connections.

  • ExpressRoute Direct Model:

    • This model allows a customer’s network to connect directly to Microsoft at peering locations strategically placed around the world, with a 10 Gbps or dual 100 Gbps connection.

    • This model supports active/active connectivity at scale

    • This model does not rely on a third party for ExpressRoute connectivity.

    • This model is good when very high bandwidth is required.

Route Advertisement

  • When Microsoft peering gets configured on your ExpressRoute circuit, the Microsoft Edge routers establish a pair of BGP sessions with your edge routers through your connectivity provider. No routes are advertised to your network by default. To enable router advertisements, you must associate a route filter.

ExpressRoute Circuit SKUs

  • ExpressRoute circuits are offered in three SKUs:

    • Local
      • Can be used to provide connectivity to vNets in one or two Azure regions in the same metro/geographical area.
      • Not all ExpressRoute locations support the ‘Local’ SKU
      • One benefit of the local SKU is there is no additional cost for transferring data out of Azure through the ExpressRoute connection (egress data).
    • Standard
      • Can provide connectivity to vNets and Azure services in Azure regions in a geopolitical area. For example, all regions in North America.
      • Egress data transfer is an added cost.
      • There are two billing models for egress data. Metered and Unlimited.
        • Metered requires that you estimate how much egress data you will use, and you only pay for that amount.
        • Unlimited allows you to use any amount of data but has a fixed monthly fee.
    • Premium
      • Can provide connectivity to vNets globally.
      • There are two billing models for egress data. Metered and Unlimited.
        • Metered requires that you estimate how much egress data you will use, and you only pay for that amount.
        • Unlimited allows you to use any amount of data but has a fixed monthly fee.
      • The premium SKU is required if you plan to use Microsoft peering to access Microsoft SaaS/PaaS services over the ExpressRoute connection.

  • ExpressRoute circuits are offered in three SKUs: Local, Standard and Premium. (Two shown below)

    FeatureExpressRoute StandardExpressRoute Premium
    Global ReachNoYes
    Increased Route LimitsNoYes
    Connectivity to Microsoft PeeringLimited to the same geopolitical regionGlobal connectivity
    Service ProvidersLimitedExpanded
    Route Advertisements4,00010,000
    Support for Azure Government and National CloudsNoYes
    BGP CommunitiesNoYes
    BGP Sessions2 per peering, per ExpressRoute circuit4 per peering, per ExpressRoute circuit
    CostLowerHigher
    AvailabilityVaries by regionVaries by region

ExpressRoute Gateway SKUs

  • When we create an ExpressRoute Gateway service, we need to specify the SKU that we want to use. We can choose from one of the following three:
    • Standard / ErGw1AZ: This option supports a max of four ExpressRoute connections and up to 1 Gbps bandwidth
    • High Performance / ErGw2AZ: This options supports a max of eight ExpressRoute connections and up to 2 Gbps bandwidth
    • Ultra Performance / ErGw3AZ: This options support a max of 16 ExpressRoute connections and up to 10 Gbps bandwidth
  • The SKUs with ‘AZ’ in the name are zone-redundant, meaning they are highly available across Azure Availability Zones.
  • We can change a SKU after the Gateway has been created
  • An ExpressRoute Gateway must be deployed in a Gateway subnet. (Named ‘GatewaySubnet’). It is recommended to use at least a /26 for the GatewaySubnet.
  • When choosing a Gateway SKU, we want to ensure the bandwidth of the SKU matches the bandwidth of the circuit. ExpressRoute Gateway SKUs

ExpressRoute FastPath

FastPath is designed to improve the data path performance between connected remote networks and Azure vNets. To understand how FastPath works, we need to understand the default behavior without it. By default, the ExpressRoute Gateway performs two main tasks: exchanging network routes with our remote networks AND routing network traffic to Azure vNet resources. Routing the network traffic adds a little processing overhead, which impacts performance metrics such as Packets per Second (PPS) and Connections per Second (CPS). When enabled, FastPath sends network traffic directly to vNet resources, bypassing the gateway. This results in higher bandwidth and better overall performance. FastPath is available for all ExpressRoute circuits, but the ExpressRoute Gateway must be either the ultra-performance or ErGw3AZ SKU.

Encryption over ExpressRoute

  • By default, ExpressRoute connections are not encrypted.
  • Microsoft offers two optional solutions for encrypting data in transit over ExpressRoute connections:
    • MACsec - a Layer 2 encryption protocol that can be used to encrypt physical links. To implement MACsec, we need a Key Vault to store the encryption keys. This key is referred to as the connectivity association key (CAK).
    • IPSec - a Layer 3 encryption protocol that can be used to encrypt data between two endpoints. To implement IPSec, we need to configure a VPN Gateway in Azure and a VPN device on-premises.

BFD

  • BFD (Bidirectional Forwarding Detection) is a network protocol that detects link failures in a network. It is used to detect failures in the forwarding plane of a network.
  • BFD is supported over private peering and Microsoft peering.
  • When you enable BFD, you can speed up failure detection between Microsoft Enterprise Edge (MSEE) devices and your equipment.
  • How it works:
    • On the MSEE devices, BGP keep-alive and hold-time are typically configured as 60 and 180 seconds, respectively. For that reason, when a link failure happens, it can take up to three minutes to detect the failure and switch traffic to an alternate connection.
    • You can control the BGP timers by configuring a lower BGP keep-alive and hold-time on your edge peering device. If the BGP timers are not the same between the two peering devices, the BGP session will establish using the lower time value. The BGP keep-alive can be set as low as 3 seconds. The hold-time can be as low as 10 seconds. However, setting these values too low isn’t recommended because the protocol is process-intensive.

Configure ExpressRoute and site to site coexisting connections

  • You can configure Site-to-Site VPN as a secure failover path for ExpressRoute or use Site-to-Site VPNs to connect to sites that are not connected through ExpressRoute.

  • Configuring Site-to-Site VPN and ExpressRoute coexisting connections has several advantages: -You can configure a Site-to-Site VPN as a secure failover path for ExpressRoute.

    • Alternatively, you can use Site-to-Site VPNs to connect to sites that are not connected through ExpressRoute.

  • You can configure either gateway first. Typically, you will incur no downtime when adding a new gateway or gateway connection.

  • Network Limits and limitations

    • Only route-based VPN gateway is supported. You must use a route-based VPN gateway. You also can use a route-based VPN gateway with a VPN connection configured for ‘policy-based traffic selectors’.
    • The ASN of Azure VPN Gateway must be set to 65515. Azure VPN Gateway supports the BGP routing protocol. For ExpressRoute and Azure VPN to work together, you must keep the Autonomous System Number of your Azure VPN gateway at its default value, 65515. If you previously selected an ASN other than 65515 and you change the setting to 65515, you must reset the VPN gateway for the setting to take effect.
    • The gateway subnet must be /27 or a shorter prefix, (such as /26, /25), or you will receive an error message when you add the ExpressRoute virtual network gateway.
    • Coexistence in a dual stack VNet is not supported. If you are using ExpressRoute IPv6 support and a dual-stack ExpressRoute gateway, coexistence with VPN Gateway will not be possible.