Communication · Information operations · Information Warfare

The secret world of microwave networks

Has anyone spanned the Atlantic with floating microwave masts yet? We’re not sure…

Stretching between London and Frankfurt there is a private, mysterious network that is twice as fast as the normal Internet. The connection, provided by a series of microwave dishes on masts, was once completely secret: only one very rich company was allowed to use it, and no one else knew about it.

A couple of years later though, a competitor completed its own microwave link between the two cities—and thus the first company, not wanting to lose out on potential business, revealed that it too had a link between the cities. If a competitor had never emerged, that first link would probably still be shrouded in secrecy today.

Similar stories can be found all over the world, but because these networks are privately owned, and because they’re often used by financial groups trying to find an edge on the stock market and eke out a few extra billions, you have to dig deep to find them.

For example, back in 2013, just as the algorithmic high-frequency trading (HFT) bubble was starting to pop, researchers in California looked at a large corpus of market trades and found that, starting in March 2011, there had been a sudden latency drop of 2.5 milliseconds between Chicago and New York. Prior to that, the previous fastest trading latency was about 7.5 milliseconds, so a reduction to 5ms was significant. The researchers then used the FCC and other records to deduce that, at the time of their 2013 study, there were 15 (!) networks licensed to operate microwave links between the two cities. The latency drop was probably caused by a new low-latency microwave network coming online.

Enlarge / A 1949 newspaper clipping showing AT&T’s “just 34 hops” microwave link from New York to Chicago.

But what about the British weather?

Microwave networks themselves are incredibly old hat. Way back in 1949, New York and Chicago (712 miles apart as the eagle flies) were connected by a 34-hop line-of-sight microwave network operated by AT&T. In the UK, from the mid-1950s until the 80s, the nation’s trunk communications network was fashioned out of microwave radio links; it carried everything from television and telephone to national defence data.

Enlarge / The UK’s proposed microwave backbone in 1956.

Deploying a microwave network was cheaper than running cables, and the link capacity was greater than other copper-wire technologies at the time. Working against microwave networks, however, is the fact that their high-frequency signals (usually between 6GHz and 30GHz) are highly directional and require line of sight between each base station. Furthermore, while there is tons of bandwidth available at the top end of the microwave spectrum, high-frequency signals suffer from severe attenuation when travelling through rain, cloud, or just about anything that isn’t clear air—an effect usually referred to as “rain fade” (older satellite TV users probably know all about that one).

Rain fade can be mitigated by boosting the transmission power, by using larger dishes and hydrophobic coatings, or a fail-over protocol where the link drops down to a lower frequency that can better penetrate the inclement atmosphere when necessary. Another option, starting in the 1980s, was to replace or augment microwave links with optical fibre, which was more reliable and provided a seemingly bottomless cup of bandwidth.

Fast forward to today and fibre has mostly superseded microwave for trunk network connections. Microwave networks are nevertheless still widely used, and the underlying technologies are being actively developed. Nowadays they’re mostly used to connect geographically remote areas to the Internet—I have a friend in Norway whose town is on the wrong side of a mountain, so their connection is bounced via a microwave tower at its summit—or for specialist applications, such as private financial networks. Up until a few years ago, almost every cell tower was backhauled via microwave, but many of those links are being replaced with fibre to keep up with the throughput requirements of busy LTE and LTE-Advanced cells.

Enlarge / A map of some of the microwave networks in southern England and continental Europe, mapped out by Alexandre Laumonier. The microwave links down to Cornwall are for the transatlantic submarine cables that land there. This map is by no means exhaustive; it was compiled by a third party.

How low can you go?

When the world is already blanketed in a dense mesh of high-speed fibre-optic cabling, the obvious question is: why use your own microwave network?

The first reason is somewhat obvious; if you have your own network connection, it’s usually easier to guarantee things like security, quality of service, bandwidth, and other factors that businesses value highly.

The second reason, as we’ve already alluded to, is that microwave networks—somewhat surprisingly—can have lower latency than fibre. With some advanced networks, that latency is only a few microseconds slower than the speed of light. Fibre can be pretty quick over short stretches, but it soon starts lagging over longer distances, such as between two stock exchanges or a multinational’s offices.

Fibre networks are hamstrung by the intertwining forces of money and geography. Laying a fibre network is incredibly expensive: you have to dig a trench that’s hundreds (or thousands) of miles long, or lease access to ducts that have already been laid by infrastructure companies such as BT Openreach. You also have to respect the geography of the land; when faced with a mountain or river, do you go straight across at great expense, or do you make a diversion to the nearest bridge or tunnel? Combine these two factors and you’ll understand why most of the world’s terrestrial fibre networks slink alongside existing roads and railways—it’s just the most sensible option.

Every time a network architect makes one of these sensible decisions, there’s a small increase to the end-to-end latency. Add them up, and you end up with a few extra milliseconds—which is when the low-latency microwave networks swoop in to pick up some business.

A map of terrestrial and submarine European backbone links, circa 2001. Finding modern maps of terrestrial cabling is tough—but for the most part, similar routes are used today—just with upgraded hardware.

Let’s take London-Frankfurt as a real-world example. As the raven flies, the two cities are 396 miles apart. A radio signal travelling through air at just under the speed of light (299,700km per second) would cover that distance in 2.126 milliseconds. Through a glass or plastic fibre, where light has to bounce along the refractive index rather than travel in a straight line, the speed of light is reduced to around 200,000km/s, resulting in a theoretical minimum latency of 3.186 milliseconds.

In reality though, the fibre network between London and Frankfurt isn’t just a single straight piece of fibre. For a start, depending on where the London server is, the packet of data might bounce around a few times until it gets to the right router for its journey across to Europe. Along the way, there are other routers and repeaters. And once the packet arrives in Frankfurt, it’s the same deal as in London: the destination server is probably a few hops away.

Add geography and infrastructure to the mix, a submarine cable crossing (probably via Calais in France or Ostend in Belgium), plus the fact that a router in London might decide to send the packet via Paris instead, and the average latency between two servers in London and Frankfurt is actually closer to 17 milliseconds.

Now, at long last, for the punchline: a private microwave network between London and Frankfurt has a latency of about 4.2 milliseconds. I say “about,” because the newest connections are still secret and their exact latencies are unknown. That’s why businesses and financial institutions which absolutely must have the fastest connection opt for microwave links instead of fibre.

Building a point-to-point network

Microwave networks have two key advantages: radio signals travel through air about 50 percent faster than light moves down fibre, and you can (usually) build microwave links in a straight line between the two end points. The latter aspect means that the total physical distance travelled by a packet can be significantly reduced, plus you have the option of building the microwave network so that it actually terminates near the user, meaning packets have to traverse fewer routers.

Enlarge / The BridgeWave FP80-3000, capable of up to 3Gbps over 80GHz. For scale, the antenna is 30cm across.

Infrastructure-wise, microwave networks are essentially a bunch of masts with two transceivers at the top, each pointing in opposite directions—kinda like a glorified semaphore system, really. The distance between the masts will vary, depending on geography, but the average distance between masts is about 25 to 40 miles. The actual maximum distance is determined by each transceiver’s height above ground level, radio frequency licensing restrictions, and the lay of the land; if you put a tall mast on top of a hill and have a powerful enough transmitter, you can comfortably achieve 50+ miles before you hit the radio horizon.

Cost-wise, you’re looking at about £10,000 to £20,000 per transceiver for something like the BridgeWave unit (pictured), plus about £100,000-£200,000 for each mast itself (including access, construction, backup power, etc).

Going back to the London-Frankfurt example, the network probably consists of about 20 masts—so, somewhere between £2.5 and £5 million to physically build the network. That’s not counting staff, ongoing operational costs (power, leases, support), and numerous other factors. The networking companies we spoke to for this story were loath to give us exact figures, but one London-Frankfurt microwave network operator gave us a ballpark figure of “between 10 and 20 million euros.”

Finally, just to bring us full circle, it’s interesting to consider how much it would cost to run a (theoretical) straight-line fibre network from London to Frankfurt. If BT Openreach had a 400-mile duct running between London and Frankfurt, it would cost about £1 per metre per year to lease space for a fibre-optic loop—and 400 miles is 644,000 metres. The price of fibre-optic cabling varies, depending on the number of cores and the shielding, but somewhere between 50p and £1 per metre is about right for humdrum terrestrial cable. And then, similar to microwave, you need signal amplifiers—which are expensive, but cheaper than microwave transceivers—every 20 or 30 miles.

Laying a new run of fibre is cheaper than rolling out your own private microwave network, then—but that only works if there’s already a duct between the two points that you’re trying to connect. If you have to build your own 400-mile duct… good luck!

Sebastian Anthony / Sebastian is the editor of Ars Technica UK. He usually writes about low-level hardware, software, and transport, but it is emerging science and the future of technology that really get him excited.



2 thoughts on “The secret world of microwave networks

  1. Interesting article, Joel. Thanks for sharing, but I am not sure it is technologically up to date. My understanding is television white space, not microwave, is where innovation is occurring in closed mesh networks plus connecting remote areas to the Internet (e.g., combined with solar-powered based stations). See, and TVWS is doing a lot of good for a lot of people, but I wonder what the national security implications are when I read about Russian spy ships of U.S. coasts per I probably don’t want to know. 😦

Comments are closed.