How I finally understood BGP by building a small network in my homelab
I work at layer 7. BGP was the part I could not explain, so I rebuilt it in my homelab with FRR, then broke it on purpose.

I work at Cloudflare, but not on this part of the network.
My days are about TLS, HTTP, bot management and WAF rules. That is layer 7. I can explain why a request fails at the edge.
But I could not explain what happens between two networks before the packet reaches the edge, because I had never worked on it.
BGP was the part I did not understand. Everyone says BGP is what makes the Internet work. It appears in every outage report, in every story about someone leaking a prefix, and every time a network disappears from the map. But I had no idea how it worked.
So I built a small one. Three LXC containers on my Proxmox, three private AS numbers and three Linux bridges. Small enough to understand in one evening, but real enough to behave like the real thing. Then I broke it on purpose.
This is the path I took, in the order I took it, with the problem that forced each step. Every command ran on those containers.
The lab
Three Debian LXC on a single Proxmox host, each with two interfaces, connected to three Linux bridges:
- Router A — first network — AS 65005 — WAN
172.16.0.1/30— LAN10.0.5.1/24 - Router B — second network — AS 65006 — WAN
172.16.0.2/30— LAN10.0.6.1/24 - Router C — the attacker — AS 65007 — WAN
172.16.0.3/24— LAN10.0.7.1/24
bgpxp is the bridge that connects the three WAN interfaces. lan-a and lan-b are the client networks.
A and B start on a /30 transit link. When C joins, I re-address that link to 172.16.0.0/24 (see Problem 6). The /24 in Router C's row is the final state. The screenshots show the /30 until then.
The AS numbers come from the private range, 64512–65534. It is the same idea as 192.168.x.x: reserved for labs, never routed on the Internet.
This is the whole topology. No DNS, no NAT, no firewall. I wanted the smallest network where BGP still does something.
Two networks that cannot talk
Ping between the two WAN interfaces works:
root@RouterA:~# ping -c 3 172.16.0.2
PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data.
64 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=0.045 ms
64 bytes from 172.16.0.2: icmp_seq=2 ttl=64 time=0.061 msPing the LAN on the other side does not:
root@RouterA:~# ping 10.0.6.1
ping: connect: Network is unreachable"Unreachable" is not the same as "timeout". The kernel is not saying that nobody answered. It is saying it does not know where that address is. Router A knows two things: 10.0.5.0/24 is behind eth1, and 172.16.0.0/30 is behind eth0. It knows nothing about 10.0.6.0/24. So the packet is dropped before it even leaves the machine.
The two networks are physically connected, but they cannot see each other. That is the gap a routing protocol fills.

A Linux box is not a router
Even if A knew where 10.0.6.1 was, it would still refuse to forward. By default, the Linux kernel only accepts packets addressed to itself. Everything else is dropped.
That is on purpose. A server is not a router.

So you tell the kernel that it is allowed to forward packets:
sysctl -w net.ipv4.ip_forward=1
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.confAnd it still does not work. A is now allowed to forward packets, but it still cannot, because it has no route. The difference matters: forwarding is a permission, routing is knowledge. A router needs both.
This is also where two words I kept reading in incident reports finally made sense. The control plane builds the map. The data plane moves the packet using that map. At this point I have a data plane and no map.

Installing a routing daemon
Time to install a routing daemon. I installed FRRouting, the open-source routing suite. In a real network it runs on routers. On Linux it is just a set of daemons.
apt update && apt install frr frr-pythontools -yOne practical note: the LXC needs a third interface on vmbr0 to reach the Internet and download packages. I added it, rebooted, installed, then left it out of the BGP design. It plays no role in what follows.
By default, FRR starts with every protocol disabled. You enable only what you need:
# /etc/frr/daemons
bgp=yessystemctl restart frrThen the shell:
root@RouterA:~# vtysh
RouterA#The prompt changes, and so does the language. Inside vtysh you are no longer in Linux. It is a Cisco-like CLI: configure terminal, router bgp, show. It feels strange for ten minutes, then it makes sense.

Declaring neighbors by hand
Both routers can now speak BGP. They still do not talk to each other, and that is on purpose: BGP has no discovery mechanism. You declare your neighbors by hand, with their IP address and their AS number.
That is a security decision. You do not want a random router to open a session with you and announce anything it likes. It also shows the main theme of this article. BGP runs on explicit trust, and trust is exactly what gets abused.
Each side declares the other:
configure terminal
router bgp 65005
bgp router-id 172.16.0.1
neighbor 172.16.0.2 remote-as 65006
address-family ipv4 unicast
network 10.0.5.0/24
exit-address-family
exit
exit
writeMirror it on B with router bgp 65006 and neighbor 172.16.0.1 remote-as 65005.
This is eBGP, external BGP: two different AS numbers. The same protocol also runs inside an AS (iBGP), with different rules. That is a story for another day.
After a few seconds, the routers exchange OPEN and KEEPALIVE messages, open a TCP session on port 179, and start talking:
RouterA# show ip bgp summary
...
Neighbor V AS MsgRcvd MsgSent ... Up/Down State/PfxRcd PfxSnt
172.16.0.2 4 65006 4 4 ... 00:01:59 (Policy) (Policy)
The session is up. The timer runs. And the last two columns say (Policy), with zero prefixes received. Nothing is being exchanged.
You may notice that the router ID in the screenshots is 171.16.0.1 while my configuration says 172.16.0.1. I made a typo. BGP did not care, because the router ID is an identifier, not an address. It is also the kind of detail that shows the screenshots are real.
This table is the first thing you look at on any router, and it looks the same everywhere. Here is the same protocol seen from a Cogent looking glass in Singapore, for 79.174.20.0/24. The path is50629 3214 34872 213638. The AS on the left is the closest to the router I queried. The AS on the right is the origin, the network that owns the addresses. The AS numbers in the middle are transit providers carrying the route. The length of the list gives the distance, and it also detects loops.localprefis internal to Cogent. TheCommunityvalues (174:...) are tags Cogent adds to its own routes.

The (Policy) state
I expected BGP to trust me. It does not, not anymore.
FRR implements RFC 8212: on an eBGP session, no route is accepted without an inbound filter, and no route is announced without an outbound filter. The option is called bgp ebgp-requires-policy, and it is enabled by default outside the datacenter profile.
The reason comes from the early design. The protocol was made when the network was small and the operators knew each other. A router that announced a prefix was believed. There was no way to say "you are not allowed to announce this". Thirty years later, that blind trust is the cause of every route leak and every hijack.
In production you would write route-maps: accept only 10.0.6.0/24 from AS 65006, and announce only the prefixes you own. In a lab, I just disabled the requirement on both sides:
configure terminal
router bgp 65005
no bgp ebgp-requires-policy
exit
exit
writeAnd then… nothing changed. Still no route.
BGP only reacts to events. It does not re-read the configuration on its own. Removing the filter does not make it re-read what the neighbor already sent. You have to reset the session:
clear ip bgp * soft"Soft" means: re-run the policy and exchange the table again without closing the TCP session. clear ip bgp * soft in and ... out target one direction. Without a direction, both are refreshed. On a real network, engineers type this fifty times a day. The hard reset (clear ip bgp *) closes the session and rebuilds everything from scratch: fine in a lab, expensive in production.
And then:
RouterA# show ip bgp
*> 10.0.5.0/24 0.0.0.0 0 32768 i
*> 10.0.6.0/24 172.16.0.2 0 0 65006 i
Two routes. The *> means valid and best: valid because the next hop is reachable and the policy allows it, best because it won the selection. And the kernel got them:
root@RouterA:~# ip route
10.0.5.0/24 dev eth1 proto kernel scope link src 10.0.5.1
10.0.6.0/24 nhid 10 via 172.16.0.2 dev eth0 proto bgp metric 20
172.16.0.0/30 dev eth0 proto kernel scope link src 172.16.0.1proto bgp: FRR installed the route in the Linux kernel through zebra. Control plane and data plane are in sync.
root@RouterA:~# ping 10.0.6.1
64 bytes from 10.0.6.1: icmp_seq=1 ttl=64 time=0.045 msThis is the moment it works. Two networks that could not see each other are now one network, and the only thing that changed is four lines of BGP and one policy decision.

Problem 6: adding a third AS
Two routers only make a demo. To see transit, I needed a third one.
I added Router C, AS 65007, on the same transit bridge. It did not work, for a reason I had not thought about: the transit link was a /30. A /30 has two usable addresses, and A and B were already using them. There was no room left for C.
So I re-addressed the transit to 172.16.0.0/24 and moved all three interfaces. In a real network you plan this. In a lab, you learn it by running into it.

Then the same three steps as before: declare the neighbor on A, declare A on C, and announce C's own LAN, 10.0.7.0/24.

Two sessions now instead of one, and each neighbor sends one prefix.
Now the interesting part. Go on Router C:
RouterC# show ip bgp
*> 10.0.5.0/24 172.16.0.1 0 0 65005 i
*> 10.0.6.0/24 172.16.0.2 0 0 65005 65006 i
*> 10.0.7.0/24 0.0.0.0 0 32768 i
C has a route to B's LAN, and C has no BGP session with B. The AS path says where the route came from: 65005 65006. A took the route it learned from B and announced it again to C. That is what an eBGP router does with its best paths: it tells every other peer.
Look at the next hop, though: 172.16.0.2, which is B's own address, not A's. Because the three routers share the same transit subnet, FRR keeps the original next hop instead of rewriting it to itself. So the control plane goes through A, and the packet goes straight from C to B. If I wanted to force traffic through A, I would need neighbor ... next-hop-self.
This matters, because it is exactly what a route server at an Internet Exchange does: it gives you the path, not the traffic.
I can check it end to end from C:
RouterC# ping 10.0.6.1It answers, and the packet reaches B directly. C's next hop for that prefix is B's own address, one hop away on the shared segment. No tunnel through A, no NAT, nothing exotic. Just a route learned from a third party, and a next hop that happens to be the destination.
The hijack
Now the part I actually wanted to understand.
BGP chooses the best path per prefix. But when two different prefixes cover the same address, BGP is no longer the one deciding: the forwarding table is. And the forwarding rule is simple and always the same: the most specific prefix wins. A /25 beats a /24, always, whatever the AS path length, the local preference, or anything else BGP computed.
So if you want to steal 10.0.6.0/24, you do not announce 10.0.6.0/24 from somewhere else. You announce two /25 that cover it, from a peer the victim already trusts. The victim keeps B's real /24 in its BGP table and still sends every packet to you.
That is the whole trick. Let me do it from C.
BGP can only announce a prefix that already exists in the router's own RIB. So first, I create two fake static routes that point nowhere:
configure terminal
ip route 10.0.6.0/25 Null0
ip route 10.0.6.128/25 Null0
router bgp 65007
address-family ipv4 unicast
network 10.0.6.0/25
network 10.0.6.128/25
exit-address-family
exit
exit
write
A accepts them, because A has no filter on that session and because the announcement comes from a neighbor I declared.
RouterA# show ip bgp
*> 10.0.5.0/24 0.0.0.0 0 32768 i
*> 10.0.6.0/24 172.16.0.2 0 0 65006 i
*> 10.0.6.0/25 172.16.0.3 0 0 65007 i
*> 10.0.6.128/25 172.16.0.3 0 0 65007 i
*> 10.0.7.0/24 172.16.0.3 0 0 65007 i
Five routes. B's real route is still there, still valid. And yet:
root@RouterA:~# ip route get 10.0.6.50
10.0.6.50 via 172.16.0.3 dev eth0 src 172.16.0.1 uid 0
A packet for B's LAN now leaves towards C. Not because BGP changed its mind, but because 10.0.6.50 is inside 10.0.6.0/25, and /25 is more specific than /24.
In my lab, C has a Null0 route, so the traffic dies there. That is a black hole, not a hijack: I am breaking the network, not spying on it. A real attacker would forward the packets to a machine that answers like the target. Nobody would notice until a certificate did not match, or until someone looked at prefix lengths in a looking glass.
Two things surprised me:
- B is not "removed" from the Internet. Its route is still in every table. It is simply no longer used for those addresses. The damage is invisible unless you look at prefix lengths.
- The attack needed nothing clever. Two static routes and two
networkstatements. A BGP hijack does not need smart tools. It needs a trust relationship you are part of.
What this looks like at Internet scale
On 24 February 2008, Pakistan Telecom wanted to block YouTube nationally. YouTube's prefix was 208.65.153.0/22. Pakistan Telecom (AS17557) announced 208.65.153.0/24, a more specific prefix, and the announcement leaked to the world through its transit provider, PCCW.
Every router that received it preferred the /24 over the /22, exactly like my /25 over B's /24. YouTube disappeared from the Internet for about two hours, worldwide, because of one missing filter on one eBGP session.
It was the same experiment at a different scale, and nobody meant to do it.
This is why RPKI exists. A resource holder signs a ROA that says "AS X is allowed to announce prefix Y".
Routers that validate (RPKI-to-Router, ROV) mark the announcement invalid and drop it. Look at FRR's output again: there is a legend for RPKI validation codes: V valid, I invalid, N Not found. In my lab there is no validator, so nothing is validated and every prefix would show up as N. In real networks, ROV is now an important part of the filtering stack, and still far from universal.
It is also why route monitoring exists. When the control plane cannot be trusted, you watch it.
This is the video that made me want to try this: youtube.com/watch?v=IzLPKuAOe50.
What this lab does not have
An honest list, because a lab is a simplified world:
- iBGP, route reflectors, confederations. I only did eBGP between three ASes. Inside a real AS, the same protocol runs with different rules: next hop unchanged, full mesh or reflectors, AS path untouched.
- Policy. In production, every session has prefix-lists and route-maps on both sides. I removed the policy requirement to make the lab work, which is the opposite of what you would do for real.
- RPKI validation. As above: the column exists in my output, the validator does not.
- IPv6. Same protocol, different address family, and my lab is IPv4 only.
- Scale. The real Internet is roughly 75,000 autonomous systems and about a million IPv4 prefixes. My three routers and five routes show how it works, not how big it is.
What changed for me
I did not become a network engineer in a weekend. But now I understand it.
When I read an incident report about a route leak, I now know what the table looked like, which column moved, and why the more specific prefix won. When I see (Policy) on a session, I know it is not a bug, it is RFC 8212 doing its job. When someone says "BGP is based on trust", I can show the two lines that prove it.
That was the point. I spend my days at layer 7, where the tools hide most of the complexity. BGP is the layer where the Internet is a set of promises between routers, and where one wrong promise can take a network off the map. I wanted to see it with my own hands. Three containers and about thirty lines of configuration were enough.
If you want to reproduce it
The three configs, cleaned up. On each LXC: sysctl -w net.ipv4.ip_forward=1, install FRR, set bgp=yes in /etc/frr/daemons, restart FRR.
! Router A — AS 65005
router bgp 65005
bgp router-id 172.16.0.1
no bgp ebgp-requires-policy
neighbor 172.16.0.2 remote-as 65006
neighbor 172.16.0.3 remote-as 65007
address-family ipv4 unicast
network 10.0.5.0/24
exit-address-family! Router B — AS 65006
router bgp 65006
bgp router-id 172.16.0.2
no bgp ebgp-requires-policy
neighbor 172.16.0.1 remote-as 65005
address-family ipv4 unicast
network 10.0.6.0/24
exit-address-family! Router C — AS 65007
router bgp 65007
bgp router-id 172.16.0.3
no bgp ebgp-requires-policy
neighbor 172.16.0.1 remote-as 65005
address-family ipv4 unicast
network 10.0.7.0/24
exit-address-familyThen the attack. Add the two Null0 routes and the two /25 networks on C, run clear ip bgp * soft on A, and watch show ip bgp grow from three routes to five.
Comments
No comments yet