It is too bad this important work needed to be done on the cheap. You'd think if the Swiss National Bank was involved, you could get a proper budget....
It would have been a lot easier to focus on the important implementation details if the server was an off the shelf Lenovo datacenter server (SD550?) with a pair of 100 gig/s NVIDIA cards in it.
(Source: last month I set up a machine like this for a colleague to do approximately the same task. I spent "copy and paste the production server config" time on it, not a week.)
Youden 20 days ago [-]
I have 25Gbps from Init7 at home. My "router" is a Minisforum MS-01 with a second-hand Mellanox ConnectX-5, running VyOS.
My main home server is a Supermicro SYS-510D-4C-FN6P. It has dual 25Gbps ports onboard but also an Intel E810-XXVDA4T with another 4x25Gbps ports.
Both of them are perfectly capable of saturating their ports using stock forwarding on Linux, no DPDK, VPP, anything, without breaking a sweat. Both of them were substantially cheaper than the machine in the article.
Is there something I'm missing? Why does this workstation need a ~$1000 motherboard and a ~$1000 Xeon CPU? Those two components alone cost more than either of my computers and seem like severe overkill.
FireBeyond 20 days ago [-]
My understanding is that the setup needs to allow them to work on packet routing at those speeds, not just send/receive, to simulate SCION.
Youden 20 days ago [-]
Ah, so they need to hold giant routing tables in memory and do lookups in them or something like that?
Veserv 20 days ago [-]
Does not look like it [1]. It appears to be a protocol that enumerates your exact path, interface by interface, on every data packet. So you can just blindly forward to the next hop written in the packet itself.
By my guess, a competent and efficient implementation should be able to run the routing logic at ~30-100 million packets per second per core. That would be ~300-1,000 Gb/s per core, so you would bottleneck on your memory bandwidth if you have even a single copy.
"SCION OSS border router performance reached a ceiling of around 400k-500k packets per second, which is roughly equivalent to 5-6 Gbit/s at a 1500-byte MTU." vs. 1.4 M PPS for IP (on an older CPU) https://toonk.io/linux-kernel-and-measuring-network-throughp...
cpach 20 days ago [-]
Ah. Thanks!
romshark 18 days ago [-]
> Is there something I'm missing? Why does this workstation need a ~$1000 motherboard and a ~$1000 Xeon CPU? Those two components alone cost more than either of my computers and seem like severe overkill.
Yes, as stated in the article, it probably could have been cheaper. But this setup is supposed to:
1. Run simulations and benchmarks of/on entire SCION topologies with multiple ASes.
2. Potentially grow beyond 25 Gbit/s into the 200 Gbit/s ranges (and more?).
3. Be available to me ASAP (can't wait months for it to arrive from China).
4. Potentially be used for CI/CD performance regression testing in the future.
The budget allowed a bit of headroom for the future.
dist1ll 19 days ago [-]
Your MS-01 routes line-rate 25Gbps in software with VyOS w/o kernel bypass? That's very surprising to me. At what packet sizes?
neutrinobro 20 days ago [-]
Nice write up! For this sort of thing, I have leaned towards AMD Epyc, Intel e810, and DPDK for the software stack. Unfortunately, lately the supermicro H13SSL line of mobo's appear to have become near-unobtainable with ridiculous 6+ month lead times.
preisschild 19 days ago [-]
Only issue I have with those smicro boards is that they dont support OpenBMC. I don't want to pay extra for a license to use the redfish api...
Melatonic 20 days ago [-]
Why that mobo specifically ?
neutrinobro 20 days ago [-]
No idea, you can still get one-off boards here and there, but buying anything in quantity has been tricky. I can only surmise supermicro's resources are largely tied up with AI data center build out, with everything else relegated to short runs.
layla5alive 19 days ago [-]
Helping to put all the bullets in net neutrality...
Pathway to even greater corporatization and splintering of the internet?
Replacing public RIRs with private organizations, securely routing between each other..
How do I peer with the big corps in a SCION world?
Security and privacy are already addressed by things like transport layer encryption, so SCION doesn't really enable a more secure internet, it enables more (largely corporate) control
romshark 18 days ago [-]
First of all, at this point, SCION is not here to replace BGP. It's here to provide a more secure way of interconnecting ASes for critical infrastructure applications (finance, defense, government, etc..) that allows path selection and verification over multiple-ISPs. It can for example, be seen as an alternative to MPLS but offering more capability.
SCION also offers more protection against DDoS attacks and other outages thanks to its multi-path routing capabilities and ability to failover quicker than BGP as it builds and stores its path knowledge in advance.
> How do I peer with the big corps in a SCION world?
You do so by joining an ISD (Isolation Domain) and inheriting TRC (Trust Root Configuration).
> so SCION doesn't really enable a more secure internet, it enables more (largely corporate) control
Much critical infrastructure is still reliant on leased lines or MPLS which is expensive and reliant on a single ISP which often reduces resilience. It often also requires assurances about where its traffic is being forwarded (e.g. through particular countries or regions) which is difficult or impossible with BGP. SCION can instead provide these assurances over the commodity Internet provided by multiple ISPs, by being able to verify paths and allowing packet senders to control how packets should be routed given the available path options.
ISDs are typically for specific use cases (e.g. Swiss Secure Finance Network) where strong assurances are needed for where traffic is sent, but ISDs can decide admission criteria for themselves and how they wish to communicate with other ISDs and the rest of the Internet.
Think of the power grid for example. Putting power plants on the internet is probably a bad idea. A better idea is to interconnect power plants through multiple ISPs over a SCION ISD. Less expensive than leased lines or MPLS, and more flexible.
markhahn 20 days ago [-]
Most of this was "enthusiasts playing with bigboy stuff", but it turns out ok in the end.
20 days ago [-]
entropyneur 19 days ago [-]
Going to such great lengths to keep the office quiet. It wouldn't even occur to me to think about the noise.
auspiv 20 days ago [-]
Wow, 249 CHF for 8x fans is insane. The grip Noctua has on people! Nice workstation.
Personally was always a fan of just going with the largest fans possible - surprised we don't see more cases designed around 140mm and larger. 200mm is much less common but has a more pleasing noise profile
Gracana 20 days ago [-]
I'm also a fan of that sort of setup. A Fractal Meshify 2 XL will fit a bunch of 140mm fans, or you can get the Torrent which is smaller but has 2x 180mm fans up front. I have both and would recommend them, though the Torrent is a tight fit for a big board, and the shield on the back of the Asus W790 motherboards interferes with the cable routing grommets on the motherboard tray, so you have to remove them.
The article says they're using 80mm fans. Am I missing something?
newsclues 20 days ago [-]
Not tiny 1u fans is the point
immibis 20 days ago [-]
Noctua makes really good fans, I'm told. Want to get on their level and make a similar amount of money? In a world of slop, quality engineering is valuable.
RachelF 20 days ago [-]
There fans are still good, but not the quietest anymore.
Noctua no longer manufactures them, they are now made by YS Tech.
Rendered at 14:25:58 GMT+0000 (Coordinated Universal Time) with Vercel.
It would have been a lot easier to focus on the important implementation details if the server was an off the shelf Lenovo datacenter server (SD550?) with a pair of 100 gig/s NVIDIA cards in it.
(Source: last month I set up a machine like this for a colleague to do approximately the same task. I spent "copy and paste the production server config" time on it, not a week.)
My main home server is a Supermicro SYS-510D-4C-FN6P. It has dual 25Gbps ports onboard but also an Intel E810-XXVDA4T with another 4x25Gbps ports.
Both of them are perfectly capable of saturating their ports using stock forwarding on Linux, no DPDK, VPP, anything, without breaking a sweat. Both of them were substantially cheaper than the machine in the article.
Is there something I'm missing? Why does this workstation need a ~$1000 motherboard and a ~$1000 Xeon CPU? Those two components alone cost more than either of my computers and seem like severe overkill.
By my guess, a competent and efficient implementation should be able to run the routing logic at ~30-100 million packets per second per core. That would be ~300-1,000 Gb/s per core, so you would bottleneck on your memory bandwidth if you have even a single copy.
[1] https://www.ietf.org/archive/id/draft-dekater-scion-dataplan...
Yes, as stated in the article, it probably could have been cheaper. But this setup is supposed to:
1. Run simulations and benchmarks of/on entire SCION topologies with multiple ASes.
2. Potentially grow beyond 25 Gbit/s into the 200 Gbit/s ranges (and more?).
3. Be available to me ASAP (can't wait months for it to arrive from China).
4. Potentially be used for CI/CD performance regression testing in the future.
The budget allowed a bit of headroom for the future.
Pathway to even greater corporatization and splintering of the internet?
Replacing public RIRs with private organizations, securely routing between each other..
How do I peer with the big corps in a SCION world?
Security and privacy are already addressed by things like transport layer encryption, so SCION doesn't really enable a more secure internet, it enables more (largely corporate) control
SCION also offers more protection against DDoS attacks and other outages thanks to its multi-path routing capabilities and ability to failover quicker than BGP as it builds and stores its path knowledge in advance.
> How do I peer with the big corps in a SCION world?
You do so by joining an ISD (Isolation Domain) and inheriting TRC (Trust Root Configuration).
> so SCION doesn't really enable a more secure internet, it enables more (largely corporate) control
Much critical infrastructure is still reliant on leased lines or MPLS which is expensive and reliant on a single ISP which often reduces resilience. It often also requires assurances about where its traffic is being forwarded (e.g. through particular countries or regions) which is difficult or impossible with BGP. SCION can instead provide these assurances over the commodity Internet provided by multiple ISPs, by being able to verify paths and allowing packet senders to control how packets should be routed given the available path options.
ISDs are typically for specific use cases (e.g. Swiss Secure Finance Network) where strong assurances are needed for where traffic is sent, but ISDs can decide admission criteria for themselves and how they wish to communicate with other ISDs and the rest of the Internet.
Think of the power grid for example. Putting power plants on the internet is probably a bad idea. A better idea is to interconnect power plants through multiple ISPs over a SCION ISD. Less expensive than leased lines or MPLS, and more flexible.
Noctua no longer manufactures them, they are now made by YS Tech.