This article is a work in progress; still updating it with more info as of July 2016.
In May 2016 I got my hands on a few Arista switches for the first time; a few 7280SE-68-R (7280SE with (48) 10gig SFP+ ports, two 100gig QSFP and reverse back-to-front airflow), and a pair of 7010T 48xGigE+4x10gig switches. All have been flashed to 4.15.6M EOS. This article is to summarize my initial thoughts coming from a Cisco and Brocade background. I’ll categorize:
- Obviously the price per port; far better, with far more features, than Cisco/Brocade competitive offerings.
- What made me look at Arista to begin with were the huge table sizes supported for MAC, ARP and v6 ND. If you are deploying a cloud infrastructure, web hosting aggregation, ISP aggregation, etc. where you have a LOT of downstream IP/IPv6 addresses and don’t feel like buying a ton of hardware just for the sake of buying a ton of hardware, then Arista is exactly what you’re looking for. I used to use a lot of Brocade FastIron SX’s for layer 2 and low end (static routing, minor ospf/ospfv3) layer 3, because even those venerable old switches would let me do 64k ARP, 32k MAC and 131k IPv6 ND. To get similar numbers out of Cisco from a layer 3 switch, you’re looking at spending an honest eight to ten times more than Arista because you have to step up to the Nexus 7k series; that is just completely ridiculous. Unfortunately Brocade no longer plays well in this space because the only thing they have that can do those kinds of numbers, or better, these days is the MLXe, and while it makes an awesome router, it makes for a very expensive layer 3 switch if that’s all you need.
- I love the ‘routing-context’ command. If you’re using VRF’s, it lets you place yourself into the relevant VRF for commands you intend to issue, so you don’t need to go playing around with dumb ‘ip tftp source-interface’ type stuff just to copy a config out the interface you want, or added arguments just to ping a bunch of stuff.
- VARP – Arista’s protocol allowing each participating switch to route inbound packets for a virtual IP. Each participating switch responds to ARP requests for the IP’s in question, and as they receive packets for those IP’s, they route them. Unlike VRRP or HSRP, there’s no ‘primary’ or ‘owner’ of the virtual IP where all the packets go, and there’s no need for all the devices to monitor the VIP and try to figure out if they need to take over for it. If you combine this with Arista’s MLAG multi-chassis LACP trunking, you can come very close to eliminating spanning tree and making use of all links via port channels with packets getting routed regardless of which device they hit. Obviously it does make troubleshooting a little bit more complicated, for example, if you have a heavy mesh and one optic of one port channel is starting to fail; so get your network monitoring in order and watch for interface errors.
- MLAG – Arista’s multi-chassis LACP (think port channel / virtual port channel for Cisco folks). Even their lower end switches support this and it works great. Connect your equipment, a few minor config changes and you can have redundancy across multiple chassis with no spanning tree and every link forwarding. I’ve been using it with Brocade’s Multi-Chassis Trunk (MCT) and LAG’s on the other side between a pair of Arista 7280’s and a pair of Brocade MLXe’s without issue; each side is cross connected to the other side, so now I’ve got 40 Gbit/sec bandwidth (4x10gig in the LAG) between the two sides, no spanning tree, no routing protocol hiccups as links are added/removed from service. If you go down this path (no pun intended) and are using OSPF/OSPFv3, make sure you familiarize yourself with all the relevant OSPF cost-related settings, including your ‘auto-cost reference-bandwidth’ setting. When you start doing trunks, it’s very easy to accidentally cause a routing problem if you aren’t explicitly setting the bandwidth (Brocade side) to get a consistent link cost.One downside is that obviously what happens at layer 2 is not in sync with layer 3 routing protocols, so your MLAG may send a packet down a specific link that causes it to end up on switch B when its destination is a next hop IP present on switch A. If you’re using static routes and VARP, you eliminate that issue. In any case, I’m willing to have that less than ideal situation in return for the added redundancy, and don’t really care about the bandwidth since I’ve got 2×100 Gbit/sec QSFP100’s between the two 7280SE’s.
- Much easier LAG configuration than Brocade. One line per physical interface, then you get a port channel interface as a result, do all your config at the port channel level. Brocade, does two lines of config for Arista’s one at the interface level (lag id and type), but then you don’t get a new interface, you just have one physical selected as primary and have to do all your config there. Get it wrong and it complains about trying to change an interface that isn’t primary, so then you waste time doing show int or show link-aggregate to figure out which one you need to be looking at, etc. Doing multi-chassis lag? One extra command adding the mlag id to the port channel interface; couldn’t be easier.
Things that REALLY annoyed me:
- I can’t stand that there are users which exist outside of the running config, and even worse, I find get really pissed there are user names which cannot be removed. For example, by default, there’s an ‘admin’ account who has no password and unrestricted access via console port, you also can’t delete it:
#no user admin % You cannot delete the 'admin' account.
If you “show run” you won’t see it. If you add more users to the config, who do show in show run, you still won’t see it. You have to show user-account to even know it’s there:
#show user-account user: admin role: network-admin privilege level: 1
So really all you can do at this point is set a strong password for it if you want to keep unauthorized people from console’ing into your switches if you don’t have 100% physical access control against them. At the very least, the admin user does show up in the running config once it has a password set.
- Per https://eos.arista.com/securing-eos-cli/ you must enable non-default strong encryption, enable timeouts for console and ssh (none by default), and LLDP is enabled by default. I don’t like when the defaults are less secure; you should default to the strongest settings and let customers knowingly downgrade their security if necessary.
- Per above, for maximum security, LLDP is on by default and must be disabled via:
no lldp run
- With Cisco and Brocade, you do not have to define connected networks at the ‘router ospf’ level if you’re redistributing connected, you simply define ‘ip ospf area 0’ at the interface level and suddenly that interface is participating in ospf and its network is being distributed. Arista is really annoying in this area; you have to define the network that a given interface is contained within at the ‘router ospf’ level before it even participates in OSPF. More on this in the wishlist section.
- I really don’t like MSTP being the default. Or if we need to quantify, how about we say I don’t like MSTP being the default on 1U fixed config switches. It’s simply a nightmare to maintain properly for small networks, and by small I mean even a multi-building campus where you have a handful of VLANs, where there’s no real harm in running rapid PVST. Sure, if you need to run MST because you have Cisco PVST+ stuff on each side you don’t want it interacting with, or are deploying a metro network, etc., that’s great, but then you probably already know all about MST and are ready. I think most people will get themselves into more trouble than good with MST, for no real gain (from their perspective).
Things that I found mildly annoying:
- There is an unremovable ‘transceiver qsftp default-mode 4x10G’ command in the running config even if your switch has no QSFP+ ports. Not a big deal, but if you have a mix of 7280’s where some are the QSFP+ 40gig models and some are the QSFP100 100gig models, you may think you’re on the wrong switch if you notice that.
- I find it mildly annoying that on a switch with QSFP ports operating in single port mode, i.e. you’re using the QSFP port at either 40 or 100 Gbit for one connector, you still have to specify it as interface 49/1 instead of just 49. The reason this is done is obviously because a QSFP+ port can operate as 40Gbit or 4x10Gbit via breakout cable, and QSFP100 can operate as 100 Gbit or 4x25Gbit copper via breakout cable, but if neither of those are inserted, no reason to make me type in the extra /1 on a fixed config switch.
- Jumbo frames are enabled by deafult and cant’ be disabled. I’m okay with that, but I think a better manner of handling this would be to have a jumbo config directive present by default, even if you can’t remove it, to make it obvious.
- Arista switches will think a 10gig interface is up if it’s plugged into a remote GigE SFP on the other end. I ran into this when connecting some Fortinet firewalls to Arista 7280SE’s. The Fortinet firewall came with 8+8 gigE copper/fiber ports, and two 10gig. They included two transceivers in the box. Since there were only two optics, and only two 10gig ports, I mistakenly assumed they were 10gig optics; the SFP’s themselves did not state speed, just a model number. I plug them in, cable them to the 7280’s, the Arista side immediately gets a green light, and at the CLI level, showed:
Ethernet23 is up, line protocol is up (connected) Hardware is Ethernet, address is 0000.0000.xxxx (bia 0000.0000.xxxx) Description: fw1_e17 Member of Port-Channel23 Ethernet MTU 9214 bytes , BW 10000000 kbit Full-duplex, 10Gb/s, auto negotiation: off, uni-link: disabled
Based on that, I made the mistaken assumption it was a config issue on the Fortinet side causing my Arista port to be ‘up’ (and at 10gig no less) while the Fortinet side showed it as down.
- I don’t like the fact that management services may not be disabled like you think they are if you use VRF’s, or even worse, may appear differently from one to the next if the default for the given service differs. For example, if you have a management VRF named ‘MGMT’ and you want to put the SSH access to the switch in that VRF, you might do this and think you’re good to go, like how it works on Brocade:
management ssh vrf MGMT !
In reality, you’ll have just enabled SSH on the management ‘MGMT’ VRF interface and left it enabled on every other interface. You need to make it look like this if you want SSH only in the MGMT VRF:
management ssh shutdown vrf MGMT no shutdown !
Worse, if you want telnet disabled everywhere, it will look like this:
management telnet vrf MGMT shutdown !
due to the fact that telnet is already disabled by default. So if you were just glancing over your config, you’d think oh, I forgot to disable telnet on the default VRF, when in reality, it is disabled, even though it doesn’t state that in the config the way disabling SSH in the default VRF does…
- I don’t like that enabling or disabling routing protocols for IPv4 or IPv6 differs in config appearance if you use VRF’s. For example, again, assuming you have a VRF named MGMT, your config will look like this if you want IPv4 and IPv6 routing in the default VRF and not in the MGMT VRF:
ip routing no ip routing vrf MGMT ! ipv6 unicast-routing
You’ll notice an absence of a ‘no ipv6 unicast-routing vrf MGMT’ statement. IPv6 routing in VRF’s is apparently disabled by default, differing from IPv4 routing, so typing it doesn’t result in anything showing up in the config.
- Although the global config command ‘ip ospf name-lookup’ shows as applying to nearly every ‘show ip ospf <…>’ command in the 4.15.3F manual, it does not actually seem to apply to anything other than the ‘show ip ospf neighbor’ command. This is unfortunate given how convenient it is to see peer router names instead of IP’s.
- I don’t like that Arista’s OSPFv2 implementation does not allow you to configure authentication at the area level. I do like the fact that they require you to use authentication if using OSPFv2 on VLAN interfaces, but they don’t require it on routed interfaces. I like to have area authentication defined as a safe guard in case passive interface default gets turned off, or if someone simply enables OSPF on an interface you don’t want it going out, without defining authentication.
- There’s a very cool command “ip ospf name-lookup” which tells the switch to resolve DNS names for IP’s found in the output of various “show ip ospf….” commands like show ip ospf neighbor. That’s very handy to immediately know what peers are connected when you see names instead of IP’s. Unfortunately, Arista only implemented this command for OSPFv2, so if you’re running IPv6 and OSPFv3, it has no effect, and there’s not an equivalent “ipv6 ospf name-lookup”. :cry:
- I wish that Arista would emulate Cisco and Brocade in allowing you to define the OSPF area of an interface at the interface level. Oddly, they have done this for IPv6 in their OSPFv3 implementation. For example, on Cisco/Brocade, you can:
interface Vlan500 ip address 192.0.2.1 ip ospf 65535 area 0
and the relevant network will be included in the advertised networks without any further work if you’re running ‘redistribute connected’. With Arista, you not only have to define the network at the ‘router ospf’ level, but until you do that, ‘show ip ospf interface’ won’t even show the relevant interface as being part of OSPF, leaving you to wonder where you made a mistake. On Cisco/Brocade, ‘show ip ospf interface’ will show you all the interfaces with addresses defined, and their state in relation to OSPF, even if that state happens to be shutdown. It makes it much easier to quickly identify where the issue is if an interface you expect to be participating isn’t.
- There’s a bug in the CLI in which it does not reveal that “ipv6 ospf bfd” is a valid command, but it is.