Have you ever wondered how far virtualization goes? How much flexibility can you implement in your environment and what is the minimum amount of hardware you still require?
These questions will hopefully be answered if you read on.
Once upon a time.. There was server virtualization, desktop virtualization, storage virtualization and cloud computing. The Software-Defined Data Center was born! Even networking could be virtualized and provided you with unimaginable flexibility in your infrastructure.
The scope for this article will basically cover the networking part and specifically the edge part of your infrastructure like public internet and DMZs. My homelab will be the subject, but the architectures that will be showcased here, can be implemented in production for sure.
In a nutshell, my homelab is configured like the diagram below.
A little explanation could be handy: The Internet is giving me everything to share my articles with you over a 180 Mbit down- and 18 Mbit up-link. The Internet Modem is connected with 1 Gigabit to a self-built hardware appliance which is running Sophos UTM and has two physical interfaces. It does however have multiple virtual interfaces to connect different logical networks to each other. The red lines indicate unfiltered, “dirty” internet, while the blue lines indicate filtered, safe traffic. The appliance is connected to a Gigabit Switch, which connects all my home devices together like my NAS and BA-ESX (which is my Bad-Ass ESXi host).
The hardware I used to build my UTM:
- Asus AT4NM10T-I motherboard with onboard Intel Atom CPU
- Antec ISK 300-150 case
- Kingston KVR800D3S8S6 2GB RAM
- Patriot Torqx 2 PT232GS25SSDR 32GB SSD
- Intel Gigabit CT Desktop Adapter NIC
So, now you wonder how far does virtualization go? As I’m already using techniques like virtual interfaces, VLANs, server and desktop virtualization, what remains? Well, you will always need some sort of physical connection to the outside world, or at least a network connection. The place where you plug it in, can be about anything: directly in a physical server, a firewall, router, or switch. The minimum amount of hardware required depends on your environment of course, but in case of a traditional setup, you need the following hardware:
- Physical up-link to a network (could be internet)
- A security device that manages your internal and external networks
- Physical switch for connecting your servers
- Physical servers for implementing a hypervisor
My case here is to create more flexibility and push virtualization to the MAX. As you can see in the list above, the only item that does not contain the word physical, is the security device. I decided to replace my physical security device with a virtual, high-available security device and see how this performs.
What is Sophos UTM?
Before I get to the details, I would like to share some information about the security solution I am using (and have been using for the past 6 years in both home and business environments): Sophos UTM (previously Astaro UTM). This piece of software is really the Swiss Army knife among networking services. Whether you need a DHCP server, NTP server or would like to provide your users and engineers a way to work remote, Sophos UTM is the way!
A summary of the major features I think the product has:
- Easy to use web-based management interface
- User Portal for accessing applications remotely through a browser
- Integration with directory services like Active Directory, RADIUS and LDAP
- Network Services
- NTP
- DHCP
- DNS
- Network Protection
- Firewall
- NAT
- Load Balancing
- Intrusion Prevention
- Wireless Protection (Wireless LAN Controller)
- Web Protection (Proxy)
- Endpoint Protection (Antivirus for clients and servers)
- E-mail Protection
- SMTP Gateway (Including support for multiple profiles in cases you have multiple domains and multiple e-mail servers)
- Anti-Spam (Including Quarantine Manager)
- Anti-Virus
- Web Server Protection (Reverse Proxy)
Sophos delivers both hardware appliances, but it’s also possible to purchase a software license and enables you to install their software in a virtual machine, or on custom-built hardware.
To give you an idea about the user-friendly and intuitive interface, you can see the screenshots below.
-
- UTM Dashboard
-
- Firewall Configuration, easy, fast and versatile
-
- Logging and reporting
To the MAX!
Now enough talking! Let’s see how the network diagram looks after removing the physical security device and moving network security to the virtualization layer.
Erhm, ok.. So where did security go? My dirty traffic is now flowing directly into my switch!
Well, let’s zoom in a bit:
I have created a seperate VLAN on my switch, especially for internet. The Internet Modem is plugged into an access port on the switch, that resides in this internet VLAN. Only devices that are connected to this VLAN will be able to access dirty traffic, so there is my segmentation. The trunk that I was already using to provide my Bad-Ass ESXi host with the ability to use all VLANs on my switch, now has an extra port group on the vSphere Distributed Switch that utilized the internet VLAN.
The only two virtual machines that are connected to this internet VLAN, are my two UTM appliances. Security is back!
Besides removing the need for a physical security device, I moved network security to the virtualization layer, giving me more flexibility and a lower possibility that physical failures impact the availability (that is, if I would have more ESXi hosts, but you get my point). The reason for deploying two UTM appliances, is that they can be put into an active-standby or active-active cluster. I applied the active-standby configuration by adding an extra virtual NIC to the appliances and connect those NICs to a dedicated heartbeat VLAN.
This setup would never be possible with my physical box, as it only has two physical interfaces and I need one for the internet uplink, one for internal virtual interfaces and finally a dedicated interface for the cluster configuration.
Performance
Now how does this perform? Really really well I must say! It even works better than I predicted.
When one of two UTMs fail, the other UTM takes over within seconds. Now one of the challenges was the internet uplink. Would the Internet Modem assign the WAN IP address to the new active UTM? Well yes, but only after disabling HA link monitoring on the WAN and internal interfaces inside the UTM. Only the heartbeat NIC should be enabled for HA link monitoring. Without this adjustment, I was getting unexpected results. Now, when one UTM fails, internet access is back online in about 5 seconds.
The way I configured this from the VMware point-of-view is displayed in the screenshots below.
-
- vSphere Distributed Port Group configuration for the virtual trunk
-
- vSphere Distributed Switch configuration
-
- Virtual Machine configuration
And the bandwidth throughput? Awesome!
I would recommend everyone trying out UTM some time. I never had any regrets implementing this solution anywhere. Be sure to check out their website at sophos.com or the UTM page.
The solution described in this article can be used with many different products. Think about VMware vCloud Networking & Security, NSX or even Microsoft Hyper-V with third-party virtual firewall appliances.
Hopefully this helps you think of different networking and virtualization strategies to provide the best possible solution for your customer (or homelab! ;-)).
SnowVM 
Hi,
I was wondering if your modem performs the connection to the Internet or if your modem is in bridge mode and your Sophos UTM is performing the connection?
Cheers,
Chris
LikeLiked by 1 person
Hi there Chris,
My modem is in bridge mode, so the public IP is attached to the virtual UTM.
Hope this answers your question.
René
LikeLike
Excellent post! This is exactly what I’m looking to do. Thank you!
LikeLiked by 1 person
Hello, this is great and something that I would like to do too! My basic set up is:
modem -> FW -> Cisco router -> Cisco switch -> ESXi host
I have Sophos installed and ready to go, I just have a hard time wrapping my head around how the VLANs will work. Would I create a vlan on all devices or just on the switch? If I wanted to use Sophos for URL filtering only I would set it up with WCCP but I would like this to essentially be “in-line” with all traffic so I can use all/most of the features.
Any thoughts? Thanks a ton!
LikeLiked by 1 person
Hey there Paul! If your goal is to replace your FW and router by the Sophos UTM, you should at least create an extra VLAN for internet traffic, as you will plug-in the modem right into your switch. But if you want to add the UTM just for URL filtering, I would suggest placing it in the same VLAN as where your Cisco router is in and set the default gateway of the UTM to your router. No need to create extra VLANs in this scenario. Does this answer your question?
LikeLike
Hello,
very nice setup, I’ve being using the utm for just over a year now and I love it.
I’m in the process of introducing vlans and a managed Cisco switch into my home lab setup.
I also have a wireless access point and guest network in my setup as well.
I’m wondering if it’s possible for you to give me some guidance with the setup based on how you did yours? any help you can provide would be very much appreciated.
LikeLiked by 1 person
Hi Donovan, currently lacking a good connection at my holiday address. Will get back to you ASAP.
LikeLike
Hey there Donovan, you could implement several scenarios depending on your requirements. What would you like to achieve? Please reach out to me on Twitter or LinkedIn, would like helping you getting things done!
LikeLike
In your setup, does that mean users and devices are in the same VLAN as the Sophos UTM LAN? i.e. when a user/device connects to the Internet, its gateway or default route always point to Sophos UTM. Or in another word, Sophos UTM isn’t aware of the VLAN you setup in the switch?
I currently have separate VLANs for users/devices/servers created in the ‘switch’ in your example. The Sophos UTM sits in the ‘servers’ VLAN. How would you protect users/devices VLAN?
LikeLike
Hi Anthony,
In my setup, I had several VLAN interfaces that were being used for client, server, test and management traffic. All devices in those VLANs are however using the UTM as default gateway.
Sophos UTM isn’t aware of the VLANs in the switch, but UTM does tag traffic with the corresponding VLAN tag that is created in the switch. You are the administrator and have to make sure those two are in sync in terms of VLAN configuration.
I would suggest connecting your UTM to a trunk port on your switch and creating VLAN interfaces on your UTM that are using your existing tags. That way you can reach the UTM from your user/devices VLANs.
Let me know if this helps!
LikeLike
Thanks. I just realised you’re using a switch so UTM is handling all the routing for each of the VLAN networks. Is that correct? I am using a router in place of the switch in your diagram, so my router is handling all the routing between VLANs. Works well in your scenario as the default gateway of each of the VLAN networks just points to the UTM. In my case, I have to setup fancy rules in the router to route certain traffic (like web traffic) to the UTM. It doesn’t seem like a good way to do it. In my case I can’t just poing the default gateway to the UTM (Tried that, but end up not being able to talk to the other VLAN networks defined in the router, only the Internet via the UTM’s WAN interface).
Not sure if you came across my scenario where routing isn’t done by UTM itself. Doesn’t seem to be designed for this scenario.
LikeLike
Hi Anthony. Yes, the UTM is handling all routing. I think in your case, you’re going to need static (or dynamic) routing to get things to work. It is possible though! In the internet case, you should set up routing between the UTM and network behind your router. Next you will need to configure SNAT to translate traffic coming from behind your router on the WAN interface. We could figure this out offline if you like? Send me a DM on LinkedIn or Twitter.
LikeLike
Very nice and well explained everything 🙂
Could you recommend some good HW for the Sophos UTM please ?
I am currently testing the Sophos UTM in my “VMware Workstation 12” to make sure, everything work well with physical stuff 🙂
I am currently struggling setting up the “Web filtering”, i just can’t get it work, so im using the Application Filtering instead.
LikeLike
Hi Venka, there’s not really a “good” and “bad” hardware configuration, it all depends on what you need. The hardware I used consisted of a mini-ITX motherboard with two NICs, a SSD and onboard CPU. I’ve added the specs to the article:
Asus AT4NM10T-I motherboard with onboard Intel Atom CPU
Antec ISK 300-150 case
Kingston KVR800D3S8S6 2GB RAM
Patriot Torqx 2 PT232GS25SSDR 32GB SSD
Intel Gigabit CT Desktop Adapter NIC
In regards to the web filtering part, I’ve not really used this function yet. Have you figured it out yet?
LikeLike
Hi,
yes i figured it out, how it works 🙂
Thanks for the info 🙂
LikeLike
Hi this is a great set up that im looking at doing. I have a few hosts and vcenter set up in my lab. As you set up is for just one host, how would you set this up with multi hosts in vwmare? I have a managed switch, just trying to figure out how to configure this with multiple hosts and storage so physical nics can talk to vm’s but all traffic go through Sophos utm for routing and firewall.
Thanks in advance
LikeLike
In case you have multiple hosts, there’s just two things that are different: You’ve got a higher availability as your UTM will be able to failover between hosts (if you’ve got a HA cluster set-up). And traffic between VLANs and traffic going out on the internet will all go through a single host (the host where the UTM appliance is running). As long as you have the internet and VM VLANs available on the right physical ports of your hosts, the UTM can basically run everywhere. Does this answer your question?
LikeLike
You say “The only two virtual machines that are connected to this internet VLAN, are my two UTM appliances. Security is back!”
Please can you give more detail as to how they are connected. for example, the Internet VLAN may be 2, so you have created 2 vNics for each VM that Connect to the WAN (switch port on which the internet modem is connected) and specificed the VLAN ID on the vNic as 2 also. Or did to leave the vNic VLAN ID alone and specify the VLAN on a the Sophos Interface called “WAN”?
By giving more detail you are helping those out just starting in the world on VLANs and VMs…
Thanks
LikeLike
Hi Andrew, so the internet VLAN is mapped to the switchport that’s connected to my modem. The trunks going to my ESXi host carry ALL VLANs. However, I only configure this VLAN on the UTM to make sure external traffic is filtered by them.
Each UTM has three vNICs/interfaces. The WAN interface on the UTM is untagged, so the tagging needs to be done on the vSphere Distributed Switch level.
The internal interface uses virtual interfaces so I can create multiple interfaces using several VLAN tags, this required a vSphere port group of type “trunk”.
And the third interface is used for HA between both UTM’s, als untagged in the UTM like the WAN interface, and tagged in the vSphere switch (distributed or standard doesn’t make a difference).
Does this answer your question?
LikeLike
Yes it does, thanks. I’ve had my network up and running for a few months now, and all is well. I’ve even had to deploy a bit of traffic bandwidth management to stop the kids from maxing out the connection, youtube, Minecraft, stuff like that.
LikeLike
Awesome! Great hearing back from you and good that the solution is working well =)
LikeLike
Also I find it hard to believe you dont have any form of WiFi AP on your network…
If you do where is it in relation to your diagram?
Thanks
LikeLike
Hi Andrew. Of course I’ve got WiFi! But it wasn’t related to the topic I was trying to describe here.
The AP is connected to the gigabit switch and is mapped to the “client” VLAN I have for all client devices like laptops, printers, phones etc. All my networks are behind the UTM.
LikeLike
Also what is the default gateway IP of your switch? is it blank or the WAN gateway…..
LikeLike
Hi again Andrew. The default gateway of the switch is set to the UTM for management purposes. If you’d like to manage your switch from, let’s say a remote VPN connection, you need L3 to get to it and set the default gateway. The configuration shown in this article is compatible with both L2 and L3 managed switches.
LikeLike
Great article, I read this awhile back and just never got around to trying this out but I think I’m going to give it a try myself. I am currently on pfSense but I have tried Sophos (UTM and the newer XM?) and will try a few others. One topic that I haven’t been able to find clarity on (i’m not a networking wiz) is vlan management. Some articles have you create the VLANs in the UTM software and others have you create them on your switch. Can you explain why you went with building them on the UTM vs the Switch (just my uneducated guess but I would think it would be faster done on the switch)? Unless it’s not a L3 switch maybe?
I was going to make a 4-5 port VLAN for my internet so that I can connect various UTM/Firewalls to the internet for testing inline.
LikeLike
Hi Rafael,
All VLANs need to exist on the switch(es) you are connecting to your router (UTM for example). They are not created on the router.
The thing you are probably referring to, is the vlan interface, or default gateway. When you have a L3 switch, you could create an interface inside a VLAN which lives on the switch (as the switch is L3, it understands IP routing). Alternatively, you could create interfaces on a router, like the UTM.
Difference is that traffic for a given subnet uses the switch as default gateway, or the UTM. Depending on your requirements, you could choose where to put the default gateway. You could even choose to put the default gateway on your L3 switch, and configure routing on the switch to forward traffic to the UTM. Having all default gateways on the same device however, makes the whole a lot less complex.
Regarding the 4-5 port internet VLAN, this could only work if there’s only one device “occupying” your external WAN address. It’s not possible to have multiple devices occupy it. You’re probably going to need multiple IP addresses and a specific configuration on your modem. If you’re starting to play around with this please keep me posted!
Good luck,
René
LikeLike
I did not realize I never replied and thanked you. Thanks! Your post cleared things up for me. I did manage to do a basic version of your setup and it worked great. My problem (home lab for noob setup) was that I had everything running on my main supermicro ESXi server and I would have to bring down the connectivity for the whole house anytime I need to power down or reboot the main server. The WAF (let alone kids) factor was not great. So i ended up getting a Dell 7010SFF running an i5. I didn’t think it would work but I tried anyway and ESXi 6.5.0a installed without a single hitch (I did add in a quad port intel card) and everything was recognized. So now I have two ESXi servers with the 7010 running just my (now) OPNsense VM. I don’t have HA but I would like to make a “copy” of that VM and re-create it so that if I ever mess up the primary VM I can boot up the “gold” copy and still get connected. I’m just not that great with ESXi networking and everytime I install ESXi I’m not quite sure if I have the networking setup properly so this would be a safety net.
LikeLike
Hello, i have the same architecture of network, but i have an issue to configure the network adatpter. any one can help me please ?
LikeLike
Hi Khaled, I’m afraid I need a bit more information around your issue. Which network adapter are you trying to configure and what settings are you applying?
LikeLike
Hey Rene hope all is well..
sent you a linkedin request.. I would like to follow-up on this topic and could use a bit of advise.
things got very busy so I’m just now trying to get my setup complete plus I’m added a few more layers.
Thank you,
LikeLike
Hi Donovan,
I remember accepting your request, but I now see I haven’t replied to your comment yet. Replying to your new comment now!
LikeLike
Hi Rene hope all is well,
Just wondering if this post is still alive?
I could use some advise setting up Sophos utm, Pfsense, ESXI 6.5 and 4 managed switches.
Thx in advance,
LikeLike
Hi Donovan,
Although getting a bit old, it still covers the basics and I don’t think UTM changed their solution that much. I must say I don’t have Sophos UTM running anymore in my lab, as it’s all running on VMware NSX-based components now. And no experience with pfSense yet, but shoot! I might be able to help you anyway 🙂
René
LikeLike
Hi Rene, thx for the reply.
I sent you a message on LinkedIn with a network diagram for your ref. I’m looking for your advise on best configuration for performance, security and inter-vlan routing with 4 managed switches, 2 firewalls, AP, NAS, ESXI6.5 and 1 public IP.
Thx in advance.
LikeLike