A few weeks ago I installed a the first firewall in the brand new “yet to be named” datacenter that we are building out for The FreeBSD Project. I wrote a quick PF ruleset to get things going and make sure we could ssh back in before I left the datacenter. I started it and immediately lost my ssh from the outside and could not get back in. I flushed the rules and decided it would have to wait for later because I had a plane to catch.
At BSDCan I finally got some time to look at the rules and figure out what was going wrong. A handy thing to do is setup a cronjob to automatically flush the rules every 5 minutes while you work on them to prevent locking yourself out. After setting up the cronjob I noticed that I could get in just fine if I explicitly added the IP to the rule I was using instead of using the following:
pass in on $ext_if proto tcp from any to ($ext_if) port ssh keep state
So I started looking at the interfaces and I noticed I had accidentally put the IPs on the wrong interfaces. I had the CARP IP on the main interface and the main IP on the CARP interface. Once I noticed this I was able to move the IPs to where they should be and everything worked like it should.