WirelessPhreak.com

travel, science, technology, and all other geeky things
Follow Me

Fun F5 Troubleshooting 

Test your HTTP keep alive from the F5 CLI:
Using curl:
     curl -vvv -H "Host: domain.com" -H "Connection: Close" -H "User-Agent:" -H "Accept:" serverip:port/uripath.html

Using Telnet:
telnet serverip port
Then copy the first half of your keep alive
     GET /uripath.html

From the above listed commands you should see exactly what the F5 is receiving  when it sends a keep alive.  From the returned http request you can determine the best data to use for a receive string.




So F5 license has always been kind of funky. I am not saying it's bad but I've just always wondered why the auto license update didn't work. Then recently we licensed ASM and again had to perform the manual license process, it went all well as it always had but we were not getting ASM signature updates?

So it was time to dive into the F5 and start troubleshooting. The first thing was to confirm that the F5 could resolve the DNS name for the service updates... Check!

Next, you need to check the routing, there are two routing tables the LTM table and the sys management routing table. The LTM routing table had a default route that was not able to access the internet. This was by design since the interface it was attempting to use was in a secure DMZ. This may not affect you if you allow your F5 to the internet directly but we did not have the luxury.

So this is where we were a little confused. One would think the License update and ASM Signature updates would be part of the sys-management routing table, unfortunately, that isn't the case. We discovered that the F5 attempts to reach out were following the LTM default route and not the defined sys management-route default.

One the issue was identified it was easily resolved by adding a route to the F5 services int he sys-management routing table, outlined in italic.

10.0.0.1 Is the internal gateway or next hop in this scenario.
104.219.104.0/21 Is the IP space for F5 services.
The rest should be self-explanatory.
sys management-route F5_Service_Route {

    gateway 10.0.0.1

    network 104.219.104.0/21



[email protected](f5-guest-01)(cfg-sync Changes Pending)(/S1-green-P::Active)(/Common)(tmos)# create sys management-route F5_Service_Route network 104.219.104.0/21 gateway 10.0.0.1

[email protected](f5-guest-01)(cfg-sync Changes Pending)(/S1-green-P::Active)(/Common)(tmos)#

[email protected](f5-guest-01)(cfg-sync Changes Pending)(/S1-green-P::Active)(/Common)(tmos)# list sys management-route                                          


sys management-route F5_Service_Route {

    gateway 10.0.0.1

    network 104.219.104.0/21


}

sys management-route tacacs2 {

    gateway 10.0.0.1

    network10.0.0.10/32

}

sys management-route tacacs1 {

    gateway 10.0.0.1

    network 10.1.1.10/32

}

sys management-route default {

    gateway 10.0.0.1

    network default

}


Lastley this article has alot of good info about setting ASM and attack signitures.
https://api-u.f5.com/support/kb-articles/K8217?pdf
After we added the sys management route were able to perform auto license retrievals and get our ASM signatures update. I hope this helps anyone also stumped with the same issue.





Enjoy!


Load Balancing Multiple Tier Applications
Many multi-tier applications use proxy servers (PS). Proxy servers often provide authentication and/or security to allow clients to connect to web application servers (WebAPP).

In this scenario an F5 virtual servers sits in front of the PS server and is using source IP persistence. This means that based on the clients source IP they will be assigned to the same back-end PS server.

The proxy servers in front of the Web APP virtual server will be using HTTP Insertion cookie to maintain persistence. What this means is the F5 will present a cookie to the clients browser even though it is going through the PS server. So when that client connects to the web application through the PS server it will present that cookie in its header and the F5 will decrypt the cookie and send it to the correct Web APP server in the back-end.

Source IP benefits and short comings
Benefit:
There is no dependency on the client’s device or software such as browser or OS.
It is very easy to troubleshoot, and view current state on the clients’ persistence on the F5 using CLI command.
It is proven persistence method and is very dependable.
Many apps i have configured to use source IP have seen fairly balanced load balancing.

Short comings:
It will not give you as evenly distributed load balancing as a session based persistence. (Sites that have users behind a NAT)
It is not the correct load balancing persistence to set on the web application since the only IP addresses it would see are the PS servers.

HTTP cookie benefits and short comings:
Benefits:
This is a session based load balancing which means it doesn’t load balance each machine or IP, but the actual web session the client has open. When the users closes their browser the cookie is destroyed.
Create a very balanced load between back-end servers.
Cons:
Depended on user machines, browsers, security policies, or OS. If there systems or software does not support encrypted cookies, or their policy is not to accept or store cookies, this would break persistence.
More difficult to troubleshoot and map users back to a machine without actually having the clients cookie.

Solution:
The perfect solution would be to find a session based persistence that does not rely on the client’s machine to provide a cookie. At the very least we will need a session based persistence between the PS servers and the Web APP virtual server since we know source IP address is not a viable solution between the PS servers and the Web APP virtual server.

Recommendation:
Load balance the PS virtual  server using source IP persistence provides good load balancing as long as there is a diverse client pool, and allows for more visibility into troubleshooting specific end point connections. Then use a session based persistence between the PS servers and the Web APP virtual servers that does not involve the client.

F5 recommended using SSL persistence for the Web APP virtual server and sticking with source IP on the PS Virtual server. What this would do is set persistence based on the SSL session ID between the PS servers and the Web APP virtual server. SSL persistence also makes a persistence table entry so we would have similar troubleshooting visibility we would have with source ip persistence.

In addition we will need to make sure least connection is set as the load balancing algorithm for both the PS and Web APP pools. This will insure we are not blindly load balancing but leveraging the F5 visibility into connections to the pool members.

FYI:
Theoretically you can still use the HTTP cookie persistence on your WebAPP server, but as I found out the hard way many  PS servers will combine multiple client HTTP sessions into one back end connection to the WebApp virtual server. The default way F5 handles http cookie persistence is during the creation of an HTTP session it looks at the first HTTP request for the http cookie header, then based on the first cookie will load balance the entire HTTP session to the designated back end HTTP server. Where this breaks is if you are combining multiple HTTP client sessions and the web application servers are not expecting that clients traffic, (web application does not share user session information.) To fix this issue you can enable a OneConnect profile on the WebAPP virtual server. One of the actions that is taken by the OneConnect profile is to look at every HTTP request and load balance based on each request not just the first HTTP request.

If SSL Persistence and Cookie is not an option:
Universal persistence profile allows you to use an irule that will look for the x-forwarded-for header and use a hash value to maintain persistence. F5 warned us there would be some coding on our part to create a custom irule and it that this type of persistence is a very CPU expensive process on the F5. This one is definitely last on the list.

The beginning of the end for traditional cable companies may be upon us. More feasible and easy to use cord-cutting options are becoming main streams like Play Station Vue, Sling TV, and YouTube TV. We mustn't forget who started it all. Netflix and Hulu pioneered the over the top streaming services, then HBO leveraged their hit show Game of Thrones to launch HBO Now. But now the big boys have come to play, Disney and Apple are launching their new streaming services.

Disney+ is quickly going to be the big boy on the block. With their purchase of Star Wars, Marvel and Fox in addition to their existing massive archive there are no other studios in Hollywood with a catalog as deep. It only makes since Disney would want to create their app. If you look at their accusations of not only intellectual properties but also technology (BAMTech, LLC) you can see Disney has been planning this move to Digital for a while. If you have been to Disneyland you understand that Disney is the master of creating a curated and controlled experience. it only makes sense they would want to bring that same experience to their video content online.

Apple TV+... Well, there isn't much to be said yet. Apple is investing in its original content but doesn't have any sort of back catalog of content. So it is yet to be seen if Apple can pull off its streaming service.