Our customer has over 1,000 users using thin client terminals at multiple offices around the UK and they rely on Internet-based systems for many core business processes.
Intermittently, response times on these systems increased to over 30 seconds.
The key components are as follows:
- Client PCs and Xen servers using Internet Explorer
- Windows Active Directory domain controllers
- Bluecoat proxy servers
All components except physical desktops are based in the two corporate data centres.
We analysed several examples of the problem and found that, while response times for web requests (HTTP/HTTPS) were good, the client PC was sending DNS requests for every element on the page – many of these requests were failing.
We then focused on the DNS performance and analysed several hours of DNS requests on the outside of the proxy servers:
And on the inside of the proxy servers:
This clearly showed that the delays were in the proxy server
The response time problem was a combination of three factors:
- IP Phones were trying to access the Internet but DNS was not fully configured for this
- A logging change on the Proxy servers caused its DNS service to become saturated
- A configuration change in the Proxy Auto-Configuration (PAC) file caused desktop PCs and Xen servers to send superfluous DNS requests which were then affected by the Proxy DNS issue
The desktop support team put in a workaround which stopped the superfluous DNS requests.
The underlying problem was then fixed by the proxy server support team.