A Post-Mortem

Over the course of the degradation, we identified an increase in traffic to several sites in the LycanHost client network.

In order to provide clarity and be transparent about our practices in mitigation, we have provided this post-mortem of the events as a breakdown of our resolution to protect our customers.

The routing and performance of one of our client sites, was impacted to a degree that the main pages of the site were redirecting to malicious URLs and providing content we would not normally permit in our network. During this we noted a significant uptick in requests performed to one specific site, at which time we enabled our attack mitigation protocols.

During the attack, we noted a peak of 56,000 requests being sent to the sites domain name, with significant portions of traffic from two countries notably visible in our analytics. After identifying these countries, we enabled steps to prevent IP addresses in their regions from accessing our network and providing interstitial challenges to our other global visitors who exhibited suspicious behavior.

After several hours we saw the request flow drop off significantly and we began to release some of the protocols in place, however a further attack was later revealed , where we applied further countermeasures to ensure the site remained online and impact free, the two countries we had identified earlier continued to be present in the connection logs, and were finally prevented access after more concise steps where taken to prevent their access, though we will not reveal the methods employed to stop the flow of traffic.

24 hours later, we were then able to confirm the traffic has returned to normal levels. But we noted a global increase in traffic of 20% across our client facing network, and our CloudFlare reporting indicated that around 40% of the traffic received was not cached.

We have taken several steps to prevent this happening again and have detailed them below.

• Firstly, the optimization of sites that are hosted within the network, ensuring that files are secure, permissions are correctly configured and that sensitive data is not web-accessible in any form.
• Ensuring all sites regardless of content are updated to the latest revisions of software if available, that any obsolete software is identified and we will work with our customers to review these and migrate to alternative software where appropriate.
• We have also conducted a full review of security across our network, and reset any tokens, login credentials or other access data out of an abundance of caution.
• We have enforced more coverage on our larger client sites for caching purposes, ensuring that we can keep our client sites online through any future incidents.

Finally, we wish to update our customers that we have implemented an off-site monitoring solution that actively checks several endpoints within our network for performance and uptime, in our current configuration we will be able to detect and respond to outages and performance degradation within 3 minutes of its occurrence and provide updates seamlessly via our StatusPage deployment, The reason for this is two-fold. While off-site, it ensures it remains online during any outage and allows us to direct all traffic to our incident site for fast and efficient dissemination of information to affected parties.

LycanHost deeply regrets any downtime our clients experience and remains committed to taking any available steps to keep our clients online.

After further review of performance data over the past 24 hours, we are confident that performance has returned to normal levels and that routing is no longer impacted.

After further investigation, we have identified several root causes of the performance and routing degradation on this site.

We have taken the appropriate steps to mitigate these causes going forward, ensuring all security updates have been deployed and out of an abundance of caution have reset all tokens/accounts and login features to secure states

We are continuing to see elevated traffic levels for sites throughout the service however are monitoring these to ensure our service remains unimpeded

Visitors may find their access challenged by our DDoS provider CloudFlare if it detects suspicious behavior, this should not impact the largest majority of visitors, however we feel it prudent to advise that these steps have been taken to ensure safe and consistent service,

We will continue to monitor the traffic into and out of the site until we are reasonably confident that no further action is necessary.

We have now located the cause of the routing issue and are working to resolve this as quickly as is possible.

We have also identified several factors that appear related to this issue and are investigating those further.

Users will continue to see mitigation screens in place across the network for the interim period.

We are aware that some users accessing this site are being directed to hijacked webpages and shown content that is not hosted within the LycanHost network.

Steps have been taken to direct traffic to the correct servers and we are looking into analytics that are indicating higher than normal traffic levels for the site.

Users accessing will be presented with a browser challenge page from our DDoS mitigation service.