17h30 EST Everything is back to normal. We no longer detect any issue. We will mark this as solved for now. If there is anything, as always, you can contact us (help at cloudrad.io) and we will be happy to help.
14h00 EST Seems to be back, still monitoring the situation.
13h35 EST Just confirmed with the data center, they are aware of the situation. Since it's pretty major, I'm expecting (and hoping) for a quick fix.
13h10 EST Network issue again between data center. Monitoring, but during that time the console and player will be hard to reach.
8h55 EST System are now back. It was a network issue between our backend system, we are monitoring the situation.
8h40 EST we are seeing error again on the backend. More to follow.
8h30 EST we have identified the problem being a server on the CDN that wasn't responding properly to the call of our backend.
8h00 EST today the player was inaccessible to all radio and the console was responding slowly.
11h42 Services are now back online. We can confirm that it was a network issue at the datacenter. We will continue to monitor the situation.
11h30 We are seeing other service degradation, our console and web player are slow to respond. We are seeing a huge increase in failed request over our API. We continue to monitor the situation.
11h15 There is an network incident in Europe right now. Most of our services in this area are unreachable. The provider is already aware of the situation, we expect a quick resolution since of the scale of this issue.
22h10 EST - Monitoring have been re-enabled on the auto DJ.
22h00 EST - The issue has been resolved. It was a router issue at our provider that was affecting the traffic on certain IP range and the IP of our master server was affected.
19h00 EST - This issue is causing a lot of false down notification from our monitoring system. We have disabled the monitoring for now.
17h30 EST - We are still investigating this issue.
16h30 EST - There is some intermittent issue with the CDN. While the end point is working for most radio, the master fails to answer to some request.
06h45 EST - Server is back online. There was a miss configured service on the server which caused a kernel panic. The server was rebooted and is now back online. We will investigate further.
06h30 EST - A server in Salt Lake City is currently down, investigating.