5h43 EST Feb 8 - Console and Player is currently unavailable. Will update with more information.
5h48 EST Feb 8 - Everything is back online.
[14 January 7h00 EST]There is an issue with the console at this and the service on 1 service location (Roubaix, Europe) that we are investigating.
[7h30 EST] There was an issue with one of our radio location (Roubaix). All the secondary IP stopped responding on the server. This caused a lot of error on our console which degraded the service for the radio player and console access. We restarted the network interface on Roubaix which resolved the issue with all the secondary IP. Services are going back to normal now. We are still monitoring the situation. More information to come.
[8h03 EST] We are adding monitoring on every secondary IP to have full visibility on this. All radio should be back online, if your radio is still offline, contact email@example.com
Incident details: Last night at around 1h40 EST, our servers in Roubaix location were doing a critical security update. One of the update caused the network interface to restart. Upon restart, the secondary network interface failed to load its configuration. Because of this, some IP were not accessible until we restarted the interface around 7h15 EST. It impacted multiple radios who became unavailable. This had for impact to cause higher load than usual on our console due to all the failed call. It explains why the console was not accessible (or very slow) for our customer. There are multiple steps we will take to address this issue. First, we added monitoring on secondary IP on all our server so we have visibility in the future if such a problem arise again. We will also improve the way we receive those critical notifications so we are alerted day and night. We are reviewing and improving our internal procedure as well so everyone in our team knows how to react and who to contact in such an event. We will also add more resource to our console so it can handle the load better in those type of situation.
We apologize for the issue again, we take your radio uptime very seriously. If you have any question, do not hesitate to contact us at firstname.lastname@example.org
[16 January 13h10 EST] Update: The issue happened again last night. While we responded quickly to avoid extended downtime, we are still investigating the root cause of this. I will update shortly on this.
[16h00 EST] After a complete review, it seems like automatic updates were causing the issue. Both day (14 and 16) there was a systemd update. We have disabled all automatic update at this time and we will perform them manually. We have also updated the firmware of the network interface on affected servers.
09h22 EST: There is some customer report of metadata not loading on the player, we are currently looking into it.
09h30 EST: Metadata issue have been resolved. We will come back with more details.
Incident details: It was a certificate that was expired causing the metadata not to load. We have renewed the cert resolving this issue. To prevent this from rehappening we also have added monitoring on all our cert to be notified before they expire.
13:05 EST: Load average is back to normal. All Airtime are back online. If you are still experiencing issue, please contact our support.
12:45 EST: Load average is coming back down. Some Airtime are coming back online.
12:30 EST: Server now responding to SSH, services slowly coming back online.
12:25 EST: There is an incident with the Airtime server. The server was not responding to SSH command. Reboot initiated.
2018-06-19 Radio monitoring is now working.
2018-06-14 We are aware that the radio monitoring is not working at the moment. We are squashing a few bugs, and we will deploy that feature shortly.
[12:30 EST 2018-06-09]
All issues have been resolved.
[09:30 EST 2018-06-09]
Preview of the player is still not loading in the console. Investigating.
[09:20 EST 2018-06-09]
We have restarted the affected service; all the players are back online. Monitoring the situation.
[09:00 EST 2018-06-09]
We got reports over the last few hours that the player is not working. We are investigating.
The error was caused by a faulty deployment. We made some change to our console but the CDN who deliver the player haven't been deployed, which caused the error. Deploying the CDN fixed the issue.
We took 2 actions to prevent this from happening again. We changed the way our deployment is done, so such thing can no longer occur. We also added external and internal monitoring on the player.