What's new on Services CloudRadio

hugues@cloudrad.io

ANNOUNCEMENT
January 16, 2019

Console & Radio server incident

[14 January 7h00 EST]There is an issue with the console at this and the service on 1 service location (Roubaix, Europe) that we are investigating.

[7h30 EST] There was an issue with one of our radio location (Roubaix). All the secondary IP stopped responding on the server. This caused a lot of error on our console which degraded the service for the radio player and console access. We restarted the network interface on Roubaix which resolved the issue with all the secondary IP. Services are going back to normal now. We are still monitoring the situation. More information to come.

[8h03 EST] We are adding monitoring on every secondary IP to have full visibility on this. All radio should be back online, if your radio is still offline, contact help@cloudrad.io

Incident details: Last night at around 1h40 EST, our servers in Roubaix location were doing a critical security update. One of the update caused the network interface to restart. Upon restart, the secondary network interface failed to load its configuration. Because of this, some IP were not accessible until we restarted the interface around 7h15 EST. It impacted multiple radios who became unavailable. This had for impact to cause higher load than usual on our console due to all the failed call. It explains why the console was not accessible (or very slow) for our customer. There are multiple steps we will take to address this issue. First, we added monitoring on secondary IP on all our server so we have visibility in the future if such a problem arise again. We will also improve the way we receive those critical notifications so we are alerted day and night. We are reviewing and improving our internal procedure as well so everyone in our team knows how to react and who to contact in such an event. We will also add more resource to our console so it can handle the load better in those type of situation.

We apologize for the issue again, we take your radio uptime very seriously. If you have any question, do not hesitate to contact us at help@cloudrad.io

[16 January 13h10 EST] Update: The issue happened again last night. While we responded quickly to avoid extended downtime, we are still investigating the root cause of this. I will update shortly on this.

[16h00 EST] After a complete review, it seems like automatic updates were causing the issue. Both day (14 and 16) there was a systemd update. We have disabled all automatic update at this time and we will perform them manually. We have also updated the firmware of the network interface on affected servers.

IMPROVEMENT
January 13, 2019

Release note

New features:

- Ability to delete a show from the schedule after it have been canceled. - Crossfade have been added between scheduled show. - Improvement of the random play so same track cannot be played twice in a row.

Bug fix:

- Fixed a bug where scheduled show could overlap and created performance problem with Auto DJ process. - Crossfade was not working as specified in Auto DJ settings. Changed the crossfade type to fix the issue. - Fixed a small bug in field requirement when creating new Auto DJ.

emoji negtive reaction emoji neutral reaction emoji positive reaction
ANNOUNCEMENT
January 04, 2019

Service Incident - Player

09h22 EST: There is some customer report of metadata not loading on the player, we are currently looking into it.

09h30 EST: Metadata issue have been resolved. We will come back with more details.

Incident details: It was a certificate that was expired causing the metadata not to load. We have renewed the cert resolving this issue. To prevent this from rehappening we also have added monitoring on all our cert to be notified before they expire.

emoji negtive reaction emoji neutral reaction emoji positive reaction
Create your own newsfeed