Active Incident

Updated a few seconds ago

Rayos database issuesOperational

Incident Status

Operational

Components

Rayos CMS

Locations

US-EAST-1


June 3, 2020 1:29AM CDT
June 3, 2020 6:29AM UTC
[Investigating] Rayos suffered database issues start around 1AM CDT in us-east-1. We're currently investigating the cause, but services have been restored at this time.

June 3, 2020 2:23AM CDT
June 3, 2020 7:23AM UTC
[Monitoring] The issue appears to be related to an automated task that optimizes tables. In this case, it caused a transaction deadlock on the database system. We're looking into changes this process to prevent the deadlocks in the future.

June 3, 2020 2:26AM CDT
June 3, 2020 7:26AM UTC
[Monitoring]

June 3, 2020 12:38PM CDT
June 3, 2020 5:38PM UTC
[Monitoring] At 1pm CDT today, we will be making a software release that should prevent the specific database deadlock from occurring again.

BLOX CMS




Operational

Job Processing




Operational

BLOX Total CMS




Operational

Network




Operational

Database




Operational

Storage System




Operational

Support Call Center




Operational

Field 59




Operational

Rayos CMS




Operational

FTP, Drive, DNS, E-Mail




Operational

E-Editions




Operational

Advertising & Behavioral Targeting




Operational

Business Directory




Operational

Forms




Operational

Subscription




Operational

Analytics & Email Reach




Operational

Calendar




Operational

BLOX CMS /APP




Operational

Sandstone Administration




Operational

OTT & NOW App




Operational

GamesPerform




Operational

Rivista CMS




Operational

History (Last 7 days)

Flex e-Edition access control paywallOperational

Incident Status

Operational


Components

BLOX CMS


Locations

US-MIDWEST-1, US-EAST-1



June 4, 2020 11:19AM CDT
June 4, 2020 4:19PM UTC
[Identified] An issue has been identified with the e-Edition Access Control paywall preventing access to content. An out-of-band release is being prepared to correct the issue.

June 4, 2020 1:05PM CDT
June 4, 2020 6:05PM UTC
[Monitoring] The package will be released today at 1:05PM Central with the fix.

June 4, 2020 3:05PM CDT
June 4, 2020 8:05PM UTC
[Resolved] Flex 3.170.1 has been released and the issue is resolved.
RAYOS - Gravity Forms upgrade issuesPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Rayos CMS


Locations

US-EAST-1, US-MIDWEST-1



June 3, 2020 2:12PM CDT
June 3, 2020 7:12PM UTC
[Identified] The RAYOS 0.12.0 release that was made earlier has an upgrade issue with Gravity Forms. Consumers attempting to submit forms are reporting issues posting. An upgrade script needs to be run but will require making an out-of-band release of 0.12.1 to address the issue. The upgrade script will be deployed and ran as soon as possible today.

June 3, 2020 5:47PM CDT
June 3, 2020 10:47PM UTC
[Resolved] Gravity forms database upgrade script has been run on all sites. Form submissions and capturing of all fields should be functioning once more on all sites.
Rayos 0.12.0 Release 10a centralOperational

Incident Status

Operational


Components

Rayos CMS


Locations

US-EAST-1, US-MIDWEST-1



June 3, 2020 9:48AM CDT
June 3, 2020 2:48PM UTC
[Identified] Rayos 0.12.0 will be released at 10am central, featuring content automation, data privacy controls, and a variety of bug fixes and performance updates.

June 3, 2020 10:25AM CDT
June 3, 2020 3:25PM UTC
[Resolved] Release of 0.12.0 is complete and without issue.
Issue with CNN Video Affiliate Network imported videosPartial Service Disruption

Incident Status

Partial Service Disruption


Components

BLOX CMS


Locations

US-MIDWEST-1, US-EAST-1



May 15, 2020 5:02PM CDT
May 15, 2020 10:02PM UTC
[Identified] Imported videos from the CNN Video Affiliate Network are no longer functioning properly. Customers with access to and have configured CNN Full Bundle imports do not appear to be affected. We are working with CNN to resolve this issue and will update this advisory when a resolution is available.

June 2, 2020 11:23AM CDT
June 2, 2020 4:23PM UTC
[Resolved] We've migrated affected customers to an alternative feed choice and are still working with CNN behind the scenes to deal with other potential feed API changes.
Outage with keyword tagging using OpenCalaisPartial Service Disruption

Incident Status

Partial Service Disruption


Components

BLOX CMS


Locations

US-MIDWEST-1, US-EAST-1



June 1, 2020 1:42PM CDT
June 1, 2020 6:42PM UTC
[Identified] Semantic keyword tagging using OpenCalais is presently down and appears to be related to an internal API issue within Reuter's systems regarding invalid SSL certificates. The actual public facing end-point is available. A support ticket has been opened with Reuter's to investigate the issue. There is no ETA on a fix at this time. Customers making use of BLOX smart tagging are unaffected as that uses another provider.

June 1, 2020 2:52PM CDT
June 1, 2020 7:52PM UTC
[Monitoring] Reuters has responded and has confirmed an issue with SSL certificates on their end. They are working on attempting to resolve the issue.

June 2, 2020 11:04AM CDT
June 2, 2020 4:04PM UTC
[Resolved] Reuters has fixed the certificate issue with their domains and keyword suggestion should now be functioning again.
Rayos 0.11.2 OOB @ 2pm centralOperational

Incident Status

Operational


Components

Rayos CMS


Locations

US-EAST-1, US-MIDWEST-1



June 1, 2020 2:16PM CDT
June 1, 2020 7:16PM UTC
[Identified] Rayos 0.11.1 released at 2pm to address customer specific data import concerns.

June 1, 2020 2:17PM CDT
June 1, 2020 7:17PM UTC
[Resolved] Release was successful and without incident.
Email password changes to become requiredOperational

Incident Status

Operational


Components

Support Call Center


Locations

US-CORP-1



May 20, 2020 1:11PM CDT
May 20, 2020 6:11PM UTC
[Identified] Beginning June 02, 2020 all email accounts hosted on Townnews will be required to have the passwords changed every 60 days. Beginning May 26th these email accounts will receive a daily reminder about the password expiration, including instructions on how to change your password. Please contact Townnews if you have questions. Users can change their email password at any time by clicking the "Change password" link on our webmail portal: https://webmail.tn-cloud.net/

June 1, 2020 10:32AM CDT
June 1, 2020 3:32PM UTC
[Resolved] This is resolved