Active Incident

Updated a few seconds ago

Field59 Player IssuesOperational

Incident Status

Operational

Components

Field 59

Locations

US-CLOUD-1


October 26, 2020 8:12PM CDT
October 27, 2020 1:12AM UTC
[Investigating] The Field59 player nodes are experiencing an issue preventing new nodes from coming online. This is preventing the player from loading on sites that rely on the player code provided by the Field59 platform. We are still investigating and working on a resolution to this issue.

October 26, 2020 8:56PM CDT
October 27, 2020 1:56AM UTC
[Monitoring] The player nodes are now operational again and serving player code. We are continuing to monitor operations.

BLOX CMS




Operational

Job Processing




Operational

BLOX Total CMS




Operational

Network




Operational

Database




Operational

Storage System




Operational

Support Call Center




Operational

Field 59




Operational

Rayos CMS




Operational

FTP, Drive, DNS, E-Mail




Operational

E-Editions




Operational

Advertising & Behavioral Targeting




Operational

Business Directory




Operational

Forms




Operational

Subscription




Operational

Analytics & Email Reach




Operational

Calendar




Operational

BLOX CMS /APP




Operational

Sandstone Administration




Operational

OTT & NOW App




Operational

GamesPerform




Operational

Rivista CMS




Operational

History (Last 7 days)

BLOX 1.54.0 "remember me" login feature issuesPartial Service Disruption

Incident Status

Partial Service Disruption


Components

BLOX CMS


Locations

US-MIDWEST-1, US-EAST-1



October 22, 2020 4:00PM CDT
October 22, 2020 9:00PM UTC
[Identified] An issue with the "remember me" login feature in BLOX 1.54.0 has been identified. The legacy "remember me" login end-point attempts to restore cookies but does not completely set cookies lifetimes to match the new authentication token login lifetime. The result is an end-user may appear to be logged in but may have lost access to content or the user admin dashboard. An out-of-band release of the core system is being worked on to correct the issue when the legacy end-point is called. A workaround is to have the user log out and in again which will properly set the cookie expirations. We apologize for any end-users that may be affected by this issue.

October 22, 2020 5:45PM CDT
October 22, 2020 10:45PM UTC
[Identified] The bug fix has passed QA testing and is being reviewed for release by the systems team - the tentative deployment time is about 6:00PM central today. This advisory will be updated when the release has been deployed.

October 22, 2020 6:05PM CDT
October 22, 2020 11:05PM UTC
[Monitoring] The bug fix for the legacy "remember me" login problem has been released and we are now monitoring for any additional issues.

October 23, 2020 11:32AM CDT
October 23, 2020 4:32PM UTC
[Identified] After working with some key partners this morning - another issue has been identified relating to remember me functionality that is similar to the issue that was fixed late yesterday but caused by another avenue of user interaction. A user that logs in and switches tabs or keeps the site open in a browser tab for about 4 hours may have a subscription access check occur behind the scenes. When this happens, service cookies may be changed such that they will be lost if the user closes and then opens the browser. In this scenario, the user will still be logged in but appear to not have services. As a workaround, the same fix that was recommended before applies - the user can log out and in again and should be able to continue accessing paywalled content. An out-of-band fix of the BLOX subscription product is being worked on and will be released as soon as possible today.

October 23, 2020 3:33PM CDT
October 23, 2020 8:33PM UTC
[Monitoring] BLOX subscription 1.29.1 has been released to address the issue found today. Any time a backend check for continued subscription access is performed the cookie lifetimes will be extended. Some changes were made to also try and restore the cookie if it is prematurely removed. We are continuing to monitor the release for anomalies.

October 26, 2020 8:06PM CDT
October 27, 2020 1:06AM UTC
[Resolved] This is resolved
RAYOS 1.0.2 release at 2:00PM centralOperational

Incident Status

Operational


Components

Rayos CMS


Locations

US-EAST-1, US-MIDWEST-1



October 23, 2020 1:53PM CDT
October 23, 2020 6:53PM UTC
[Monitoring] RAYOS 1.0.2 will be released OOB at 2:00PM central to resolve an elections race priority bug that overwrites the priority on feed ingestion. This release also includes CSS improvements for elections front-end display. There are no other changes in this release.

October 23, 2020 2:31PM CDT
October 23, 2020 7:31PM UTC
[Resolved] RAYOS 1.0.2 has been released.
RAYOS 1.0.1 release at 9:00AM centralOperational

Incident Status

Operational


Components

Rayos CMS


Locations

US-EAST-1, US-MIDWEST-1



October 23, 2020 8:13AM CDT
October 23, 2020 1:13PM UTC
[Monitoring] RAYOS 1.0.1 will be released OOB at 9:00AM central to resolve a elections image bug that overwrites images on feed ingestion. This release also includes an improvement to inline featured image tags to help the BLOXCMS importer identify related child images. There are no other changes in this release.

October 23, 2020 9:21AM CDT
October 23, 2020 2:21PM UTC
[Resolved] RAYOS 1.0.1 has been released.
Flex 3.182.2 out-of-bandOperational

Incident Status

Operational


Components

BLOX CMS


Locations

US-MIDWEST-1, US-EAST-1



October 21, 2020 3:05PM CDT
October 21, 2020 8:05PM UTC
[Identified] A Flex 3.182.2 package will be released out-of-band Thursday the 22nd @ 9AM Central. The release contains improvements for NowApp registration.

October 22, 2020 10:28AM CDT
October 22, 2020 3:28PM UTC
[Resolved] The Flex 3.182.2 out-of-band was released this morning at 9AM Central
BLOX OTT 0.2.4 update to be rolled back at 9:15AMPartial Service Disruption

Incident Status

Partial Service Disruption


Components

OTT & NOW App


Locations

US-CLOUD-1



October 21, 2020 9:10AM CDT
October 21, 2020 2:10PM UTC
[Identified] BLOX OTT 0.2.4 changed the device connection mechanism to allow for a way to "logout" from the device. Unfortunately, some Roku devices are crashing randomly as a result of the change - so this is going to be rolled back.

October 21, 2020 9:56AM CDT
October 21, 2020 2:56PM UTC
[Resolved] The issue has been resolved for all Roku OTT apps.