Active Incident

Updated a few seconds ago

BLOX Subscription 1.51.5 re-release at 12:00 PM CDTOperational

Incident Status

Operational

Components

Subscription

Locations

US-EAST-1, US-MIDWEST-1


September 28, 2023 11:45AM CDT
September 28, 2023 4:45PM UTC
[Monitoring] BLOX Subscription 1.51.5 will be re-released at 12:00 PM CDT. The release includes performance improvements to the renewal process.
Storage system disruptionOperational

Incident Status

Operational

Components

BLOX CMS, Job Processing, Storage System

Locations

US-MIDWEST-1


September 27, 2023 9:32AM CDT
September 27, 2023 2:32PM UTC
[Monitoring] At approx 8:50AM CST a large data removal on one of our storage clusters caused a cascading reset of four storage blades in that cluster. This reset of multiple blades may have caused some sites to display 500 server errors for up to 1 minute while those blades reloaded. These errors would primarily be within the BLOX/Rayos admins and the Job system. Website pages would have been delivered from our cache during this short outage. At this time all services are operational and we have opened a high priority case with our storage vendor to find root cause and resolve the issue going forward. Updates will be added to this advisory as needed.

September 27, 2023 11:36AM CDT
September 27, 2023 4:36PM UTC
[Monitoring] This issue was related to an undisclosed software bug that caused an OOM (Out of memory) condition during very large data delete operations. We will be performing a minor non disruptive software update on our storage clusters to address this issue as an emergency update. A final update will be posted when the software update has completed.
Rayos Platform Password ResetOperational

Incident Status

Operational

Components

Rayos CMS

Locations

US-EAST-1, US-MIDWEST-1


September 22, 2023 11:41AM CDT
September 22, 2023 4:41PM UTC
[Monitoring] Beginning Monday, October 2nd we will be performing a bulk password and two-factor authentication(2FA) reset on all Rayos Platform sites. This is necessary to ensure that weak passwords are removed to protect against any potential threats.
Email hosting and Webmail end-of-life announcement Service Disruption

Incident Status

Service Disruption

Components

FTP, Drive, DNS, E-Mail

Locations

US-EAST-1, US-MIDWEST-1


September 6, 2023 11:05AM CDT
September 6, 2023 4:05PM UTC
[Identified] After careful consideration, we have made the decision to discontinue our email hosting services, including our Webmail product, effective September 26, 2023. This end-of-life announcement only encompasses BLOX Digital's email hosting and webmail services. Our integrated email marketing suite, BLOX Email Reach, will not be affected. For more information and migration assistance, please read our official announcement: https://www.bloxdigital.com/releases/software_releases/end-of-life-announcement-for-email-hosting-service-and-webmail/article_cf2a1b98-3857-11ee-85c9-274b137fb32f.html
Twitter Rate Limit ErrorsPartial Service Disruption

Incident Status

Partial Service Disruption

Components

BLOX CMS

Locations

US-EAST-1, US-MIDWEST-1


July 18, 2023 4:01PM CDT
July 18, 2023 9:01PM UTC
[Identified] Twitter has implemented a daily cap on the number of tweets that can be published through their platform. This cap has the potential to cause disruptions in your ability to post content seamlessly via our BLOX Notifer integration. However, we want to assure you that we are actively exploring solutions to mitigate any potential interruptions. Should you encounter any errors posting tweets via Notifier, we encourage you to post manually to their platform.
Constant Contact Delivery IssuesDegraded Performance

Incident Status

Degraded Performance

Components

Analytics & Email Reach

Locations

US-EAST-1, US-MIDWEST-1


January 6, 2023 5:13PM CST
January 6, 2023 11:13PM UTC
[Identified] TownNews is currently experiencing an issue impacting email delivery via our Constant Contact partnership. The issue is specifically relating to some automated mailings not correctly sending, as well as open rate information not correctly tracking. The issues revolve around changes to Constant Contact’s API. We are diligently working with Constant Contact to find a quick solution to both issues. We are also working with Constant Contact on long-term API updates that will improve stability and deliverability for the future.

BLOX CMS




Operational

Job Processing




Operational

BLOX Total CMS




Operational

Network




Operational

Database




Operational

Storage System




Operational

Support Call Center




Operational

BLOX VMS




Operational

Rayos CMS




Operational

FTP, Drive, DNS, E-Mail




Service Disruption

E-Editions




Operational

Advertising & Behavioral Targeting




Operational

Business Directory




Operational

Forms




Operational

Subscription




Operational

Analytics & Email Reach




Degraded Performance

Calendar




Operational

BLOX CMS /APP




Operational

Sandstone Administration




Operational

OTT & NOW App




Operational

GamesPerform




Operational

Data Insights




Operational

History (Last 7 days)

BLOX 1.72.2 release at 4PM centralOperational

Incident Status

Operational


Components

BLOX CMS


Locations

US-EAST-1, US-MIDWEST-1



September 27, 2023 4:06PM CDT
September 27, 2023 9:06PM UTC
[Identified] BLOX 1.72.2 will be released at 4PM central to address the following issues: * Task system performance improvements * Fix for robots.txt crash with large disallow lists * Fix for missing archive content in sitemaps from older years

September 28, 2023 11:47AM CDT
September 28, 2023 4:47PM UTC
[Resolved] Software released without incidents.
RAYOS 1.18.1 Release Rollback Operational

Incident Status

Operational


Components

Rayos CMS


Locations

US-EAST-1, US-MIDWEST-1



September 26, 2023 4:56PM CDT
September 26, 2023 9:56PM UTC
[Monitoring] RAYOS release 1.18.1 was rollback at 5:00PM central due to a white label visibility issue.

September 27, 2023 1:28PM CDT
September 27, 2023 6:28PM UTC
[Monitoring] RAYOS 1.18.1 will be re-released out-of-band at 2pm CST. The release will still include an update to the WordPress login page, displaying an alert to inform of upcoming password and 2FA resets.

September 27, 2023 2:39PM CDT
September 27, 2023 7:39PM UTC
[Resolved] Software was re-released without incidents.
BLOX Subscription 1.51.5 release at 3:30PM centralOperational

Incident Status

Operational


Components

Subscription


Locations

US-EAST-1, US-MIDWEST-1



September 21, 2023 3:31PM CDT
September 21, 2023 8:31PM UTC
[Identified] BLOX subscription will be receiving an out-of-band update at 3:30PM central to improve the performance of the renewal and notice background tasks that are used to notify and process customer credit cards for subscription. The improvement should allow for larger batches of both notices and cards to be processed.

September 21, 2023 5:09PM CDT
September 21, 2023 10:09PM UTC
[Monitoring] This release has been rolled back due to issues introduced by the performance change. We are reviewing alternative changes to address the initially reported issue.

September 21, 2023 5:30PM CDT
September 21, 2023 10:30PM UTC
[Monitoring] We've applied a temporary patch to the renewal process so that it can continue to handle the increase in renewals for affected sites.

September 27, 2023 10:46AM CDT
September 27, 2023 3:46PM UTC
[Resolved] This will be re-released at a future date
Flex Templates 3.237.2 release at 3:00PM CSTOperational

Incident Status

Operational


Components

BLOX CMS


Locations

US-EAST-1, US-MIDWEST-1



September 26, 2023 2:02PM CDT
September 26, 2023 7:02PM UTC
[Monitoring] Flex Templates 3.237.2 will be released out-of-band at 3:00PM central. This release fixes an issue with purchasing subscriptions via PayPal. There are no other changes in this release.

September 27, 2023 9:23AM CDT
September 27, 2023 2:23PM UTC
[Resolved] Software released without incidents.
RAYOS 1.18.1 release at 10AM centralOperational

Incident Status

Operational


Components

Rayos CMS


Locations

US-EAST-1, US-MIDWEST-1



September 26, 2023 9:16AM CDT
September 26, 2023 2:16PM UTC
[Monitoring] RAYOS 1.18.1 will be released out-of-band at 10:00AM central. This release includes an update to the WordPress login page. An alert has been added which includes information about the upcoming Rayos Platform Password reset. There are no other changes in this release.

September 26, 2023 10:04AM CDT
September 26, 2023 3:04PM UTC
[Resolved] Software released without incidents.