tag:status.9line911.com,2005:/history9Line Status - Incident History2024-03-28T19:06:07-05:009Linetag:status.9line911.com,2005:Incident/194821812023-12-23T07:56:28-06:002023-12-23T07:56:28-06:00Global Load Balancing Change for Webex HELD<p><small>Dec <var data-var='date'>23</var>, <var data-var='time'>07:56</var> CST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>23</var>, <var data-var='time'>07:00</var> CST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>12:41</var> CST</small><br><strong>Scheduled</strong> - <b>Overview</b><br />We will introduce Cloudflare's Global Load Balancing with dynamic steering capabilities to replace the existing load balancing solution. By leveraging Cloudflare's dynamic steering technology, we intend to ensure that end-user requests are routed to the most efficient and responsive endpoint based on real-time internet conditions and endpoint health. <br /><br /><b>Service Impact</b><br />Impact to end-users is expected to be minimal as DNS records are controlled by Cloudflare today. However, the possibility exists that a user may attempt to access the Webex App E911 Settings, which utilizes HELD, and encounter a network timeout during the DNS record update.<br /><br /><b>Duration of Impact</b><br />The expected impact, if any, will be less than one minute while Cloudflare updates DNS records.<br /><br /><b>Exclusions</b><br />The following services are specifically excluded from impact:<br />(1) Call servers or geolocation routing for non-HELD services (Cisco UCM, Jabber, BroadWorks, Teams, etc.);<br />(2) Users that have previously set a location in the Webex App before the maintenance window.<br /><br /><b>Who to Contact</b><br />To contact 9Line regarding this scheduled maintenance, please open a ticket: https://support.9line911.com</p>tag:status.9line911.com,2005:Incident/159085682023-01-22T21:16:18-06:002023-01-22T21:16:18-06:00Welcome Emails sent in error<p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>21:16</var> CST</small><br><strong>Resolved</strong> - We have identified a set of approximately 430 users that received welcome emails inadvertently. This issue was resolved immediately, and the welcome emails can be deleted.</p>tag:status.9line911.com,2005:Incident/141972472022-12-08T00:00:35-06:002022-12-08T00:00:35-06:00Portal Upgrade<p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>00:00</var> CST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>19:00</var> CST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'> 2</var>, <var data-var='time'>15:41</var> CST</small><br><strong>Scheduled</strong> - 9Line will upgrade the production portal during this maintenance window to introduce new features and resolve open caveats. Interruptions to accessing the portal during the maintenance window will be brief, and you may need to log into the portal again during the maintenance window. Call processing and geolocation routing are unaffected during this maintenance window.<br /><br />If you have questions about this maintenance window, please raise a ticket with the support team - https://9line.zendesk.com/</p>tag:status.9line911.com,2005:Incident/135116462022-11-20T15:33:54-06:002022-11-20T15:33:54-06:00SC - SBC Cluster<p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>15:33</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>15:33</var> CST</small><br><strong>Update</strong> - Service is restored and calls are operational through the SC cluster.</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>15:24</var> CST</small><br><strong>Update</strong> - The SC cluster proxy has been restarted after a software crash, and the system is now coming online.</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>15:15</var> CST</small><br><strong>Investigating</strong> - We are currently investigating an issue with the SC - SBC cluster. The W2 cluster is fully operational. We will post updates as we have additional information.</p>tag:status.9line911.com,2005:Incident/109718962022-08-27T02:40:34-05:002022-08-27T02:40:34-05:00Intermittent Connectivity Issues<p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>02:40</var> CDT</small><br><strong>Resolved</strong> - All workloads are fully operational and have been stable during monitoring.</p><p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>23:11</var> CDT</small><br><strong>Investigating</strong> - The secondary SBC cluster is experiencing intermittent connectivity issues. The engineering team is migrating the workload to backup resources to work around.</p>tag:status.9line911.com,2005:Incident/100247472022-05-20T21:00:39-05:002022-05-20T21:00:39-05:00Call Server Updates<p><small>May <var data-var='date'>20</var>, <var data-var='time'>21:00</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>18:00</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>18</var>, <var data-var='time'>19:45</var> CDT</small><br><strong>Scheduled</strong> - Overview<br />9Line will be performing maintenance to our call servers to support new features.<br /><br />Service Impact<br />Due to load balancing and high availability, we do not anticipate any service disruption to critical services during this maintenance.<br /><br />Duration of Impact<br />While we expect no impact to services, if a 911 call occurred at the same time the service endpoint is switched, your equipment will automatically attempt the other service endpoints. There may be a small delay in call signaling/setup as your SBC/voice gateway switches peers.<br /><br />To contact 9Line regarding this scheduled maintenance, please open a ticket: https://9line.zendesk.com</p>tag:status.9line911.com,2005:Incident/96134062022-03-27T20:00:23-05:002022-03-27T20:00:23-05:00Call server updates<p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>20:00</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>11:01</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>14:12</var> CDT</small><br><strong>Scheduled</strong> - Overview<br />9Line will be performing maintenance to our call servers to support new features.<br /><br />Service Impact<br />Due to load balancing and high availability, we do not anticipate any service disruption to critical services during this maintenance.<br /><br />Duration of Impact<br />While we expect no impact to services, if a 911 call occurred at the same time the service endpoint is switched, your equipment will automatically attempt the other service endpoints. There may be a small delay in call signaling/setup as your SBC/voice gateway switches peers.<br /><br />To contact 9Line regarding this scheduled maintenance, please open a ticket: https://9line.zendesk.com</p>tag:status.9line911.com,2005:Incident/90128812022-01-08T22:00:36-06:002022-01-08T22:00:36-06:00Database Schema Update<p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>22:00</var> CST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>19:01</var> CST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>18:12</var> CST</small><br><strong>Scheduled</strong> - Overview: 9Line will be performing maintenance that will update the production database schema to support new features.<br /><br />Service Impact: Due to load balancing and high availability, we do not anticipate any service disruption to critical services during this maintenance.<br /><br />Duration of Impact: While we expect no impact to services, if a 911 call occurred at the same time the database is switched, your equipment will automatically attempt the other service endpoints. There may be a small additional delay in call signaling/setup as your voice gateway switches peers.<br /><br />To contact 9Line regarding this scheduled maintenance, please open a ticket: https://9line.zendesk.com</p>tag:status.9line911.com,2005:Incident/89077432021-12-16T18:17:00-06:002021-12-21T18:17:51-06:00Website updates<p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>18:17</var> CST</small><br><strong>Resolved</strong> - Scheduled updates for the website.</p>