Status updates | Ponlponl123 Labs Incidents and maintenance reported on status page for Ponlponl123 Labs https://status.ponlponl123.com/ https://d1lppblt9t2x15.cloudfront.net/logos/d7dba4f3cefa2af24a2d3cb667210e1e.png Status updates | Ponlponl123 Labs https://status.ponlponl123.com/ en Maintenance: Scheduled Internal Firewall Maintenance https://status.ponlponl123.com/maintenance/1081287 Sun, 04 Oct 2026 14:20:00 -0000 https://status.ponlponl123.com/incident/1081287#47ba32863934182c6ef5bed10ed81eff985fa9f3ca9d9220b61b81e5164de08e Maintenance Maintenance completed Maintenance: Scheduled Internal Firewall Maintenance https://status.ponlponl123.com/maintenance/1081287 Sun, 04 Oct 2026 11:20:00 -0000 https://status.ponlponl123.com/incident/1081287#4a6773b236d9aaebd0fa1a86b8815818942b3e3a5a62c49217d4ef4040142ec1 Maintenance Expected Impact: Brief intermittent connectivity interruptions (5–10 minutes) across internal network segments. Maintenance: Cluster vertical scalling https://status.ponlponl123.com/maintenance/1079299 Fri, 02 Oct 2026 08:30:00 -0000 https://status.ponlponl123.com/incident/1079299#ca0e15a2dddab193a4ad3571a62c67fb281a345801c34a0076da9e6fb71b270e Maintenance Maintenance completed Maintenance: Moving from mailcow to Stalwart. https://status.ponlponl123.com/maintenance/1079360 Fri, 02 Oct 2026 05:05:00 -0000 https://status.ponlponl123.com/incident/1079360#0d0e9d50a2501ea06d2143863b4c37eb32b90875507c8b88080e40bb4f5cfb51 Maintenance Maintenance completed Maintenance: Moving from mailcow to Stalwart. https://status.ponlponl123.com/maintenance/1079360 Thu, 01 Oct 2026 15:05:00 -0000 https://status.ponlponl123.com/incident/1079360#b6e223d985e9b05264c67e7f4c74188ac9b16b1cae35809cd2fb93e0d2e658ac Maintenance We're recommend to not send any important email to us during this maintenance, but you still can be send to us by postman to our HQ. Maintenance: Cluster vertical scalling https://status.ponlponl123.com/maintenance/1079299 Thu, 01 Oct 2026 13:50:00 -0000 https://status.ponlponl123.com/incident/1079299#bda304cfeaf281720ce36d8df2c391db71cec1ca05d1619b90c00245c56063c3 Maintenance Services might be temporary not working. Maintenance: Scheduled Infrastructure Vertical Scalling. https://status.ponlponl123.com/maintenance/1076336 Mon, 28 Sep 2026 16:00:49 -0000 https://status.ponlponl123.com/incident/1076336#9cf0cc3a4b11af8392197da886ce82c86bb64e6f29cdf4ac34ed12c13aa34c7d Maintenance Maintenance completed Maintenance: Scheduled Infrastructure Vertical Scalling. https://status.ponlponl123.com/maintenance/1076336 Mon, 28 Sep 2026 11:00:49 -0000 https://status.ponlponl123.com/incident/1076336#600e350335c4dbeed2fc86b5b29d177b5970c527fb682c724bf016042ad9fe37 Maintenance We will be performing scheduled infrastructure maintenance to scale up system resources. During this window, a small percentage of users may experience brief periods of dashboard unavailability or slower response times. Our team is actively monitoring the update to minimize disruption. Maintenance: Scheduled Internal Firewall Infrastructure Maintenance https://status.ponlponl123.com/maintenance/1068365 Fri, 18 Sep 2026 20:00:19 -0000 https://status.ponlponl123.com/incident/1068365#80c6f59926a44ecc252da8af548018117ba01289279a66ad06b88134f6c6caa5 Maintenance Maintenance completed Maintenance: Scheduled Internal Firewall Infrastructure Maintenance https://status.ponlponl123.com/maintenance/1068365 Fri, 18 Sep 2026 17:15:19 -0000 https://status.ponlponl123.com/incident/1068365#1297fbc11db21ea6705c78d1212887473566e33ad4ba6488b5ba913d0d809a0f Maintenance We are performing a scheduled infrastructure upgrade on the **Internal Firewall**. * **Start Time:** 9/19/2026 12:12 AM * **End Time:** 9/19/2026 03:00 AM **Expected Impact:** During this maintenance window, users may experience temporary disconnections or brief periods of unavailability when attempting to access local network segments, internal resources, and corporate intranets. We are working to complete the updates as quickly as possible and appreciate your patience. Maintenance: [Scheduled Maintenance] Database Security Hardening & Zero-Trust mTLS Migration https://status.ponlponl123.com/maintenance/1060022 Fri, 11 Sep 2026 16:16:12 -0000 https://status.ponlponl123.com/incident/1060022#1c246656784b205eaa0d3a7feaef55abe8766c6e0bf5d87d653ef1b7453645d6 Maintenance **Summary**: Migrating database access from static passwords to zero-trust mTLS client certificate authentication (Vault PKI) and enforcing multi-tenant privilege boundaries. **Scope**: MariaDB cluster, backend API services, and CI/CD migration runners. **Impact**: * Brief intermittent connection drops (~5–10 min) during rolling restarts. * Permanent deprecation of static database passwords and remote TCP `root` access. * DB engine will reject direct `DELETE` queries (soft-delete enforcement). **Action Required**: None for end-users; internal services will automatically transition to ephemeral X.509 certificates. Maintenance: Schedule maintenance https://status.ponlponl123.com/maintenance/1053444 Sun, 06 Sep 2026 08:00:00 -0000 https://status.ponlponl123.com/incident/1053444#accfecf983a4eddbf1aeaa4ba6a6b62d673235e4e7ffccc7b8748a07f6b9dc0c Maintenance ### Scheduled Maintenance Notice - **Title**: Infrastructure Maintenance - **Status**: Scheduled - **Impact Level**: High (Service Interruption) - **Affected Services**: Website, API, Kubernetes, Redis, Database #### Description - **Objective**: Core infrastructure GitOps modernization, cluster workload migration, HA tuning. - **Operations**: - Kubernetes workload sync & controller adoption - Database cluster (MariaDB Galera) & cache (Redis Sentinel) configuration update - Ingress routing & internal proxy optimization - **User Impact**: - Temporary intermittent latency / connection timeouts on APIs & Websites - Active sessions may terminate; auto-reconnect on service recovery - **Rollback Plan**: Instant revert to baseline manifests; zero data modifications. Maintenance: Mail server maintenance. https://status.ponlponl123.com/maintenance/830468 Sun, 22 Feb 2026 05:40:00 -0000 https://status.ponlponl123.com/incident/830468#d137c520d512b35a3f7c63f7b2f366ab21ec7f6937fadcb21f2141034c0e62d6 Maintenance Mail server is currently maintenancing for better performance and high-availability. Maintenance: Storage Maintainance https://status.ponlponl123.com/maintenance/814288 Thu, 29 Jan 2026 10:00:00 -0000 https://status.ponlponl123.com/incident/814288#1edc26acf9f5ebcfb7419fbeb700ed943482599c31be882ecf1fc3df1252800e Maintenance Storage system (included CDN) will temporary unreachable during maintainance. Maintenance: Internal Firewall Maintenance https://status.ponlponl123.com/maintenance/723573 Sun, 07 Sep 2025 17:00:21 -0000 https://status.ponlponl123.com/incident/723573#8356382da60cface7fa718dc638be7d6c027ca4f85956a4068266bc8642a62f9 Maintenance We are maintaining our Internal Firewall for maximum performance, availability and stability. Maintenance: Migrating from rke2 to k3s... https://status.ponlponl123.com/maintenance/715795 Sun, 31 Aug 2025 12:20:00 -0000 https://status.ponlponl123.com/incident/715795#8edc0b3034989937212f6fa099422845ecda18b49cc5f01358a0948644b92a64 Maintenance We're currently migrating our infrastructure from rke2 to k3s cuz rke2 required more resources but our each machine have a limit budget to handle it. Maintenance: We are migrating our infrastructure from microk8s to RKE2 for greater stability. https://status.ponlponl123.com/maintenance/708605 Tue, 19 Aug 2025 16:00:00 -0000 https://status.ponlponl123.com/incident/708605#08facbfd082cbde4db311ad38fc7d67cb497a239b15768444d1bb82b3de4fb8c Maintenance We apologize for the inconvenience. We are migrating our infrastructure from microk8s to RKE2 for greater stability. As we expanded our infrastructure, microk8s began to malfunction. After rebooting each node, our microk8s no longer functioned properly. Therefore, we are taking this opportunity to migrate to a more stable, production-grade infrastructure than microk8s. Database cluster degraded. https://status.ponlponl123.com/incident/706151 Fri, 15 Aug 2025 11:16:00 -0000 https://status.ponlponl123.com/incident/706151#1c1672290a35785d5a02d3de7eb93629e55dfbb718761acc0bc922ab201316ed Incident Fixed! Maintenance: Database maintenance https://status.ponlponl123.com/maintenance/706154 Thu, 14 Aug 2025 16:57:00 -0000 https://status.ponlponl123.com/incident/706154#0cc31491e2c16bd1658245afcb7069a1f3e3d7995c93b7c93e1de10e6d643a1d Maintenance We're temporary degraded our database cluster to only node to maintenance. Database cluster degraded. https://status.ponlponl123.com/incident/706151 Thu, 14 Aug 2025 16:55:00 -0000 https://status.ponlponl123.com/incident/706151#55c1d8733f62ff1f03b6c80a380eb17ca98238c0e9c7e908a05984e53e7389f3 Incident We're temporary degraded our database cluster to only node to maintenance. Pona! Discord Application are temporary down for a while or a day. https://status.ponlponl123.com/incident/706071 Thu, 14 Aug 2025 16:55:00 -0000 https://status.ponlponl123.com/incident/706071#39484c0bc423910e116744785b161255effb3d956746c3b78b3adf3a18bfaacf Incident Fixed! Pona! Discord Application are temporary down for a while or a day. https://status.ponlponl123.com/incident/706071 Thu, 14 Aug 2025 14:21:00 -0000 https://status.ponlponl123.com/incident/706071#3cfe1200504dda80ab911b9faf7ad620cfadc24c0341375b0f13faaf5646996a Incident We're coding more error handlers for handle our down time as soon as possible, sorry for the outage service. Internal Firewall not responding https://status.ponlponl123.com/incident/706037 Thu, 14 Aug 2025 13:26:00 -0000 https://status.ponlponl123.com/incident/706037#87d7bb69e945546ec4eb5191eea1e71b303b60f41ca6376c80290284650ea4bc Incident All service recovered! Static Storage server are down. https://status.ponlponl123.com/incident/705568 Thu, 14 Aug 2025 12:30:00 -0000 https://status.ponlponl123.com/incident/705568#d50fca46e00a87aaaa3cc12a9a0fd67712a4aaa4b66055cbe7983d49c1c8eda4 Incident Static storage server recovered! Internal Firewall not responding https://status.ponlponl123.com/incident/706037 Thu, 14 Aug 2025 04:35:00 -0000 https://status.ponlponl123.com/incident/706037#16343b1d0fb77a9bfccbfb506744c8c11b9653f67f87f281e0432d162c14ff62 Incident We're trying to restarting it as possible (internal remote is not working too) Internal Firewall not responding https://status.ponlponl123.com/incident/706037 Thu, 14 Aug 2025 04:34:00 -0000 https://status.ponlponl123.com/incident/706037#654009b7dcda1a41596677fd2b30d633428c32dd8f742f0e8e61fa45e5848d0d Incident Our Master K8S Node not responding Internal Firewall not responding https://status.ponlponl123.com/incident/706037 Thu, 14 Aug 2025 04:07:00 -0000 https://status.ponlponl123.com/incident/706037#c962b32551fe6693c7b14210561289769b72b0d57f8dfcda9b2de3271c7422ba Incident Our Internal Firewall not response to any requests Static Storage server are down. https://status.ponlponl123.com/incident/705568 Wed, 13 Aug 2025 15:57:00 -0000 https://status.ponlponl123.com/incident/705568#1fce892b96760db2c7438dc29424465bb0d29626ee52d388d02a7787d3cfefd5 Incident Sorry for outage service, i'm now going to recovery it. this will take time for a while or a few days. Infrastructure is down for a day https://status.ponlponl123.com/incident/626199 Sat, 26 Jul 2025 17:11:00 -0000 https://status.ponlponl123.com/incident/626199#5bf8613ea11a553ba4e0cd6f438bd2d453cb981b536154b4970414d96369cd99 Incident Issue solved Infrastructure is down for a day https://status.ponlponl123.com/incident/626199 Sat, 26 Jul 2025 03:33:00 -0000 https://status.ponlponl123.com/incident/626199#634f72f352ae9b677e19acdccff3bf76686e4821943e211735791e8079a563b0 Incident Due to the current conflict on the Thailand-Cambodian border, in order for all relevant officials to be able to work effectively, we reserve the right to temporarily suspend our services, including software and radio services, as well as satellite networks. We understand that the inability to access and/or use these services may cause damage and loss at the very least. However, in order to protect Thailand's sovereignty to the best of our ability, we must migrate our services and infrastructure systems before everything returns to normal. With deep concern and sorrow for the current conflict on the Thai-Cambodian border, which has resulted in injuries, loss of life, and damage to communities, including civilian targets, the lives and peace of the people must not be used as a tool in any conflict. The team condemns all violence and attacks on innocent people, regardless of the reason or demand, and reaffirms the voice of the public: We do not support war and violence in any form. We call on both sides to return to negotiations and find a peaceful and sustainable solution for the safety and well-being of the people of both countries. We would like to send our encouragement and condolences to all the families of the victims. Team Pattarapong.Dev Network and infrastructure service providers, open source developers Ponlponl123 Labs and Ponlponl123. Internal Health Check API Issue https://status.ponlponl123.com/incident/598669 Sat, 07 Jun 2025 16:59:00 -0000 https://status.ponlponl123.com/incident/598669#5f79f3b28d59b538a9400074a9943ee04c4dac53551c59f9f2174ea62f4e0b7a Incident Resolved Internal Health Check API Issue https://status.ponlponl123.com/incident/598669 Sat, 07 Jun 2025 15:00:00 -0000 https://status.ponlponl123.com/incident/598669#ae24069a532c66122a1ba7afc9b5ee492f5ec5efbefb708b08e19f88d7cb4dcb Incident Our Internal-Health-Check-API is currently having some issues, that will show up services is down in Status Page but actually that service is still operational. Maintenance: System maintenance https://status.ponlponl123.com/maintenance/559983 Sun, 11 May 2025 05:00:00 -0000 https://status.ponlponl123.com/incident/559983#41e47cd85ae624f04fd17d3f4e69066aa63bd3035888659b3baa0bec6a32f4b3 Maintenance We're currently maintenance some of our k8s nodes and our proxy on 1st networking layer may temporary unavailable. Maintenance: ⚠️ Electrical Down! https://status.ponlponl123.com/maintenance/559685 Sat, 10 May 2025 10:55:55 -0000 https://status.ponlponl123.com/incident/559685#a8d64ba753f36ddd95886e2f70d003a79364ae0851b953743e0afa5f80981933 Maintenance We're currently booting up our machine back to operate