Sorry about the outage folks. That was a big enough PITA I figured I'd post an explanation directly on all the active boards.

The datacenter the servers are in had a total power failure around 11:30 AM ET. My host is investigating what happened and why battery backup failed as well.

When the servers cam back online, PHP, the programming language the boards are written in, was "f***ed good and proper" as an industrial mechanic I know would say. It took me another hour or so to beat on PHP hard enough to get it to perform well.

So ... all is back to normal. Apologies for your ~4 unexpectedly productive hours this afternoon.

Now back to your regularly scheduled sh*tposting.

(02-24-2017 03:22 PM)georgia_tech_swagger Wrote:  Sorry about the outage folks. That was a big enough PITA I figured I'd post an explanation directly on all the active boards.

The datacenter the servers are in had a total power failure around 11:30 AM ET. My host is investigating what happened and why battery backup failed as well.

When the servers cam back online, PHP, the programming language the boards are written in, was "f***ed good and proper" as an industrial mechanic I know would say. It took me another hour or so to beat on PHP hard enough to get it to perform well.

So ... all is back to normal. Apologies for your ~4 unexpectedly productive hours this afternoon.

Now back to your regularly scheduled sh*tposting.

(1) Thanks for posting an explanation, as I was curious about what had happened.

(2) Thanks for all you do with keeping this forum going. I know it's quite a monumental task, and I thoroughly appreciate having the many boards I post on available and functioning. Outages like this morning help remind me that I shouldn't be so blasé about having these boards. Thanks, again.

(02-24-2017 03:22 PM)georgia_tech_swagger Wrote:  Sorry about the outage folks. That was a big enough PITA I figured I'd post an explanation directly on all the active boards.

The datacenter the servers are in had a total power failure around 11:30 AM ET. My host is investigating what happened and why battery backup failed as well.

When the servers cam back online, PHP, the programming language the boards are written in, was "f***ed good and proper" as an industrial mechanic I know would say. It took me another hour or so to beat on PHP hard enough to get it to perform well.

So ... all is back to normal. Apologies for your ~4 unexpectedly productive hours this afternoon.

Now back to your regularly scheduled sh*tposting.

Thanks GTS. Just curious as to what exactly happened due to this inadvertent reboot. Did the reboot make Apache unhappy with alternate PHP env that wasn't tuned for the BB? I saw the 5** status return and assumed something was up with Nginx not being able to talk to Apache. Anyway Thanks for bringing back the time killer.

(02-24-2017 04:39 PM)12thmonarch Wrote:  Thanks GTS. Just curious as to what exactly happened due to this inadvertent reboot. Did the reboot make Apache unhappy with alternate PHP env that wasn't tuned for the BB? I saw the 5** status return and assumed something was up with Nginx not being able to talk to Apache. Anyway Thanks for bringing back the time killer.

Some libraries got changed out underneath PHP for security updates. This apparently didn't have much impact on PHP while still running. After reboot, it caused PHP to immediately segfault when starting. This led to a nasty string of backtracking to find the problem, then updating the whole of PHP, and then finding and fixing the performance slowdowns from changes that happened in PHP itself.

Not using Apache at all. ModPHP cannot handle these kinds of workloads. Every single request to the site gets handed off to at least four, maybe as many as five different daemons (nginx, php fpm, varnish, memcache, mysql). That's a little insane. But that's what it takes to make PHP not suck at this scale. Things will be much nicer on Aurora when the request pipe simplifies to just GoLang -> MySQL.

(02-24-2017 04:58 PM)georgia_tech_swagger Wrote:  

(02-24-2017 04:39 PM)12thmonarch Wrote:  Thanks GTS. Just curious as to what exactly happened due to this inadvertent reboot. Did the reboot make Apache unhappy with alternate PHP env that wasn't tuned for the BB? I saw the 5** status return and assumed something was up with Nginx not being able to talk to Apache. Anyway Thanks for bringing back the time killer.

Some libraries got changed out underneath PHP for security updates. This apparently didn't have much impact on PHP while still running. After reboot, it caused PHP to immediately segfault when starting. This led to a nasty string of backtracking to find the problem, then updating the whole of PHP, and then finding and fixing the performance slowdowns from changes that happened in PHP itself.

Not using Apache at all. ModPHP cannot handle these kinds of workloads. Every single request to the site gets handed off to at least four, maybe as many as five different daemons (nginx, php fpm, varnish, memcache, mysql). That's a little insane. But that's what it takes to make PHP not suck at this scale. Things will be much nicer on Aurora when the request pipe simplifies to just GoLang -> MySQL.

I was always puzzled how the site was setup to withhold such high loads. Now i know. All the non-blocking goodness of nginx, two different caching mechanisms(let me guess you have also setup memcache for mysql to improve query performance )

I've been getting network errors using the mobile apps (both Tapatalk & CSNbbs apps) ever since the outage. Anyone else, or just me?

Exact error is "Network error, please try again later (get_config)"

Thanks for all you do for this board.

He had to get a new rat for the wheel.

(02-26-2017 07:46 AM)NoodleOwl Wrote:  I've been getting network errors using the mobile apps (both Tapatalk & CSNbbs apps) ever since the outage. Anyone else, or just me?

Exact error is "Network error, please try again later (get_config)"

CSNbbs app is back working again. I had the same issues, as did other users throughout CSNbbsland.

I'm putting together some feedback for streamlining the mobile view on the browser when the app is phased out -- as it's scheduled to be later this year.

I appreciate the hell out of all he does to keep this board up.

It's one of the slicker ones that I've posted on and I've been posting since message boards came out on the internet.

Don't get hammered with popups or too many ads. Pretty easy to link or post an image of embed a video. Nice formatting.

Makes it an easy place to come to and it was missed the other day when it was down. The only gathering place we have as a conference.