How do I fix Passenger application startup problem

Recent days I was working on deploying one of my Rails project on a complete new VPS. I had operated according to my experience for deploying sites before, but at the last step, after I have deployed the site, it always raised error message "An error occurred while starting up the preloader: it did not write a startup response in time." when I try to visit the site. But, thanks to much hard work and retry, I found the source of the problem and finally fix it.

I will spend short time to show the main steps to resolve the problem. Let's GO!

What Does The Error Message Mean?

According the official wiki of Passenger on Github:

Phusion Passenger reports this error if the application did not finish initializing within a time limit, or if it exited without sending Phusion Passenger a message that says "I've initialized successfully!"

That is, TWO things are expected that our application should:

• response a message to notify Passenger that itself is ready
• response within a time limit

otherwise Passenger will regard the application has been failed to startup and hence report error.

Possible Causes Of Problem

According "Possible causes of problems" section in the above wiki, we can conclude that there will be the below four causes:

• Stdout redirection
• Early termination in bash
• Application startup freeze
• Server too busy

How I Refused Impossible Causes

For the first possible cause, I can not find any STDOUT redirection in all my bash init scripts, so I trust there is no redirection in my server.

For the second one, I had tried to config to disable the loading of bashrc, but it still didn't work, so the second cause is not the cause.

The third one is impossible because I had built the application on other servers successfully, it seems that the cause was not from the application itself.

The True Evil Found

For the last one possible cause, I tried to run top command in shell of the server to determine if CPU or disk is busy, and I found a interesting thing: everytime I visit the site, the CPU usage was up to > 80%, even 90% - 100%, and after about one and half minutes, the browser rendered a rails "Something went wrong" page and meantime the CPU usage was back to <10%.

I then check the explanation from Passenger again:

It could also be that your server is so busy doing something (either CPU-wise or disk-wise) that it fails to start an application process within a reasonable amount of time. The default startup limit is 90 seconds.

Aha, looks like the time limit is the evil. So I decided to try to change the timeout to a longer time, such as 300 seconds.

In my passenger module config file /etc/apache2/mods-available/passenger.conf(more details about why this file, see Passenger Guide: Working with the Apache configuration file), I explicitly append:

PassengerStartTimeout 300

After that, I tried to restart my Apache2 server and deploy again, and Passenger waited for startup within a longer time this time, and hence my application was lastly deployed successfully.