Home > nginx > Nginx Retry of Requests ( Nginx - Haproxy Combination )

Nginx Retry of Requests ( Nginx - Haproxy Combination )

March 11Hits:18
Advertisement

I wanted to ask about Nginx Retry of Requests. I have a Nginx running at the backend which then sends the requests to HaProxy which then passes it on the web server and the request is processed. I am reloading my Haproxy config dynamically to provide elasticity. The problem is that the requests are dropped when I reload Haproxy. So I wanted to have a solution where I can just retry that from Nginx. I looked through the proxy_connect_timeout, proxy_next_upstream in http module and max_fails and fail_timeout in server module. I initially only had 1 server in the upstream connections so I just that up twice now and less requests are getting dropped ( only when ) have say the same server twice in upstream , if I have same server 3-4 times drops increase ).

So , firstly I wanted to now , that when a request is not able to establish connection from Nginx to Haproxy so while reloading it seems that conneciton is seen as error and straightway the request is dropped .

So how can I either specify the time after the failure I want to retry the request from Nginx to upstream or the time before which Nginx treats it as failed request.

( I have tried increaing proxy_connect_timeout - didn't help , mail_retires , fail_timeout and also putting the same upstream server twice ( that gave the best results so far )

Nginx Conf File

upstream gae_sleep {

server 128.111.55.219:10000; 

}

server {

listen 8080; server_name 128.111.55.219; root /var/apps/sleep/app; # Uncomment these lines to enable logging, and comment out the following two #access_log  /var/log/nginx/sleep.access.log upstream; error_log  /var/log/nginx/sleep.error.log; access_log off; #error_log /dev/null crit;  rewrite_log off; error_page 404 = /404.html; set $cache_dir /var/apps/sleep/cache;    location / {   proxy_set_header  X-Real-IP  $remote_addr;   proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;   proxy_set_header Host $http_host;   proxy_redirect off;   proxy_pass http://gae_sleep;   client_max_body_size 2G;   proxy_connect_timeout 30;   client_body_timeout 30;   proxy_read_timeout 30; }  location /404.html {   root /var/apps/sleep; }  location /reserved-channel-appscale-path {   proxy_buffering off;   tcp_nodelay on;   keepalive_timeout 55;   proxy_pass http://128.111.55.219:5280/http-bind; } 

}

Answers

So after trying to find the answer to retrying requests at nginx, I haven't found a clean way of retrying requests but have come up with sort of a hacky way for it. So within the upstream section in the nginx conf , you should put multiple copies of the same upstream server as retrying in nginx is at upstream server level. If one upstream server fails then nginx tries the request at another upstream server. If you have only 1 upstream server like I have it wont retry the request. So to overcome that I put multiple copies of the same upstream server , so that by the time nginx goes through the list of servers and sends requests , the upstream server ( haproxy in this case ) would have reloaded and request will go to. It is also essential to go through the various timeouts that nginx provides at http module and server module. "fail_timeout" - says that if a upstream server is not available decommission it for x secs , but if all of them are not available then it doesn't decommission ( I am mentioning this as by the time nginx goes through the entire list haproxy may not have come up but this wont be a problem because of this property ) PS : this is a hacky solution and i had to have some 100 - 150 entries of upstream in my nginx file for reducing errors to in significant number. Better solutions are welcome :)

Not exactly what you're asking, but maybe more to the point: How are you restarting HAProxy? With the SF option you should be able to restart HAProxy w/out dropping any connection requests. I don't believe nginx can retry requests, but if you really wanted to you could have an HAProxy front end to another HAProxy instance. Then HAProxy would retry requests to the second instance. But that's seems really silly. Check out the SF option first.

Restart HAProxy script (from comments) The greps are just for my warm-fuzzy feelings:

#!/bin/sh

ps -ef | grep haproxy
haproxy -f haproxy.cfg -sf $(cat /var/run/haproxy.pid)
ps -ef | grep haproxy

Tags:nginx, haproxy

Related Articles

  • Nginx Retry of Requests ( Nginx - Haproxy Combination )March 11

    I wanted to ask about Nginx Retry of Requests. I have a Nginx running at the backend which then sends the requests to HaProxy which then passes it on the web server and the request is processed. I am reloading my Haproxy config dynamically to provide

  • can nginx retry request if unicorn is overloaded and returns a 502?December 26

    Is it possible to have nginx retry a second backend before returning a 502 to the client? Would something like this work? Front end: # haproxy:85 => [a few app servers]:8000 # more specifically: # haproxy => [nginx => unicorn (502 when busy)] # W

  • How to use limit_conn and limit_req modules of Nginx when it is behind HAProxyApril 13

    I'm using limit_conn_zone (with limit_conn) and limit_req_zone (with limit_req) in Nginx which sits behind a HAProxy. They both run on the same host. I see whenever a limit is breached in Nginx (resulting in 503 responses), the client IP is 127.0.0.1

  • Nginx and passenger request timeout (nginx status code 499)March 19

    DISCLAIMER: I have read numerous posts here, on stackoverflow and other pages and I could not come up with a solution, so hopefully this helps. We are running a Rails App on Passenger together with nginx and HAProxy. Passenger: 4.0.7 Nginx: 1.4.1 The

  • Nginx, awstats and request time

    Nginx, awstats and request timeOctober 4

    How can I tune nginx to write request time to logs such that awstats can show time taken for each request? Also, how can I see requests by time taken in awstats? This is the current format log_format main '$remote_addr - $remote_user [$time_local] "$

  • Apache and Nginx - Forwarding .htaccess requests to Apache?December 3

    A really simple one but it's alluding me entirely. I've set up nginx as a proxy for Apache and at the moment it's only set to allow requests for certain files through to Apache, the config line is below. Problem I'm having, I need it to send .htacces

  • nginx proxy redirecting request to different proxyMay 26

    I have three LXC containers (CN1, CN2, CN3). They each run their own Apache instance and have their own networking, so each container has their own private IP. I set up all containers to listen on the private IP only for Apache. I also set up Nginx t

  • Nginx forward all requests to another domainOctober 24

    I have been struggling to get this whole setup to work. I've found bits and pieces on the net but nothing that solves all of my requirements. I need to do this to make IE8 work, so unfortunately my hands are tied a little. I can move some endpoints a

  • How do I tell Nginx to pass requests to different apps based on the incoming url?February 8

    How do I tell Nginx to pass requests to different apps based on the incoming url? For example, mybudgetapp.com and myshoppingapp.com both point to my server where both apps (8001, 8002) and Nginx (80) are running. When a user goes to mybudgetapp.com

  • Reverse SSL Proxy: NGINX in front of NGINX and JoomlaDecember 2

    My question is similar to this one, except that I have another nginx server behind the frontend nginx ssl proxy server: -------------------------- -------------------------------- ---------- | nginx: https://bla.com | -----> | nginx: http://localhost

  • Why does passenger-install-nginx-module neither install Nginx nor implement Passenger support on Centos7 while the output indicates the contrary?January 1

    Introduction At the moment there are approximately 100 Puppet agents that connect to the Puppetmaster. Sometimes it takes a while before implemented changes on the Puppetmaster are picked up by the agents. Some information has been found that Passeng

  • NGINX Permissions: 'sudo nginx' vs 'sudo service nginx start'January 5

    I'm doing a capistrano Setup here with nginx 1.6.2 and Unicorn. But Under my current setup nginx doesn't create the server i have written in con file. Im sure its a permission error for my user's directory as thats where the conf files are located un

  • nginx proxy pass on nginx 1.1.19February 11

    I am trying to get images not found on our staging server to automatically load from the production server. I tested this locally with the following configuration on nginx 1.6.2: And this worked as expected server { root /var/www/html/test; index ind

  • nginx invalid parameter cookie nginx,wordfence pluginSeptember 11

    I'm trying to install wordfence on my wordpress website and it requested me to update my nginx.conf for caching, I created a new file name wp.conf and pointed included it in nginx.conf but when trying to restart nginx I get the following error: nginx

  • Error installing nginx with passenger-install-nginx-module on ubuntu 11.10 & rails 3.1.0November 17

    Here is the error message from installing nginx with passenger-install- nginx-module (rvmsudo). The nginx is 1.0.6 installed under /opt/nginx (default). gem install passenger successfully prior. /usr/bin/ld: /home/dtt/.rvm/gems/ruby-1.9.2-p290/gems/p

  • How can I upgrade nginx after adding ppa:nginx/development?May 6

    I've installed nginx via the ppa stable repository via  add-apt-repository ppa:nginx apt-get install nginx which adds the stable release. But now I'd like to switch to the development branch, so I added apt-add-repository ppa:nginx/development and n

  • FastCGI - error - Your request timed out. Please retry the request.December 29

    Hello, I made upgrade to new isp config 3 version. After that I get every time on every website on server this error in 30 second after page is loaded (to browser - plain text, white background). Code: Your request timed out. Please retry the request

  • Locate the nginx.conf file my nginx is actually usingNovember 11

    Working on a client's server where there are two different versions of nginx installed. I think one of them was installed with the brew package manager (its an osx box) and the other seems to have been compiled and installed with the nginx packaged M

  • How to remove Nginx and compile new Nginx from sourceAugust 26

    I've installed Nginx using apt-get install nginx but now I need to add SPDY and PageSpeed extensions. This is why I need to remove the current Nginx installation and install the new one from source. How should I proceed? Do I have to uninstall the cu

  • Apache won't start while Nginx is running, but Nginx can start while Apache is running?January 31

    I have Nginx and Apache (through XAMPP) installed on my Ubuntu 14.04 LTS. When the computer starts, only Nginx will be running. Trying to start Apache via: sudo /etc/lampp/lampp start but I get an error: XAMPP: Another web server is already running.

Copyright (C) 2017 ceus-now.com, All Rights Reserved. webmaster#ceus-now.com 14 q. 0.400 s.