本文主要是介绍记一次由gzip引起的nginx转发事故,希望对大家解决编程问题提供一定的参考价值,需要的开发者们随着小编来一起学习吧!
故事背景
书接前几篇文章,仍然是交付甲方遇到的一个特殊诉求,从而引发了本期的事故。甲方的诉求是前端的请求过来,需要加密,但是要经过waf,必须要求是请求明文,那就要在waf和nginx之间做一个解密前置应用处理。大致架构图如下:
本次事故的起因是因为,经过waf的请求响应头信息增加了一个Content-Encoding:gzip导致的数据无法返回前端。
技术栈
nginx:1.16.1
springboot:2.5.14
hutool:5.8.15
NGINX下载:NGINX下载链接
情景再现
我们一点一点还原下,当时遇到的问题。我们这里需要两个java应用和一个nginx三个工程。注意下述不是完整代码,都是核心代码片段。只为说明问题产生的过程,所以不会大面积贴出所有代码!!!
前置服务
http调用目标服务核心代码
注意:此处代码重点在于使用hutool HttpRequest.execute()调用目标服务,其余方法理性观看!
public static HttpResponse executeAndGetResponse(HttpServletRequest request, String forwardAddr, String decryptData, String wafHost) {if (decryptData == null) {return executeAndGetResponse(request, forwardAddr, wafHost);}HttpRequest httpRequest = getHttpRequest(request, forwardAddr, wafHost);Map<String, Object> copyForm = null;if (httpRequest.form() != null) {copyForm = new HashMap<>(httpRequest.form());}httpRequest.body(decryptData, request.getContentType());//重新设置form,设置body时会将form设为null,所以需重新设置form//TODO body和form二者只能存在一个,当form存在时,则body会被置为null,除了Get请求其他参数都要放到body体中if (Objects.equals(RequestMethodEnum.GET.getMethod(), request.getMethod())&& httpRequest.form() == null && CollectionUtil.isNotEmpty(copyForm)) {httpRequest.form(copyForm);}HttpResponse response = httpRequest.execute();return response;}
响应流返回给ng核心代码
public static void returnStream(HttpResponse forwardResponse, HttpServletResponse response) {//拷贝转发响应头携带的信息Map<String, List<String>> headers = forwardResponse.headers();if (CollectionUtil.isNotEmpty(headers)) {log.info("----------------------------------------------");for (Map.Entry<String, List<String>> entry : headers.entrySet()) {if (CollectionUtil.isNotEmpty(entry.getValue())) {response.addHeader(entry.getKey(), entry.getValue().get(0));log.info("响应头信息:[{}:{}]", entry.getKey(), entry.getValue().get(0));} else {response.addHeader(entry.getKey(), "");}}}//输出响应日志String contentType = forwardResponse.header("Content-Type");if (StringUtils.isNotBlank(contentType) && contentType.toLowerCase().startsWith(MediaType.APPLICATION_JSON_VALUE)) {String body = forwardResponse.body();if (body.length() > 1000) {log.info("请求响应长度大于1000,只打印前1000个字符,详情可查看转发服务:{}", StrUtil.cleanBlank(body).substring(0, 1000));} else {log.info("请求响应:{}", StrUtil.cleanBlank(body));}}OutputStream outputStream = null;GZIPOutputStream gzipOut = null;try {response.setCharacterEncoding("UTF-8");outputStream = response.getOutputStream();IoUtil.copy(forwardResponse.bodyStream(), outputStream);outputStream.flush();} catch (IOException e) {log.error("流读取IO异常", e);throw new RuntimeException(e);} finally {if (gzipOut != null) {try {gzipOut.close();} catch (IOException e) {throw new RuntimeException(e);}}if (outputStream != null) {try {outputStream.close();} catch (IOException e) {throw new RuntimeException(e);}}}}
启动配置文件
spring.application.name=router
server.port=7070waf.host=127.0.0.1
forward.url=http://${waf.host}:9090
目标服务
接口实现
import cn.hutool.core.io.IoUtil;
import com.alibaba.fastjson.JSON;
import lombok.extern.slf4j.Slf4j;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.OutputStream;
import java.util.HashMap;
import java.util.Map;
import java.util.zip.GZIPOutputStream;@RestController
@RequestMapping("/hello")
@Slf4j
public class HelloController {@GetMapping("/test")public Map getTest() {Map<String, String> ng = new HashMap<>();ng.put("2", "333333333333333333333");return ng;}@GetMapping("/test3")public void getTest3(String a, String b, HttpServletResponse response) {Map<String, String> ng = new HashMap<>();ng.put("2", "Hello,received:" + a + b);response.addHeader("Content-Encoding", "gzip");OutputStream outputStream = null;GZIPOutputStream gzipOut = null;try {response.setCharacterEncoding("UTF-8");outputStream = response.getOutputStream();log.info("--响应开始压缩--");gzipOut = new GZIPOutputStream(outputStream);IoUtil.write(gzipOut, false, JSON.toJSONBytes(ng));gzipOut.finish();log.info("--响应压缩完成--");outputStream.flush();} catch (IOException e) {log.error("流读取IO异常", e);throw new RuntimeException(e);} finally {if (gzipOut != null) {try {gzipOut.close();} catch (IOException e) {throw new RuntimeException(e);}}if (outputStream != null) {try {outputStream.close();} catch (IOException e) {throw new RuntimeException(e);}}}}@GetMapping("/test2")public Map getTest2() {Map<String, String> ng = new HashMap<>();ng.put("2", "333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333");ng.put("3", "333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333");return ng;}@GetMapping("/gzip-test")public void gzipTest(HttpServletRequest request, HttpServletResponse response) {response.addHeader("Content-Encoding", "gzip");OutputStream outputStream = null;GZIPOutputStream gzipOut = null;try {response.setCharacterEncoding("UTF-8");outputStream = response.getOutputStream();log.info("--响应开始压缩--");gzipOut = new GZIPOutputStream(outputStream);Map<String, String> data = new HashMap<>();data.put("name", "Kevin");IoUtil.write(gzipOut, false, JSON.toJSONBytes(data));gzipOut.finish();log.info("--响应压缩完成--");outputStream.flush();} catch (IOException e) {log.error("流读取IO异常", e);throw new RuntimeException(e);} finally {if (gzipOut != null) {try {gzipOut.close();} catch (IOException e) {throw new RuntimeException(e);}}if (outputStream != null) {try {outputStream.close();} catch (IOException e) {throw new RuntimeException(e);}}}}@GetMapping("/gzip-test2")public void gzipTest2(HttpServletRequest request, HttpServletResponse response) {response.addHeader("Content-Encoding", "gzip");OutputStream outputStream = null;GZIPOutputStream gzipOut = null;try {response.setCharacterEncoding("UTF-8");outputStream = response.getOutputStream();log.info("--响应开始压缩--");gzipOut = new GZIPOutputStream(outputStream);Map<String, String> data = new HashMap<>();data.put("name", "Mary");IoUtil.write(gzipOut, false, JSON.toJSONBytes(data));gzipOut.finish();log.info("--响应压缩完成--");outputStream.flush();} catch (IOException e) {log.error("流读取IO异常", e);throw new RuntimeException(e);} finally {if (gzipOut != null) {try {gzipOut.close();} catch (IOException e) {throw new RuntimeException(e);}}if (outputStream != null) {try {outputStream.close();} catch (IOException e) {throw new RuntimeException(e);}}}}}
启动配置文件
spring.application.name=demo2
server.port=9090
nginx配置
#user nobody;
worker_processes 1;error_log logs/error.log;
#error_log logs/error.log notice;
error_log logs/error.log info;pid logs/nginx.pid;events {worker_connections 1024;
}http {include mime.types;default_type application/octet-stream;log_format main '$remote_addr - $remote_user [$time_local] "$request" ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent" "$http_x_forwarded_for"';access_log logs/access.log main;sendfile on;#tcp_nopush on;keepalive_timeout 65;upstream backend {server 127.0.0.1:7070;keepalive 50;}#gzip on;server {listen 6060;server_name 127.0.0.1;access_log logs/host.access.log main;client_max_body_size 20m;client_header_buffer_size 32k;location / {proxy_pass http://backend;#proxy_http_version 1.1;#proxy_set_header Connection "";}}}
问题集锦
错误1:NS_ERROR_NET_RESET
解决方案
增加配置
proxy_http_version 1.1;
proxy_set_header Connection “”;
#user nobody;
worker_processes 1;error_log logs/error.log;
#error_log logs/error.log notice;
error_log logs/error.log info;pid logs/nginx.pid;events {worker_connections 1024;
}http {include mime.types;default_type application/octet-stream;log_format main '$remote_addr - $remote_user [$time_local] "$request" ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent" "$http_x_forwarded_for"';access_log logs/access.log main;sendfile on;#tcp_nopush on;keepalive_timeout 65;upstream backend {server 127.0.0.1:7070;keepalive 50;}#gzip on;server {listen 6060;server_name 127.0.0.1;access_log logs/host.access.log main;client_max_body_size 20m;client_header_buffer_size 32k;location / {proxy_pass http://backend;proxy_http_version 1.1;proxy_set_header Connection "";}}}
错误2:net::ERR_CONTENT_DECODING_FAILED 200 (OK)
解决方案
修改前置应用返回ng的响应流处理方法,检测到响应头中含有Content-Encoding:gzip对报文内容做压缩处理,再返给ng
public static void returnStream(HttpResponse forwardResponse, HttpServletResponse response) {//拷贝转发响应头携带的信息Map<String, List<String>> headers = forwardResponse.headers();if (CollectionUtil.isNotEmpty(headers)) {log.info("----------------------------------------------");for (Map.Entry<String, List<String>> entry : headers.entrySet()) {if (CollectionUtil.isNotEmpty(entry.getValue())) {response.addHeader(entry.getKey(), entry.getValue().get(0));log.info("响应头信息:[{}:{}]", entry.getKey(), entry.getValue().get(0));} else {response.addHeader(entry.getKey(), "");}}}//输出响应日志String contentType = forwardResponse.header("Content-Type");if (StringUtils.isNotBlank(contentType) && contentType.toLowerCase().startsWith(MediaType.APPLICATION_JSON_VALUE)) {String body = forwardResponse.body();if (body.length() > 1000) {log.info("请求响应长度大于1000,只打印前1000个字符,详情可查看转发服务:{}", StrUtil.cleanBlank(body).substring(0, 1000));} else {log.info("请求响应:{}", StrUtil.cleanBlank(body));}}OutputStream outputStream = null;GZIPOutputStream gzipOut = null;try {response.setCharacterEncoding("UTF-8");outputStream = response.getOutputStream();String contentEncoding = forwardResponse.contentEncoding();if(StringUtils.isNotBlank(contentEncoding) && contentEncoding.equalsIgnoreCase("gzip")) {log.info("--响应开始压缩--");gzipOut = new GZIPOutputStream(outputStream);IoUtil.write(gzipOut, false, forwardResponse.bodyBytes());gzipOut.flush();gzipOut.finish();log.info("--响应压缩完成--");} else {IoUtil.write(outputStream, false, forwardResponse.bodyBytes());outputStream.flush();}} catch (IOException e) {log.error("流读取IO异常", e);throw new RuntimeException(e);} finally {if (gzipOut != null) {try {gzipOut.close();} catch (IOException e) {throw new RuntimeException(e);}}if (outputStream != null) {try {outputStream.close();} catch (IOException e) {throw new RuntimeException(e);}}}}
高版本Nginx
最开始我本地复现并没有关注到ng的版本,所以开始使用的是1.25.3的版本,还衍生出了新的问题也贴出来供大家参考下。
#user nobody;
worker_processes 1;error_log logs/error.log;
error_log logs/error.log notice;
error_log logs/error.log info;pid logs/nginx.pid;events {worker_connections 10000;
}http {include mime.types;default_type application/octet-stream;log_format main '$remote_addr - $remote_user [$time_local] "$request" ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent" "$http_x_forwarded_for"';access_log logs/access.log main;sendfile on;tcp_nopush on;keepalive_timeout 65;gzip on;upstream backend {server 127.0.0.1:7070;keepalive 50;}server {listen 6060;server_name 127.0.0.1;access_log logs/host.access.log main;client_max_body_size 20m;client_header_buffer_size 32k;location / {proxy_pass http://backend;proxy_http_version 1.1;proxy_set_header Connection "";}# redirect server error pages to the static page /50x.html#error_page 500 502 503 504 /50x.html;location = /50x.html {root html;}}}
错误1:502 Bad Gateway
解决方案
通过配置ng的错误日志可以查到如下错误:
2024/03/28 01:16:07 [error] 8352#5480: *43 upstream sent duplicate header line: "Transfer-Encoding: chunked", previous value: "Transfer-Encoding: chunked" while reading response header from upstream, client: 127.0.0.1, server: 127.0.0.1, request: "GET /noauth/captcha/slide-image HTTP/1.1", upstream: "http://127.0.0.1:7070/noauth/captcha/slide-image", host: "localhost:6060"
意思是请求头中行Transfer-Encoding: chunked重复导致的;
这个只需要在后端服务中返回ng之前移除请求头就可以了。
注意:这是因为proxy_http_version 1.1;
proxy_set_header Connection “”; 这两个配置导致的,高版本nginx中会自动加入Transfer-Encoding: chunked,所以从后端传过来的response中也存在就会重复,nginx1.16.1版本就不会出现这个问题。
总结
一个gzip引发的案件,原因是因为过waf的时候,waf会自动引入gzip压缩处理,导致前置应用没有处理,解决此问题的方案有2。
方案一
前置应用获取到目标服务的响应结果后,已经是解压后的数据,这是因为hutool是一个http客户端,如果服务端返回的response中带有gzip的标志,hutool获取到的结果已经是解压过后的数据,可以继续移除hutool获取到的响应头中的Content-Encoding:gzip往外继续抛即可,这样抛到ng的时候也是不带gzip头信息的,数据也刚好搭对。但是注意这就失去了压缩的意义了,会损失一些传输损耗,达不到压缩的积极意义。
方案二
那就是检测到响应头中带有gzip标识,返回响应流的时候,做压缩处理,同样是响应头和响应体搭对即可。
未解之谜
情景复现中难以模拟我们真实的一种场景是,我们做了压缩处理,但是在请求中只有一个接口是成功的,其余接口均无返回,我们也不得其解,如果能找到原因再来公布。
这篇关于记一次由gzip引起的nginx转发事故的文章就介绍到这儿,希望我们推荐的文章对编程师们有所帮助!