Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignoring parsing errors of parseHeaders in client.go #1917

Closed
prepaser opened this issue Dec 11, 2024 · 5 comments · Fixed by #1953 · May be fixed by #1921
Closed

Ignoring parsing errors of parseHeaders in client.go #1917

prepaser opened this issue Dec 11, 2024 · 5 comments · Fixed by #1953 · May be fixed by #1921

Comments

@prepaser
Copy link
Contributor

I have to deal with the wrong response header (Referrer Policy, not Referrer-Policy maybe a typo?) sent by the server, but fasthttp causes an error, the moment I receive the wrong header field, and everything goes down. To deal with this, there is nothing I can do but change the fasthttp code. Simply deleting the validHeaderFieldByte in parseHeaders works fine. Browsers also generally seem to just ignore those fields.

Could you please provide a way to ignore the wrong header field? I think it would be enough to optionally disable header field validation, or provide a way to modify validHeaderFieldByteTable.

@erikdubbelboer
Copy link
Collaborator

A pull request would be welcome. Preferably matching net/http in behavior.

@ksw2000
Copy link
Contributor

ksw2000 commented Dec 13, 2024

In net/http, they cut the header by : and convert the header key by calling canonicalMIMEHeaderKey. When canonicalizing the header key, if there is space in the key, they just skip it. See https://go.dev/issue/34540.

That is to say, when processing headers, there is no error if the header key contains "spaces" in net/http. However, if we change the example, it will still result in an error.

See the example below:

header with the key include space

func main() {
	rawRequest := "HTTP/1.1 200 OK\r\nRefer Policy: origin\r\n\r\n"

	fmt.Println("net/http:")
	reader := strings.NewReader(rawRequest)
	res, err := http.ReadResponse(bufio.NewReader(reader), nil)
	if err != nil {
		log.Println(err)
	} else {
		fmt.Printf("%#v\n", res.Header)
	}

	fmt.Println("fasthttp:")
	req2 := fasthttp.AcquireResponse()
	defer fasthttp.ReleaseResponse(req2)

	reader.Seek(0, io.SeekStart)
	err = req2.Read(bufio.NewReader(reader))
	if err != nil {
		log.Println(err)
	} else {
		req2.Header.VisitAll(func(key []byte, value []byte) {
			fmt.Println(string(key), string(value))
		})
	}
}
net/http:
http.Header{"Refer Policy":[]string{"origin"}}
fasthttp:
2024/12/13 08:57:12 error when reading response headers: invalid header key "Refer policy". Buffer size=41, contents: "HTTP/1.1 200 OK\r\nRefer policy: origin\r\n\r\n"

header with empty key

func main() {
	rawRequest := "HTTP/1.1 200 OK\r\n: empty key\r\n\r\n"

	fmt.Println("net/http:")
	reader := strings.NewReader(rawRequest)
	res, err := http.ReadResponse(bufio.NewReader(reader), nil)
	if err != nil {
		log.Println(err)
	} else {
		fmt.Printf("%#v\n", res.Header)
	}

	fmt.Println("fasthttp:")
	req2 := fasthttp.AcquireResponse()
	defer fasthttp.ReleaseResponse(req2)

	reader.Seek(0, io.SeekStart)
	err = req2.Read(bufio.NewReader(reader))
	if err != nil {
		log.Println(err)
	} else {
		req2.Header.VisitAll(func(key []byte, value []byte) {
			fmt.Println(string(key), string(value))
		})
	}
}
net/http:
2024/12/13 08:58:36 malformed MIME header line: : empty key
fasthttp:
2024/12/13 08:58:36 error when reading response headers: invalid header key "". Buffer size=32, contents: "HTTP/1.1 200 OK\r\n: empty key\r\n\r\n"

@slicingmelon
Copy link

Yes, I can confirm the issue too. Temporarily, I wrote an error handler and allowed it to pass any "invalid header" error(s). Then, I used fasthttp.ResponseHeader.VisitAll() to get all HTTP response headers k:v, as shown in the code snippet above.

@ksw2000
Copy link
Contributor

ksw2000 commented Feb 12, 2025

Hi, I am currently fixing this issue. Here are a few points to note:

  1. net/http will accept header keys that contain spaces, but it will not canonicalize them.
  2. net/http applies the same mechanism for trailer fields that contain spaces.
  3. Unit tests need to be modified.

@erikdubbelboer
Copy link
Collaborator

It's ok to modify test to match the behaviour of net/http.

ksw2000 added a commit to ksw2000/fasthttp that referenced this issue Feb 12, 2025
Make behavior consistent with net/http by allowing header keys and trailers containing spaces without canonicalizing them
erikdubbelboer pushed a commit that referenced this issue Feb 19, 2025
* fix: accept invalid headers with a space #1917

Make behavior consistent with net/http by allowing header keys and trailers containing spaces without canonicalizing them

* fix: lint paramTypeCombine

* fix: #1953 (comment)

* fix: golangci-lint nestingReduce
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants