Skip to content

Parse headers in HTTPUtil without commons-httpclient 3 - #542

Open
NicoPiel wants to merge 3 commits into
OpenIntegrationEngine:mainfrom
NicoPiel:modernise/httpclient3-removal
Open

NicoPiel wants to merge 3 commits into
OpenIntegrationEngine:mainfrom
NicoPiel:modernise/httpclient3-removal

Conversation

@NicoPiel

@NicoPiel NicoPiel commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

What

HTTPUtil.parseHeaders no longer uses commons-httpclient 3's HttpParser. It now delegates to httpcore 4.4's AbstractMessageParser.parseHeaders (httpcore is already on the server classpath), instead of using a custom parser.

Why

This is one step toward dropping commons-httpclient 3.0.1. Following review, it uses an existing, maintained parser rather than hand-written parsing code.

Behaviour changes against httpclient 3

HTTPUtil is user-script API, so these are worth a line in the release notes. Well-formed headers (Name: value lines separated by CRLF or LF, ending at a blank line or EOF) parse the same as before. These corner cases change:

Input httpclient 3 httpcore 4.4
Malformed header (no colon, status line, empty name) org.apache.commons.httpclient.ProtocolException (an IOException), Unable to parse header: <line> org.apache.http.ProtocolException (not an IOException), Invalid header: <line>
Empty header name (:v) accepted as name "" rejected
Folded line before the first header ( orphan) silently ignored rejected
Whitespace-only line (\t) ends the header block treated as a folded continuation, so parsing continues
Line that is only a bare CR (\r\r\n) ends the header block rejected (Invalid header: [0x10])
Folded continuation of an empty value (X: then b) " b" "b"
Trailing whitespace on a folded line (a, b , \tc) "a b c" "a b c" (only leading whitespace of each fold is dropped)
Control characters around a value (\u000ba\u0001) trimmed (String.trim) kept; only spaces and tabs are trimmed

Reviewer notes

  • The first commit adds characterization tests that pass against the old httpclient 3 implementation. The last commit updates them to pin httpcore's behaviour; each changed assertion is one of the rows above. 21 tests now.
  • str.getBytes() still uses the platform charset and the bytes are decoded as UTF-8, as before.
  • No header count or line length limit is applied (-1), matching the old unlimited behaviour.
  • This does not remove the jar yet. That needs the HttpStatus change and the WebDAV rewrite merged too.

🤖 Generated with Claude Code

Nico Piel added 2 commits October 29, 2026 19:30
Pin the behaviour of the current commons-httpclient 3 based
implementation (folding, trimming, duplicates, termination, errors)
before replacing it.

Signed-off-by: Nico Piel <nico.piel@hotmail.de>
Replace HttpParser.parseHeaders with an equivalent inline parser so the
userutil API no longer needs the EOL httpclient 3 jar. Line splitting,
folding, trimming, UTF-8 decoding and the error message are unchanged;
the parse error is now a java.net.ProtocolException (still an
IOException) instead of the httpclient 3 ProtocolException.

Signed-off-by: Nico Piel <nico.piel@hotmail.de>
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Test Results

130 files  ± 0  130 suites  ±0   3m 58s ⏱️ +47s
810 tests +21  810 ✅ +21  0 💤 ±0  0 ❌ ±0 
936 runs  +21  930 ✅ +21  6 💤 ±0  0 ❌ ±0 

Results for commit 90c8e09. ± Comparison against base commit 4589238.

This pull request removes 2 and adds 23 tests. Note that renamed tests count towards both.
com.mirth.connect.donkey.server.data.jdbc.OracleJdbcDaoTest ‑ testBatchInsertMessageContentKeepsStatementOpenUntilExecuted
com.mirth.connect.donkey.server.data.jdbc.OracleJdbcDaoTest ‑ testFailedBatchInsertClearsAndClosesTheStatement
com.mirth.connect.donkey.server.data.jdbc.JdbcDaoTest ‑ insertMessageContentBatchBatchesEveryRowThenExecutesOnce
com.mirth.connect.donkey.server.data.jdbc.JdbcDaoTest ‑ insertMessageContentBatchClosesTheStatementWhenBindingFails
com.mirth.connect.server.userutil.HTTPUtilTest ‑ testBareCarriageReturnLineIsRejected
com.mirth.connect.server.userutil.HTTPUtilTest ‑ testCarriageReturnHandling
com.mirth.connect.server.userutil.HTTPUtilTest ‑ testContinuationOfEmptyValue
com.mirth.connect.server.userutil.HTTPUtilTest ‑ testCrlfLines
com.mirth.connect.server.userutil.HTTPUtilTest ‑ testDuplicateNamesLastWins
com.mirth.connect.server.userutil.HTTPUtilTest ‑ testEmptyNameIsRejected
com.mirth.connect.server.userutil.HTTPUtilTest ‑ testEmptyString
com.mirth.connect.server.userutil.HTTPUtilTest ‑ testEmptyValue
…

♻️ This comment has been updated with latest results.

gibson9583
gibson9583 previously approved these changes Oct 30, 2026

@gibson9583 gibson9583 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks safe to merge.

@mgaffigan mgaffigan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not be implementing an HTTP parser. This is how you get more request smuggling vulnerabilities.

Change to some existing parser we're already using. That may break some corner case where a client is sending nonstandard requests - but such things are not reasonably avoidable.

Signed-off-by: Nico Piel <nico.piel@hotmail.de>
@NicoPiel

Copy link
Copy Markdown
Contributor Author

Agreed. parseHeaders now delegates to httpcore 4.4's AbstractMessageParser.parseHeaders, so there's no custom parsing code left. I updated the characterization tests to pin httpcore's behaviour and listed every corner case that changes against httpclient 3 in the PR description. They're all malformed or unusual inputs: an empty name, an orphan folded line, whitespace-only and bare-CR lines, fold whitespace, control-character trimming, and the exception type and message.

@mgaffigan mgaffigan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks better - can we deprecate this method? Seems like it should not be used - doesn't handle repeated headers - and how is someone getting a set of http headers as a string?

@tonygermano tonygermano left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not approved yet. Switching to httpcore is the right call, and the characterization tests make the behaviour changes easy to follow. Three asks before merge:

  1. Encode the string as UTF-8. The new decoder rejects malformed input instead of replacing it, so on a JVM whose default charset isn't UTF-8 (e.g. Java 17 on Windows), any non-ASCII header value now throws MalformedInputException. httpclient 3 returned a mangled value instead. With UTF-8 on both ends, the round trip is exact on every platform:
    buffer.bind(new ByteArrayInputStream(str.getBytes(StandardCharsets.UTF_8)));
    testNonAsciiValue can then drop its assumeTrue, and the description's "platform charset behaviour unchanged" note should be updated.
  2. Squash the inline-parser commit into the httpcore commit. We rebase-merge, so otherwise the hand-written parser lands on main with a message that no longer matches what ships. Since HTTPUtil is user-script API, please give the remaining commit a body listing the behaviour changes, for example:
    Parse HTTPUtil headers with httpcore instead of httpclient 3
    
    Delegate HTTPUtil.parseHeaders to httpcore 4.4's
    AbstractMessageParser.parseHeaders, which is already on the server
    classpath, so the userutil API no longer needs the EOL
    commons-httpclient 3 jar.
    
    Well-formed headers parse the same. Corner cases that change:
    - A malformed header throws org.apache.http.ProtocolException
      ("Invalid header: ..."), which is not an IOException, instead of
      httpclient 3's ProtocolException ("Unable to parse header").
    - Empty header names, a folded line before the first header and
      bare-CR lines are rejected.
    - A whitespace-only line is treated as a continuation instead of
      ending the header block.
    - Folding onto an empty value no longer adds a leading space.
    - Only spaces and tabs are trimmed from values, not other control
      characters.
    - The string is encoded as UTF-8 instead of the platform charset.
    
    Keeping the characterization tests as their own first commit is good.
  3. In HTTPUtilTest, use JUnit 4.13's assertThrows in assertParseFailure instead of try/fail/catch. If the wrong exception is thrown, it reports the expected and actual types and keeps the original as the cause:
    private static void assertParseFailure(String input, String badLine) {
        ProtocolException e = assertThrows(ProtocolException.class, () -> HTTPUtil.parseHeaders(input));
        assertEquals("Invalid header: " + badLine, e.getMessage());
    }
    This replaces the assertTrue and fail imports with import static org.junit.Assert.assertThrows;.

Maintainer follow-up

Mitch's question about deprecating parseHeaders deserves its own issue.

Claude checked:

  • The PR merges cleanly onto current main, and HTTPUtilTest passes (21 tests), both as-is and with the assertThrows helper from ask 3.
  • With -Dfile.encoding=ISO-8859-1 and the header X-Name: Müller, httpclient 3.0.1 returns M?ller and httpcore 4.4.13 throws MalformedInputException. With UTF-8 as the default, both return Müller.

🤖 Drafted with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants