haproxy.log containing ": " messages work now

[jonico]

haproxy.log files containing ": " as part of the payload resulted in a stacktrace

Traceback (most recent call last):
  File "update-stats.py", line 113, in <module>
    main()
  File "update-stats.py", line 78, in main
    ReportAPIRequests(configuration, dataDirectory, metaStats).update()
  File "/Users/jonico/hubble/hubble/updater/reports/Report.py", line 189, in update
    self.updateData()
  File "/Users/jonico/hubble/hubble/updater/reports/ReportDaily.py", line 35, in updateData
    self.updateDailyData()
  File "/Users/jonico/hubble/hubble/updater/reports/ReportAPIRequests.py", line 16, in updateDailyData
    sum(map(lambda x: int(x[3] if len(x) > 2 else 0), newData))]
  File "/Users/jonico/hubble/hubble/updater/reports/ReportAPIRequests.py", line 16, in <lambda>
    sum(map(lambda x: int(x[3] if len(x) > 2 else 0), newData))]
IndexError: list index out of range

This was because

hubble/updater/scripts/api-requests.sh

Line 8 in b55e5fe

perl -ne 'print if s/.*: ([^:]+).*\/api\/v3\/([^\/\? ]+)\/([^\/\? ]+?(\/[^\/\? ]+)).*/\1 \2 \3/' |

did a too greedy search for a ":" to figure out the source ip.

Valid messages inside haproxy.log containing ": " in the payload like

Nov 23 15:12:08 octodemo-com-primary haproxy[27650]: 127.0.0.1:60001 [23/Nov/2017:15:12:08.418] https_protocol~ web_unicorns/localhost 0/0/0/79/79 200 45447 - - ---- 13/6/1/1/0 0/0 {octodemo.com||Operation: California Auto-Deploy} "GET /api/v3/repos/lukeartorg/reading-time-demo-lukeart/deployments HTTP/1.1"

lead to a too greedy parsing, missing the source IP.

Hence, made regexp searching for ".*:" non-greedy so that haproxy.log lines containing ": " as part of the payload are processed correctly.

[larsxschneider]

This works, thank you 👍 ! However, I wonder if the following might be easier to understand (untested):

perl -ne 'print if s/.*haproxy\[\d+\]: ([^:]+).*\/api\/v3\/([^\/\? ]+)\/([^\/\? ]+?(\/[^\/\? ]+)).*/\1 \2 \3/' 

[jonico]

@larsxschneider: I am pretty busy this week, but will try to carve out some time to try your snippet as well.

[larsxschneider]

superseded by #54 . Thanks Johannes :)