What it is

GoldenGate's restapi.log records every REST API call made to a deployment, but its format defeats normal log tooling: each entry spans multiple lines, and what follows the header is not one JSON document but two back to back: a Request #reqno object and a Response object, with a trailing comma Python's own json module won't parse. This script turns each entry into a single valid JSON object per line (NDJSON), so the whole log becomes filterable with jq.

You'll find a restapi.log under var/log/ in your Service Manager home and in every deployment home. There are always at least two active files (one for the Service Manager, one per deployment), and rotated ones (restapi.log.1, restapi.log.2, …) get picked up automatically when you point the script at a directory.

What the script does

  1. Lists the target's restapi.log* files and orders them from oldest to newest (rotated files first, by rotation number, then the live restapi.log last).
  2. Reads each file line by line, watching for the line that starts a new entry (Request #reqno: after a timestamp and service header).
  3. Buffers every line belonging to that entry until the next entry's start line appears, then hands the buffered block off for parsing.
  4. Pulls the timestamp, status, service name, and request number out of the header line, and converts the timestamp to a Unix epoch for easy jq range filtering.
  5. Splits the block on the literal Response: marker, then extracts the first complete, brace-balanced JSON object from each half (handling the trailing commas that make these blocks invalid JSON on their own).
  6. Writes one JSON object per line to the output file, combining the parsed request, the parsed response, and the header metadata. A block that fails to parse is counted as invalid and, if you passed an invalid-file path, written there verbatim instead of being dropped silently.
  7. Logs a running total per file and a grand total (found / written / invalid) once every file has been processed.

Quick start

# Single file analysis
python3 restapi_log_reader.py /path/to/var/log/restapi.log restapi.ndjson

# Whole log directory (rotated files included, from oldest to newest)
python3 restapi_log_reader.py /path/to/var/log restapi.ndjson

Run against a whole deployment's log directory:

> python3 restapi_log_reader.py /u01/app/oracle/product/ogg_test_01/var/log /home/oracle/restapi_test_01.ndjson
2026-02-14 07:34:14,934 INFO Input is a directory: /u01/app/oracle/product/ogg_test_01/var/log
2026-02-14 07:34:14,934 INFO Using log files (from oldest to newest): restapi.log.1, restapi.log.2, restapi.log.3, restapi.log
2026-02-14 07:34:17,673 INFO DONE: 2363 records found total, 2363 written, 0 invalid

Output keys

Each line of restapi.ndjson carries these top-level keys:

  • request - original Request #reqno element of the log.
  • response - original response element of the log.
  • restapi_datetime - date string of the log entry.
  • restapi_epoch - jq-readable timestamp, useful for filtering.
  • restapi_status - status displayed in the log header (INFO, ERROR, etc.).
  • restapi_service - GoldenGate service displayed in the log header (adminsrvr, distsrvr, etc.).
  • restapi_reqno - request number.

The interesting part of a response usually lives in response.content.response.items.

jq recipes

A working set to start from - the full blog post has more:

Count events by status

jq -r '.restapi_status' restapi.ndjson | sort | uniq -c

Everything that isn't INFO

jq -c 'select(.restapi_status != "INFO")' restapi.ndjson

Logs for one service

jq -c 'select(.restapi_service == "adminsrvr")' restapi.ndjson

Count by HTTP verb

jq -r '.request.context.verb' restapi.ndjson | sort | uniq -c

Non-200 responses

jq -c 'select(.response.context.code != "200 OK")' restapi.ndjson

Last hour only

jq -c --argjson now $(date +%s) '. | select(.restapi_epoch >= ($now - 3600))' restapi.ndjson

Specific request number

jq 'select(.restapi_reqno == 689)' restapi.ndjson

Timeline view (date, status, verb, URI)

jq -r '"\(.restapi_datetime) \(.restapi_status) \(.request.context.verb) \(.request.context.uri)"' restapi.ndjson | head

Most-called endpoints

jq -r '.request.context.uri' restapi.ndjson | sort | uniq -c | sort -nr | head -10

Calls from a specific IP

jq -c 'select(.request.context.headers["X-Real-IP"] == "10.0.0.2")' restapi.ndjson

The script

#!/usr/bin/env python3
import re
import sys
import json
import logging
from pathlib import Path
from datetime import datetime

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
)

REQUEST_START_RE = re.compile(
    r"^\d{4}-\d{2}-\d{2} "
    r"\d{2}:\d{2}:\d{2}\.\d{3}\+\d{4}\s+"
    r"\w+\s*\|\s*RestAPI\..*?\|\s*Request #\d+:"
)


def find_restapi_logs(directory: Path):
    """
    Return restapi logs ordered from the highest rotation number
    to the lowest.
    """
    def sort_key(p: Path):
        m = re.search(r"\.(\d+)$", p.name)
        if m:
            return -int(m.group(1))  # Negative for descending order
        elif p.name == "restapi.log":
            return float('inf')  # Ensure restapi.log comes last
        else:
            pass  # Any other files are skipped

    files = [f for f in directory.glob("restapi.log*") if f.is_file()]
    return sorted(files, key=sort_key)


def parse_json_block(text: str):
    try:
        text_clean = re.sub(r",\s*([}\]])", r"\1", text)
        return json.loads(text_clean)
    except json.JSONDecodeError as e:
        raise ValueError(f"JSON parse error: {e}") from e


def extract_first_json_object(s: str) -> str:
    depth = 0
    start = None
    in_string = False
    escape = False

    for i, ch in enumerate(s):
        if in_string:
            if escape:
                escape = False
            elif ch == "\\":
                escape = True
            elif ch == '"':
                in_string = False
            continue

        if ch == '"':
            in_string = True
        elif ch == "{":
            if depth == 0:
                start = i
            depth += 1
        elif ch == "}":
            depth -= 1
            if depth == 0 and start is not None:
                return s[start:i + 1]

    raise ValueError("No complete JSON object found")


def extract_request_response(text_block: str):
    # Match log header (timestamp, status, service)
    log_header_match = re.search(
        r"^(\d{4}-\d{2}-\d{2} "
        r"\d{2}:\d{2}:\d{2}\.\d{3}\+\d{4})\s+"
        r"(\w+)\s*\|\s*RestAPI\.(.*?)\|",
        text_block,
        re.MULTILINE
    )
    restapi_datetime = log_header_match.group(1) if log_header_match else None
    restapi_status = log_header_match.group(2) if log_header_match else None
    restapi_service = log_header_match.group(3).strip() if log_header_match else None

    # Convert datetime to epoch
    restapi_epoch = None
    if restapi_datetime:
        try:
            dt = datetime.strptime(restapi_datetime, "%Y-%m-%d %H:%M:%S.%f%z")
            restapi_epoch = int(dt.timestamp())
        except Exception as e:
            logging.warning(f"Failed to parse restapi_datetime '{restapi_datetime}': {e}")

    # Extract request number
    reqno_match = re.search(r"Request #(\d+)", text_block)
    restapi_reqno = int(reqno_match.group(1)) if reqno_match else None

    parts = text_block.split("Response:", 1)

    request_raw = extract_first_json_object(parts[0])
    request = parse_json_block(request_raw)

    response = None
    if len(parts) > 1:
        try:
            response_raw = extract_first_json_object(parts[1])
            response = parse_json_block(response_raw)
        except Exception as e:
            logging.warning(
                f"Response parse error for reqno {restapi_reqno}: {e}, storing raw"
            )
            response = {"raw": parts[1].strip()}

    return {
        "request": request,
        "response": response,
        "restapi_datetime": restapi_datetime,
        "restapi_epoch": restapi_epoch,
        "restapi_status": restapi_status,
        "restapi_service": restapi_service,
        "restapi_reqno": restapi_reqno,
    }


def parse_logs(log_files, output_file: Path, invalid_file: Path = None):
    total_found = total_written = total_invalid = 0
    invalid_out = invalid_file.open("w", encoding="utf-8") if invalid_file else None

    buffer = []
    in_record = False

    with output_file.open("w", encoding="utf-8") as out:
        for log_file in log_files:
            file_found = file_written = file_invalid = 0
            logging.info(f"Processing {log_file}")

            with log_file.open("r", encoding="utf-8", errors="ignore") as f:
                for line in f:
                    if REQUEST_START_RE.match(line):
                        if buffer:
                            total_found += 1
                            file_found += 1
                            record_text = "".join(buffer)
                            try:
                                record = extract_request_response(record_text)
                                out.write(json.dumps(record, ensure_ascii=False) + "\n")
                                total_written += 1
                                file_written += 1
                            except Exception:
                                total_invalid += 1
                                file_invalid += 1
                                if invalid_out:
                                    invalid_out.write(record_text + "\n")
                        buffer = [line]
                        in_record = True
                    elif in_record:
                        buffer.append(line)

            logging.info(
                f"{log_file.name}: {file_found} found, {file_written} written, {file_invalid} invalid"
            )

        # Final flush
        if buffer:
            total_found += 1
            try:
                record = extract_request_response("".join(buffer))
                out.write(json.dumps(record, ensure_ascii=False) + "\n")
                total_written += 1
            except Exception:
                total_invalid += 1
                if invalid_out:
                    invalid_out.write("".join(buffer) + "\n")

    if invalid_out:
        invalid_out.close()

    logging.info(
        f"DONE: {total_found} records found total, "
        f"{total_written} written, {total_invalid} invalid"
    )


def main():
    if len(sys.argv) < 3:
        print(f"Usage: {sys.argv[0]} <log_directory> <output_ndjson> [invalid_file]")
        sys.exit(1)

    log_dir = Path(sys.argv[1])
    output_file = Path(sys.argv[2])
    invalid_file = Path(sys.argv[3]) if len(sys.argv) > 3 else None

    log_files = find_restapi_logs(log_dir)

    logging.info(f"Input is a directory: {log_dir}")
    logging.info(
        "Using log files (from oldest to newest): "
        + ", ".join(f.name for f in log_files)
    )

    parse_logs(log_files, output_file, invalid_file)


if __name__ == "__main__":
    main()

Blog posts using this script