Receive logs from Stackdriver Logging

Posted: 2017-01-02, Modified: 2017-01-02   gcp log go

Summary

In order to obtain log data from Google Cloud Platform, using the Stackdriver client which logadmin package provides is one of the easiest ways. But, the type of log entries returned by the client is logging.Entry, and the type of payloads is interface{}, which means you need to cast payloads to access their fields. This article introduces how to cast them with a pre-defined structure.

logadmin package

logadmin provides interfaces to access logs in Google Cloud Platform.

The following code obtains log entries matching a query in variable filter:

import (
    "cloud.google.com/go/logging/logadmin"
    "golang.org/x/net/context"
    "google.golang.org/api/iterator"
)

func GetLogEntries(filter string) error{
    ctx := context.Background()
    client, err := logadmin.NewClient(ctx, "your-project-id")
    if err != nil {
        return err
    }
    defer client.Close()

    iter := client.Entries(ctx, logadmin.Filter(filter))
    for {
        e, err := iter.Next()
        if err == iterator.Done {
            return nil
        } else if err != nil {
            return err
        }
        // Use log entry `e`, of which type is logging.Entry.
    }
}

logging.Entry

The type of log entries which the logadmin client returns is logging.Entry, and the type of their payload is interface{}. To access each field of those payloads, you need to case them. In most cases, you can cast those payloads to pointers of structpb.Struct. Since structpb.Struct is a kind of meta-structure, you also need to convert an instance of structpb.Struct to another structure.

My structpbconv package provides such conversion.

structpbconv

The following code converts a log payload of a VM instance’s activity_log in Compute Engine from a structpb.Struct instance.

import (
    "github.com/golang/protobuf/ptypes/struct"
    "github.com/jkawamoto/structpbconv"
)


type ActivityPayload struct {
    EventTimestampUs string `structpb:"event_timestamp_us"`
    EventType        string `structpb:"event_type"`
    TraceID          string `structpb:"trace_id"`
    Actor            struct {
        User string
    }
    Resource struct {
        Zone string
        Type string
        ID   string
        Name string
    }
    Version      string
    EventSubtype string `structpb:"event_subtype"`
    Operation    struct {
        Zone string
        Type string
        ID   string
        Name string
    }
}

func NewActivityPayload(payload *structpb.Struct) *ActivityPayload {
    var res ActivityPayload
    structpbconv.Convert(payload, &res)
    return &res
}

I defined ActivityPayload structure according to an actual payload of activity_log. To give a mapping of field names, use a structpb tag.

Finally, we can access each field like that:

// Getting log entries of which event types are GCE_OPERATION_DONE.
iter := client.Entries(
  ctx, logadmin.Filter("jsonPayload.event_type = \"GCE_OPERATION_DONE\""))
for {
    e, err := iter.Next()
    if err == iterator.Done {
        return nil
    } else if err != nil {
        return err
    }

    // Converting the payload.
    if s, ok := e.Payload.(*structpb.Struct); ok {
        payload := NewActivityPayload(s)

        // Printing trace_id.
        fmt.Println(payload.TraceID)
    }

}