Receive log entries from Stackdriver Logging by logging package

Posted: 2017-05-05, Modified: 2017-05-05   go Google Cloud Platform Stackdriver

Summary

I’ve introduced logadmin package to read log entries from Stackdriver Logging in Google Cloud Platform, but there is another package logging version 2 to provide read log entries functions.

Although the logging package is still in experimental stage, it seems to provide more functionality than logadmin package. This post introduces a basic usage of this logging package.

logging package (Ver.2)

The following sample code creates a logging client and retrieves log entries matching a filter in variable filter:

import (
	"context"

	"cloud.google.com/go/logging/apiv2"
	"google.golang.org/api/iterator"
	loggingpb "google.golang.org/genproto/googleapis/logging/v2"
)

func GetLogEntries(ctx context.Context, filter string) (err error){

	client, err := logging.NewClient(ctx)
	if err != nil {
		return
	}
	defer client.Close()

	iter := client.ListLogEntries(ctx, &loggingpb.ListLogEntriesRequest{
		ResourceNames: []string{
			fmt.Sprintf("projects/%v", "your-project-id"),
		},
		Filter: filter,
	})

	for {
		e, err := iter.Next()
		if err == iterator.Done {
			break
		} else if err != nil {
			return err
		}

		// Use log entry `e`...

	}

	return nil
}

Next, I’ll introduce differences between logadmin package in this scenario. While the client in logadmin package requires a project ID when it is created, the logging package’s client requests a project ID as a resource name. The resource name allows four types of resources including Google Cloud Platform’s projects; and from the document of loggingpb.ListLogEntriesRequest, the following formats are acceptable for resource names:

• “projects/[PROJECT_ID]”
• “organizations/[ORGANIZATION_ID]”
• “billingAccounts/[BILLING_ACCOUNT_ID]”
• “folders/[FOLDER_ID]”

which means we can retrieve log entries not only associated with a project but also associated with an organization, a billing account, and a folder.

Another difference is iterators return LogEntry type log entries, and we can access text payloads via GetTextPayload and JSON payloads via GetJsonPayload, while iterators in logadmin package returns interface{} for any payloads.

However, GetJsonPayload returns a structpb.Struct type object and we still need to convert it to other structure which represents an actual log entry’s structure. For example, GCE_OPERATION_DONE event writes a log entry in the following structure:

type ActivityPayload struct {
    EventTimestampUs string `structpb:"event_timestamp_us"`
    EventType        string `structpb:"event_type"`
    TraceID          string `structpb:"trace_id"`
    Actor            struct {
        User string
    }
    Resource struct {
        Zone string
        Type string
        ID   string
        Name string
    }
    Version      string
    EventSubtype string `structpb:"event_subtype"`
    Operation    struct {
        Zone string
        Type string
        ID   string
        Name string
    }
}

and we need to convert objects which GetJsonPayload returns and written in structpb.Struct to objects in the above ActivityPayload. My structpbconv package helps this conversion, and the following is a sample code:

import (
 	"github.com/golang/protobuf/ptypes/struct"
	"github.com/jkawamoto/structpbconv"
)

func NewActivityPayload(payload *structpb.Struct) (res *ActivityPayload) {
    res = new(ActivityPayload)
    structpbconv.Convert(payload, res)
    return
}

Finally, the following is a sample to retrieve log entries of GCE_OPERATION_DONE events in Google Compute Engine by logging package:

import (
	"context"

	"cloud.google.com/go/logging/apiv2"
	"google.golang.org/api/iterator"
	loggingpb "google.golang.org/genproto/googleapis/logging/v2"
   	"github.com/golang/protobuf/ptypes/struct"
	"github.com/jkawamoto/structpbconv"
)

func GetLogEntries(ctx context.Context) error{

	client, err := logging.NewClient(ctx)
	if err != nil {
		return
	}
	defer client.Close()

	iter := client.ListLogEntries(ctx, &loggingpb.ListLogEntriesRequest{
		ResourceNames: []string{
			fmt.Sprintf("projects/%v", "your-project-id"),
		},
		Filter: `jsonPayload.event_type = "GCE_OPERATION_DONE"`,
	})

	for {

		e, err := iter.Next()
		if err == iterator.Done {
			return nil
		} else if err != nil {
			return err
		}

		payload := NewActivityPayload(e.GetJsonPayload())
		fmt.Println(payload.TraceID) // For example, print the trace ID.

	}

}